Spoilt for choice:

The most important tool in marketing may need sharpening

 

Questionnaires are the bread and butter of the researcher investigating consumer behaviour. But a new study sounds a note of caution about how the design of surveys themselves may affect the reliability of the data.

Marketing researchers routinely use questionnaires to understand, explain and predict the behaviour of consumers.

But designing a survey to yield watertight intelligence about markets is no mean feat: how many categories for each answer should researchers offer the individuals they quiz, and what is the best way of labelling these?

A new study by Bert Weijters, Elke Cabooter and Niels Schillewaert weighs up the pros and cons of the different options surveys employ to rate the answers of respondents.

“The Effect of Rating Scale Format on Response Styles: the Number of Response Categories and Response Category Labels” examines how differences in the format of  questionnaires can affect their results.

The researchers provide key insights into the best way to rate answers in an effort to sharpen this key business tool, and propose guidelines to help marketing teams choose the most appropriate format.

DILEMMAS: Questionnaire formats

Much of what we know about consumers is derived from questionnaire data but those devising surveys face dilemmas, particularly over what format they should use to enable respondents to rate their answers to questions.

A key issue is how many categories of possible responses a questionnaire should offer and whether these should all be labelled.

Commonly used questionnaire formats offer people being interviewed five, six or seven categories which are all labelled, or labelled at the endpoints (for example, with the first category being “strongly disagree” and the last “strongly agree”).

This issue has received little attention in research but is of importance because response categories may have a bearing on how people actually answer. Moreover, can data generated by a questionnaire using one format be adequately compared with data using another?

Weijters, Cabooter and Schillewaert compared the ways in which questionnaires are formulated in terms of three common forms of bias that can creep into how people answer:

• the extent to which respondents show a greater tendency to agree rather than disagree with items, irrespective of what they are;
• a tendency to answer by disproportionately using the endpoints in a scale of possible responses (such as “strongly disagree”);
• and the extent to which people give the same answer to two items that are opposite in meaning.

 

Different types of response scales may affect how people answer because they may enhance the importance or otherwise of certain options and nudge ambivalent or indifferent respondents to choose sides.

The researchers wanted to find out the impact on the common forms of bias of a range of permutations that combined different forms of labelling, varying numbers of categories or gradations in a response scale, and the use of a midpoint.

They focused on the two most common ways of labelling (labelling all the categories versus the endpoints only) and the most popular numbers of response options (five and seven). They also considered scales with four and six options to see whether having a midpoint makes a difference.

The researchers made a number of predictions about how these permutations would affect the common forms of bias found in responses based on how people behave when they answer questionnaires.

RESEARCH: Tests and findings

To test their ideas, the researchers conducted an online survey among panel members of a Belgian internet market research company. They manipulated the format of the rating scale by varying the number of response categories from four to seven and applying different forms of labelling. They found that:

• labelling has a significant effect on how people respond, and in particular on all three common forms of bias found in questionnaires. Where all the response categories are labelled, the tendency to agree rather than disagree rises, there is less of a tendency for respondents to answer by disproportionately using the endpoints, and the extent to which people give the same answer to items that are opposite in meaning decreases.
• including a midpoint increases the tendency to agree and also decreases the tendency to answer by disproportionately using the endpoints, more so when further gradations are also included in the scale. A midpoint also reduces the degree to which people give the same answer to items that are opposite in meaning, even more so when all the response categories are labelled.
• adding extra options to a scale does not affect the tendency to agree and the extent to which people give the same answer to two items that are opposite in meaning, but does lower a tendency to answer by disproportionately using the endpoints. When only the endpoints are labelled, adding gradations increases the extent to which people give the same answer to two items that are opposite in meaning.

 

The findings suggested that scholars should be particularly cautious when comparing results derived from questionnaires that use different types of scales for responses. An analyst wanting to report on the intention of consumers to buy a product might, for example, use an absolute percentage of those who agree with a question to argue that “the majority of respondents agree” - but this study suggests that such conclusions can be shaky because its findings also revealed that the distribution of responses was affected by labelling and the inclusion of a midpoint.

The questionnaire format that market researchers tend to use by default – a seven-point scale labelled at its endpoints - does not necessarily provide the best data compared to five-point scales. Further testing revealed that, for a five-point format, the endpoints-only scale performs better than the fully labelled scale in terms of how reliably variables can predict outcomes and variance.

IMPLICATIONS: Formats and frameworks

 The study has important implications for marketing research:

• The format of the scale on which questionnaire respondents can answer questions affects the trend and consistency of the data, and this means researchers should be cautious when comparing information obtained from surveys using different formats.
• Researchers need to choose the format they intend to use in a questionnaire carefully, and broader analyses aggregating or comparing data from several studies need to take into account different formats.
• The practice of reporting survey results by means of percentages of respondents who agree with a statement has to be treated with caution because distributions can vary considerably depending on the format used.
• In formats with endpoint labels only there is a stronger relationship between attitudes and intentions than in fully labelled formats.
• The phenomenon by which people give the same answer to two items that are opposite in meaning can occur during the process by which respondents select a category for their answer, but including a midpoint reduces this effect.

 

Based on their findings, Weijters, Cabooter and Schillewaert set out guidelines to help marketing researchers choose the best format to use in questionnaires:

• Choosing the ideal number of options in a questionnaire requires a trade-off between maximizing the information that can be gained against minimizing the demands on those individuals being questioned. As students tend to be better equipped to answer questionnaires than the general population, it may be better to use scales with seven categories for them but five for the general public.
• To reduce the extent to which people give the same answer to two items that are opposite in meaning, researchers should label all the answer categories.
• If a researcher wants to summarize responses in terms of means or percentages (for example, when reporting opinions), it may be better to use a fully labelled five-point scale (seven for students) which is easier for respondents to interpret.
• Five- or seven-point scales with endpoint labels are more reliable if a researcher wants to examine or estimate relationships between variables and predict trends.
• Analysis across several studies needs to take into account the formats that have been employed.

 

Related Academic Paper

Weijters B. Cabooter E. Schillewaert N. 2010. The effect of rating scale format on response styles: the number of response categories and response category labels. International Journal of Research in Marketing. 27 : 236 - 247

 

Published on 13/05/2011