What Effect Has Reversing Scales In A Grid Have On The Result?

Recently we tested the concern that the result of a scale question will vary depending on the direction in which you construct the scale.

The test was small but revealing.

Two independent, general population samples, each of 50 respondents, were matched on gender and age.

Each group was shown a grid of products (randomized order) and asked to rate them on “Healthy for you” – a five-point bipolar scale ranging from “Very healthy for you” to “Very Un-healthy for you”.

The only difference between the two groups was a reversal of the scale –

The Left to right group scale runs from “Very Healthy for you” to “Very Un-healthy for you”, and,
The Right to left group scale runs from “Very Un-healthy for you” to “Very Healthy for you”.

SMP_N_MAR14_1_1

SMP_N_MAR14_1_2

 

The chart below summarizes the result for the ratings of Coca Cola – the ‘Left to right’ group are in blue and the ‘Right to left’ group are in red.

SMP_N_MAR14_1_6

We can see that the Right_to_Left group have an increased incidence of ‘Very Unhealthy for you”, compared to the Left_to_Right group.

It appears that respondents are more likely to select the end of the scale when it is on the left, rather than on the right.

Analysis across all products.

Table 1:

SMP_N_MAR14_1_3
Across all products, we can see that the Right_to_Left group have an increased incidence of ‘Very Unhealthy for you”, compared to the Left_to_Right group.

It appears that respondents are more likely to select the end of the scale when it is on the left, rather than on the right.

A further analysis embellishes this proposition.

If we designate each of the nine products as being either Healthy or Unhealthy (based on their average rating of -2 for’ Very Unhealthy for you’, through to +2 for ‘Very Healthy for you’) we find the following:

Table 2:

SMP_N_MAR14_1_4

Averaged over the two groups, only three products averaged on the healthy side of the scale. (bottled water, milk and orange juice).

If we split the analysis by this health designation we find:

Table 4:

SMP_N_MAR14_1_5

For Healthy products, ‘Very healthy’ scores best in the left_to_right group where it appears on the left.

For unhealthy products ‘Very Unhealthy’ scores best in the right_to_left group, where it appears on the left.

This suggests that some respondents, (in reading from left to right), tend to click on the first item that matches their overall position (Healthy versus Un-healthy), regardless of whether it is ‘Very’ or ‘Somewhat’.
One clear implication of these results is that it is very important to control for order effects. For instance, many researchers place great emphasis on DWB (Definitely Would Buy) in Purchase Intent Scales. The DWB percentage would be a prime candidate for the bias identified in the data above.

The simplest way to control for order effects of this nature is to randomly reverse the scale across the sample.

SMP allows this as an option on all grids with a feature called “Random Rotate“.  Select if from the list of randomization options found in question parameters “Randomize Answers:” dropdown shown below.

SMP_N_MAR14_1_7

1 Comment

  1. Forex Review May 10, 2019 Reply

    A critical aspect of how this bias can come to affect the responses of participants relates to the norms of the society in which the research is taking place.

Leave a reply

Your email address will not be published. Required fields are marked *

*