How to get to the truth by minimising response error
You may think you have drafted a fool-proof survey but beware of response error. This refers to any error introduced into the survey results due to respondents providing untrue or incorrect information. This blog entry covers two of the main types of response error – namely acquiescence bias and social desirability bias – giving some examples and advice on how these can be minimised.
This is the tendency for respondents to agree or be positive when answering questions, as it is often harder for people to say "No" rather than "Yes". Response options such as "Yes/No", "Agree/Disagree" or "True/False" can increase acquiescence bias, and thereby decrease honesty levels. Here are some ways to minimise this bias:
- Try to avoid the simple "Yes/No" type responses if you can. So, for example, the question: "Would you ever consider shopping at Store X?" Yes/No would become: "How likely are you to ever shop at Store X?" Very unlikely/Unlikely/Not sure/Likely/Very likely.
- Rather than using "Yes/No", you can present options as alternatives. For example, "Do you generally prefer paying for items with cash?" Yes/No would become: "How do you generally prefer to pay for items? Cash/Debit/credit card…
- If you are using scales, present the negative option first, so start with "Very dissatisfied" or "Strongly disagree", for example. There is a tendency to go with the first option in an online survey (called the Primacy effect) so these two biases will counteract each other somewhat.
Social Desirability Bias
Respondents will often try to answer questions in a way that puts them in a good light. It can take the form of over-reporting "good" behaviour or under-reporting "bad" behaviour. This type of bias can be particularly prevalent for questions with a social agenda. Here are some ways to reduce this type of bias:
- A simple statement to appeal to their honesty can help – for example, "Please answer these questions as truthfully as possible". Also reassure them about issues of confidentiality and security of data, especially if you are asking sensitive questions.
- Think carefully about the wording of your questions. Direct questions can make people less willing to answer truthfully. For sensitive issues you can try face-saving wording. So the question: "Have you read the Finance report you were sent last week?" can be reworded to: "Have you had a chance yet to read the Finance report you were sent last week?"
- An alternative is to prefix the question with a statement to show that you are not judging them and understand if they say no, such as: "We all lead busy lives and sometimes don't get around to doing things as quickly as we would like. With this in mind, have you had a chance yet to read the Finance report you were sent last week?"
- Use projection techniques to avoid a difficult direct question – such as asking respondents what someone else with similar values thinks about a topic. An example would be: "What would your friends do if they saw someone breaking the law?" Their answer is likely to reflect their own opinion without directly stating it. But a word of caution when it comes to analysis: you can't relate the answers back to the respondents themselves.
- Extend the answer options to imply that extreme behaviour is not unusual. This makes respondents more comfortable about telling the truth. So for the question: "How many cigarettes do you smoke per day?", the initial answer options may have ended at "2 packs or more" a day. However, if you add in extra extreme options, such as "2-3 packs" and "4 or more packs", then heavy smokers are more likely to select the true option.
Follow these tips to minimise these common types of response error as much as possible. This will enable you go get closer to the truth when it comes to the results.