Why read this? : Bias in market research is something you want to avoid. It makes your research results less reliable. This week, we look at where bias can appear in each of the 3 types of research approach. Learn how to prepare for it to reduce its impact, and who’s likely to cause it. Read this to get better at managing bias in market research.
A bias is an influence or prejudice that leads you to think in a particular way. It makes your thinking subjective rather than objective.
In market research, you want the results to be objective. So you try to reduce the influence of bias in the process.
That’s easier said than done. Bias is everywhere. It influences how we perceive the world around us. It influences what we like, what we don’t like, how we make decisions and what we do.
To reduce the bias that can influence market research, you first need to identify where it occurs and then plan what to do about it. Bias appears in many places. It affects the researcher and the respondents. It comes in when setting up research and when analysing the results. It’s hard to find any part of the market research process where you don’t have to think about bias.
Bias in market research approaches
You choose the approach after you’ve defined the business problem and written the research brief. It’s usually your market research company who make the recommendation on which approach to use.
Each approach serves a different research purpose and each has its own particular types of bias to look out for.
Let’s look at each of them in turn :-
Bias in secondary research
Secondary research uses research that’s been done for some other purpose. You analyse it to see if it can answer your research questions.
It’s often used to get a background view on a research topic, and where you don’t have much budget to spend on research.
These can be published reports like national statistics such as the Australian Bureau of Statistics publish. They can also be special topic reports like Mary Meeker of Bond Capital’s well known annual Internet Trends report.
They can also come from online data sources, notably Google. For example, you can find keywords for inspiration to write content that people are searching for. That’s secondary research because the data wasn’t captured for that reason.
Secondary research holds many attractions. It’s usually easy to find and often free (or low cost). It gives you a fast and broad overview. That’s helpful for new subjects you don’t know a lot about.
However, there’s an obvious challenge with it. It was done for some other purpose, not yours. You have to ask how well someone else’s research can answer your research questions.
There are many ways bias can come into this type of market research. Let’s start with a fairly obvious one.
Self-interest (source) bias
Self-interest bias is where the source of the research has some sort of interest in shaping what the research results show.
For example, there’s this research study from Think TV. It tells you it was done by a “leading independent marketing analytics firm”. Its headline result shows TV is the most efficient media channel across FMCG, Automotive, Finance and E-Commerce.
They talk about how they ran econometric modelling across 3 years of data on 21 top advertisers in Australia.
There’s lots of specific (and therefore believable) sounding ROI numbers about different media channels.
But just hold on a minute. This research is sponsored by an organisation called Think TV? And its research shows TV is better than other media channels. How likely is it that there’s no bias here?
They claim the research is unbiased, but their stakeholders are all the companies who sell TV advertising in Australia. It’s obviously in their interest to make TV advertising sound like a good media buying option.
So always check the the source of your secondary research. Ask why they did the research in the first place. If they’re using it to promote their own purpose, the results may be biased and less helpful in answering your research questions.
There’s also a potential sampling bias in the Think TV research. They don’t talk about how they chose the 21 companies used in the survey.
How do you know they didn’t selectively pick companies who used TV advertising successfully? How do you know the way those companies do TV advertising would be the same way you’d do it? Obviously, you don’t. So, there’s a potential sampling bias if you were to use this research.
This doesn’t necessarily mean the research is wrong. But it may not be giving you the full picture.
Google Analytics works using cookies. These are small pieces of code it places in the user’s browser so it can collect data about their visit to your site.
But if your website has to be GDPR compliant, it has to ask consent for these third-party cookies. If users don’t accept, you don’t get their data. And that means you get a biased view of your visitors.
You’ll only see data on the ones who accept the cookies. That’s a bias. For an unbiased view of all visitors, you need to to set up a first-party data gathering system.
Availability and confirmation bias
There’s also a danger of bias in how you analyse and use secondary research. It’s often tempting to overvalue its importance. You may be tempted to use it to answer your full research question even if it only answers part of your question.
This is an availability bias. That’s where you make decisions on what you have or what most easily comes to mind. You don’t think about whether you need more data to make a better decision. You go with what you’ve got, even if what you’ve got isn’t enough to make a good decision.
There’s also a danger of confirmation bias. This is where you have pre-conceived ideas about the answer. So, you focus only on the data that confirms what you already thought, and discount anything that says the opposite. The purpose of market research is to give you market led insights than you then make decisions on. Deciding the answer first, then looking for data to prove it is a biased way to use market research.
How to reduce bias in secondary market research
Dealing with bias in secondary market research is hard. You’ve no control over how the research itself. The main area you can manage bias is in how you analyse and use it.
Try to gather data from multiple sources for example. Look at who’s behind the research and what its purpose is. Look at the methodology. The best research will share details of how it managed bias.
If it’s lacking in those details, or you see signs of self-serving or sampling bias for example, use that research with caution.
Be careful as you analyse and interpret it. Check you’re being open-minded about what it’s telling you. Make sure you’re not using it to give you an answer you can use to prove your case.
Secondary research is appealing because it’s fast and cheap. But that means you have no control over the quality of it. Be careful bias doesn’t creep in to how you use it to make decisions.
Bias in qualitative research
With qualitative research, you speak directly to customers. You use it to ask “why” led questions. Why do customers do what they do? What influences sit behind their motivations, attitudes and behaviours.
There’s not the same source bias you get with secondary research. You’re the source. You know why you’re asking the questions and the purpose of the research.
You decide the questions, and you adapt them to the responses you get.
It’s a flexible, adaptive process, driven by the responses of the respondents.
However, setting up that flexibility and adaptiveness introduces a new set of biases to worry about in this type of market research.
Bias in the interviewer
Market researchers who run one-to-one and focus group interviews should be trained to reduce bias in the research.
They need to stay neutral and impartial through each stage of the process.
To get the unbiased thoughts of the respondents, the interviewer needs to make them feel at ease.
They should create a sense of trust where respondents feel comfortable sharing what they really think and do.
Interviewers need to be non-judgemental and not let their reactions influence the answers. This can be as basic as thinking about how they dress, how they speak and the body language they use.
Sensitive or emotional topics
They also need to take care if the topic is sensitive or emotional. They need to make sure their own thoughts and feelings don’t creep in.
Bias often appears in research on political or social topics for example.
For topics like taxation, welfare support or abortion, the interviewer has to learn to stay neutral. To not react positively to opinions they agree with. To not react negatively to those they disagree with. The interviewer’s reactions can add bias to the results.
It can also creep into what seem like innocuous topics. How much TV someone watches or what shows they watch for example. How many books they read and which authors they follows. Some respondents might feel the interviewer will judge their answers and so give answers that make them look better.
Confirmation bias can also creep into how the interviewer listens to the answers. They may only hear what they want to hear. The interviewer needs to stay neutral, whatever the respondents say.
Then, there’s the anchoring bias. This is where you overemphasise the value of the first piece of information you hear, over other pieces of information that follows.
So, for example, you’re researching price options in qualitative research. Customers will react differently if you start from the lowest price and go up, compared to starting with the highest price and going down.
You need to mix up how you ask this type of question to reduce the bias. Interviewers should vary the order in which they ask questions. This is especially true if asking respondents to choose from a list of options. Anchoring bias makes people over-focus on the first option given. This can bias your research results.
Bias in the respondents
You also have to look out for bias in the respondents during qualitative research.
There’s the mix of respondents you choose for example. This again is sampling bias.
You set recruitment criteria so they match the segment you’re researching. But they’re not statistically representative of the whole segment.
When you screen only on demographics (age, gender, income etc) for example, you may end up with people with different behaviours and attitudes.
They’re there as individuals and won’t see themselves as speaking for your segment. That means you get all their opinions, ideas, quirks and foibles. But these may be unique to them. They might not give you an idea of what the wider segment might actually think or do.
Groupthink and social acceptance bias
In focus groups, you also have to watch out for groupthink bias. This is where one person puts out an opinion first and the rest of the group go along with it to keep the group feeling harmonious. Even if they secretly disagree with that view.
The respondents in focus groups usually don’t know each other. There’s some social pressure in sharing your opinion in front of strangers. You don’t want to feel judged. You say things you think are socially acceptable, rather than what you really think.
It’s part of the interviewer’s job to make it comfortable to share honest opinions to reduce this type of social acceptance bias.
False memory bias
There can also be false memory bias. You’re often asking respondents about things they’ve done in the past. When they last bought a product in your category and how they made the decision for example. But, the longer ago that happened, the less reliable their memory will be.
It may not be an issue with regular purchases, but it can be with bigger ticket items like cars or electronics. It could be years since their last purchase. That can make it hard to remember details, so they guess or make things up. That adds bias to the research results.
Most of all, the interviewer needs good listening skills. They need to feel the dynamics of the group to make sure they’re getting honest answers out of respondents.
Bias in your observations
With qualitative research, you often observe some of the research (especially if it’s focus groups) to get a feel for how it’s going.
But you rarely observe all of it, because that would take up too much time.
However, that means you might over-value the bits you saw in your decision-making.
It’s easier for you to picture those than only hearing about what happened in the bits of the research you didn’t see. This is another availability type of bias.
First-hand experiences are vivid and stick in our heads. They’re more mentally available than second-hand experiences which we only hear or read about.
In the research debrief, you have to listen to the summary of the whole research project. Don’t let the sessions you observed bias how you look at the results.
There’s also a danger of confirmation bias when you observe research. You’re listening out for respondents to say certain things that’ll show you were right all along. You don’t want that. You need to live by the same “rules” as the interviewer when you observe – stay neutral and impartial.
How to reduce bias in qualitative research
Staying neutral and impartial to reduce bias is easier said than done though.
It’s especially hard in qualitative research. Biases are mainly driven by people, and qualitative research is full of people-driven interactions. You’re asking people to give their subjective thoughts on a topic. Subjectivity is by nature biased.
You can put some structure and process in place to reduce the bias. How you select respondents and organise the questions for example. But the approach itself depends on open questions and free-flowing responses. It’s difficult to plan ahead and prepare for all eventualities. You just don’t know what respondents will say. If you did, you wouldn’t need to do the research.
Processes to help reduce bias
Your market research company should also have processes to help reduce bias. For example :-
- using a team of researchers and rotating them around different roles (e.g interviewer, observer) as the research progresses.
- having multiple people (including you) review the question structure for biased questions before the research starts.
- using counter biasing statements – eg. saying a certain behaviour is common, then ask the question about it.
- making indirect statement refer to what “other people” think or do. The assumption is that the respondent’s answer reflect what they think or do.
- using labeled response categories. If the answers are sensitive, the interviewer shows cards with each answer, but labelled with an identifier eg letter or number. The respondent doesn’t have to say the answer out loud, they just state the letter or number.
Observing qualitative research is always worthwhile. But remember, the debrief from the market research company is more important. You’ll get a more rounded view of the results, that should have most bias stripped out.
And finally, remember qualitative research doesn’t replicate “real” buying conditions. You’re asking people about their motivations and buying decisions in a relatively artificial environment.
In a real-life situation, most people don’t put much thought into what they buy. Make sure your interest in what they do doesn’t bias you to over-interpret what they say in the research results.
Bias in quantitative research
Quantitative research is more structured and process driven. That helps reduce a lot of bias.
You use a questionnaire to ask a large group of people the same questions.
If you run the research online (as most surveys are these days), you don’t even need an interviewer. That takes out a big source of bias right away.
In the days when using interviewers for quantitative research eg. on-street surveys or phone interviews, you’d have to check they noted responses correctly.
You’d also have to make sure they didn’t get fatigued so errors would creep into the process. The automated approach of online quantitative research takes away a lot of these sources of bias.
Biased questions in quantitative research
The main bias challenge in quantitative research is usually in the questions themselves. Biased questions try to overly influence the direction of answers. This leads to incorrect results.
So, for example, there’s the self-serving bias we mentioned earlier. That’s when you try to sway a response to get the answer you want. For example, imagine you came up with this question :-
“on a scale of 1 to 5, how effective do you think the award-winning advert from (Product X) was?”.
In that example, “award-winning” is clearly self-serving. It introduces bias. Respondents will think, “oh this must be a good advert if it won awards.” So, you’d get an unrealistically positive view of the advert.
Then there’s false-consensus bias. This is where you believe your views are the same as most other people, even if they aren’t. This is because you spend the most time with people who have similar views. You follow consumer media and news from sources that share your view of the world. You overestimate these views as “what everyone else thinks”.
In questionnaires, this bias can appear in questions like :-
“do you agree with the majority of people that Product X is the best on the market?”.
In that case “majority of people” introduces bias. When respondents consider Product X normally, they don’t know what the majority thinks.
The way this question is worded will produce overly positive responses. It’s biased.
Question fatigue or confusion
Finally, there’s also a danger if you try to ask too many questions or your questions are confusing. Both will make respondents more tired.
Tired respondents may stop reading questions properly, and pick quick, easy answers to get to the end of the research faster.
You have to be cared of getting tired biased answers from the end of the questionnaire. Ideally, questionnaires shouldn’t take too long to complete. 20 to 30 minutes is about as long as most people can give a questionnaire their full attention.
Data quality will decline for surveys that run longer than that.
Biased analysis in quantitative research
You also have to be careful of bias in the analysis of quantitative research.
For example, you assume respondents understand the question the same way you do. That might not be the case.
They can’t really ask for clarification, especially if it’s online. If they don’t understand it, they may just guess an answer. That can bias the research results.
There’s also potential bias with the statistics that sit behind the research.
For example, when you see a result, but the agency tells you it isn’t statistically significant.
Say you’re comparing campaign A vs campaign B. The research might tell you 44% of respondents preferred A, 42% preferred B. But the agency tells you this difference isn’t statistically significant. Your natural tendency is just to go for A because it’s slightly bigger, right? But because it’s not statistically significant, the real answer could still be that B is better.
In these cases, you’re making a biased decision based on what Daniel Kahnemann in his book Thinking, Fast and Slow calls WYSIATI. What You See Is All Their Is. Your brain likes likes easy answers. In this case, 44 is bigger than 42 so you pick that, right? But the “statistically insignificant” difference here isn’t enough for you to make an unbiased decision.
If you eat that, you need to gather more information. That means more time, effort and money and our brains are reluctant to do that. So we pick with easy but biased answer. What we should do is what other information you have. In this case, what did they specifically like about each advert? How memorable was each advert? How close does the message fit to your brand identity and so on. This type of deeper analysis stops you making a biased decision.
Biased sampling in quantitive research
The final bias area to consider in quantitative research is around your approach to sampling.
Quantitative research demands a representative sample of respondents, from which you can make statistically valid decisions about your total universe of respondents.
For some research topics, this can be a challenge.
For example, there are certain topics where respondents may fear their answers will have consequences, or they’ll be judge on them.
This can bias the answers people give, even if you tell them their responses will be anonymised.
Take Employee Satisfaction Surveys as an example. These can be tricky. Usually these are analysed and reported down to department and team levels. People may avoid giving honest answers if they feel it’ll impact their jobs.
You may also run into topics where it’s difficult to find people willing to answer questions on that topic. If it’s particularly sensitive or emotional for example. So you only get a sample of people willing to take part in the research, not a representative sample of everyone in the target audience.
How to reduce bias in quantitative research
Quantitative research is usually the type of market research where you can most reduce bias. Large samples, and repeated structured questionnaires mean you have a lot of control over the process. You can plan ahead and structure the process to reduce bias.
The 3 key areas to focus on are the questions, the analysis and the sampling.
Make sure you review the questionnaire before it goes out. Raise any concerns you have over bias. Ask the market research company to talk through their processes for reducing bias. Do they follow quality control standards such as AS : ISO 20252, the international quality standard for market and social research (See the Market Research Society website for more on this).
Look for things like how many people review the questions and how often they rotate the order of the questions. These are the types of good practice that reduce bias.
Try not to rush to snap judgments based on the first chart you see. Listen to the whole debrief. Ask questions to understand what the research company are telling you.
Don’t make immediate decisions. Give yourself some time to process the information, and re-read the deck again the next day. Get your team together and listen to their thoughts. You wan to make sure your own biases don’t overly influence the decisions you make.
Think about what you’ll do if the answers aren’t definitive. It’s helpful to have back-up plans and options if the research results take you in an unexpected direction.
Check the sample size is big enough to statistically represent your target audience. Your research company should explain how they calculate this. Ask them how confident you can be in the validity of the results. Quantitive research is the most expensive of the 3 types of research, but it’s usually the one that has the least amount of bias. Make sure you’re getting your money’s worth from the research company.
Bias in the people involved in market research
So far we’ve talked about how to reduce bias in the market research process and the different approaches. But remember, it’s people who set the process, and that’s where most bias starts. So to finish, we’ll review biases that people involved in the research might have.
Bias in market research companies
Market researchers who sign up to the standards and policies of the The Research Society commit to following best practice in eliminating bias from market research.
For example, in their Code of Professional Behaviour, under Data Provision and Reporting, clauses 34-36 spell out their obligations to eliminate bias :-
- 34 : When reporting on a project, Members must make a clear distinction between the findings, the Member’s interpretation of those findings and any conclusions drawn or recommendations made.
- 35 : Members must provide their clients with appropriate methodological details of any project carried out for the clients to enable them to assess the validity of the results and any conclusions drawn.
- 36 : Members must take reasonable steps to ensure that findings from a project, published by themselves or in their company name, are not incorrectly or misleadingly presented.
A distinction between the findings and their interpretation of those findings. That’s about reducing bias because the findings are objective, and the researcher’s interpretation is subjective.
Details of the methodology so you can assess the validity of the results. That’s also about reducing bias.
And steps to ensure findings are not incorrectly or misleadingly presented. Those too are about presenting results that have the last amount of bias.
Bias in marketing agencies
The market research company helps you find the answers you need, but it’s usually your marketing agency who convert those answers into actions.
Most agencies will be used to working with market research companies. You’d normally include them in the process.
For example, you might ask them to comment on the research brief. You’d invite them to observe qualitative research focus groups. And you’d definitely invite them to the debrief.
They will bring some biases though. They’ll certainly have some self-interest in the outcomes. Whatever service they provide you, they’ll be looking for signs in the research that you need more of that service. New adverts or more media for example.
You on the other hand need to lead the way in being objective. You need to think about your whole marketing mix and what the research is showing you about what to do.
Bias in you
Which brings us to the hardest bias to get rid of. That’s the bias in you.
Don’t be surprised by that.
Most people aren’t aware of their own biases. Think about what yours could be and what you can do to reduce them.
For example, make sure you involve people from your team and agencies in the process. Ask them if they think you have biases from other work you’ve done with them.
This makes you more aware of your biases and gives you a more diverse range of thinking about the results. That can mean less biased decision-making.
Conclusion - Managing bias in market research
The best way to reduce bias is to be prepared for it.
Look at your research plan and note anything that sounds like it could have a bias. Look for the most obvious biases first. For example, self-serving bias from the source of your secondary research, confirmation bias from interviewers and false-consensus bias in the questions you ask.
Think about who’s involved in the research. Consider what biases they’ll have.
Work with the market research company to reduce these (and their own biases) but be aware they’re hard to completely get rid of.
To find out more about bias, we recommend reading Richard Shotton’s The Choice Factory which we discuss in our article on business books that stand out. Also, Daniel Kahnemann’s Thinking, Fast and Slow which we cover in our how to use emotions in creative work article has some interesting chapters on bias.
Handshake : Cytonn Photography on Pexels