Why read this? : We explore how to handle bias in market research. Learn who and what causes bias, and how to reduce its impact on research. Read this to learn how to manage bias in market research.
A bias is an influence or prejudice which leads you to think in a subjective rather than objective way.
Market research should be driven by objectivity. It should aim to minimise bias as this leads to better marketing decision-making.
Easier said than done though.
Bias is everywhere. It influences how we perceive the world around us. It influences what we like, what we don’t like, how we make decisions and what we do.
To reduce bias in market research, you start by identifying where it happens.
In the market research process, most biases appear when you do the research. That then impacts your analysis and action plan.
Doing the research covers one or more of these approaches :-
Bias in secondary research
Secondary research uses existing research. You try to find answers from research originally carried out for some other purpose.
It’s often used to familiarise yourself with a new topic. As it’s often free, it’s also used where you don’t have much to spend.
For example, using free published reports from national bodies like the Australian Bureau of Statistics. There are also many specialist publishers. For example, the annual Internet Trends report from Mary Meeker of Bond Capital.
Much of this is available online. (See our keyword research article for more on how to look for it).
Secondary research has many benefits. There’s a lot of it out there. It’s usually easy to find. And it gives you a broad overview of a new topic for little or no spend.
But, it has limits. Its answers often don’t answer your specific questions. The temptation to fit its answers to your questions brings in the first bias in market research we’ll cover.
Self-interest (source) bias
Self-interest bias with the research source is where whoever’s behind the research wants the results to support their own agenda.
That leads to subjective results. Or in other words, biased results.
For example, we recently looked at this research study from Think TV. Read the research summary, and on the surface, it sounds unbiased. Done by a “leading independent marketing analytics firm”.
It used econometric modelling across 3 years of data on 21 top advertisers in Australia. There are lots of specific-sounding ROI numbers. Specific numbers always sound more believable, right?
But park the methodology for a second. Consider who paid for this research which shows TV as the most efficient media channel across FMCG, Automotive, Finance and E-Commerce.
An organisation called Think TV. Their stakeholders are all TV companies who sell TV advertising. It’s hard not to be sceptical. Especially when there are other independent sources on media and marketing effectiveness e.g. the Ehrenberg Bass Institute,
So always check your sources. Ask why they did the research in the first place. Could the purpose have led to bias in the results? The more the chance of bias, the less reliable the results.
Sampling bias
Also, consider if the sample used represents the total market. If it’s been selected to deliver a certain result, that’s sampling bias.
For example, the Think TV research mentions the data from 21 companies. But we don’t know which companies. And why they chose only those 21 companies. Maybe they only picked companies that used TV advertising successfully. And ignored those whose TV campaigns didn’t work.
We don’t know how they picked the sample. However, research should always be done on randomly selected, representative samples. Otherwise, you won’t get the full picture. The research will be biased.
Availability and confirmation bias
Bias also appears when you analyse secondary research. If it’s all you have, you may overvalue its importance. You may stretch answers to fit your questions, even if those answers aren’t correct.
This is called an availability bias. You make decisions on what you have, without doing further research or thinking. This can lead you in the wrong direction. The results could be out of date, inaccurate or incomplete. You make worse decisions as you rely on poorer-quality data.
Similarly, confirmation bias can lead to poor decisions. Here you have some idea about what the research answers are (or what you want them to be) before you start researching.
So, you cherry-pick results which back up that thinking, rather than staying objective. You ignore anything which contradicts your view. But that’s subjective. Remember, subjectivity creates bias in market research. You have to stay objective.
How to reduce bias in secondary market research
As the research has already been carried out, your only way to reduce bias in secondary market research is in how you analyse and use it.
For example, gather data from multiple independent sources. Look at who’s behind the research and what its purpose is.
Check the methodology. High-quality research shares how it manages bias.
If it doesn’t do this, or you see obvious signs of bias, be wary of using those results.
Be careful as you analyse and interpret it. Are you staying unbiased? Make sure you’re not using it because it’s too hard to find out more. Or just to confirm what you already thought.
Secondary research sounds great because it’s fast and cheap. But you get what you pay for, and the quality can vary. Be careful of being biased.
Bias in qualitative research
You use qualitative research to speak directly to customers. To find out why they think and do what they do.
Why is a key question. You want to know what influences customer attitudes and behaviours.
The source and purpose of the research are clear. It’s your research questions asked to your target audience. You (or your research company) ask questions, listen to the answers and adjust future questions based on those answers.
It’s a flexible, adaptive process. But that flexibility and adaptiveness bring new market research biases to worry about.
Bias in the interviewer
Market researchers who run one-to-one and focus group interviews are trained to reduce bias.
For example, they build rapport and listen with empathy to respondents.
People are more open (and less biased) if they trust the researcher.
Researchers work hard to put respondents at ease. Even down to how they dress, how they speak and their body language.
The researcher should stay neutral and impartial throughout. They can’t let their own beliefs or opinions creep in. To get the unbiased thoughts of the respondents, the researcher must make them feel safe.
Sensitive or emotional topics
Researchers should also take care if the topic is sensitive or emotional.
For example, they should keep their own opinions out of research on political or social topics.
The interviewer has to stay neutral on divisive topics like taxation, welfare support and abortion.
No nodding at opinions they agree with. No head-shaking at things they disagree with. These types of reactions bias the results.
It can also creep into more innocuous-sounding topics. How much TV someone watches, or what shows they watch, for example. Some respondents might feel the interviewer will judge their answers. They might claim to watch documentaries but actually mostly watch reality TV. That would bias the results.
Anchoring bias
Then, there’s the anchoring bias. This is where you overemphasise the value of the first piece of information you hear. Everything which follows is anchored to what you heard first.
For example, say you’re researching price. If you say the lowest price first, everything after that sounds expensive. But if you say the highest price first, everything after sounds like better value. You need to vary the order to reduce bias.
Any time you ask respondents to choose from a list of options, be conscious the first item in the list gets the most attention. Switch the order of lists around to reduce this bias.
Bias in the respondents
It’s not just the researcher who can bring bias. The respondents can bring bias too.
For example, there’s the mix of respondents you choose. Like before, this is sampling bias.
You recruit respondents because they’re in the segment you’re researching. But they don’t statistically represent the whole segment.
And though they might fit the demographic (age, gender, income etc) profile, it’s usually attitudes and behaviours you’re more interested in.
There’s no guarantee the people you speak to have the same attitudes and behaviours as the total segment. So, the research results may be subjective to that small group, not an objective view of the total market.
Groupthink and social acceptance bias
In focus groups, you also have to watch for groupthink bias. This is where one person shares an opinion and everyone is too polite to disagree. So, they nod along to keep the group happy, even if they disagree.
Few people like to stand out with different opinions when in a group of strangers. This is known as social acceptance bias. You feel social pressure to go along with the group. You don’t want to express a different opinion and be judged for it. Often the only way to identify these types of beliefs is in one-to-one interviews, where there’s no group influence.
False memory bias
Similarly, there’s false memory bias. In categories where the purchase you’re researching doesn’t happen very often, respondents might not remember how they decided. But it can be embarrassing to say you can’t remember. So, they make things up.
It’s rarer with regular purchases, but it happens a lot with bigger ticket items like cars or electronics. It could be years since their last purchase. If it’s hard to remember details, you guess or say what you think the researcher wants to hear. That adds bias to the research results.
Bias in your observations
Often you’re able to observe the qualitative research as it happens (especially with focus groups). But it’s usually only 1 or 2 sessions, not the whole thing.
And that can bias how you interpret the research. Because you may well over-value the sessions you saw in your decision-making.
You have an availability bias. You can picture the sessions you observed easily. But the ones you didn’t observe, you can’t really visualise.
So you favour what you observed and don’t stay objective. You have to step back and look at the research as a whole. Don’t let your observations bias your interpretation.
There’s also a danger of confirmation bias when you observe research. You listen for respondents giving the answer you want. You’re biased to listen for answers which validate your opinions. That’s being subjective. Remember, unbiased research means staying objective.
Bias is mainly driven by people
Staying objective to reduce bias is easier said than done though.
It’s especially hard in qualitative research. Biases are mainly driven by people. Qual is full of people-driven interactions. You’re trying to stay objective listening to the subjective thoughts of the respondents. That’s hard to do.
Clear processes help reduce the bias. For example, sample selection and the order of the questions. But qual depends on open questions and free-flowing responses. It’s hard to prepare for all eventualities. You never know what respondents will say. If you did, you wouldn’t need the research.
Processes to help reduce bias in qual
Work with your market research company on processes to reduce bias e.g. :-
- use a team of researchers and rotate them around different roles (e.g. interviewer, observer) as the research progresses.
- have multiple people (including you) review the question structure for biased questions before the research starts.
- use counter biasing statements – eg. saying a behaviour is common, then asking the question about it.
- make indirect statements i.e. refer to what “other people” think or do. The assumption is the respondent’s answer reflects what they think or do.
- use labelled response categories for sensitive answers. The answers are given a label, and respondents call out the label, not the actual answer.
Bias in quantitative research
Quant research is normally driven by questionnaires. You start by reviewing these for bias to screen out biased questions.
Respondents generally fill in the questionnaire themselves online, which removes interviewer bias. (There are few on-street or phone interview surveys these days).
That also means the answers come direct and unfiltered from the respondents, so no bias creeps in before the analysis stage.
Biased questions in quantitative research
Let’s start with where bias can come into the questions you ask. Remember, bias tries to lead you to think in a subjective rather than objective way. Biased questions don’t get you objective answers.
Self-serving questions
For example, some questions may have a self-serving bias. They try to get the respondent to give an answer which supports your existing opinion. Take this as an example :-
“On a scale of 1 to 5, how much did you like the award-winning advert from (Product X)?”.
What’s wrong with this? It’s that clearly wedged-in “award-winning”. If you mention something is award-winning, your gut tells you it must be good. It’s won awards after all. But that’s leading respondents to be more positive about the advert. It’s biased.
False-Consensus bias
Then there’s false-consensus bias. This is where you assume or make out an opinion to be what the majority of people think, when that may not be the case. It can bias respondents to agree with that view, even if they don’t really believe it. Showing “what everyone else thinks” can bias answers and stop you from understanding what specific respondents think.
Take this example :-
“Do you agree with the majority of people that Product X is the best on the market?”.
See how “majority of people” introduces bias? You’re implying the respondent should also think Product X is the best on the market because everyone else does. But that may not be true. You’re biasing the results.
Question fatigue or confusion
Finally, bias can also creep in if you ask too many questions. Answering questions is tiring. It’s tiring because it takes brainpower, and our brains use a lot of energy.
As per our 5Ws of idea generation article, our brains are 2% of our body weight and 20% of our daily calorie consumption.
If respondents get tired, they’ll stop reading questions properly. They’ll give less well-thought-out answers. That’s more bias.
In general, attention spans peak at around 20-30 minutes. The longer than that it takes to answer the questions, the more the quality of the answers will drop.
Similarly, confusing questions also drain people’s energy. If they have to think too hard, they end up guessing answers. Again, more bias in the market research.
Keep questions clear, and questionnaires short. The better you do this, the less bias in the results.
Biased analysis in quantitative research
Bias can also come into quant research analysis.
For example, there’s the whole area of statistical significance. This should help you stay objective.
But because it’s often misunderstood, people misuse it or make subjective decisions anyway.
For example, say we quant tested two advertising campaigns. A and B. We want to know which one customers prefer. And let’s say 44% prefer A, and 42% prefer B. You go with A, right?
But then the agency tells you the result isn’t statistically significant. It’s inconclusive. It could still be more people prefer B to A. You should do more research. Ask more questions. But of course, you don’t. Because that would take time, money and effort. So, you go with what you’ve got.
But this is technically biased. It’s what Daniel Kahneman in his book Thinking, Fast and Slow calls WYSIATI. What You See Is All There Is. (See more about Kahneman’s work in our thoughts about thinking article).
Your brain likes easy answers. Like 44 is bigger than 42. Trying to persuade it to do more research to find a “statistically significant” (and correct) answer is too much hard work. You take the easy (but biased) option.
Biased sampling in quantitive research
You should also aim to avoid sampling bias. Sample size and selection matter a lot in quant research.
You need the sample to statistically represent the market. That statistical back-up is a core part of your decision-making.
For some research topics though, this can be a challenge. It can be hard to find a good sample if the topic is sensitive or emotional. Or the respondent believes their answers might have consequences.
The sample you do get (people willing to answer questions about a difficult subject) may not represent the total market.
You need to find ways to make sure your sample is more representative to reduce the bias. For example, making respondent answers more anonymous, or incentivising a wider range of people to take part.
How to reduce bias in quantitative research
The way quant research works gives you opportunities to reduce bias. It’s usually the research approach with the least bias as it’s more structured and process-driven. There’s less people bias to manage.
There are 3 areas to focus on :-
The questions
Always review the questions before they go out. Look for any you think might be biased. Ask the market research company to share their processes for reducing bias. Do they follow quality control standards such as AS : ISO 20252, the international quality standard for market and social research? (See the Market Research Society website for more on this).
For example, they should make sure several people check the questions for bias. They should also rotate the order of questions. That helps reduce bias. The less bias in the questions, the more reliable the results.
The analysis
Beware rushing to snap judgments. Don’t assume the answer’s on the first chart you see. Listen to the whole debrief. Stay objective. Ask the research company questions. It’s what you’re paying them for.
Don’t make snap decisions. Give yourself time to process what you hear. Read the debrief again the next day with a fresh mind. Get your team together and listen to their thoughts. Make sure your own biases don’t overly influence the decisions you make.
Also, have a backup plan if the research results aren’t definitive or take you in an unexpected direction. You want to make sure you’re making good decisions.
The sampling
Check the sample size. Is it big enough to statistically represent your target audience? Ask the research company how they calculate the sample size, and how confident you can be in the results.
Quant research costs the most to do, so you want the results to be clear, reliable and bias-free. Make sure you get your money’s worth from the research company.
Bias in the people involved in market research
In all 3 approaches, we’ve mentioned how people bring bias into the market research process. Let’s close with a closer look at the people involved, and why those biases might occur.
Bias in market research companies
Market researchers who sign up to the standards and policies of The Research Society commit to following best practice in eliminating bias from market research.
For example, in their Code of Professional Behaviour, under Data Provision and Reporting, clauses 34-36 spell out their obligations to eliminate bias :-
- 34 : When reporting on a project, Members must make a clear distinction between the findings, the Member’s interpretation of those findings and any conclusions drawn or recommendations made.
- 35 : Members must provide their clients with appropriate methodological details of any project carried out for the clients to enable them to assess the validity of the results and any conclusions drawn.
- 36 : Members must take reasonable steps to ensure that findings from a project, published by themselves or in their company name, are not incorrectly or misleadingly presented.
A distinction between the findings and their interpretation of those findings. That’s about reducing bias. The findings are objective. The researcher’s interpretation is subjective.
Details of the methodology to assess the validity of the results. That’s also about reducing bias.
And steps to ensure findings are not incorrectly or misleadingly presented. Those too are about presenting less biased results.
Bias in marketing agencies
Your marketing agency helps you convert research answers into actions. They have an inherent interest in what the research shows. It impacts what and how much you do with them.
For that reason, you should include them in the process. Share the brief, and ask them for feedback. Invite them to observe focus groups. Make sure they’re at the debrief.
But, take care with how they analyse the results, and what they recommend you do next.
They have biased self-interest in the outcomes. They’ll look for signs in the research you need more of the services they provide. So, you’ll get a biased view that you need more advertising, media, digital, or whatever it is they do for you.
You’re the one who needs to stay the most objective. It’s your brand and your marketing mix overall. You need to find the best right answer for you.
Bias in you
This brings us to the hardest bias to get rid of. That’s the bias in you.
What you think is subjective to you. Hard as you try to stay objective, you need to constantly watch out for your own biases. They’re often hard to spot.
So, ask for feedback from others in the research. Try to be aware of areas where you know you’re biased.
Use some of the techniques we’ve described here to try to reduce the impact of those biases.
Acknowledging they exist is the first step in getting rid of them. The more you can screen out bias, the more objective your decision-making will be.
Conclusion - Managing bias in market research
The best way to reduce bias is to prepare for it.
Look at your market research plan. Where could bias creep in? Look for obvious bias areas first.
Is there self-serving bias from your secondary research source? Confirmation bias from qual interviewers? False-consensus bias in the quant questions you ask?
Think about who’s involved in the research. What biases will the research company bring? What biases will your marketing agency bring? And of course, think about your own biases.
To learn more about bias, there are a couple of books we like. First, Richard Shotton’s The Choice Factory. It talks a lot about biases, and we share a specific example in this article. Also, Daniel Kahneman’s Thinking, Fast and Slow as per this article has lots on bias.
Plus, check out our behavioural science article or our market research guides for more on this topic. Or get in touch if you need help managing bias in market research.
Photo credits
Trash sign : Photo by Gary Chan on Unsplash
Attention sign : Photo by Markus Spiske on Unsplash
Google Tablet : Photo by PhotoMIX Ltd. from Pexels
Google Analytics : Photo by Edho Pratama on Unsplash
Glasses : Photo by Josh Calabrese on Unsplash
Conversation Image : Photo by Joshua Ness on Unsplash
Legal scales : Photo by Tingey Injury Law Firm on Unsplash
Man crying : Photo by Tom Pumford on Unsplash
Frustrated Man (adapted) : Photo by Usman Yousaf on Unsplash
Eye : Photo by Daniil Kuželev on Unsplash
Abacus : Photo by Crissy Jarvis on Unsplash
Question mark sign : Photo by Emily Morter on Unsplash
Bored in front of computer : Photo by Magnet.me on Unsplash
Calculator : Photo by Ray Reyes on Unsplash
Hand / Stop : Photo by Nadine Shaabana on Unsplash
Two men working together at a desk : Photo by Nik MacMillan on Unsplash
Handshake : Cytonn Photography on Pexels
Surprised Monkey : Photo by Jamie Haughton on Unsplash
