The Problem with Online Panels

Introduction

The COVID pandemic has disrupted many traditional research processes resulting in more and more companies relying on consumer panels and communities to get information from consumers.

Here, we have compiled some of the most common issues that online panels have so that you can make an informed decision on whether it’s the right approach for your business.

What is an online research panel?

When it comes to conducting surveys and getting feedback from consumers, there are several ways to get started such as:

• Face-to-face interviews: consumers are surveyed in person in public areas (streets, malls, etc.) or at home

• Focus group discussions: a targeted segment of consumers are gathered in a place and asked questions about products or services

• Phone Interviews: consumers are getting called by research companies over the phone

• Online Interviews: consumers are found and surveyed online

Online Survey Panels are one of the most common alternative when it comes to reaching out to consumers online. They consist of groups of pre-recruited and pre-screened respondents who agreed to participate in online surveys about various types of topics. Each respondent took the time to register an account in the system (website or app) of the panel, and fill in a free survey to categorize and pre-screen them. They will, later on, be compensated for each survey they participate in (from USD 0.1 to USD 50 on average depending on the topic and market).

Using online panels is a practical option because they are low cost, have a high response rate, and are fast. There is no doubt about that because we are all spending more and more of our time as individuals online, it makes sense to convert a paper-and-pencil questionnaire into an online survey and reach hundreds of people instantly. BUT, the truth is that there are numerous factors in play, and they should not be ignored.

The Problem: Quality

"Acknowledging that there is a problem is the first step toward solving it."

Panels are an interesting solution: they provide speed and access to population anywhere in the world. However, we should not forget the quality dimension of that methodology.

Why is the quality of online panels bad?

Every insight methodology has challenges that need to be addressed. Research companies must handle those challenges by minimizing the errors and maximizing the solutions without tossing out the insights. Yet, online panels seem to have given up on the quality of respondents.

#1 - Your audience is simply not there

The panel advertising industry, which recruits panel members, encourages respondents to focus on monetary benefits. In other words, respondents will be receiving payments (often below USD1) to download their app, create a profile on it, and take surveys. This approach is quite effective to attract a lot of respondents.

However, we may all agree that individuals with higher purchasing power are obviously not swayed by this type of incentive and mechanics. This actually means that if you are trying to get insights on your market or industry, the individuals in those panels may very likely not be your target audience at all.

#2 - Some respondents may become Professional Survey Takers

One concern with online research is that it often attracts professional respondents who participate in a variety of surveys because it is a way to make money. Individuals who are motivated by financial gain are more likely to be dishonest about their answers in order to receive the reward as soon as possible. Also, professional participants who are familiar with online surveys will answer questions more strategically which means they give poor responses and are less likely to be engaged. As a result, they are more interested in receiving some sort of incentive rather than in providing honest and quality answers, impacting the research's validity.

#3 - Using third-party data comes with a lot of natural limitations

The vast majority of online respondents are recruited through the internet, based on their declared qualifications, however, some may misrepresent themselves, or login information from users be misused or sold to other parties, who could then use it to take surveys. Many panels use different third-party partners to collect respondents, resulting in a lack of direct control and traceability over the sourcing.

A respondent can be a member of two or more panels, and this will cause the quality of the data to suffer. When multiple surveys are given to the same participants it is likely that the participants will learn the patterns and expectations of a specific survey.

In addition, they're more likely to receive identical surveys from multiple panel providers. That instance, if they answer two or more of the same survey and get the same responses, the data is duplicated; if they answer two or more of the same survey and get different answers, the data contradicts itself and is biased. In any case, obtaining data from the same individual repeatedly does not enhance the accuracy of the research findings.

But why don't the panel companies do anything about it?

While panel companies are the ones conducting the surveys, they're almost always under intense pressure from their clients to provide data faster and cheaper than ever before, and that does little to ensure quality. Clients often choose speed and affordability over quality, providers will not invest in increasing quality and prioritize providing you the respondents' answers as soon as possible at the lowest possible cost.

How do we fix this?

The accuracy of your data depends on two main components: finding the right respondents and asking them the right questions. Though we believe survey design is the most important component, having the right panel is also of utmost importance.

If you do not want to drive your business with poorly collected data then there are a few things you can do:

1. Always try to recruit new participants for each project,

2. Refrain from frequently giving respondents a reward for completing your survey, and especially not through payment,

3. Investigate your panel partner's sources and methodology to make sure it's a good fit for your business. Do not use a source whose data you can't verify (a panel partner that has lots of panel partners is probably not a good panel partner),

4. Include qualification questions to screen out respondents who do not match your target profile, and red-herring (trap) questions to make sure you are only keeping the right respondents in your survey.

At The Method Research, there are many other steps we take in order to ensure quality both on the respondents' selection and on the survey design - check our other article on how to ask the right question here. However, those first steps will help you get better results.

Conclusion

Online survey panels are a powerful tool to build relationships and gather information in these fast-changing times. However, as we have seen it, relying on this source of information is by definition biased and might not be what you need to make your decisions properly.

At The Method Research, we do not rely on any survey panels but recruit respondents for each project we conduct for our clients and ourselves. We run advertising campaigns on multiple online channels and invite participants to give their opinion for free or to get a chance to participate in a lucky draw. Our advertising knowledge and skills allow us to bring the insights cost at a competitive range with survey panels while ensuring a way higher quality.

To learn more about what we do and how The Method Research could be the right insights partner for you, feel free to visit our website and other previous reports.