You can spend a lot of time, money, and effort developing a consumer or research survey, so you want to be sure it’s effective and easily understood. But how can you find this out without actually sending your survey to potential participants? By conducting survey testing. So how do you test a survey, and what method of survey testing is right for you?
Create your test survey, questionnaire, or form now!
What is Survey Testing?
Survey testing means running your survey through a series of tests to check for potential problems. Testing can help catch any issues with a survey before you send it out to your key participants. Potential problems include:
- Confusing or highly sensitive questions
- Poor question logic and flow
- Agitation due to survey length
- Poor survey design or layout
- Sources of survey bias
- Technical or security glitches
By weeding out problems in a survey test, you can avoid a lot of headaches down the road and ensure that your final data is accurate and actionable. Of course, testing a survey can be a pretty simple or a highly complex affair. If you’re a small company, for example, it may mean just testing a draft of the survey with a few employees. On the other hand, if you’re conducting a large research study, you may need to run a pilot test and go through every method of testing which we cover next.
8 Ways To Conduct Survey Testing
There are a number of ways to test your survey before sending it out to the masses. Depending on the complexity of your survey, you may choose one method, a combination of methods, or engage in all of them.
1. Pretesting
A pretesting survey, also known as skirmishing or respondent debriefing, involves informally testing an unstructured survey or questionnaire with a sampling of potential respondents who can shed light on any potential problems. Getting their feedback requires adding several evaluation questions to the end of your survey. These can be open-ended or closed-ended questions that ask about survey content, comprehension, acceptability, difficulty, design, and length.
2. Pilot Testing
Unlike pretesting, a pilot testing survey involves formally testing your complete, structured survey with a small sample of respondents. Rather than ask evaluation questions, respondents are given the survey as-is and are asked to complete it. Pilot testing your survey, sometimes known as a dry-run, a dress rehearsal, or a soft launch, provides a sense of the kind of responses you will receive and any issues that may arise during the real survey period.
3. Data Analysis
After you’ve pretested or pilot tested, it can help to complete some data analysis. This involves looking at patterns in responses to see where confusion, hesitation, disengagement, or drop-out has occurred. You can often be discovered by identifying straight-lining (the same answer is always checked regardless of the question), unanswered questions, and inconsistent or unrealistic responses. Read more about these analysis techniques in our Data Cleaning Blog.
4. Cognitive Interviewing
What’s really going on in your respondents’ minds as they answer your survey questions? Cognitive interviewing can get to the bottom of this. These interviews are conducted in person with a small sample of respondents (generally about 5-15). As they answer each survey question, a facilitator asks them to think aloud, describing their thought processes, emotional responses, and understanding of what each question means. These interviews can help determine if a question is ambiguous, confusing, or makes people uncomfortable due to its content. They can also spot drop-out risks by identifying if respondents are getting bored or agitated by the survey.
5. Expert Evaluation
Sometimes it helps to have an expert opinion, especially if your survey is asking questions about a complex subject matter. Topic experts can tap into their deep well of knowledge to evaluate your questions in order to help shape or reshape the content. Survey methodologists may also be called upon to help determine the best ways to collect accurate data for research questions.
6. Focus Groups
A focus group is a semi-structured discussion with a small group of people, usually around ten or so, led by a moderator. The members of the group can be very helpful in forming questions and addressing some of the following issues:
- Subject matter relevancy to the group
- Characteristics of the target population
- Understanding of questions and concepts
- Difficulty of the survey and abilities of the group
- General group reaction to the survey wording and design
While focus groups can be very helpful, there is the danger of “group speak,” wherein problems can be built up and exaggerated as group members “feed off” one another during discussions. So, focus groups should never be the sole means of survey testing.
7. A/B Testing Experiments
Another great method of survey testing is A/B testing experiments. This requires splitting your sample of respondents in two and providing different variations of the survey to each. This method can be a powerful way to understand how changes in question wording or order, survey design or layout, and other methodological factors may influence respondents and the data you collect.
8. Observational Studies
This method of survey testing involves having a sampling of respondents complete your survey while an observer watches over them. The observer is generally someone who is an expert at observing body language and facial expressions. They can often determine whether a person is feeling uncomfortable, bored, agitated, and other emotions. For observational studies to work, respondents should be told that the survey is being tested, not their reactions, so that they don’t put on an act.
7 Benefits of Survey Testing
There are numerous benefits to survey testing, some of which we’ve already discussed. Here are our top seven benefits.
1. Reducing Waste
Pretesting helps avoid the costly and time-consuming process of scrapping a flawed survey and starting all over again. Plus, you’d need to recruit new respondents (using the same ones could introduce survey bias) which requires additional effort, and more than likely, expense.
2. Improving Reputation
If you release a flawed survey, it can reflect poorly on your company or research and waste the time and effort of respondents. Plus, if you release inaccurate conclusions based on data accumulated through a flawed survey, you’ll be looked at as unreliable in your research by your peers and the public.
3. Improving Decision Making
Inaccurate data gathered through a flawed survey can lead your company to make poor decisions or can lead researchers to incorrect conclusions. Testing a survey to be sure respondents understand it will result in more accurate responses when the final version is sent out, improving decision making.
4. Maintaining Compliance
Is there a security issue with your survey? When it comes to people’s personal information, this is more important than ever, especially with the introduction of GDPR compliance. By testing your survey before releasing it in large numbers, technical glitches and security issues can be caught and repaired before any damage is done.
5. Saving Money
If you employ physical marketing strategies, such as direct mail coupons, newsletters, or magazines, and send these out to uninterested people based on flawed data, you’ll be wasting a lot of money. Developing a new product based on flawed data can cost you even more!
6. Uncovering Problems
We’ve covered this, but to reiterate, survey testing can identify all types of problems, from intimidating survey length to ambiguously worded questions, poor design to flawed survey logic. Of course, once problems have been uncovered, it’s much easier to find solutions!
7. Improving Productivity
If you begin acting upon data collected from a flawed survey, whether it’s creating a marketing plan or building a research paper, you’ll ultimately be wasting your time because respondents didn’t understand or engage well with your survey. By collecting accurate responses following survey testing, productivity is improved regardless of what you intend to do with the data.
Conclusion
It’s always important to test your survey before sending it to the masses. Of course, as with any type of feedback process, testing is useless unless you act on the findings. Whether that requires redesigning the survey, rewriting the questions, or retooling the security, it’s important that you don’t disregard what you’ve learned. And, if you try one method of survey testing and don’t identify any issues, you can call it a day or you may want to employ a different method just to be on the safe side. Either way, survey testing can be a big time and money saver!
Ready to create a test survey? SurveyLegend makes it easy with our pre-designed, secure, and responsive templates. Get started today!
Do you regularly run a survey test? What method of survey testing is your preferred approach? Let us know in the comments!
Create your test survey, questionnaire, or form now!
FAQs
Survey testing involves running your survey through a series of tests to check for issues such as confusing questions, poor design, or technical glitches.
By testing a survey, marketers and researchers can catch potential problems before a survey is sent to the main group of participants. This can save time, money, effort, and embarrassment.
A pretest survey is informal and asks a small sample of participants to provide feedback about the survey upon completion to identify issues. A pilot survey is formal and doesn’t ask for feedback; instead, results are analyzed internally to attempt to uncover problems.
