What’s the Difference Between Survey Validity and Survey Reliability?

A conclusion is only as good as the data that informs it. Researchers understand this, so those conducting surveys strive to obtain the most accurate results. This means survey data should be both reliable and valid. But aren’t those two terms the same? While often used interchangeably, survey reliability and survey validity are two different things. In this blog, we’ll look at each so that you can create surveys that tick both boxes.

Create Your FREE Survey, Poll, or Questionnaire now!

The Difference Between Survey Reliability and Survey Validity

While survey reliability and validity are closely linked, each plays a different role when it comes to survey results. Validity is about question accuracy. It looks at how well your survey conveys its questions (and possibly answers, if they are provided). Validity asks, will participants understand exactly what you’re surveying them about?

Reliability, on the other hand, looks at whether your survey questions would consistently result in the same responses from participants if the survey was repeated in the same situation. How likely are survey participants to respond the same way if they retook the survey later on?

Example of Survey Reliability vs. Survey Validity

The best way to highlight the difference between survey reliability and validity is with an example. Here’s a question on a survey for students created using SurveyLegend.

To determine validity, ask yourself: Is my question going to measure what is supposed to be measured? In the example above, the question seems pretty straightforward. However, it’s not necessarily clear on what engages the student. For example, a student could rate their level of engagement based on all factors pertaining to school (classes, teachers, socializing with friends, extracurricular activities). Or, they might respond based on their studies only, or a particular class in which the survey was conducted. A simple rewording could ensure understanding of the question’s intent.

To determine reliability, ask yourself: Will this question consistently elicit the same type of response? It’s very possible that a student in a bad mood may respond “0” or very unengaged one day, and a week later – now in a better mood – respond with a “10,” or highly engaged. That makes responses pretty unreliable. Rewording the question to give a time frame (e.g., today, this week, this year) could help to get better reliability. 

As you can see, while survey validity and survey reliability are closely related, they are not the same. A valid survey doesn’t necessarily equate to a reliable survey, and vice versa.

7 Types of Survey Validity

There are a couple of different factors to analyze when it comes to the validity and reliability of a survey. Here are seven types of survey validity methods.

1. External validity

This method of validity assesses sampling and looks for sampling errors. To be valid, a sample must be representative of the greater population. In our example, let’s say the population is all senior students at a particular school. Let’s say there are 300 seniors. A 10% sample size of 30 would be acceptable. 

However, what if the researcher simply surveyed one class of 30 students? The class could be one that most students dislike, so the majority may feel unengaged at the time. Or, it could be an AP (advanced placement) class with high achievers who are likely to be very engaged. Either way, the results are skewed. The research would be better off random sampling 30 students to improve representativeness.

2. Internal Validity

This validity tests whether the survey questions actually explain the outcome that is being researched. In our student engagement example, the researcher needs to ask questions that help identify factors that influence engagement. So, you would want to focus on the relationship between independent variables (e.g., grade point average, participation in extracurricular activities, level of truancy, disciplinary action taken, and so on) and the dependent variable (e.g., likelihood of engagement).

3. Statistical Validity

This type of survey validity refers to representativeness. In the student survey, for example, let’s say you conclude that the average level of engagement is a 7. But, how many students responded? If there’s 300 students, and only 10 responded, statistical validity is poor. To improve statistical validity, more students must participate.

4. Construct Validity

This looks at survey design, e.g. the wording of the questions and the flow of questions. It also seeks to eliminate survey bias and leading or loaded questions, the former which, intentionally or unintentionally, persuades the participant to answer in a particular way, the latter which asks multiple questions within one question.

5. Face Validity

As the name implies, this form of validity takes your questions at face value: Do the items in the survey look like they’ll measure what they’re intended to measure? No mathematical formulas, no in-depth hypothesizing, just a good old fashioned judgment call. If a researcher doesn’t trust their judgment, or wants a second opinion, a survey expert may be asked to conduct a face validity check.

6. Content Validity

Content validity helps to ensure you get the full picture when it comes to respondent answers. Consider a follow up question to the previous student engagement survey. If the student selects 5 or below, considered unengaged or very unengaged, using survey logic the questionnaire leads to this question:

See a problem? The researcher may not get a clear picture of student engagement as it leaves out other possibilities. For example, “personal issues” or “home life” could be leading to disengagement. Or, perhaps the classes are “too easy” or “too hard” rather than just “boring.” It’s always a good idea to think of all potential answers, and at the very least, consider an “other” bucket.

7. Criterion Validity

This form of validity looks at whether answers fit with behaviors. In our example, let’s say a student says they are highly engaged. The researcher then looks at the student’s history and sees that the student has straight As and is involved in athletics and drama club. This is evidence of criterion validity, as performing well in studies and participating in extracurricular activities would fit with high levels of engagement. If criterion validity doesn’t exist, the researcher then looks for the possible disconnect.

2 Types of Survey Reliability

Focused on survey consistency over time, there are just two types of survey reliability.

1. Test-Retest Reliability

To determine if a survey is reliable, researchers need to consider test-retest reliability. In other words, if 100 students today say they are highly engaged, and they were surveyed again in a month, would the results be similar? Sure, a student may pick up or lose interest in their studies or activities over time, but for the most part, it is reasonable to expect most students’ engagement to be steady.

2. Internal Reliability

For internal reliability, once again looking at our example, researchers need to examine whether students who tend to be disengaged also tend to be negative in response to questions about classes, teachers, and activities. The research would also look to see if highly engaged students are generally positive about some of all of these items.

Conclusion

To be sure a survey provides accurate data, researchers should always consider both survey reliability and survey validity. Sometimes, hard analysis is needed; other times, an educated guess is acceptable. It’s all up to how valid and reliable you want your data to be. Ready to start surveying? SurveyLegend has you covered. Our online surveys are easy to use, not to mention easy on the eyes, with the ability to include survey images and picture questions. Begin your survey journey with SurveyLegend today!

Are you concerned with survey reliability and validity? How do you ensure your data is accurate? We’d love to hear from you in the comments!

Create Your FREE Survey, Poll, or Questionnaire now!

Frequently Asked Questions (FAQs)