Different types of questions & responses

Close-ended vs. open-ended questions

Sometimes you need answers that are easy to quantify. In such cases, you can easily go ahead and create a question with several predefined choices. Examples of questions with predefined answers that you can make with SurveyLegend are single selection or multiple selection. Results from such questions are really easy to illustrate, and very easy to understand.

Example:

When presenting the statistics for a question like the example above, you can easily illustrate that e.g. 70% of male participants liked brand X or Y. Such data is easier to use when making  decisions, and easier to share.

However, pros recommend that in a multiple choice question, the choices should cover all possible answers. Sometimes, this will mean including an option for “Other,” or “Don’t know,” or even “Don’t wish to say” for sensitive questions.

Not only will this get you more accurate data, but it builds trust. If respondents feel you’re trying to make them give you an answer they don’t agree with, they may just skip the question, or stop answering questions altogether.

When you give the chance to participants to provide you with their own choices (which are not included in your predefined choices), you are making an open-ended question.

Example:

This gives participants some freedom to express and share more deliberate answers.

Example:

Limit the respondents to single selection

Sometimes you need to give your respondents the flexibility to answer by making more than a single choice. Although SurveyLegend’s multiple selection is a very useful survey field, it can be tricky to analyze the results you collect with it. If your participant chooses more than one of the answers, does that mean they are equally important?

Try to think whether you really need them to choose the answer that matters the most,, or just want to be nice and give them more choices and more flexibility?

It’s not wrong to use multiple selections, but it’s harder in most cases to evaluate the results.

Make sure answers don’t overlap

When using single selection questions, choices should be mutually exclusive and should not overlap. For example, if a question asks:

“Which kind of food is your favorite?”, the answers shouldn’t include both Indian food and vegetarian food, because there might be Indian foods which are vegetarian too. If someone’s favorite food is one of those foods, which response is appropriate?

For scaled questions, use words not numbers

People think with words, not numbers. Although numbers in the analyzed survey data (e.g. average) can make abstract meanings easier to grasp for survey creators, they are hard for survey participants to relate to, when answering.

Let’s say you want to ask them:

“On a scale of one to seven, with 1 being extremely likely, and 7 being not likely at all, how likely are you to pay for our product again?”

First of all, you just made an unnecessarily loooong question for no reason. Then, you expect your respondents to answer while remembering all the abstract things you mentioned and calculate some number in their head that is perhaps close to what they think; and after that, answer three or four or whatever. You make so much cognitive work for them and don’t know if they will write a number that really represents what they are thinking.

Instead, you can easily verbalize the answers for them, put them in a scaled order, and then quickly convert the collected data to numbers in your final calculations.

Example:

Unipolar vs. bipolar questions

Speaking of questions with scaled answers, how many choices/answers should you give the participants? Survey methodologists have come to the conclusion that:

For unipolar questions

When you ask a unipolar question that asks the respondents to rate only one aspect of something, a common 5-point scale gives the most reliable data.

Example: Unipolar question with 5-points

For bipolar questions

When you ask a bipolar question, you are asking the respondent to describe their opinions or feelings based on two opposite notions, for example “Liking” ↔ “Disliking” something. Therefore, it’s good to add a middle choice too, being “Neutral”. So the scale will be something like this:

Like ↤ Neutral ↦ Dislike

Then you can decide to include 1, 2, or 3 options on each side on the “Neutral” choice. So, your final bipolar question will have 5-points scale or 7-points scale.

In this case, it makes sense to think about having odd-numbers of choices, but just keep in mind that giving too many choices to participants might make them confused.

Example: Bipolar question with 5-points

Use Yes/No questions carefully

If you ask your survey participants: “Do you like to play computer games?” and give them only two choices of “Yes” and “No”, what do you expect them to say? Sometimes they might have time and interest, and during some periods they might not have the time for it. Or they might like some specific sorts of computer games, but not all…

By limiting your respondents to only two choices, you ask them to see the subject as black and white. But if your aim is to analyze how people really think within the spectrum from black to white, then this is not a well thought-out question.

So, sometimes it’s better to give them more choices.

Example:

Measuring quantities

When you want to measure how much your survey respondents spend time, money, or energy on something, you can use a scaled question with 5 to 7 choices. If you believe your respondents probably can’t give you exact numbers, just use words like in the example below:

Example: Verbal scale

However, if you think it is possible for them to answer with more exact numbers, and the numbers are important to you, go ahead and make some numeric intervals. It mostly depends on your data needs.

Example: Numeric intervals

Beware when making ranges

Remember not to make overlapping ranges. This makes people confused when they are trying to chose the right answer.

For example lets say you ask them:  “About how long do you play games in a typical day?” and give them these choices: A) 1 hour or less, B) 1 to 2 hours, C) 2 to 3 hours, D) more than 3 hours. Which one should a person choose if he/she plays 2 hours per day? Choice B or C?

Example:

Therefore, you should try to keep your ranges roughly equal in size and make sure they do not overlap one another.

Include a way out

Let’s say you ask a question about your respondents’ favorite flavor of ice cream. You may include some flavors in the options that you provide, but will never be able to include all flavors. In such cases, it’s better not to force the participants to choose only from your choices. Try to include an “Other” option.

You can even let them write down their own favorite flavors, with an open-ended question and collect the data for further analysis. There may be very interesting data in such responses.

Example: