I’m always surprised (and frankly, a little alarmed) when clients don’t want to pre-test a questionnaire. This strikes me as a little bit like launching an ad campaign without testing the content. Maybe scarier: you could make all kinds of decisions based on data that came from questions that your respondents didn’t understand or couldn’t answer.
Pre-testing can take many forms, but my favorite is cognitive interviewing, where a respondent completes a questionnaire and then goes through it with an interviewer. In this “think aloud” exercise, I’ll ask respondents things like: What did this question mean to you? How did you decide what to answer? How did you come up with that number?
Pre-testing is almost always eye-opening. Not only can we see what words or questions need to be revised, but respondents often mention things that we hadn’t thought of including in the questionnaire. I recently tested the following question among college students:
Which of the following was the most important factor when you were selecting a college?
Price/availability of financial aid
- Availability of classes I wanted to take
- Availability of extracurricular activities I wanted to be involved in
- Location
- Availability of a social scene I am interested in (e.g. Greek life)
- Opinions of my family members
Several pre-test respondents mentioned that the opinions of their family members about where they went to school were less important than the opinions of their peers—an option we hadn’t considered, but quickly added.
Three important things to remember about pre-testing:
- Pre-test with the audience who will be answering the questionnaire! Don’t use your highly educated colleagues to test a survey that will be going to the general public.
- A pre-test will take time. A cognitive interview for a 10-minute questionnaire could easily take an hour.
- Be ready and willing to make changes based on the results of a pre-test.
