Survey design is a linguistically intense undertaking. Every respondent should get the same meaning from your questions—and it should be the meaning you had in mind when you wrote the questions. A client wanted to know about feelings of security in different neighborhoods, and wrote the question “How safe do you feel in the area where you live?” In pre-tests, we quickly learned that an area is not an area is not an area! Some respondent quotes:
“I feel extremely safe in my building and on my block, but not on the next block over.”
“I feel somewhat safe in this town. There is some crime, but not really in my neighborhood.”
So what can you do to standardize question meaning? A few things:
- Pretest! Without pretests, my client would have plowed forward with “area—“ and respondents would have understood it in all different ways. Ideally, pre-tests are conducted by a trained interviewer with a number of respondents to test for consistency of understanding.
- Expert interviews: Your expert could be your mom, a dentist, or your fifth-grade neighbor—as long as they’re typical of the people you want to take your survey. Do they understand the questions? Are these the words they would use to talk about these ideas? Even a quick conversation could shed some light on potential problems with your wording.
- Test the grade level of your questions with a tool like the Flesch Reading Ease test: http://www.readabilityformulas.com/flesch-reading-ease-readability-formula.php (keep in mind that the average reading level in the US is around 8th grade, and one in five Americans reads at or below a 5th grade level).