Fourteen principles for questionnaires that produce usable data
Most surveys do not fail in the analysis. They fail in the wording, long before anyone has seen a number. These are the principles I keep coming back to, gathered in one place.
Before you write a single question
Start with the decision, not the question
Write down which decision the survey is meant to inform before you write anything else. If an answer cannot change what you actually do, you are collecting numbers that are nice to have rather than numbers you need. It is the most common reason reports get read once and never used.
Decide in advance what each answer will trigger
Go through the questions and write down: if the answer is high, what do we do? If it is low, what do we do? If you cannot answer that, the question probably does not belong in the questionnaire. This exercise typically removes a third of them.
Be clear about who you actually need to ask
A representative sample of the population is rarely the same as a representative sample of your customers, or of the people who make the decision. Choose the audience by who influences the outcome, not by who is easiest to reach.
The wording
One question at a time
"How satisfied are you with price and quality?" is impossible to answer precisely and impossible to analyse. The respondent has to merge two judgements into one number, and you never learn which one the answer was about. Split it in two.
Avoid leading wording
"How do you find our easy to use solution?" has already told the respondent what to think. Milder versions do the same: words like simple, fast and safe bleed into the answer. Describe, do not praise.
Ask about behaviour, not hypotheticals
"How likely are you to consider becoming a customer?" measures a hypothetical thought about a thought. "How likely are you to buy again?" measures an intention. "What did you do last time?" measures something that actually happened. The closer to real behaviour, the more the answer is worth.
Use the respondent's words, not yours
Internal terms, product names and industry jargon produce either guesswork or drop off. If you have to explain a word in order to ask the question, ask the question without the word.
Scales and answer options
Keep the scale visually linear
Breaking a one to six scale into a grid fights how we expect to read it. Instead of moving from low to high, the respondent has to hunt for the right number. That means slower answers, more misclicks and worse data, with nothing gained in return.
Make a deliberate choice about a neutral midpoint
A scale with an even number of points forces people to pick a side. Sometimes that is what you want. Often it just extracts an answer from someone who genuinely sits in the middle, and then you are measuring noise. Choose with your eyes open rather than by habit.
Give people a way out
Without a "don't know" or "not relevant", people guess. In the data file a guess looks exactly like an opinion. An honest "don't know" is often the most useful answer you can get, particularly when measuring awareness.
Be careful with smileys
Smileys make a scale quicker to read, but they are also emotional signals that shape how the options are interpreted. If you use them, use them consistently throughout, and do not switch between text and symbols partway.
The questionnaire as a whole
Remove friction at the start
Once someone has clicked through from an email, they should glide straight on. An extra "are you willing to answer a few more questions?" forces a fresh decision, and that is where you lose people you already had. Every extra choice costs you respondents.
Every extra question costs you data quality
Long questionnaires produce drop off, and carelessness among those who finish. The last questions in a long survey are almost always the worst data you hold. Be ruthless: short and precise beats long and thorough.
Read the data for response style, not just content
Some respondents agree with everything. In one company study we reanalysed, nearly half showed clear yea saying, against under seven percent in a national benchmark sample. Correct for it or you are measuring who is positive by nature rather than what they think. Look at the spread per respondent before you interpret any average.
The short version
A good questionnaire is not about asking many questions. It is about asking the right ones, and making it easy enough to answer honestly that people actually do.