A survey can look flawless on paper and still lead you to the wrong decision. The questions were clear, the form worked, the spreadsheet filled up. Then someone averages a few columns, filters by segment, and a story appears. Too often that story comes from the analysis rather than from the world. Below are seven mistakes that surface again and again in survey work, with the practical habits that prevent them.
1. Analyzing the people who answered instead of the people you asked
A customer survey with a 9% response rate tells you a lot about those 9% and very little about everyone else — yet the numbers get reported as if they cover the whole base. People with strong opinions respond more often, and the direction of that bias depends on how you asked. A feedback link inside a confirmation email reaches satisfied buyers; a phone call to lapsed accounts reaches people with a complaint ready.
Coverage matters too. If you interview people through a mobile app, you have excluded everyone who never installed it, which is precisely the group you may care about. Before trusting a percentage, ask who could not possibly have appeared in your data. Then check your respondent profile against something you know independently — age, region, tenure, spend band — and either weight the results to match or state plainly who is overrepresented.
2. Asking questions that push people toward an answer
Wording quietly decides results. Consider the difference between "How much did you enjoy our streamlined checkout?" and "How would you rate the checkout process?" The first assumes enjoyment and tells the respondent what the right answer is.
Is our support team fast and friendly?
That question is double-barreled: someone who got a fast reply but a curt tone cannot answer it honestly. The fix is one idea per question. Common traps to watch for:
- Leading framing — "Don't you agree that…" invites agreement rather than measurement.
- Loaded language — "the proposed job-killing tax" tells people what to think.
- Double-barreled items — two judgments crammed into one answer option.
- Vague frequency words — "often" means weekly to one respondent and daily to another.
- No honest escape route — without a "don't know" or "not applicable" option, people guess.
Pilot the questionnaire with five people and ask them to explain each question in their own words. You will hear the ambiguity immediately.
3. Averaging numbers that aren't really numbers
Satisfaction scales are ordinal: the gap between "1" and "2" isn't necessarily the same size as the gap between "4" and "5". Reporting that "the average customer is 2.4 satisfied" sounds precise and means almost nothing. Use distributions, medians, or the share choosing the top two boxes. Means are still useful for tracking the identical question over time across waves, but they make poor stand-alone headlines.
Two more traps sit here. First, changing your scale anchors — a 1–5 scale behaves differently from a 0–4 scale, so never compare scores across versions. Second, only combine items into an index after checking they move together and measure the same underlying idea.
4. Ignoring how the sample was actually drawn
Most real surveys are not simple random samples. They are clustered by store, school, or account manager; stratified by region; or controlled by quotas. That design affects how much uncertainty you really have. If your sample is 12 stores with 40 customers each, you have roughly 12 independent pieces of information about store-level variation, not 480. Survey-weighted analysis handles this properly, and the design effect tells you how much your confidence intervals should widen. Software defaults that assume simple random sampling will make your results look sharper than they are.
5. Cutting the data until something shows up
Twenty subgroups multiplied by ten questions produces two hundred comparisons. At a conventional significance threshold, several will look "significant" purely by chance, and the most dramatic one will end up in the slide deck. Decide which comparisons matter before you look at results — your primary segments, your primary outcomes — and treat everything else as exploration. When you do report a subgroup finding, show the subgroup's size and its confidence interval, and describe it as a lead worth testing rather than a conclusion.
6. Letting messy responses through
Raw survey files need cleaning, and skipping that step quietly manufactures findings. Watch for straight-lining (identical answers down a long grid), speeding, duplicate submissions, and contradictions — someone who reports using a product they earlier said they had never heard of. Flag these responses, decide a rule for excluding them, and document it. The same goes for open-text answers: if one analyst codes them one way and another codes them differently, your percentages move. Write a short codebook with examples, and have a second person code a sample to check agreement.
7. Reading a cause into a snapshot
"Customers who enabled alerts are more loyal" is a correlation. It may also be that committed customers are the ones who enable alerts. Survey data recorded at a single moment rarely supports causal language, especially when participation was voluntary. Choose verbs carefully: "associated with", "more likely to report", "correlates with". If you need a causal claim, you need a design built for it — an experiment, a controlled rollout, or panel data with the right adjustment variables — not a stronger sentence.
Before you publish: a five-minute checklist
- State the response rate and describe who is missing from the sample.
- Paste the exact question wording next to every number you report.
- Check that the summary statistic matches the scale — medians and shares for ratings, means for counts.
- Confirm the confidence intervals account for weighting or clustering.
- Mark each causal claim and either soften it or point to a design that earns it.
None of this is exotic. It is mostly the discipline of asking what your data can and cannot support before someone builds a decision on top of it.
Photo: athree23 / Pixabay
