In 2011 AIrbnb faced a common problem: the conversion rate was low, but quantitative data did not explAIn the reason. They had Google Analytics, heatmap, funnel, but no clear answer. They then decided to send a survey to users who had visited an apartment card without booking, with a simple question: “What prevented you from booking?”
The answers revealed invisible problems to behavioral data: poor photos, price confusion, few reliable reviews. AIrbnb intervened on these aspects and significantly improved the results. This example explAIns why, despite the era of big data, online surveys remain an irreplaceable tool: the data tell what people do, but only the words explAIn why.
However, many online surveys suffer from statistical errors: hidden biases, too small samples, ambiguous questions. The result is apparently valid but actually misleading data.
Basics of sampling: probabilistic vs non-probabilistic
In order to obtAIn generalizable results, it is serious to understand sampling.
Probabilistic sampling
Here each individual has a known probability and nothing to be selected. The main techniques are:
- Simple Random Champion: each person has the same probability. It is the strictest but difficult to apply online., Strategic Champion: the population is divided into layers (e.g. age, gender) and randomly sample by each, ensuring representativeness., Systematic Champion: you select each K-th unit from an ordered list. Simple to implement, valid if there are no hidden patterns.
These methods make it possible to generalize the results with calculable error margins, e.g.: ” 58% of customers prefer the X method with ±5% error margin and 95% confidence.”
Non-probabilistic sampling
It is based on the convenience or willingness of the participants, and includes:
- Convenience: interview who is available, as customers via emAIl. Quick but not representative., Quota: preset numbers are reached by categories, but without randomization., Snowball: respondents recruit others, useful for hidden communities but strongly biased.
The data describe the sample, not the population. In Italy, most online surveys use this approach, limiting generalization.
Calculation of sample size
The key question is: how many people are interviewing for reliable results?
The standard formula for probabilistic sampling is:
n = (Z2 × p × (1-p)) / E2
Where:, Z = critical value for confidence level (e.g. 1.96 by 95%), p = estimated proportion (0.5 if unknown, for maximum variability), E = desired margin of error (e.g. 0.05 by ±5%)
Example: NPS survey
For an e-commerce with 50,000 customers, without preliminary estimates, with 95% confidence and ±5% margin of error, about 400 responses are needed.
If you want a narrower margin (±3%), you need about 1,100 responses.
Reference table
| Error Margin | Sample Size | Typical Use |
|---|---|---|
| ±10% | 96 | Exploratory surveys, limited budget |
| ±5% | 384 | Industry standard |
| ±3% | 1.067 | Policy decisions |
| ±2% | 2.401 | National Benchmarks |
| ±1% | 9.604 | High Precision Search |
Scales of measurement and encoding of applications
The choice of scale influences the possible analysis.
- Nominal: categories without order (e.g. gender). Quotes and chi-square tests are calculated., Ordinal: categories ordered but not uniform distances (e.g. satisfaction from 1 to 5). Median and non-parametric tests such as Spearman are used., Intervals: identical order and distances, but arbitrary zero (e.g. NPS scale 0-10). You can calculate aveRAGes and use parametric tests., Report: order, equal distances and natural zero (e.g. expenditure in euro). All statistical analyses are valid.
Special stAIrs such as the Semantic Differential and the Analog Visual Scale (VAS) measure perceptions and intensity in a more nuanced way.
Five common mistakes to avoid
- Double questions: asking for two things in one makes the answer ambiguous. 2. Suggestive questions: formulations that address the answer. 3. Social desirability bis: responses influenced by what is thought to be socially acceptable. 4. Ambiguity in the options: vague terms such as “recently” or “frequently” without precise definitions. 5. Too many mandatory or open questions: increase the abandonment of the survey.
Statistical analysis: chi-square and spearman correlation
-
Chi-Quadro: Verifies relationships between categorical variables. If the p-value is below 0.05, the relationship is significant.
-
Spearman: measure correlations between ordinal variables. Values over 0.7 indicate strong correlation.
Case study: customer satisfaction survey for e-commerce
An online fashion company wants to understand why customers leave the checkout.
- Defines the objective: to identify friction points. 2. Population: 12,500 customers who have visited the checkout in the last 3 months. 3. Sample size: about 400 responses for ±5% margin of error. 4. Survey structure with screening questions, open, quantitative and demographic. 5. Target sending to customers who have abandoned the cart, 24-48 hours later. 6. Analysis with frequencies, chi-square tests and Spearman correlations to identify intervention priorities.
Speciality of the italian market
- Use of the formal “You” in B2B, “You” in B2C depending on the brand., Privacy sensitivity: mentioning GDPR increases confidence., Timing: Tuesday and Wednesday morning are the best times; August is to be avoided., Incentives: Amazon vouchers or aggregated reports increase the response rate., Length: maximum 5-7 minutes to avoid abandonment.
The point a well-designed survey takes time and attention, but it returns reliable data for conscious decisions. do not give in to the temptation to launch a survey immediately without testing and calibration. remember that the survey is the bridge between observed behavior and deep motivations. use it to truly understand your customers and improve your product or service.
Next time you read a survey that “shows” a preference, ask yourself: who answered? How was the question formulated? These answers can radically change your decisions.