In the world of data analysis, the biggest challenge is not to collect numbers, but to interpret correctly what those numbers tell. Often, impeccable technical analysis can lead to wrong conclusions because of reasoning errors that ignore the complexity of causality and intrinsic limitations of data.
Real problem
An emblematic example comes from Facebook in 2012: an internal study showed that users clicking on the “Like” buttons had higher retention rates. The conclusion seemed obvious: increasing the Like would improve the retention. Actually, causality was reversed: it was the most loyal users to use the Like. Optimizing the Like would not increase the retention.
This type of error is common and costs millions to companies. The discipline of making decisions under uncertainty requires to recognize and avoid these traps.
Conceptual model
The most frequent traps in the data analysis include:
- Confound correlation and causality, Survival bias, Simpson’s paradox, Sampling bias, Confirmation bias, Negligence of the base rate, Goodhart’s Law, Extrapolation bias, Anchoring bias, Recence bias
Each of these represents a way in which our interpretation can deviate from reality, leading to wrong decisions.
Strict formalisation
1. correlation vs causality
The correlation between two variables does not imply that one causes the other. Without an experimental design or a control group, causality cannot be affirmed.
How to avoid it: Use Randomized Controlled Trials (RCT) or, if impossible, techniques such as Difference-in-Differences to isolate the causal effect.
2. survival bias
Analyzing only those who “survive” a process can distort the perception of data. Examples include return planes with bullet holes and active customers without considering abandonment.
How to avoid it: Collect data also on those who have abandoned or are not present in the main dataset.
3. paradox of simpson
The aggregation of data may hide adverse effects in subgroups. An example is Berkeley’s university admission, where the aggregate data suggested discrimination, but faculty analysis showed otherwise.
How to avoid it: Always segment data for key dimensions before drawing conclusions.
4. sampling bias
A non-representative sample leads to erroneous conclusions. The 1936 Literary Digest survey is a classic example.
How to avoid it: Evaluate who is missing in the sample and whether the dataset reflects the real population.
5. confirmation bias
Look for data that confirm your hypothesis and ignore the opposite.
How to avoid it: actively seek data that can refute its conclusion.
6. base rate neglect
Ignoring the probability a priori can lead to incorrect interpretations of the results.
How to avoid it: Always compare with the historical baseline of the specific segment.
7. goodhart’s law
Optimizing a proxy metric can damage the real business if the metric does not reflect the final goal.
How to avoid it: Always monitor the main metrics together with the proxy ones.
8. extrapolation bias
Assume that the past preaches the future without considering changes in context.
How to avoid it: include scenarios analysis and not rely on linear extrapolations.
9. anchoring bias
The first issue presented influences the perception of subsequent data.
How to avoid it: Always provide the context and historical comparisons.
10. receipt bias
Overstimate the importance of recent events compared to a broader vision.
How to avoid it: Analyze data on time windows long enough to capture natural cycles.
Example or case study
The amazon case and latency
Amazon has proven with controlled experiments that every 100 milliseconds of latency increase costs 1% of sales. This is an example of proven causality experimentally, unlike many uncontrolled correlations.
Simpson parachute in A/B testing
A test shows that version B wins on desktop and mobile, but loses in total because traffic is distributed asymmetrically. Without segmentation, the conclusion would be wrong.
Sampling bias in satisfaction surveys
An example of Python code simulates how survey respondents are a non-representative subpopulation, overestimateing average satisfaction.
Lab / exercise
Basic level: Identify a correlation in a simple dataset and assess whether it can be causal or not.
** Intermediate level:** Perform a segmentation of data to verify the presence of Simpson’s paradox.
Research-grade level:* Designs a controlled experiment (RTT) to test the effect of a marketing campaign.
Seats and recommended materials: Datasets of real marketing campaigns, partial response survey data, examples of segmented A/B tests.
Typical error to avoid
Presenting aggregated results without segmenting data, ignoring the control group or not considering missing data are common errors leading to wrong decisions.
Quiz or checkpoint
What is the difference between correlation and causality?, What is survival bias and how does it manifest?, How does Simpson’s paradox occur in an A/B test?, Why is it important to consider the base rate?, How can you avoid confirmation bias?