In 2012, the Obama For America team was faced with a delicate decision: which subject line to use for an email sent to 4 million subscribers to raise funds. Instead of relying on instinct, they conducted 18 A/B tests on the subject line. The winner, “I will be outspent,” generated 2.67 million dollars more than the initial variant. A clear example of how a rigorous test can translate into millions in additional revenue.
However, many companies with lower resources replicate tests with too small samples, obtaining apparently significant results (p-value < 0.05 ) in a few days, and then discover that those results do not translate into any real benefit. The difference is not in the tools or budget, but in the deep understanding of the statistical limitations of the A/B tests.
Error 1: sample too small
Ron Kohavi, former head of Microsoft’s trial team, estimates that only one third of the A/B tests in large companies produce reliable results, a percentage that falls below 10% in smaller companies. The main reason is the insufficient sample size, which makes the results indistinguishable from the case due to the natural variability of human behavior.
For example, a difference between 10 conversions out of 100 visitors (10%) and 12 out of 100 (12%) may seem a 20% improvement, but with such a small sample it is probably just noise.
Calculate the size of the sample needed
To detect a 5% increase on a 3% base conversion rate, it takes about 52,000 visitors per variant with an 80% statistical power and a 95% significance level. This means that a site with 1,000 visitors per day should run the test for 52 days to have reliable results.
| CR Base | Relative Lift | Sample Size for Variant | Days (1,000 vis/day) |
|---|---|---|---|
| 1% | 10% | ~150.000 | 150 |
| 3% | 10% | ~52.000 | 52 |
| 5% | 10% | ~30.000 | 30 |
Python for sample size calculation
For those who want a precise calculation, here is a method based on the Z-test by proportions, which allows to estimate the required sample:
- Define the base conversion rate and the minimum lift to be detected., Set the level of significance (α) and power (1-β)., Calculate the sample size per variant.
If traffic is limited, it is better to focus on tests with higher lift expectations, otherwise the test may last months without any useful results.
Error 2: peeking, or looking at data before time
Checking the results of a test several times before the planned end dramatically increases the probability of false positives, from 5% theoretical to 40% real. This is because each early check is equivalent to performing multiple tests without correcting the level of significance.
How to avoid peeking
- Decide the duration of the test in advance and keep it., If you work with large volumes, evaluate sequential tests with always valid p-values, which allow you to stop the test in advance only when the evidence is overwhelming.
Error 3: contemporary multiple test
Testing multiple variants simultaneously without correcting the level of significance increases the probability of false positives. For example, with 5 variants the probability of at least one false positive is about 18.5%.
Statistical corrections
- Bonferroni: divides α by the number of comparisons, but is very conservative., Benjamin-Hochberg: controls the rate of false positive expected (FDR), less conservative and more suitable for many tests.
Error 4: ignore seasonality and weekly cycles
Many businesses have different behaviors between weekdays and weekends. Conducting a test only on weekdays can lead to incorrect conclusions, because it optimizes for one segment of users ignoring the other.
Example case
An e-commerce clothing has a conversion rate twice over the weekend compared to weekdays, with different purchasing behaviours. A variant that adds urgency can be negative in weekdays but positive over the weekend.
Rule
Turn each test for at least two full weekly cycles (2-4 weeks) to capture these changes.
Error 5: optimize proxy metrics and not real result
Optimize surrogate metrics like clicking on a button may not result in increased revenue or actual conversions. You need to choose primary metrics that reflect the real economic impact.
| Level | Metrica | Use |
|---|---|---|
| Primary | Revenue, Conversions | Direct impact measurement |
| Secondary | Add-to-cart, Inscriptions | Reasonable Proxy |
| Guardrail | Bounce rate, Error rate | Monitoring to avoid damage |
| Surrogate | CTR, Scroll Depth | Hypothesis, not final decisions |
Error 6: ignore the A/A test
Before launching an A/B test, an A/A test (two identical versions) is used to verify that the testing system does not have bias or randomization problems. A significant p-value in an A/A test indicates problems to be corrected before proceeding.
Error 7: confuse statistical significance with practical importance
A statistically significant result can have a negligible economic impact. Defining the Minimum Detectable Effect (MDE) in economic terms before the test helps avoid investment in tests with too low returns.
Error 8: new effect
New variants can show a temporary improvement due to the interest of users in the novelty, which disappears over time. Segregating the analysis for new and returning users and waiting at least 3-4 weeks helps to distinguish the real effect from the novelty phenomenon.
Error 9: ignoring guardrail metrics
Monitoring metrics such as error rates, bounce rates and loading times during the test is essential to avoid introducing regressions that damage the user experience or the performance of the site.
Pre-test checklist
Before you launch a test, answer these questions in writing:
- What is the clear hypothesis? 2. What is the primary metric? 3. What is the economically relevant MDE? 4. What is the sample size and the duration of the test? 5. How will you segment the data? 6. What metrics will you monitor? 7. What decision criterion will you adopt (p-value)?
Never change the threshold of significance after seeing the data.
Prioritize testing with ICE score
Evaluate every test on Impact, Confidence and Ease, and choose the ones with the highest score to optimize resources and results.
The point
The discipline of making decisions under uncertainty requires patience, rigour and statistical understanding. Obama’s tests worked because they respected these rules: adequate samples, no peeking, just metrics, and rigorous documentation. Without this discipline, tests risk providing false security and directing the business in the wrong direction.
Investing time in the design and analysis of an A/B test allows you to avoid unnecessary optimization months and make really data-based decisions.