In the book “Growth Engineering” growth is described as an integrated system: product, data, code, experiments and operational responsibility. This article takes that perspective back with a practical approach: a test is only valid if it measures people actually exposed to the possibility of change.
Include users who have never seen the variant dilutes the effect. The result appears neutral even when the intervention works for those who meet it.
Real problem
Measuring all users in a checkout test is like evaluating a road sign by asking even those who have not passed from that road. The problem is that the assignment to the variant is confused with the actual exposure.
Conceptual model
A good growth system reduces the cost of uncertainty, does not promise certainties. It needs a mechanism that makes the transition from signal to decision faster, linking each modification to an observable hypothesis and transformable in choice: release, iterate, stop or deepen.
Strict formalisation
The basic operating scheme shall be:
- eligible population 2. trigger event 3. variant assignment 4. recorded exposure 5. analysis of correct population
Simplicity is essential. If the flow does not stand in five or six clear steps, the team risks automating a process that has not yet understood.
Example or case study
A test on the error message in the payment form must trigger only those who arrive at that form, not all visitors to the site. Only in this way you avoid dilute the effect and you get meaningful learning.
Lab / exercise
Basic level: Identify in an existing experiment which event represents the actual trigger and verify whether all the users analysed were exposed.
Intermediate Level: Design a data collection system that clearly separates the assignment from the exposure, adding an “experiment_exposed” event.
Research-grade level: Analyze a complex experiment with multiple segments and channels, ensuring that the analysis only considers the population actually exposed and assessing the impact of any dilutions.
Datasets and recommended materials: Dataset of A/B experiments with detailed event logs, documentation on exposure metrics and trigger rate.
Typical error to avoid
Confounding assignment and exposure. Being in group B does not mean seeing B. This error generates activities and graphs, but does not improve the quality of the learning cycle.
Quiz or checkpoint
- Why is it important to distinguish between assignment and exposure in a test?, What are the key steps to ensure that an experiment measures only those who could be affected?, Which metrics help to monitor the quality of exposure in a test?