MODULE 17

Statistical Significance, A/B Testing, and Experimentation Science

Take experimentation to a serious level with causal design, power planning, variance reduction, and governance.

Foundations of statistical significance

Null and alternative hypotheses, sampling distribution, test statistic, and p-value. From Ronald Fisher's tea lesson (1925) to numerical A/A testing, including limitations of frequentist testing and operational validation templates.

Prerequisites: 1

Go to lesson

Business Hypotheses and Experimental Questions - official lesson image on GinnyTech, created by AD

Advanced 18 min

Well-Formulated Causal Questions and Business Hypotheses

Well-Formulated Causal Questions and Business Hypotheses. Core lesson of the Statistical Significance, A/B Testing, and Experimentation Science module with real problem, conceptual model, rigorous formalization, applied case, 3-level lab, and final checkpoint.

Prerequisites: 1

Go to lesson

Experimental Design, Randomization, and Metrics - official lesson image on GinnyTech, created by AD

Advanced 18 min

Experimental design, randomization, and unit of analysis

Experimental design, randomization, and unit of analysis. Core lesson of the module Statistical Significance, A/B Testing, and Experimentation Science with a real problem, conceptual model, rigorous formalization, applied case, 3-level lab, and final checkpoint.

Prerequisites: 1

Go to lesson

A/A Test and System Reliability - official lesson image on GinnyTech, created by AD

Advanced 18 min

A/A testing and measurement system validation

A/A testing and measurement system validation. Core lesson of the Statistical Significance, A/B Testing, and Experimentation Science module with real problem, conceptual model, rigorous formalization, applied case, 3-level lab, and final checkpoint.

Prerequisites: 1

Go to lesson

A/B Test: From Variant to Decision - official lesson image on GinnyTech, created by AD

Advanced 18 min

Complete flow of a professional A/B test

Complete flow of a professional A/B test. Core lesson of the Statistical Significance, A/B Testing, and Experimentation Science module with a real problem, conceptual model, rigorous formalization, applied case, 3-level lab, and final checkpoint.

Prerequisites: 1

Go to lesson

Intermediate 18 min

P-value, Errors, and Correct Interpretation

What the p-value really measures and how to read it together with effect size, intervals, power, and business decision.

Prerequisites: 1

Go to lesson

Sample Size, Power, MDE, and Duration - official lesson image on GinnyTech, created by AD

Advanced 18 min

CUPED and variance reduction

CUPED and variance reduction. Core lesson of the Statistical Significance, A/B Testing, and Experimentation Science module with a real problem, conceptual model, rigorous formalization, applied case, 3-level lab, and final checkpoint.

Prerequisites: 1

Go to lesson

Peeking, Multiple Testing, and Experiment Governance - official lesson image on GinnyTech, created by AD

Advanced 18 min

Peeking, multiple testing and sequential testing

Peeking, multiple testing, and sequential testing. Core lesson of the Statistical Significance, A/B Testing, and Experimentation Science module with a real problem, conceptual model, rigorous formalization, applied case, 3-level lab, and final checkpoint.

Prerequisites: 1

Go to lesson

Case Study: Redoing Checkout Without Being Misled - official lesson image on GinnyTech, created by AD

Advanced 22 min

Bayesian A/B, switchback test, and geo-test

Bayesian A/B, switchback test, and geo-test. Core lesson of the Statistical Significance, A/B Testing, and Experimentation Science module with real problem, conceptual model, rigorous formalization, applied case, 3-level lab, and final checkpoint.

Prerequisites: 1

Go to lesson

Cheat Sheet: Statistical Significance and A/B Testing - official lesson image on GinnyTech, created by AD

Advanced 28 min

Experimental governance + end-to-end case study

Experimental governance + end-to-end case study. Core lesson of the Statistical Significance, A/B Testing, and Experimentation Science module with real problem, conceptual model, rigorous formalization, applied case, 3-level lab, and final checkpoint.

Prerequisites: 1

Go to lesson

Module connections

Continue the path with these modules

Natural prerequisiteAnalytics Engineering with dbt and Semantic Layer13 lessons

Transform raw data into reliable, reusable, and governed models with layering, tests, contracts, and semantic layer.

Same stage of the pathMathematics for Data Analysis10 lessons

Introduces the mathematical rigor that makes models, inference, forecasting, and applied regressions more readable.

Same stage of the pathAdvanced SQL for Analytical Systems14 lessons

Use SQL as a reasoning language, not just extraction, with patterns for cohorts, funnels, experiments, and performance.

Previous moduleCourse Overview and Study Method for Data Work8 lessons

Transform the course from catalog to cognitive system, with study method, technical memory, and professional output plan.

Previous moduleData Collection & Tracking Systems8 lessons

Move from isolated tags and tools to reliable, governed, and observable measurement systems.

Previous moduleMetrics, KPI Trees, and Analytical Foundations6 lessons

Learn to read numbers as systems, with KPI trees, unit economics, correct baselines, and causal signal interpretation.