Data Science and Machine Learning 101: Model Deployment and MLOps AB Testing and Online Experimentation for ML Models

What This Is

A/B testing (also called online experimentation) is a controlled, randomized comparison of two (or more) variants of a product, model, or decision rule. In a data‑science workflow it lets you measure the causal impact of a new ML model or feature on a live metric (e.g., conversion, churn, click‑through). For example, an e‑commerce site may serve a new churn‑prediction model to 10 % of users (variant B) while the existing model continues for the remaining 90 % (variant A) and then compare the resulting retention rates.

Key Terms & Formulas

• Treatment / Control – The treatment (variant B) receives the new model/feature; the control (variant A) keeps the status‑quo.
• Randomization – Assign users to treatment or control uniformly at random; guarantees unbiased estimates of the treatment effect.
• Lift (Δ) – Absolute change in a metric: Δ = ??????_B – ??????_A.
• Relative Lift – Percentage change: ????? = Δ / ??????_A.
• Statistical Power (1‑β) – Probability of detecting a true effect; higher power → larger sample size or larger effect size.
• Sample Size Formula (two‑sided test):

[ n = \frac{2\,\sigma^2\,(Z_{1-\alpha/2}+Z_{1-\beta})^2}{\Delta^2} ]

where σ² = variance of the metric, Δ = minimum detectable lift, Z‑scores from the normal distribution.
- p‑value – Probability of observing a lift at least as extreme as the one measured, assuming the null hypothesis (no lift) is true.
- Confidence Interval (CI) – Range that contains the true lift with a chosen confidence (e.g., 95 % CI = Δ̂ ± Z_{1‑α/2}·SE).
- Sequential Testing / Peeking – Re‑evaluating results before the pre‑planned sample size is reached; inflates Type I error unless corrected (e.g., using O’Brien‑Fleming boundaries).
- Uplift Modeling – Predictive approach that estimates the individual treatment effect (ITE) rather than the average lift; often built with two‑model or meta‑learner strategies.
- A/B Test Metric Types – Binary (conversion, churn), Rate (click‑through rate), Continuous (revenue per user), Time‑to‑event (survival). Choose the appropriate statistical test (e.g., chi‑square for binary, t‑test for continuous, log‑rank for survival).
- Multiple Testing Correction – When running many experiments, control the family‑wise error rate (Bonferroni) or false discovery rate (Benjamini‑Hochberg).

Step‑by‑Step / Process Flow

1. Define the business goal & metric
   python
# Example: increase 30‑day retention
metric = 'retention_30d' # binary outcome
2. Design the experiment
3. Choose treatment (new model) and control (current model).
4. Randomly split users: np.random.rand(N) < 0.1 → treatment.
5. Pre‑compute required sample size using the formula above (or statsmodels.stats.power.tt_ind_solve_power).
6. Implement the online allocation (e.g., via a feature flag service).
   python
if random() < 0.1:
model = new_model # B
else:
model = baseline # A
7. Collect data
8. Log user‑level outcomes (user_id, variant, metric, timestamp).
9. Store in a data warehouse (e.g., Snowflake) for later analysis.
10. Analyze results
    python
import pandas as pd, statsmodels.api as sm
df = pd.read_sql('SELECT * FROM ab_log', con)
lift = df.groupby('variant')[metric].mean().diff().iloc[-1]
se = sm.stats.proportion.proportion_effectsize(...)
ci = sm.stats.proportion_confint(...)
p = sm.stats.proportions_ztest(...)
11. Compute Δ, CI, p‑value, and power.
12. Decision & rollout
13. If lift is statistically and practically significant, promote the new model; otherwise, iterate.

Common Mistakes

• Mistake: Using the same users for multiple experiments (overlap).
  Correction: Enforce mutual exclusivity or use hierarchical randomization; overlapping users break the independence assumption and bias lift estimates.

• Mistake: Stopping the test early because the p‑value looks “good”.
  Correction: Pre‑specify a stopping rule (e.g., O’Brien‑Fleming) or use a Bayesian monitoring approach; otherwise you inflate Type I error.

• Mistake: Ignoring variance heterogeneity (treating binary and continuous metrics the same).
  Correction: Choose the right statistical test (chi‑square for binary, Welch’s t‑test for unequal variances, or non‑parametric bootstrap).

• Mistake: Relying on a single metric while other key KPIs deteriorate.
  Correction: Track a dashboard of primary and secondary metrics; a “win” on one metric may be a “loss” on revenue or user experience.

• Mistake: Not accounting for multiple comparisons when running many experiments in parallel.
  Correction: Apply Bonferroni or Benjamini‑Hochberg adjustments to keep the overall false‑positive rate under control.

Data Science Interview / Practical Insights

1. “Explain the difference between an A/B test and an uplift model.” – Interviewers expect you to discuss average treatment effect vs. individualized ITE estimation, and when each is appropriate.
2. “How do you decide the sample size for a binary metric?” – Mention the two‑proportion power calculation, required lift, baseline conversion rate, and the use of statsmodels.stats.power.NormalIndPower.
3. “What are the risks of peeking, and how can you mitigate them?” – Talk about inflated α, sequential analysis methods, and the need for pre‑registered analysis plans.
4. “When would you use a Bayesian A/B test instead of a frequentist one?” – Discuss scenarios with sparse data, the desire for a posterior probability of lift > 0, and the ability to stop early based on credible intervals.

Quick Check Questions

1. Scenario: Your baseline churn model yields a 5 % churn rate. The new model (treatment) shows a 4.8 % churn rate with a p‑value = 0.12.
   Answer: Do not roll out. The lift (0.2 %) is not statistically significant (p > 0.05) and may be due to random variation.

2. Scenario: You need 80 % power to detect a 2 % absolute lift in a conversion rate of 10 % with α = 0.05. Roughly how many users per variant are required?
   Answer: ≈ 30 k users per arm (use the two‑proportion sample‑size formula).

3. Scenario: After 3 days you see a p‑value of 0.03 and stop the test.
   Answer: Wrong. Early stopping inflates Type I error; you need a pre‑planned sequential test or adjust α for interim looks.

Last‑Minute Cram Sheet (10 one‑liners)

1. Lift = metric_B – metric_A (absolute) and Relative Lift = Δ / metric_A.
2. Two‑sided sample size: (n = 2σ²(Z_{1‑α/2}+Z_{1‑β})²/Δ²).
3. Binary metric test: Use Chi‑square or Z‑test for proportions; continuous → t‑test (Welch if variances differ).
4. Power ↑ by increasing sample size, effect size, or α; ↓ by higher variance.
5. p‑value < 0.05 ≠ “practical significance”; always check effect size and business impact.
6. Sequential testing requires α‑spending (e.g., O’Brien‑Fleming) or Bayesian monitoring.
7. Uplift modeling = predict individual treatment effect; useful when heterogeneity is high.
8. Multiple experiments → apply Bonferroni (α/​k) or Benjamini‑Hochberg to control FDR.
9. ⚠️ Peeking without correction dramatically raises false‑positive risk; never do it in production.
10. Confidence interval = Δ̂ ± Z_{1‑α/2}·SE; if CI includes 0, the lift is not statistically distinguishable from zero.