Principles of Product Management: Statistical Thinking for PMs (Correlation vs Causation, Simpson’s Paradox, Sampling Bias)

Statistical Thinking for PMs (Correlation vs Causation, Simpson’s Paradox, Sampling Bias)

Statistical Thinking for PMs: Correlation vs. Causation, Simpson’s Paradox, Sampling Bias

What This Is

Statistical thinking helps PMs avoid costly mistakes when interpreting data to make product decisions. Misunderstanding correlation vs. causation, ignoring sampling bias, or falling for Simpson’s Paradox can lead to misguided feature launches, wasted engineering effort, or even product failures.
Example: A fintech app notices that users who enable push notifications have 2x higher retention. The team assumes notifications cause retention and doubles down on aggressive alerts—only to later discover that highly engaged users were simply more likely to opt into notifications in the first place (correlation ≠ causation). Proper statistical thinking would have led them to run an A/B test to isolate the true impact.

Key Terms & Frameworks

• Correlation (r): A statistical measure (-1 to +1) of how two variables move together. r = 0 means no linear relationship; r = 1 means perfect positive correlation.

• Example: "Users who complete onboarding in <2 mins have higher 7-day retention (r = 0.45)."

• Causation: When one variable directly influences another. Requires:

• Correlation (variables move together),
• Temporal precedence (cause happens before effect),

• No confounding variables (no hidden third factor).

• A/B Testing (Randomised Controlled Trial): The gold standard for proving causation. Randomly split users into control (no change) and treatment (new feature) groups to measure impact.

• Formula: Lift = (Treatment Metric – Control Metric) / Control Metric × 100%

• Simpson’s Paradox: A trend appears in different groups of data but disappears or reverses when the groups are combined.

• Example: A drug appears more effective for men and women separately but less effective overall when data is aggregated.

• Sampling Bias: When your data isn’t representative of the population you’re trying to understand. Types:

• Selection bias (e.g., surveying only power users),
• Survivorship bias (e.g., studying only retained users),

• Non-response bias (e.g., users who opt into feedback are more engaged).

• Confounding Variable: A hidden factor that influences both the independent and dependent variables, creating a false correlation.

• Example: "Ice cream sales and drowning deaths are correlated" → Confounder: hot weather.

• Hill’s Criteria for Causation: A checklist to assess if correlation implies causation (e.g., strength of association, consistency, plausibility).

• Power Analysis: Determines the minimum sample size needed to detect a meaningful effect in an A/B test.

• Formula: n = (Z² × p × (1–p)) / E², where:

  
  
  • Z = Z-score (e.g., 1.96 for 95% confidence),
  • p = baseline conversion rate,
  • E = minimum detectable effect.

• P-value: Probability that the observed effect is due to random chance. p < 0.05 is typically considered statistically significant (but not always meaningful!).

• Effect Size: Measures the magnitude of a difference (e.g., Cohen’s d for continuous data). A small p-value doesn’t mean the effect is large or important.

• Stratified Sampling: Dividing a population into subgroups (strata) and sampling from each to reduce bias.

• Example: Testing a feature on equal numbers of new vs. returning users.

• Multi-Armed Bandit (MAB): An A/B testing alternative that dynamically allocates traffic to better-performing variants to optimise for a goal (e.g., click-through rate).

Step-by-Step / Process Flow

How to apply statistical thinking to a product decision:

1. Define the Hypothesis
2. Write a clear, testable statement: "Adding a progress bar to checkout will increase conversion by 5% by reducing perceived effort."

3. Avoid vague claims like "improve UX."

4. Identify Confounders & Biases

5. List potential hidden variables (e.g., time of day, user segment, device type).

6. Check for sampling bias: "Are we only testing on iOS users?"

7. Choose the Right Method

8. For causation: Run an A/B test (or quasi-experiment if randomisation isn’t possible).
9. For correlation: Use observational data (e.g., SQL queries, analytics tools) but flag it as exploratory.

10. For complex relationships: Use stratified analysis or regression models.

11. Design the Experiment

12. Randomise users to control/treatment groups.
13. Set a primary metric (e.g., conversion rate) and guardrail metrics (e.g., NPS, error rates).

14. Calculate sample size using power analysis to avoid false negatives.

15. Analyse Results

16. Check for Simpson’s Paradox: "Does the effect hold across all user segments?"
17. Calculate effect size: "Is the 2% lift meaningful, or just statistically significant?"

18. Look for unintended consequences (e.g., higher conversion but lower AOV).

19. Make a Decision

20. If the effect is causal and meaningful, ship the change.
21. If not, iterate or kill the idea. Document learnings for future tests.

Common Mistakes

• Mistake: Assuming correlation = causation.

• Correction: Always ask, "Could there be a confounding variable?" Use A/B tests to isolate effects.

• Mistake: Ignoring Simpson’s Paradox.

• Correction: Break down results by key segments (e.g., new vs. returning users, device type). If trends reverse, dig deeper.

• Mistake: Using small sample sizes.

• Correction: Run a power analysis before launching a test. A 100-user test can’t detect a 5% lift.

• Mistake: Over-indexing on p-values.

• Correction: Focus on effect size and business impact. A p-value of 0.04 with a 0.1% lift isn’t worth shipping.

• Mistake: Testing too many variants at once.

• Correction: Stick to 1–2 treatment groups to avoid false positives (e.g., "peeking" at results early).

PM Interview / Practical Insights

1. Interviewer Trap: "Our data shows that users who watch tutorial videos have 3x higher retention. Should we force all users to watch them?"
2. What they’re testing: Can you spot correlation vs. causation?

3. Answer: "Not necessarily. Highly engaged users may self-select into tutorials. We should A/B test mandatory tutorials to isolate the effect."

4. Stakeholder Pushback: "Why do we need an A/B test? The data is clear!"

5. How to respond: "The data shows a correlation, but we don’t know if the feature causes the outcome. An A/B test will give us confidence to invest engineering resources."

6. Tricky Distinction: Statistical significance vs. practical significance.

7. Example: A test shows a 0.5% lift in conversion with p = 0.03. Is it worth shipping?

8. Answer: "Statistically significant, but not practically significant. The lift is too small to justify the risk."

9. Real-World Scenario: "Our churn model predicts that users who don’t use Feature X in the first 7 days are 2x more likely to churn. Should we force them to use it?"

10. Answer: "Not yet. The model shows correlation, not causation. We should A/B test nudges to see if encouraging use of Feature X reduces churn."

Quick Check Questions

1. Scenario: Your team notices that users who complete a "profile strength" quiz have 40% higher 30-day retention. The engineer suggests making the quiz mandatory during onboarding. What’s your next step?

2. Answer: Run an A/B test to isolate the quiz’s causal impact. The correlation could be driven by more engaged users self-selecting into the quiz.

3. Scenario: An A/B test shows that a new checkout flow increases conversion by 3% (p = 0.02) but decreases average order value (AOV) by 2%. How do you decide whether to ship it?

4. Answer: Calculate the net revenue impact (e.g., 3% more conversions × 98% AOV). If the trade-off is negative, don’t ship. If positive, consider segmenting (e.g., only show to low-AOV users).

5. Scenario: Your data shows that users in California have 2x higher engagement than users in Texas. The marketing team wants to double down on California ads. What’s a potential flaw in this plan?

6. Answer: Sampling bias. The data might not account for differences in user acquisition channels (e.g., California users came from a high-intent campaign). Check for confounders like traffic source or device type.

Last-Minute Cram Sheet

1. Correlation ≠ Causation – Always ask: "What’s the confounder?"
2. A/B tests prove causation – Randomisation eliminates bias.
3. Simpson’s Paradox – Trends can reverse when data is aggregated. Always segment!
4. Sampling bias – If your data isn’t representative, your conclusions are wrong.
5. P-value ≠ importance – A small p-value doesn’t mean the effect is meaningful.
6. Effect size matters – Calculate lift % or Cohen’s d to assess practical impact.
7. Power analysis – Determine sample size before running a test to avoid false negatives.
8. Stratified sampling – Test across key segments (e.g., new vs. returning users).
9. ⚠️ "Peeking" at A/B test results early – Increases false positives. Wait for statistical significance.
10. Multi-Armed Bandit – Dynamically optimises for a goal (e.g., CTR) without waiting for full test results.