Principles of Product Management: Assumptions Mapping & Riskiest Assumption Testing (RAT)

Assumptions Mapping & Riskiest Assumption Testing (RAT)

Assumptions Mapping & Riskiest Assumption Testing (RAT) – Study Guide

What This Is

Assumptions Mapping is the process of identifying and categorizing the unknowns in your product strategy—what you believe to be true but haven’t validated. Riskiest Assumption Testing (RAT) is a lean, hypothesis-driven approach to systematically test the most critical assumptions before investing heavily in execution. This matters because ~90% of startups fail due to poor product-market fit (CB Insights), often because they built features based on untested assumptions. Example: A fintech startup assumed users would trust AI-driven investment advice, but RAT revealed they preferred human advisors—saving months of wasted dev effort.

Key Terms & Frameworks

• Assumption: A belief about your product, users, or market that hasn’t been validated (e.g., “Users will pay for this feature”).
• Riskiest Assumption: The assumption that, if wrong, would cause the most damage to your product’s success (e.g., “Users will share their bank data with us”).
• Assumptions Map: A 2x2 grid categorizing assumptions by certainty (known vs. unknown) and impact (high vs. low). Steps: List assumptions → Plot on grid → Prioritize high-impact/low-certainty.
• RAT (Riskiest Assumption Test): A lightweight experiment to validate the riskiest assumption with minimal effort. Formula: Hypothesis → Test → Learn → Decide (e.g., “We believe users will upload receipts for expense tracking. Test: Fake door button + 50 user interviews.”).
• ICE Score (Impact, Confidence, Ease): Prioritizes assumptions to test. Formula: Impact × Confidence × Ease (higher = test first).
• Fake Door Test: A low-cost way to validate demand by simulating a feature (e.g., a “Buy Now” button that shows a “Coming Soon” message).
• Concierge MVP: Manually delivering the value proposition to test demand (e.g., a PM manually curating playlists before building an AI algorithm).
• Leap of Faith Assumption (LOFA): A core assumption that, if wrong, invalidates your entire product (e.g., “Users will invite friends” for a social app).
• Validation Threshold: The minimum success metric for a test to be considered “validated” (e.g., “30% of users click the fake door button”).
• Type 1 vs. Type 2 Errors:
• Type 1: False positive (validating a bad assumption).
• Type 2: False negative (rejecting a good assumption).
• North Star Metric (NSM): The single metric that best captures your product’s value (e.g., “Daily active users” for a messaging app). Why it matters: Aligns assumptions with business outcomes.
• Jobs to Be Done (JTBD): Framework to uncover why users “hire” your product (e.g., “I hire TurboTax to avoid IRS penalties, not to file taxes”). Use case: Identifies hidden assumptions about user needs.

Step-by-Step / Process Flow

1. List All Assumptions
2. Action: Brainstorm with your team (PM, design, eng, data) and list every assumption about users, tech, business model, and market.
3. Example: For a meal-kit startup: “Users will pay $12/meal,” “Chefs will partner with us,” “Users trust our ingredient sourcing.”

4. Tool: Use a shared doc or Miro board. Include desirability (do users want it?), viability (can we make money?), and feasibility (can we build it?).

5. Map Assumptions on a 2x2 Grid

6. Axes: Impact (high/low) vs. Certainty (known/unknown).
7. Action: Plot assumptions. Focus on high-impact/low-certainty (top-right quadrant).

8. Example: “Users will pay $12/meal” = high impact (revenue), low certainty (no data).

9. Prioritize with ICE or RICE

10. Action: Score assumptions using ICE (Impact, Confidence, Ease) or RICE (Reach, Impact, Confidence, Effort).

11. Example: “Users will upload receipts” (ICE: 8 × 5 × 7 = 280) vs. “Users will refer friends” (ICE: 6 × 3 × 9 = 162). Test the first.

12. Design the RAT

13. Action: For the top assumption, design the cheapest, fastest test to validate it.
14. Examples:
    • Fake Door Test: Add a “Save to Wishlist” button; measure clicks.
    • Concierge MVP: Manually email users personalized recommendations before building an AI engine.
    • Landing Page Test: Drive traffic to a page describing the feature; measure sign-ups.

15. Pro Tip: Use “Wizard of Oz” testing (manual behind-the-scenes work to simulate automation).

16. Run the Test & Measure

17. Action: Execute the test with a small sample (e.g., 50–100 users). Define success metrics before running (e.g., “20% click-through rate”).

18. Example: For a “subscription upsell” feature, run a fake door test for 2 weeks. If <10% of users click, kill the idea.

19. Learn & Decide

20. Action: Analyze results. If validated, proceed to build. If not, pivot or kill the idea.
21. Example: A SaaS tool tested a “team collaboration” feature via a fake door test. Only 5% of users clicked → killed the feature, saving 3 months of dev work.

Common Mistakes

• Mistake: Testing easy assumptions first (e.g., UI preferences) instead of risky ones (e.g., “Will users pay?”).

• Correction: Always test the riskiest assumption first (high impact + low certainty). Use the 2x2 grid to prioritize.

• Mistake: Running tests without a clear hypothesis or success metric.

• Correction: Write hypotheses in the format: “We believe [user segment] will [action] because [reason]. We’ll know we’re right if [metric] reaches [threshold].”

• Example: “We believe freelancers will pay $20/month for our invoicing tool because it saves 5 hours/week. We’ll know we’re right if 15% of trial users convert.”

• Mistake: Overbuilding the test (e.g., coding a full feature for a fake door test).

• Correction: Use the minimum viable test (e.g., a button, a survey, a manual process). Example: Zappos started by posting photos of shoes from local stores—no inventory.

• Mistake: Ignoring Type 2 errors (false negatives) by setting validation thresholds too high.

• Correction: Set realistic thresholds based on industry benchmarks (e.g., 10% conversion for a fake door test is often sufficient for early-stage startups).

• Mistake: Confusing correlation with causation in test results.

• Correction: Use A/B tests or holdout groups to isolate variables. Example: If users click a fake door button, it doesn’t mean they’ll pay—test willingness to pay separately.

PM Interview / Practical Insights

1. “How would you test if users want a feature before building it?”
2. Trap: Saying “build an MVP” (too slow/costly). Better: “Run a fake door test or concierge MVP to validate demand first.”

3. Pro Tip: Mention “smoke tests” (e.g., pre-selling a product before it exists) for early-stage validation.

4. “What’s the difference between an MVP and a RAT?”

5. Trap: Saying they’re the same. Distinction:
   • MVP: A minimal product to learn about the market (e.g., a basic ride-hailing app).
   • RAT: A test to validate a single assumption (e.g., “Will drivers sign up?” via a landing page).

6. Example: Dropbox’s MVP was a video demo (RAT) before building the actual product.

7. “How do you handle a stakeholder who wants to build a feature based on a hunch?”

8. Trap: Saying “just say no.” Better: “I’d map their hunch as an assumption, then propose a lightweight test (e.g., fake door) to validate it before investing resources.”

9. Pro Tip: Use “disagree and commit” if the test fails—show data to depersonalize the decision.

10. “What’s a Leap of Faith Assumption (LOFA) for [X product]?”

11. Trap: Giving a generic answer (e.g., “users will like it”). Better: Tie it to the core value prop.
12. Examples:
    • Uber: “Drivers will accept rides from strangers.”
    • Airbnb: “Hosts will trust strangers to stay in their homes.”
    • Slack: “Teams will switch from email to a chat app.”

Quick Check Questions

1. Your team wants to build a “social sharing” feature for a productivity app. How do you decide whether to proceed?
2. Answer: Map assumptions (e.g., “Users want to share their progress”), then run a fake door test (e.g., add a “Share” button and measure clicks). If <5% of users click, kill the feature.

3. Why: Social features are often overbuilt—validate demand first.

4. A stakeholder insists on building a feature because “competitors have it.” How do you respond?

5. Answer: “Let’s treat this as an assumption (‘Users want this because competitors have it’) and test it with a concierge MVP or survey before building.”

6. Why: Competitor features may not fit your users’ needs—avoid “me-too” product decisions.

7. Your RAT for a new subscription tier shows 12% conversion (target was 15%). Do you proceed?

8. Answer: Depends on the validation threshold and business context. If 12% is close to breakeven and the test had high confidence (e.g., 1,000 users), proceed with caution. If the threshold was arbitrary, run a follow-up test (e.g., A/B test pricing).
9. Why: Avoid binary thinking—assess risk vs. reward.

Last-Minute Cram Sheet

1. Assumptions Map: 2x2 grid (Impact vs. Certainty). Focus on high-impact/low-certainty.
2. RAT = Hypothesis → Test → Learn → Decide. Always test the riskiest assumption first.
3. ICE Score: Impact × Confidence × Ease. Prioritize assumptions to test.
4. Fake Door Test: Simulate a feature to measure demand (e.g., “Coming Soon” button).
5. Concierge MVP: Manually deliver value to test demand (e.g., human curation before AI).
6. Leap of Faith Assumption (LOFA): The assumption that, if wrong, kills your product.
7. Validation Threshold: Define success metrics before running the test (e.g., “20% click-through”).
8. ⚠️ Type 1 Error: False positive (validating a bad assumption). Fix: Set strict thresholds.
9. ⚠️ Type 2 Error: False negative (rejecting a good assumption). Fix: Don’t set thresholds too high.
10. JTBD: “Users hire your product to get a job done.” Use to uncover hidden assumptions.