Principles of Product Management: User Research Methods (Generative vs Evaluative, Qualitative vs Quantitative, Mixed Methods)

User Research Methods (Generative vs Evaluative, Qualitative vs Quantitative, Mixed Methods)

User Research Methods: Generative vs Evaluative, Qualitative vs Quantitative, Mixed Methods

What This Is

User research is the systematic study of users to uncover needs, behaviors, and pain points—so you can build products they actually want. It’s not just "asking users what they want" (they often don’t know); it’s about observing, measuring, and interpreting why they do what they do. Generative research (e.g., interviews, ethnography) helps you discover problems and opportunities early, while evaluative research (e.g., usability tests, A/B tests) helps you validate solutions later. Qualitative (words, stories) and quantitative (numbers, metrics) methods answer different questions, and mixed methods combine both for deeper insights.

Real-world example:
When Stripe redesigned its dashboard, the team used generative research (interviews with small-business owners) to uncover that users struggled with cash-flow visibility. They then ran evaluative research (usability tests on prototypes) to validate that a new "Cash Flow" tab reduced time-to-insight by 40%. Finally, they used quantitative data (clickstream analytics) to confirm the feature increased retention by 12%.

Key Terms & Frameworks

• Generative Research:
  Research done before you have a solution, to discover problems, needs, or opportunities. Examples: interviews, diary studies, ethnography.
  Goal: "What should we build?"

• Evaluative Research:
  Research done after you have a solution (or prototype), to test if it works. Examples: usability tests, A/B tests, surveys.
  Goal: "Does this work? How can we improve it?"

• Qualitative Research:
  Non-numerical data (words, stories, observations) that answers why or how. Examples: user interviews, open-ended survey questions, session recordings.
  Strength: Depth, context, uncovering "unknown unknowns." Weakness: Hard to scale; subjective interpretation.

• Quantitative Research:
  Numerical data that answers what, how much, or how many. Examples: analytics (e.g., Google Analytics), A/B tests, closed-ended surveys (e.g., NPS, CSAT).
  Strength: Scalable, statistically significant, objective.
  Weakness: Lacks context; can’t explain why something happens.

• Mixed Methods:
  Combining qualitative and quantitative research to offset each method’s weaknesses. Example: Run a survey (quant) to identify pain points, then interview users (qual) to understand why those pain points exist.

• Jobs-to-be-Done (JTBD):
  A framework for uncovering why users "hire" a product (the "job" they’re trying to get done). Example: People don’t buy a drill; they buy a hole in the wall.
  Key question: "What progress is the user trying to make in a given circumstance?"

• Double Diamond (Design Council):
  A 4-phase process for problem-solving:

• Discover (generative research to explore the problem).
• Define (synthesize insights into a problem statement).
• Develop (ideate solutions).

• Deliver (evaluate and refine solutions).

• ICE Score (Impact, Confidence, Ease):
  A prioritization framework for research questions or features:

• Impact: How much will this move the needle?
• Confidence: How sure are you of the impact? (1–10 scale)

• Ease: How easy is it to execute? (1–10 scale) Formula: ICE = Impact × Confidence × Ease

• NPS (Net Promoter Score):
  A quantitative metric for customer loyalty: "How likely are you to recommend [product] to a friend?" (0–10 scale).

• Promoters (9–10): Loyal enthusiasts.
• Passives (7–8): Satisfied but unenthusiastic.

• Detractors (0–6): Unhappy customers.
  Formula: NPS = % Promoters – % Detractors

• System Usability Scale (SUS):
  A 10-question survey to measure usability (quantitative). Example question: "I found the system unnecessarily complex" (1–5 scale, strongly disagree to strongly agree).
  Score range: 0–100 (68+ is "good," 80+ is "excellent").

• Triangulation:
  Using multiple research methods to validate a finding. Example: If interviews (qual) suggest users struggle with checkout, confirm with analytics (quant) showing high drop-off rates.

• False Consensus Effect:
  The cognitive bias where PMs assume users think/behave like they do. Correction: Always validate assumptions with research.

Step-by-Step / Process Flow

How to run a user research project (end-to-end):

1. Define the Goal
2. Ask: What decision will this research inform? (e.g., "Should we build a dark mode?" or "Why is our onboarding drop-off rate 60%?")
3. Align with stakeholders on success metrics (e.g., "Reduce drop-off by 20%").

4. Example: A fintech app wants to improve its savings feature. Goal: "Understand why users abandon the savings goal setup flow."

5. Choose the Right Method(s)

6. Generative phase (discovery):
   • Qualitative: User interviews (5–10 users), diary studies, ethnography.
   • Quantitative: Surveys (100+ responses), analytics (e.g., funnel analysis).
7. Evaluative phase (validation):
   • Qualitative: Usability tests (5–7 users), prototype feedback.
   • Quantitative: A/B tests, clickstream data, NPS/CSAT surveys.
8. Mixed methods: Start with qualitative to explore, then use quantitative to confirm scale.

9. Example: For the fintech app, start with interviews (qual) to uncover pain points, then analyze funnel data (quant) to see where users drop off.

10. Recruit Participants

11. Target the right users (not just "anyone"). Use segmentation (e.g., "new users who signed up in the last 30 days").
12. Avoid "professional testers" (they’re not representative).
13. Tools: UserInterviews.com, Respondent, or your own user base (e.g., email lists).

14. Example: Recruit 10 users who abandoned the savings goal setup flow in the last 7 days.

15. Conduct the Research

16. Qualitative:
    • Ask open-ended questions (e.g., "Walk me through the last time you tried to set a savings goal").
    • Avoid leading questions (e.g., "Don’t you think the flow is confusing?" → "How did you feel about the flow?").
    • Take notes and record (with consent).
17. Quantitative:
    • Ensure sample size is statistically significant (use a calculator like Evan’s Awesome A/B Tools).
    • Track the right metrics (e.g., completion rate, time on task).

18. Example: In interviews, users say, "I don’t understand what ‘round-ups’ are." In analytics, you see 70% drop-off at the "round-ups" step.

19. Synthesize Insights

20. Qualitative: Look for patterns (e.g., "3/5 users mentioned confusion about round-ups"). Use affinity mapping (group similar quotes).
21. Quantitative: Run statistical tests (e.g., t-tests for A/B tests). Look for correlations (e.g., "Users who complete onboarding are 2x more likely to retain").

22. Example: Combine interview quotes ("I don’t get round-ups") with analytics (70% drop-off) to conclude: "Users don’t understand the ‘round-ups’ feature."

23. Translate Insights into Action

24. Map insights to opportunities (e.g., "Simplify the round-ups explanation").
25. Prioritize using a framework like ICE or RICE.
26. Share findings with stakeholders in a 1-pager (problem + data + recommendation).
27. Example: Recommendation: "Add a 10-second explainer video for round-ups in the setup flow. Test with an A/B experiment."

Common Mistakes

• Mistake: Running research after building the product (e.g., usability testing a live feature).
  Correction: Start with generative research before building. Use evaluative research to iterate. Why: Fixing problems early is 10x cheaper than post-launch.

• Mistake: Relying only on quantitative data (e.g., analytics) without qualitative context.
  Correction: Use mixed methods. Why: Analytics show what users do; interviews show why. Example: Analytics show high drop-off at checkout, but interviews reveal users abandon because shipping costs are hidden.

• Mistake: Asking users, "Would you use this feature?" (They’ll say yes to be nice.) Correction: Ask about past behavior ("Tell me about the last time you tried to [do X]") or observe behavior (e.g., usability tests). Why: People are bad at predicting future behavior.

• Mistake: Recruiting the wrong users (e.g., testing a B2B feature with consumers).
  Correction: Screen participants carefully (e.g., "Must be a small-business owner who uses accounting software"). Why: Wrong users = wrong insights.

• Mistake: Ignoring small sample sizes in qualitative research (e.g., drawing conclusions from 2 interviews).
  Correction: Aim for saturation (stop when you’re not hearing new insights). Typically 5–10 interviews for qualitative. Why: Small samples miss edge cases.

PM Interview / Practical Insights

1. "How would you decide whether to use qualitative or quantitative research for a problem?"
2. Trap: Saying "qualitative for discovery, quantitative for validation" is too simplistic.

3. Better answer: "It depends on the question. If I need to explore (e.g., ‘Why are users churning?’), I’d start with qualitative (interviews). If I need to measure (e.g., ‘Does this new onboarding flow reduce drop-off?’), I’d use quantitative (A/B test). Often, I’d use both: qualitative to generate hypotheses, quantitative to test them at scale."

4. "A stakeholder says, ‘We don’t need research—we already know what users want.’ How do you respond?"

5. Trap: Dismissing their opinion or being confrontational.

6. Better answer: "I’d ask, ‘What evidence do we have for that?’ Often, assumptions are based on anecdotes or personal experience. Research helps us validate (or invalidate) those assumptions with real user data. For example, at [Company X], we assumed users wanted [Feature Y], but research showed they actually struggled with [Problem Z]. Would you be open to a small experiment to test this?"

7. "How do you handle conflicting research findings (e.g., interviews say users hate a feature, but analytics show high usage)?"

8. Trap: Picking one data source over the other.

9. Better answer: "I’d triangulate the data. High usage might mean the feature is necessary but frustrating (e.g., users have no alternative). I’d dig deeper with follow-up interviews to understand the why behind the usage. For example, at [Company Y], analytics showed high usage of a support chat, but interviews revealed users only used it because the self-service docs were confusing. We fixed the docs, and chat usage dropped by 30%—a win for both users and the support team."

10. "What’s the difference between generative and evaluative research?"

11. Trap: Saying generative is "before" and evaluative is "after" (true, but not the why).
12. Better answer: "Generative research is about discovering problems and opportunities (e.g., ‘What jobs are users hiring our product for?’). Evaluative research is about testing solutions (e.g., ‘Does this new feature solve the problem?’). Generative answers what to build; evaluative answers how well it works."

Quick Check Questions

1. Scenario: Your team wants to add a social sharing feature to increase engagement, but a small usability test (5 users) shows it confuses users and hurts NPS. The head of growth insists on shipping it because "competitors have it." How do you decide?

2. Answer: Run a quantitative A/B test with a larger sample to measure the feature’s impact on both engagement and NPS. If the data shows a net negative (e.g., engagement up 5% but NPS down 10%), kill the feature. Why: Small qualitative tests can surface problems, but quantitative data is needed to measure trade-offs at scale.

3. Scenario: You’re launching a new mobile app. Your PM wants to skip generative research and go straight to A/B testing the onboarding flow. What’s the risk?

4. Answer: You might optimize for the wrong problem. Without generative research, you don’t know why users drop off. Example: If analytics show 50% drop-off at Step 3, but you don’t know why, you might A/B test the wrong fix (e.g., changing button color when the real issue is confusing copy). Why: Generative research uncovers the root cause; evaluative research optimizes the solution.

5. Scenario: A survey shows 80% of users say they "love" your product, but retention is only 20%. What’s likely happening, and how would you investigate?

6. Answer: The survey is likely biased (e.g., only happy users responded) or misleading (e.g., "love" ≠ actual usage). Investigate with:
   • Qualitative: Interview churned users to understand why they left.
   • Quantitative: Analyze retention cohorts to see when users drop off.
   • Why: Surveys measure sentiment; behavior (retention) is what matters.

Last-Minute Cram Sheet

1. Generative research = Discover problems (e.g., interviews, ethnography). Evaluative research = Test solutions (e.g., usability tests, A/B tests).
2. Qualitative = Why/how (words, stories). Quantitative = What/how much (numbers, metrics).
3. Mixed methods = Qual + quant to offset weaknesses (e.g., interviews + analytics).
4. JTBD = "What progress is the user trying to make?" (Not "what features do they want?").
5. Double Diamond: Discover → Define → Develop → Deliver.
6. ICE Score = Impact × Confidence × Ease (prioritize research questions).
7. NPS = % Promoters – % Detractors (loyalty metric).
8. Triangulation = Use multiple methods to validate a finding (e.g., interviews + analytics).
9. ⚠️ False consensus effect = Assuming users think like you. Fix: Always validate with research.
10. ⚠️ Small sample size = Qualitative: 5–10 users (saturation). Quantitative: 100+ (statistical significance).