Principles of UX / UI (Product Design): Generative vs Evaluative Research

Generative vs Evaluative Research

Generative vs. Evaluative Research: A Portfolio-Ready Study Guide

For aspiring product designers, UX bootcampers, and graphic designers transitioning to UX

What This Is

Generative and evaluative research are two core UX research phases that answer different questions: - Generative research (aka "discovery" or "exploratory" research) helps you understand problems before designing solutions. It answers: Who are our users? What are their pain points? What do they need? Example: Interviewing nurses to uncover frustrations with a hospital’s patient portal before redesigning the appointment-booking flow.
- Evaluative research tests whether your designs work for users. It answers: Can users complete tasks? Where do they get stuck? Does the solution solve the problem? Example: Running a usability test on a checkout flow prototype to see if users abandon their carts at the payment step.

Why it matters: Skipping generative research leads to "solutions in search of a problem" (e.g., adding a chatbot because it’s trendy, not because users need it). Skipping evaluative research risks shipping a product that looks good but fails in real use (e.g., a "beautiful" dashboard no one can navigate).

Key Terms & Principles

• Generative Research
• User Interviews: Open-ended conversations to uncover needs, behaviors, and pain points. Example: Asking small-business owners, "Walk me through the last time you filed taxes—what was frustrating?" to inform a tax-prep app.
• Contextual Inquiry: Observing users in their natural environment (e.g., watching a barista use a POS system during a rush). Example: Noticing they struggle to find the "void order" button mid-transaction.
• Diary Studies: Users log their experiences over time (e.g., tracking how often they use a fitness app and why). Example: A user records, "Day 3: Forgot to log my run because the app crashed."
• Jobs to Be Done (JTBD): A framework to uncover the "job" users "hire" a product to do. Example: People don’t "hire" a drill; they hire it to "make a hole in the wall to hang a picture."

• Affinity Diagramming: Grouping qualitative data (e.g., interview notes) into themes. Example: Clustering sticky notes from user interviews into themes like "Payment Confusion" or "Slow Load Times."

• Evaluative Research

• Usability Testing: Observing users completing tasks with a prototype or live product. Example: Asking a user to "Find and book a doctor’s appointment" in a patient portal while you note where they hesitate.
• A/B Testing: Comparing two versions of a design to see which performs better. Example: Testing a green vs. blue "Sign Up" button to see which gets more clicks.
• Heuristic Evaluation: Reviewing a design against Nielsen’s 10 Usability Heuristics (e.g., "Does the system match real-world language?"). Example: Flagging that a "Submit" button should say "Pay Now" in a checkout flow.
• System Usability Scale (SUS): A 10-question survey to measure perceived usability. Example: After testing, users rate statements like "I found the system unnecessarily complex" on a 1–5 scale.
• First-Click Testing: Tracking where users click first to complete a task. Example: Testing if users click "Menu" or "Search" first to find a product on an e-commerce site.
• Hick’s Law: The time it takes to make a decision increases with the number of choices. Example: Reducing a form’s dropdown options from 20 to 5 speeds up completion.
• Miller’s Law: The average person can hold 7±2 items in working memory. Example: Limiting a navigation menu to 5–7 items to avoid cognitive overload.

Step-by-Step / Process Flow

1. Generative Research: Uncover the Problem

Goal: Define what to design before designing it.
Actions:
- Figma/Whiteboard: Create a research plan (objectives, questions, methods). Example: - Objective: Understand why patients abandon the hospital portal.
- Questions: "What’s the hardest part of booking an appointment?" "What do you wish the portal did?" - Method: 5 user interviews + 1 contextual inquiry.
- Recruit Participants: Use tools like User Interviews or Respondent to find 5–8 target users (e.g., patients aged 30–60 who’ve used the portal).
- Conduct Interviews: Ask open-ended questions (e.g., "Tell me about the last time you booked an appointment online"). Record sessions (with consent) and take notes.
- Synthesize Data: Use affinity diagramming in Figma or Miro to group insights. Example themes: - "Patients forget their login info" → Need password recovery improvements.
- "Can’t find available slots" → Need better calendar filtering.
- Share Findings: Create a 1-page summary with quotes, themes, and opportunity areas (e.g., "Improve login flow" or "Redesign the calendar UI").

Figma Tip: Use the Sticky Notes plugin to quickly cluster insights.

2. Design the Solution (Low-Fidelity)

Goal: Translate insights into rough designs.
Actions:
- Sketch Wireframes: Draw 3–5 variations of key screens (e.g., login, calendar, confirmation) on paper or in Figma. Focus on layout, not visuals.
- Apply Heuristics: Check designs against Nielsen’s 10 Heuristics. Example: - "Does the calendar show available slots clearly?" (Visibility of system status) - "Can users undo an action?" (User control and freedom) - Create a Prototype: Link screens in Figma to simulate the flow (e.g., login → calendar → confirmation).

3. Evaluative Research: Test the Solution

Goal: Validate if the design solves the problem.
Actions:
- Plan the Test: Write a test script with 3–5 tasks (e.g., "Book an appointment with Dr. Smith for next Tuesday").
- Recruit Users: Aim for 5–7 participants (Nielsen’s rule: 5 users find 85% of issues).
- Run Usability Tests: Use Figma’s Prototype Mode or Maze to observe users completing tasks. Note: - Where they hesitate (e.g., "I don’t see the ‘Book’ button").
- What they say (e.g., "This calendar is confusing").
- Analyze Results: Tally issues (e.g., "4/5 users couldn’t find the ‘Book’ button"). Prioritize fixes using severity ratings (1 = minor, 4 = critical).
- Iterate: Redesign based on feedback (e.g., move the "Book" button to the top-right, add a tooltip).

Figma Tip: Use Figma’s Observation Mode to record user sessions and share clips with stakeholders.

4. Repeat (Generative → Evaluative Cycle)

• If the design fails: Return to generative research to dig deeper (e.g., "Why do users ignore the ‘Book’ button?").
• If the design succeeds: Move to high-fidelity and test again (e.g., add colors, micro-interactions).

Common Mistakes

Design Interview / Portfolio Tips

What Interviewers Look For

1. Clear Distinction Between Generative and Evaluative
2. Tricky Question: "How would you research a new feature for a meditation app?"

3. Strong Answer: "First, I’d run generative research (e.g., interviews with users who’ve tried meditation apps) to uncover pain points like ‘I can’t find short sessions for my commute.’ Then, I’d design a prototype and test it with evaluative research (e.g., usability tests to see if users can find the ‘5-minute sessions’ filter)."

4. Evidence of Synthesis

5. Show affinity diagrams, themes, or insight summaries in your portfolio. Example:

   
   
   • Before: "We interviewed 5 users."
   • After: "We found 3 key themes: 1) Users forget their login info, 2) They want shorter sessions, 3) They dislike intrusive ads. We prioritized fixing logins first because 4/5 users mentioned it."

6. Iteration Stories

7. Interviewers love "We tested this, found X, and changed Y" stories. Example:

   
   
   • "In usability tests, users ignored the ‘Save’ button because it was gray. We made it blue and added a tooltip, which increased saves by 30%."

8. Tool Proficiency

9. Mention Figma (for prototyping), Miro/Mural (for affinity diagrams), UserTesting/Maze (for remote testing), and Optimal Workshop (for card sorting).

Portfolio Tips

• Include a Research Case Study: Dedicate 1–2 slides to:
• Research Goal (e.g., "Understand why users abandon the checkout flow").
• Methods Used (e.g., "5 user interviews + 10 usability tests").
• Key Insights (e.g., "Users were confused by the ‘Guest Checkout’ option").
• Impact (e.g., "Redesigned the flow, reducing drop-off by 20%").
• Show Your Process: Include photos of sticky notes, whiteboard sessions, or Figma prototypes with annotations.
• Avoid Jargon: Instead of "We conducted a heuristic evaluation," say "We reviewed the design against best practices and found 3 issues."

Quick Check Questions

1. Scenario: A stakeholder says, "We need to add a live chat feature to our banking app because our competitor has one." How do you respond?

2. Answer: Use generative research to validate the need first. Ask users: "Have you ever wished you could chat with a bank rep while using the app? What for?" If the data shows demand, then explore solutions. (Key principle: Don’t design based on assumptions.)

3. Scenario: You’re testing a prototype of a food-delivery app. Users complete the task ("Order a pizza") but take 3 minutes and look frustrated. What do you do next?

4. Answer: Dig deeper with follow-up questions (e.g., "What was confusing?") and observe their clicks. Likely issues: Hick’s Law (too many choices) or poor information hierarchy (key info buried). (Key principle: Usability testing reveals "why," not just "what.")

5. Scenario: Your team wants to A/B test two versions of a "Subscribe" button (green vs. red). What’s missing from this plan?

6. Answer: Generative research to define the goal. Are you testing for clicks (red might win) or trust (green might feel safer)? Also, ensure the test runs long enough to reach statistical significance. (Key principle: A/B tests need a hypothesis.)

Last-Minute Cram Sheet

1. Generative research = What’s the problem? (Interviews, diary studies, JTBD).
2. Evaluative research = Does the solution work? (Usability tests, A/B tests, heuristics).
3. Nielsen’s 10 Heuristics = 10 rules for usable design (e.g., "Match between system and real world").
4. Hick’s Law = More choices = slower decisions. ⚠️ Avoid overwhelming users.
5. Miller’s Law = 7±2 items in working memory. ⚠️ Limit menu items, form fields.
6. Fitts’s Law = Bigger + closer = easier to click. ⚠️ Make key buttons thumb-friendly.
7. First-Click Testing = Where users click first reveals IA issues.
8. SUS Score = 10-question survey to measure usability (68+ = good).
9. ⚠️ User Interviews ≠ Usability Tests – Interviews uncover problems; tests observe task completion.
10. ⚠️ Always test with 5 users (Nielsen’s rule: 5 users find 85% of issues).