Principles of Product Management: AI/ML Basics (Supervised vs Unsupervised Learning, Training/Inference, Prompt Engineering, LLM Concepts)

AI/ML Basics (Supervised vs Unsupervised Learning, Training/Inference, Prompt Engineering, LLM Concepts)

AI/ML Basics for Product Managers

What This Is

AI/ML is the backbone of modern product innovation—from fraud detection in fintech (e.g., Stripe Radar) to personalized recommendations (e.g., Netflix’s "Because You Watched" row). As a PM, you don’t need to code models, but you must understand how AI/ML works to scope features, prioritize experiments, and communicate trade-offs. For example, a neobank might use supervised learning to predict loan defaults (training on historical data) and LLMs to power a chatbot for customer support (inference with prompt engineering). Misunderstanding these concepts leads to misaligned roadmaps, wasted engineering cycles, or features that fail in production.

Key Terms & Frameworks

1. Supervised Learning
2. Definition: Training a model on labeled data (input-output pairs) to predict outcomes (e.g., spam/not-spam, price of a house).
3. Example: A ride-hailing app predicting ETAs using historical trip data (inputs: time, location, traffic; output: ETA).

4. PM Relevance: Use when you have clear, structured data and a defined target variable (e.g., "Will this user churn?").

5. Unsupervised Learning

6. Definition: Finding patterns in unlabeled data (no predefined outputs). Common techniques: clustering (grouping similar items) or dimensionality reduction (simplifying data).
7. Example: Spotify’s "Discover Weekly" uses clustering to group users with similar listening habits.

8. PM Relevance: Use for exploratory analysis (e.g., segmenting users without predefined categories) or anomaly detection (e.g., fraud).

9. Semi-Supervised Learning

10. Definition: A mix of labeled and unlabeled data (e.g., 10% labeled, 90% unlabeled). Useful when labeling is expensive.
11. Example: Google Photos auto-tagging faces (only a few labeled examples needed).

12. PM Relevance: Cost-effective when you can’t label all data but need high accuracy.

13. Training vs. Inference

14. Training: The process of teaching a model using data (e.g., feeding historical transactions to detect fraud). Expensive (compute, time, data).
15. Inference: Using the trained model to make predictions (e.g., flagging a new transaction as fraudulent). Cheap (runs in real-time).

16. PM Trap: ⚠️ Teams often underestimate training costs (e.g., GPU hours) but overestimate inference latency.

17. Prompt Engineering

18. Definition: Crafting inputs (prompts) to guide an LLM (e.g., ChatGPT) to generate desired outputs. Includes techniques like:
    • Zero-shot: No examples (e.g., "Summarize this article").
    • Few-shot: 1–3 examples (e.g., "Translate these 3 phrases to French, then translate this new one").
    • Chain-of-Thought (CoT): "Explain your reasoning step-by-step" to improve accuracy.
19. Example: Notion AI’s "Summarize" feature uses prompt engineering to extract key points from meeting notes.

20. PM Relevance: Critical for LLM-powered features (e.g., chatbots, content generation). Poor prompts = poor UX.

21. Large Language Model (LLM) Key Concepts

22. Tokens: Basic units of text (words, subwords, or characters). LLMs process text in tokens (e.g., "hello" = 1 token, "hello world" = 2 tokens).
    • PM Impact: Token limits (e.g., 4K tokens for GPT-3.5) constrain input/output length. Cost scales with tokens.
23. Temperature: Controls randomness of outputs (0 = deterministic, 1 = creative). Use low temp for factual tasks (e.g., 0.2), high temp for brainstorming (e.g., 0.8).

24. Fine-Tuning: Adapting a pre-trained LLM to a specific domain (e.g., legal, medical) using additional data.

    
    
    • PM Trade-off: Fine-tuning improves accuracy but requires labeled data and compute.

25. Bias-Variance Tradeoff

26. Definition: A model’s error comes from:
    • Bias: Underfitting (oversimplified model, e.g., predicting house prices with only square footage).
    • Variance: Overfitting (model memorizes noise, e.g., predicting prices based on a single outlier sale).

27. PM Relevance: High bias = poor accuracy; high variance = poor generalization. Balance via cross-validation.

28. Precision vs. Recall

29. Formulas:
    • Precision = TP / (TP + FP) (Of all predicted positives, how many are correct?).
    • Recall = TP / (TP + FN) (Of all actual positives, how many did we catch?).
30. Example: For a fraud detection model:
    • High precision = Few false alarms (good for user trust).
    • High recall = Few missed frauds (good for risk management).

31. PM Relevance: Choose metrics based on business goals (e.g., recall for medical diagnoses, precision for spam filters).

32. A/B Testing for ML Models

33. Framework: Treat models as "features" in experiments. Key steps:
    1. Define success metrics (e.g., CTR, conversion).
    2. Randomly split users into control (old model) and treatment (new model).
    3. Measure statistical significance (p-value < 0.05).

34. Example: LinkedIn A/B tested a new "People You May Know" model and saw a 10% lift in connections.

35. MLOps (Machine Learning Operations)

    
    
    • Definition: Practices to deploy, monitor, and maintain ML models in production. Includes:
    • Data pipelines: Automated data collection/cleaning.
    • Model versioning: Track model performance over time (e.g., "v2.1 reduced latency by 20%").
    • Monitoring: Detect drift (data or concept drift) when model performance degrades.
    • PM Relevance: MLOps is the "DevOps for ML"—without it, models rot in production.

36. Embeddings

    
    
    • Definition: Converting data (text, images) into numerical vectors that capture semantic meaning. Used for similarity searches (e.g., "Show me products like this").
    • Example: Airbnb uses embeddings to recommend similar listings.
    • PM Relevance: Enables features like "Find Similar" or personalized feeds.

37. Hallucination (LLMs)

    
    
    • Definition: LLMs generating plausible-sounding but incorrect/fabricated information.
    • Example: A legal chatbot citing a non-existent case.
    • PM Mitigation: Use retrieval-augmented generation (RAG) (fetch facts from a database before generating responses).

Step-by-Step: Applying AI/ML to a Product

Scenario: You’re a PM at a SaaS company building a feature to auto-generate meeting summaries using an LLM.

1. Define the Problem & Success Metrics
2. Action: Align with stakeholders on the goal (e.g., "Reduce time spent writing meeting notes by 30%").

3. Metrics: Time saved, user satisfaction (NPS), accuracy of summaries (human-reviewed sample).

4. Choose the Right ML Approach

5. Action: Decide between:
   • Fine-tuning (if you have domain-specific data, e.g., legal meetings).
   • Prompt engineering (cheaper, faster to iterate; e.g., "Summarize this transcript in 3 bullet points").

6. Example: Start with prompt engineering (low effort), then fine-tune if needed.

7. Design the Data Pipeline

8. Action: Work with engineers to:

   
   
   • Collect/clean data (e.g., meeting transcripts).
   • Label data (if fine-tuning; e.g., "This summary is 8/10 accurate").
   • Set up monitoring for drift (e.g., if meeting formats change).

9. Build & Test the MVP

10. Action:

    
    
    • Start with a small user segment (e.g., internal teams).
    • Use A/B testing to compare prompt variations (e.g., "Summarize" vs. "Extract action items").
    • Measure precision/recall (e.g., "Did the summary capture all key decisions?").

11. Monitor & Iterate

12. Action:

    
    
    • Track inference latency (users won’t wait 10 seconds for a summary).
    • Set up alerts for hallucinations (e.g., "This summary mentions a project that doesn’t exist").
    • Gather qualitative feedback (e.g., "The summary missed the deadline discussion").

13. Scale & Optimize

14. Action:
    • Expand to more users if metrics improve.
    • Optimize costs (e.g., switch to a cheaper LLM if quality is similar).
    • Add guardrails (e.g., "Don’t include sensitive data in summaries").

Common Mistakes

1. Mistake: Assuming ML is a "silver bullet" for any problem.

2. Correction: First validate the problem with non-ML solutions (e.g., rule-based systems, heuristics). ML is expensive—use it only when necessary.

3. Mistake: Ignoring data quality.

4. Correction: Garbage in = garbage out. Audit data for bias (e.g., underrepresented groups), missing values, and label errors. Example: A hiring tool trained on biased historical data will perpetuate discrimination.

5. Mistake: Over-optimizing for accuracy without considering business impact.

6. Correction: A 99% accurate model is useless if it’s too slow or expensive. Prioritize latency, cost, and UX alongside accuracy. Example: A fraud model with 95% precision but 10-second latency will frustrate users.

7. Mistake: Treating LLMs as "black boxes" without testing prompts.

8. Correction: Iterate on prompts like you would on UI copy. Test zero-shot vs. few-shot, temperature, and output length. Example: A customer support chatbot might need different prompts for technical vs. billing questions.

9. Mistake: Not planning for model decay.

10. Correction: Models degrade over time (e.g., user behavior changes, data drifts). Set up monitoring (e.g., track precision/recall weekly) and retraining pipelines.

PM Interview / Practical Insights

1. Interview Question: "How would you decide whether to use supervised or unsupervised learning for a user segmentation feature?"

2. Answer: Use supervised if you have labeled data (e.g., "high-value vs. low-value users") and a clear target variable. Use unsupervised if you’re exploring patterns (e.g., "What natural groups exist in our user base?"). Always start with unsupervised to validate hypotheses before labeling data.

3. Stakeholder Trap: "We need to build an AI feature—let’s use the latest LLM!"

4. Response: Push back with cost-benefit analysis. LLMs are expensive (training, inference, hallucination risks). Ask:

   
   
   • Can we solve this with rules (e.g., regex for email validation)?
   • Do we have enough data to fine-tune?
   • Is latency acceptable (e.g., LLMs add ~1–2 seconds to response time)?

5. Tricky Distinction: "Prompt Engineering vs. Fine-Tuning"

6. Prompt Engineering: Cheap, fast, but limited by the LLM’s base knowledge. Good for generic tasks (e.g., summarization).

7. Fine-Tuning: Expensive, requires labeled data, but higher accuracy for domain-specific tasks (e.g., medical diagnosis).

8. Interview Question: "Your team wants to launch a feature that uses an LLM to generate product descriptions. How do you mitigate hallucinations?"

9. Answer:
   1. Retrieval-Augmented Generation (RAG): Fetch facts from a product database before generating text.
   2. Human-in-the-Loop: Flag low-confidence outputs for review.
   3. Prompt Constraints: "Only use the following product attributes: [list]."

Quick Check Questions

1. Scenario: Your team built a fraud detection model with 95% precision but 70% recall. The CFO wants to reduce false positives (precision), but the Risk team wants to catch more fraud (recall). How do you decide?

2. Answer: Prioritize recall if the cost of missed fraud (e.g., chargebacks) is higher than false positives (e.g., manual reviews). Use cost-benefit analysis (e.g., "Each false positive costs $10 in support time; each missed fraud costs $100").

3. Scenario: You’re launching a chatbot for customer support. Users complain the responses are too verbose. What do you do?

4. Answer: Adjust the prompt to include constraints (e.g., "Answer in 2 sentences or less") and lower the temperature (to reduce creativity). Test variations with A/B testing.

5. Scenario: Your LLM-powered feature is slow (5-second latency). Engineers say it’s because the model is too large. What are your options?

6. Answer:
   1. Distill the model (train a smaller model to mimic the large one).
   2. Use caching (store frequent responses).
   3. Switch to a smaller LLM (e.g., GPT-3.5 → GPT-4 Turbo).
   4. Optimize prompts (shorter inputs = faster inference).

Last-Minute Cram Sheet

1. Supervised Learning = Labeled data (input → output). Use for prediction (e.g., churn, price).
2. Unsupervised Learning = Unlabeled data. Use for clustering (e.g., user segments) or anomaly detection.
3. Training = Teach the model (expensive). Inference = Use the model (cheap).
4. Prompt Engineering = Crafting inputs for LLMs. Zero-shot (no examples), few-shot (1–3 examples), CoT ("explain your reasoning").
5. LLM Tokens = Basic text units. Cost scales with tokens (e.g., 1,000 tokens ≈ 750 words).
6. Temperature = 0 (deterministic) to 1 (creative). Use low temp for factual tasks.
7. Precision = TP / (TP + FP). Recall = TP / (TP + FN). Trade-off: High precision = fewer false alarms; high recall = fewer misses.
8. Hallucination = LLM fabricates info. Mitigate with RAG or human review.
9. MLOps = DevOps for ML. Monitor for drift (data/concept changes).
10. ⚠️ Avoid ML for ML’s sake—start with rules or heuristics. ⚠️ Latency matters—users won’t wait 5 seconds for a prediction.