Computer Science - ICT Grade 9 Introduction to Machine Learning Training and Testing

Study Guide: Introduction to Machine Learning – Training and Testing
Grade 9 | Computer Science – ICT

1. The Driving Question

How does a computer learn to recognize your voice, predict the weather, or even recommend the next song you’ll like—without being explicitly programmed for every single case? And once it thinks it’s learned, how do we know it’s not just memorizing answers like a student cramming for a test?

2. The Core Idea – Built, Not Listed

Imagine you’re coaching a basketball team for the first time. You don’t just throw the players onto the court and hope they win—you split them into two groups. One group (the training set) practices drills, learns plays, and makes mistakes while you give feedback. The other group (the testing set) sits out, watches, and only plays in scrimmages after the first group has practiced. If the testing group performs well, you know your coaching worked. If they flop, you realize the first group just memorized the drills instead of learning the game.

Machine learning works the same way. A computer isn’t "taught" rules like a traditional program. Instead, you give it a dataset—say, 1,000 labeled photos of cats and dogs. The computer studies 800 of them (the training set), looking for patterns (e.g., "cats have pointy ears" or "dogs have longer snouts"). Then, you test it on the remaining 200 photos (the testing set) it’s never seen before. If it guesses correctly 95% of the time, great! If it only gets 60% right, it probably overfit—meaning it memorized quirks in the training data (like "all cats in this set have collars") instead of learning general rules.

Key Vocabulary:
- Training Set
Definition: The portion of data used to teach a machine learning model by adjusting its internal parameters.
Example: A spam filter trained on 10,000 emails labeled "spam" or "not spam" to learn which words (e.g., "FREE," "urgent") usually appear in junk mail.
College Note: In advanced ML, the training set is further split into mini-batches for faster learning, and techniques like cross-validation are used to avoid overfitting.

• Testing Set
  Definition: The portion of data held back during training and used only to evaluate how well the model generalizes to new, unseen data.
  Example: After training a model to predict house prices using 80% of a city’s sales data, you test it on the remaining 20% to see if it accurately estimates prices for homes it’s never "seen" before.
  College Note: In research, the testing set is often called the holdout set, and its size is carefully chosen to balance statistical significance with the need for enough training data.

• Overfitting
  Definition: When a model learns the training data too well, including its noise and random quirks, making it perform poorly on new data.
  Example: A student memorizes the answers to last year’s math test but fails this year’s because the questions are worded differently—even though the concepts are the same.
  College Note: Overfitting is a major challenge in deep learning, where models with millions of parameters can "memorize" training data without learning meaningful patterns.

• Generalization
  Definition: A model’s ability to perform well on new, unseen data—not just the data it was trained on.
  Example: A voice assistant that understands your commands and your friend’s accent, even though it was only trained on your voice.
  College Note: Generalization is tied to bias-variance tradeoff; models with high bias (too simple) underfit, while models with high variance (too complex) overfit.

3. Assessment Translation

How This Appears on Assessments:
- Classroom Formative (Exit Tickets/Short Response):
Prompt: "Explain why splitting a dataset into training and testing sets is necessary for machine learning. Use an analogy to support your answer." - Proficient Response: "Splitting data is like studying for a test with practice questions. If you only study the exact questions that will be on the test (training set), you might memorize the answers without understanding the concepts. The testing set is like the real test—it checks if you actually learned the material. For example, a model trained to recognize handwritten digits might memorize the exact shapes in the training set but fail on new handwriting styles if it wasn’t tested on unseen data." - Developing Response: "You need training and testing sets so the computer can learn. The training set teaches it, and the testing set checks if it’s right." (Lacks analogy, doesn’t explain why the split matters.)

• State Standardized Tests (Multiple Choice):
  Question: A machine learning model is trained to predict whether a fruit is an apple or an orange based on weight and color. The model achieves 100% accuracy on the training set but only 60% on the testing set. What is the most likely explanation?
  A) The model is underfitting because it’s too simple.
  B) The model is overfitting because it memorized the training data.
  C) The testing set is too small to be reliable.
  D) The model needs more features to learn from.
• Distractor Patterns:
  • A: Underfitting would mean low accuracy on both sets (e.g., 60% on training, 55% on testing).
  • C: While possible, the question doesn’t mention the testing set size, and 60% is a clear drop from 100%.
  • D: More features might help, but the issue here is generalization, not missing data.

• Correct Answer: B (Overfitting: the model memorized the training data’s quirks instead of learning general patterns.)

• SAT/ACT Framing (Logical Reasoning):
  Question: A researcher trains a model to classify news articles as "fake" or "real" using 10,000 articles. The model achieves 98% accuracy on the training set but only 70% on a new set of 1,000 articles. Which of the following is the least likely explanation for this discrepancy?
  A) The model overfit to specific phrases in the training articles.
  B) The new articles use different vocabulary than the training set.
  C) The model’s algorithm is inherently flawed and cannot learn patterns.
  D) The training set contained biased or unrepresentative examples.

• Why C is Correct: If the algorithm were inherently flawed, it would perform poorly on both sets. The discrepancy suggests the model can learn (98% on training) but fails to generalize (70% on testing).

Model Proficient Response (Short Answer):
Prompt: "A model is trained to predict whether a student will pass or fail a math test based on study hours and previous grades. The training accuracy is 95%, but the testing accuracy is 70%. What does this suggest about the model, and how could you improve it?" Response: "This suggests the model is overfitting—it memorized patterns in the training data (like specific student IDs or outliers) instead of learning general trends. For example, it might have learned that 'students who studied 5 hours always pass' because that was true in the training set, but this isn’t a reliable rule. To improve it, I’d: 1. Use more data to dilute the impact of outliers.
2. Simplify the model (e.g., reduce the number of features or use a less complex algorithm).
3. Add regularization (a technique to penalize over-complexity).
4. Cross-validate by splitting the data into multiple training/testing sets to ensure consistency."

4. Mistake Taxonomy

Mistake 1: Confusing Training and Testing Purposes
- Prompt: "Why do we need a testing set in machine learning?" - Common Wrong Response: "The testing set is used to train the model so it can learn better." (Misunderstands the testing set’s role—it’s for evaluation, not learning.) - Why It Loses Credit: The response conflates training and testing. The testing set is never used to adjust the model; it’s only for final evaluation.
- Correct Approach: - The training set teaches the model by letting it adjust its parameters.
- The testing set evaluates the model’s performance on new data to check for overfitting.
- Analogy: The training set is like practice problems; the testing set is like the real exam.

Mistake 2: Ignoring Overfitting in Analysis
- Prompt: "A model trained to detect fraudulent credit card transactions has 99% accuracy on the training set but 80% on the testing set. What does this tell you, and what should you do next?" - Common Wrong Response: "The model is doing well because 80% is still good. Maybe the testing set is just harder." (Ignores the red flag of a 19% drop in accuracy.) - Why It Loses Credit: The response fails to recognize overfitting. A 19% drop suggests the model memorized the training data instead of learning general patterns.
- Correct Approach: - The gap between 99% and 80% indicates overfitting.
- Next steps: Simplify the model, add more training data, or use techniques like dropout (randomly ignoring some data during training to prevent memorization).

Mistake 3: Misinterpreting "Accuracy"
- Prompt: "A model predicts whether a tumor is benign or malignant with 90% accuracy. Is this model reliable? Explain." - Common Wrong Response: "Yes, 90% accuracy is great!" (Ignores that accuracy alone can be misleading, especially for imbalanced datasets.) - Why It Loses Credit: The response doesn’t consider what the model is getting wrong. If 90% of tumors are benign, a model that always predicts "benign" would have 90% accuracy but be useless.
- Correct Approach: - Check precision (of all "malignant" predictions, how many are correct?) and recall (of all actual malignant tumors, how many did it catch?).
- For medical diagnoses, high recall is critical—you’d rather have false alarms than miss a malignant tumor.
- Example: A model with 90% accuracy but 50% recall for malignant tumors is dangerous.

5. Connection Layer

1. Within Computer Science:
   [Training/testing split] → [Debugging in programming]
   Just like you test a program with different inputs to find bugs, you test a machine learning model with unseen data to check if it’s "buggy" (overfitting or underfitting). Both processes rely on systematic evaluation to separate "it works on my machine" from "it works in the real world."

2. Across Subjects:
   [Overfitting] → [Confirmation bias in psychology]
   Overfitting is like confirmation bias: the model (or person) latches onto patterns that seem to work in the training data (or past experiences) but fail in new situations. For example, a student who only studies examples from one textbook chapter might ace that chapter’s quiz but fail the final exam because they didn’t learn the underlying concepts.

3. Outside School:
   [Generalization] → [Sports scouting]
   A basketball scout doesn’t just watch a player dominate in one game—they evaluate performance across different opponents, venues, and conditions to see if the player’s skills generalize. Similarly, a machine learning model’s "scouting report" (testing set) tells you if it’s a one-hit wonder or a consistent performer.

6. The Stretch Question

If a model is trained on 100% of the available data (no testing set), can it still be useful? Why or why not?

Pointer Toward the Answer:
- In theory, the model might perform well on the data it was trained on, but you’d have no way to know if it’s overfitting. It’s like a student who memorizes the answers to a practice test but has never taken a real exam—you can’t trust their "perfect score" because they might fail when the questions change.
- In practice, some models are deployed without a testing set (e.g., in low-stakes applications like movie recommendations), but this is risky. Techniques like cross-validation (splitting the data into multiple training/testing sets) can help, but they’re not a perfect substitute for a true holdout set.
- The deeper question: How do we define "useful"? If the model’s only job is to regurgitate the training data (e.g., a chatbot that repeats phrases from its training set), it might be "useful" in a limited way. But if it needs to generalize (e.g., diagnose diseases or drive cars), a testing set is non-negotiable.