Computer Science - ICT Grade 10 Web Scraping Automating Data Collection

Study Guide: Web Scraping – Automating Data Collection
Grade 10 | Computer Science – ICT

1. The Driving Question

"If a website has thousands of product prices, sports scores, or weather updates, how can you grab all that data without copying and pasting for hours? And why can’t you just ‘save as’ the webpage—what’s actually hiding behind the text you see?"

2. The Core Idea – Built, Not Listed

Imagine you’re running a fantasy basketball league, and you need the latest player stats from ESPN’s website to update your team rankings. The stats are right there on the page—but they’re buried in HTML code, like a treasure map where the X’s are invisible unless you know how to read it. Web scraping is the tool that lets you write a short program (like a robot assistant) to automatically find those X’s, extract the numbers, and save them in a spreadsheet or database. It’s not magic; it’s just teaching your computer to recognize patterns in the website’s structure (like how a <table> tag always wraps around data in rows and columns) and pull out the parts you care about.

But here’s the catch: websites don’t want to be scraped. They’re designed for humans, not robots, so they might block your program if it acts too greedy (like refreshing the page 100 times a second). That’s why scrapers need to be polite—adding delays, mimicking human behavior, and respecting a website’s robots.txt file (a set of rules that says, “Hey, don’t scrape this page!”).

Key Vocabulary:
- HTML (HyperText Markup Language)
Definition: The "skeleton" of a webpage, made of tags like <div> or <p> that tell your browser how to display content.
Example: The <h1> tag in <h1>Welcome to My Blog</h1> tells the browser to show that text as a big, bold heading.
College Note: In web development, HTML5 introduces semantic tags (<article>, <nav>) that make scraping and accessibility easier.

• API (Application Programming Interface)
  Definition: A "waiter" between your program and a website’s data—it gives you structured data (like JSON) without needing to scrape.
  Example: Twitter’s API lets you fetch tweets in a clean format, while scraping Twitter’s HTML would be messy and unreliable.
  College Note: APIs often require authentication (API keys) and rate limits, which become critical in large-scale data projects.

• Selector (CSS or XPath)
  Definition: A "GPS coordinate" for finding specific elements in HTML, like a button or a price.
  Example: The CSS selector div.product > span.price finds all <span> tags with class price inside a <div> with class product.
  College Note: XPath is more powerful than CSS for complex scraping (e.g., finding the 3rd <li> in a list), but it’s slower.

• Rate Limiting
  Definition: The practice of slowing down your scraper to avoid overwhelming a website (like waiting 2 seconds between requests).
  Example: If you scrape Amazon’s product pages too fast, your IP address might get temporarily blocked.
  College Note: Advanced scrapers use proxies (fake IP addresses) to bypass rate limits, but this raises ethical and legal questions.

3. Assessment Translation

How This Appears on Assessments:
- Classroom Tasks (Grade 10 ICT):
- Exit Ticket: Write a CSS selector to extract all book titles from a sample HTML snippet (e.g., <div class="book"><h2>Dune</h2></div>). Proficient: div.book > h2 or .book h2. Developing: h2 (too broad) or div.book (misses the title).
- Short Constructed Response: "Explain why a website might block a scraper that sends 100 requests per second. What’s one way to avoid this?" Proficient: Mentions rate limiting, IP bans, or robots.txt. Developing: Says "it’s rude" without technical detail.
- Coding Task: Write a Python script using BeautifulSoup to scrape the top 5 headlines from a news site. Proficient: Uses correct selectors, handles errors (e.g., missing elements), and saves data to a CSV. Developing: Hardcodes selectors or crashes if the HTML structure changes.

• Standardized Tests (e.g., AP CSP, State ICT Exams):
• Multiple Choice: "Which of these is a legal reason a website might block a scraper?" (Distractors: "The scraper is written in Python," "The website doesn’t like the user’s browser," "The scraper didn’t ask permission.")

• Short Answer: "Compare web scraping to using an API. Give one advantage of each." Proficient: Scraping = no API needed; API = structured data. Developing: Confuses scraping with hacking.

• Model Proficient Response (Coding Task):
  ```python import requests from bs4 import BeautifulSoup import csv

url = "https://example-news-site.com" response = requests.get(url, headers={"User-Agent": "Mozilla/5.0"}) soup = BeautifulSoup(response.text, "html.parser")

headlines = [] for article in soup.select("div.article > h2.headline")[:5]:
headlines.append(article.text.strip())

with open("headlines.csv", "w", newline="") as file:
writer = csv.writer(file)
writer.writerow(["Headline"])
for headline in headlines:
writer.writerow([headline]) `` *Why This Works:* Uses a realistic selector, limits to 5 headlines, handles headers (to avoid blocks), and saves to CSV. A "perfect" response might add error handling (e.g.,try/except` for failed requests).

4. Mistake Taxonomy

Mistake 1: Overly Broad Selectors
- Prompt: "Write a CSS selector to extract all product prices from this HTML: <div class="product"><span class="price">$19.99</span></div>" - Common Wrong Response: span or div span - Why It Loses Credit: Grabs all <span> tags on the page, not just prices. Might include unrelated text (e.g., shipping costs).
- Correct Approach: Use specificity: .product .price or div.product > span.price. Test the selector in browser dev tools first.

Mistake 2: Ignoring Dynamic Content
- Prompt: "Your scraper works on your local HTML file but fails on the live website. Why?" - Common Wrong Response: "The website is broken" or "My code is wrong." - Why It Loses Credit: Doesn’t consider JavaScript-rendered content (e.g., React/Angular sites). Scrapers like BeautifulSoup only see the initial HTML, not data loaded later.
- Correct Approach: Use a tool like Selenium or Playwright to control a real browser, or check if the data is embedded in a <script> tag as JSON.

Mistake 3: Ethical/Legal Oversights
- Prompt: "Is it legal to scrape a website’s data for a school project? Explain your answer." - Common Wrong Response: "Yes, because it’s public data" or "No, because it’s stealing." - Why It Loses Credit: Misses nuance (e.g., robots.txt, terms of service, copyright law). "Public" ≠ "free to scrape." - Correct Approach: Check robots.txt (e.g., https://example.com/robots.txt). Mention that scraping for personal use is often tolerated, but republishing data may violate terms of service. Cite hiQ Labs v. LinkedIn (2019) as a legal precedent.

5. Connection Layer

• Within CS: Web scraping → Data Cleaning Why: Scraped data is often messy (e.g., prices with "$" symbols, dates in different formats). Understanding scraping helps you anticipate what needs cleaning before analysis.

• Across Subjects: Web scraping → Economics (Market Research) Why: Companies scrape competitor prices to adjust their own (e.g., Amazon vs. Walmart). The same logic applies to tracking supply/demand trends in microeconomics.

• Outside School: Web scraping → Fantasy Sports Why: Apps like DraftKings scrape player stats from ESPN or Yahoo to update live scores. If you’ve ever wondered how your fantasy team auto-updates, now you know.

6. The Stretch Question

"If a website changes its HTML structure, your scraper breaks. How could you design a scraper that ‘adapts’ to small changes—like a <div> becoming a <section>—without needing constant updates?"

Pointer Toward the Answer:
Think about semantic patterns instead of exact tags. For example, if you’re scraping book titles, look for text that’s bold, inside a container with "book" in its class name, and longer than 3 words. Tools like machine learning (e.g., training a model to recognize "price-like" text) or fuzzy matching (e.g., "find the number closest to the word ‘price’") can help. This is how companies like Diffbot build "AI scrapers" that work even when websites redesign.