TECH **Python for Data Science: Lists, Tuples, Sets, Dictionaries – Use Cases & Performance**

Python for Data Science: Lists, Tuples, Sets, Dictionaries – Use Cases & Performance

A hyper-practical, zero-fluff guide for real projects and certifications (PL-300, DP-100, etc.)

1. What This Is & Why It Matters

You’re building a data pipeline, cleaning a dataset, or optimizing a machine learning feature store. Your code is slow, memory-hungry, or crashes with KeyError/IndexError. The root cause? You picked the wrong Python data structure.

• Lists are your Swiss Army knife—flexible but slow for lookups.
• Tuples are immutable and memory-efficient—perfect for fixed data (e.g., coordinates, database rows).
• Sets are blazing fast for membership tests—ideal for deduplication or filtering.
• Dictionaries are O(1) lookup powerhouses—critical for feature engineering, JSON parsing, and caching.

Real-world scenario:
You’re processing 10M user records. A for loop with if x in list takes 30 seconds. Switching to if x in set drops it to 0.003 seconds. That’s a 10,000x speedup—enough to turn an overnight batch job into a real-time API.

What breaks if you ignore this?
- Performance: Your ETL pipeline times out in production.
- Memory: Your Docker container OOM-kills because you used lists instead of tuples for static data.
- Correctness: Your deduplication logic fails silently because you used a list instead of a set.

2. Core Concepts & Components

? List (list)

• Definition: Ordered, mutable sequence. [1, 2, 3]
• Production insight:
• Slow lookups: O(n) for x in list. Use only for small datasets or when order matters.
• Memory overhead: Lists pre-allocate extra space (like a stretchy backpack). Good for dynamic growth, bad for static data.

? Tuple (tuple)

• Definition: Ordered, immutable sequence. (1, 2, 3)
• Production insight:
• Faster and lighter: Tuples use ~25% less memory than lists (critical for large datasets).
• Hashable: Can be used as dictionary keys (e.g., (lat, lon) for geospatial lookups).
• Thread-safe: Immutable = no race conditions in parallel processing.

? Set (set)

• Definition: Unordered, mutable collection of unique elements. {1, 2, 3}
• Production insight:
• O(1) membership tests: x in set is 1000x faster than x in list for large datasets.
• Deduplication: set(list_with_dupes) removes duplicates instantly.
• No indexing: Can’t do set[0]—use only for membership/filtering.

? Dictionary (dict)

• Definition: Unordered key-value pairs. {"name": "Alice", "age": 30}
• Production insight:
• O(1) lookups: dict["key"] is as fast as a set for membership.
• Memory tradeoff: Dictionaries use ~3x more memory than lists for the same data (but worth it for lookups).
• JSON-native: Directly maps to/from JSON (critical for APIs and NoSQL databases).

3. Step-by-Step Hands-On: Optimizing a Data Pipeline

Task: You’re processing a CSV of 1M user records. Your goal: 1. Deduplicate users by email.
2. Count unique country values.
3. Store results in a dictionary for fast lookups.

Prerequisites

• Python 3.8+ (for := walrus operator, but not required).
• A CSV file (users.csv) with columns: email, name, country.

Step 1: Load Data (Bad vs. Good)

❌ Bad (slow, memory-heavy):

import csv

emails = []
countries = []
with open("users.csv") as f:
    reader = csv.DictReader(f)
    for row in reader:
        emails.append(row["email"])  # O(1) append, but O(n) lookups later
        countries.append(row["country"])

✅ Good (optimized):

emails = set()  # O(1) membership tests
countries = set()
with open("users.csv") as f:
    reader = csv.DictReader(f)
    for row in reader:
        emails.add(row["email"])  # Deduplicates automatically
        countries.add(row["country"])

Step 2: Count Unique Countries (Bad vs. Good)

❌ Bad (O(n²) time):

unique_countries = []
for country in countries:
    if country not in unique_countries:  # O(n) lookup
        unique_countries.append(country)
print(len(unique_countries))

✅ Good (O(1) time):

print(len(countries))  # Sets are already unique

Step 3: Build a Fast Lookup Dictionary

❌ Bad (O(n) lookups):

user_data = []
with open("users.csv") as f:
    reader = csv.DictReader(f)
    for row in reader:
        user_data.append(row)  # List of dicts

# Later: Find user by email (O(n) time)
for user in user_data:
    if user["email"] == "[email protected]":
        print(user)

✅ Good (O(1) lookups):

user_dict = {}
with open("users.csv") as f:
    reader = csv.DictReader(f)
    for row in reader:
        user_dict[row["email"]] = row  # Key: email, Value: full row

# Later: Instant lookup
print(user_dict["[email protected]"])

Step 4: Verify Performance

import time

# Test list vs. set membership
big_list = list(range(1_000_000))
big_set = set(big_list)

# Time list lookup
start = time.time()
_ = 999_999 in big_list
print(f"List lookup: {time.time() - start:.6f} sec")  # ~0.02 sec

# Time set lookup
start = time.time()
_ = 999_999 in big_set
print(f"Set lookup: {time.time() - start:.6f} sec")  # ~0.000001 sec

Output:

List lookup: 0.021456 sec
Set lookup: 0.000001 sec  # 20,000x faster

4. ? Production-Ready Best Practices

Performance

• Use tuples for static data: Coordinates, database rows, or configuration constants.
  ```python # Bad (mutable, memory-heavy) LAT_LON = [37.7749, -122.4194]

# Good (immutable, memory-efficient) LAT_LON = (37.7749, -122.4194) - Replace `list.count(x)` with `collections.Counter` for frequency counting.python from collections import Counter counts = Counter(countries) # {"US": 1000, "IN": 500, ...} - Use `dict.get(key, default)` to avoid `KeyError` in production.python user = user_dict.get("[email protected]", {"name": "Unknown"}) ```

Memory

• For large datasets, prefer generators (()) over lists ([]) to avoid loading everything into memory.
  ```python # Bad (loads all rows into memory) rows = [row for row in csv.DictReader(f)]

# Good (streams rows one at a time) rows = (row for row in csv.DictReader(f)) - Use `__slots__` in classes to reduce memory overhead (advanced).python class User:
slots = ["email", "name"] # Saves ~40% memory vs. dynamic attributes ```

Correctness

• Never modify a list while iterating over it. Use a copy or a set.
  ```python # Bad (skips elements) for x in my_list:
  if x % 2 == 0:
  my_list.remove(x)

# Good my_list = [x for x in my_list if x % 2 != 0] - Use `frozenset` for immutable sets (e.g., as dictionary keys).python key = frozenset({"US", "IN"}) # Can be a dict key ```

5. ⚠️ Common Mistakes & Traps

6. ? Exam/Certification Focus

Typical question patterns:
1. Performance: "Which data structure is fastest for checking if an element exists in a collection?"
- Answer: set (O(1) lookup).
- Trap: list (O(n)) or tuple (O(n)).

1. Memory: "You need to store 1M latitude-longitude pairs. Which structure uses the least memory?"
2. Answer: tuple (immutable, compact).

3. Trap: list (pre-allocates extra space).

4. Deduplication: "How do you remove duplicates from a list while preserving order?"

5. Answer:
   python
from collections import OrderedDict
list(OrderedDict.fromkeys(my_list))

6. Trap: set(my_list) (loses order).

7. Dictionary keys: "Which of these can be a dictionary key?"

8. Answer: tuple (immutable), str, int, frozenset.
9. Trap: list (mutable, unhashable).

7. ? Hands-On Challenge

Problem:
You have a list of 100,000 user IDs (some duplicates). Write a function that: 1. Returns the count of unique IDs.
2. Returns the 10 most frequent IDs.
3. Runs in < 0.1 seconds.

Solution:

from collections import Counter

def analyze_user_ids(user_ids):
    unique_count = len(set(user_ids))  # O(n) time, O(n) space
    top_10 = Counter(user_ids).most_common(10)  # O(n log n) time
    return unique_count, top_10

# Test
user_ids = [1, 2, 2, 3, 3, 3] * 100000
unique_count, top_10 = analyze_user_ids(user_ids)
print(f"Unique IDs: {unique_count}, Top 10: {top_10}")

Why it works:
- set deduplicates in O(n) time.
- Counter is optimized for frequency counting (faster than manual loops).

8. ? Rapid-Reference Crib Sheet

Key defaults:
- dict iteration order is insertion-ordered (Python 3.7+).
- set and dict resize dynamically (like lists).
- tuple is ~25% more memory-efficient than list.

9. ? Where to Go Next

1. Python collections module – defaultdict, Counter, deque.
2. Time Complexity in Python – Official cheat sheet for all operations.
3. High Performance Python (Book) – Chapter 2 covers data structures in depth.
4. Real Python: Dictionaries – Practical examples and performance tips.