TECH **Python for Data Science: File I/O & Context Managers (`with` Statement) – Zero-Fluff Study Guide**

Python for Data Science: File I/O & Context Managers (with Statement) – Zero-Fluff Study Guide

1. What This Is & Why It Matters

What It Is

File I/O (Input/Output) is how your Python scripts read from and write to files—CSVs, JSON, Parquet, logs, or even raw binary data. Context managers (the with statement) are Python’s way of ensuring files are properly opened, used, and closed—even if your code crashes mid-operation.

Why It Matters in Production

• Memory leaks: If you forget to close a file, your script holds resources hostage. In a long-running data pipeline, this can crash your server.
• Corrupted data: If a script fails mid-write, you might end up with a half-written file. Context managers prevent this.
• Security risks: Open file handles can be exploited (e.g., log files with sensitive data left unclosed).
• Performance: Unclosed files slow down your system. In big data workflows, this adds up fast.

Real-World Scenario

You’re building a daily ETL pipeline that: 1. Reads a 10GB CSV from S3.
2. Cleans the data with pandas.
3. Writes the result to a Parquet file.
4. Uploads it back to S3.

If you don’t use a context manager, your script might: - Crash mid-write, leaving a corrupted Parquet file.
- Leak file handles, causing the next run to fail with Too many open files.
- Leave temporary files on disk, filling up your server’s storage.

Context managers fix all of this automatically.

2. Core Concepts & Components

1. File Handles

• Definition: A "pointer" to an open file in memory. Like a bookmark in a book—it tells Python where to read/write.
• Production insight: Every open file handle consumes system resources. If you don’t close them, your script (or server) will eventually crash with OSError: [Errno 24] Too many open files.

2. open() Function

• Definition: The built-in Python function to open a file. Takes a filename and a mode ('r', 'w', 'a', etc.).
• Production insight: Always specify the encoding (e.g., encoding='utf-8'). If you don’t, you’ll get UnicodeDecodeError when reading non-ASCII text (e.g., emojis, Chinese characters).

3. File Modes

• Production insight: 'w' erases the file immediately when opened. If you’re debugging, use 'a' to avoid accidental data loss.

4. with Statement (Context Manager)

• Definition: A Python construct that ensures a file is automatically closed when the block ends, even if an error occurs.
• Production insight: This is non-negotiable in production. Manual file.close() is error-prone (what if an exception happens before it runs?).

5. read() vs readlines() vs readline()

• Production insight: read() on a 10GB file will crash your script. Use readline() or chunked reading instead.

6. pandas.read_csv() / to_csv()

• Definition: Pandas’ optimized functions for reading/writing CSV files. Under the hood, they use Python’s file I/O but with better performance and error handling.
• Production insight: Always use index=False when writing to avoid an extra "Unnamed: 0" column.

7. json.load() / json.dump()

• Definition: Python’s built-in JSON parser. load() reads a JSON file into a Python dict/list, dump() writes a Python object to JSON.
• Production insight: Use indent=2 for human-readable JSON (e.g., config files). Omit it for machine-readable (smaller file size).

8. pathlib.Path (Modern File Handling)

• Definition: A Python 3+ library for object-oriented file paths (better than os.path).
• Production insight: Use Path('file.txt').read_text() instead of open(). It’s cleaner and handles encoding automatically.

3. Step-by-Step Hands-On: Building a Robust File I/O Pipeline

Prerequisites

• Python 3.8+ (for pathlib and modern open() defaults).
• A sample CSV file (e.g., data.csv with columns id, name, value).
• Pandas installed (pip install pandas).

Task

Build a script that: 1. Reads a CSV file.
2. Filters rows where value > 100.
3. Writes the result to a new CSV.
4. Handles errors gracefully (e.g., missing file, corrupt data).

Step 1: Basic File I/O (Without Context Manager – ❌ Bad Practice)

# ❌ DON'T DO THIS IN PRODUCTION
file = open('data.csv', 'r')
data = file.read()
file.close()  # What if an error happens before this line?

Problem: If file.read() fails, file.close() never runs → resource leak.

Step 2: Using with (✅ Correct Way)

# ✅ Safe and clean
with open('data.csv', 'r', encoding='utf-8') as file:
    data = file.read()
# File is automatically closed here, even if an error occurs

Step 3: Reading CSV with Pandas (Best for Data Science)

import pandas as pd

try:
    df = pd.read_csv('data.csv')
    print("Original data shape:", df.shape)
except FileNotFoundError:
    print("❌ Error: File not found. Check the path.")
    exit(1)
except pd.errors.EmptyDataError:
    print("❌ Error: File is empty.")
    exit(1)
except Exception as e:
    print(f"❌ Unexpected error: {e}")
    exit(1)

Why this matters:
- pd.read_csv() handles encoding, missing values, and data types automatically.
- The try/except block prevents crashes in production.

Step 4: Filtering and Writing CSV

filtered_df = df[df['value'] > 100]

# ✅ Safe write with context manager
with open('filtered_data.csv', 'w', encoding='utf-8') as file:
    filtered_df.to_csv(file, index=False)  # index=False avoids extra column

Key details:
- index=False prevents pandas from adding an extra "Unnamed: 0" column.
- encoding='utf-8' ensures special characters (e.g., é, ñ) are saved correctly.

Step 5: Handling Large Files (Chunking)

# Process a 10GB file without loading it all into memory
chunk_size = 10_000  # Rows per chunk
with pd.read_csv('large_data.csv', chunksize=chunk_size) as reader:
    for i, chunk in enumerate(reader):
        filtered_chunk = chunk[chunk['value'] > 100]
        # Append to output file (mode='a' for append)
        with open('filtered_large_data.csv', 'a', encoding='utf-8') as f:
            filtered_chunk.to_csv(f, header=(i == 0), index=False)

Why this matters:
- Memory efficiency: Only loads chunk_size rows at a time.
- Header handling: header=(i == 0) ensures the header is only written once.

Step 6: Using pathlib for Cleaner Code

from pathlib import Path

input_path = Path('data.csv')
output_path = Path('filtered_data.csv')

if not input_path.exists():
    print(f"❌ Error: {input_path} does not exist.")
    exit(1)

df = pd.read_csv(input_path)
filtered_df = df[df['value'] > 100]
filtered_df.to_csv(output_path, index=False)

Advantages:
- Cross-platform: Works on Windows, Linux, and macOS.
- Readable: Path('file.txt').read_text() is cleaner than open('file.txt').read().

Step 7: Logging for Debugging

import logging

logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(levelname)s - %(message)s',
    filename='pipeline.log'
)

try:
    df = pd.read_csv('data.csv')
    logging.info(f"Read {len(df)} rows from data.csv")
    filtered_df = df[df['value'] > 100]
    filtered_df.to_csv('filtered_data.csv', index=False)
    logging.info(f"Wrote {len(filtered_df)} rows to filtered_data.csv")
except Exception as e:
    logging.error(f"Pipeline failed: {e}", exc_info=True)
    raise  # Re-raise to fail the job (e.g., in Airflow)

Why this matters:
- Debugging: Logs help you trace failures in production.
- Audit trail: You can track how many rows were processed.

4. ? Production-Ready Best Practices

Security

• Never hardcode paths: Use environment variables or config files.
  python import os input_path = os.getenv('INPUT_PATH', 'data.csv') # Fallback to 'data.csv'
• Validate file extensions: Prevent malicious file uploads (e.g., .php files disguised as .csv).
  python if not input_path.endswith('.csv'):
raise ValueError("Only CSV files are allowed.")
• Use tempfile for sensitive data: If processing PII, write to a temporary file and delete it after use.
  python import tempfile with tempfile.NamedTemporaryFile(delete=True) as tmp:
df.to_csv(tmp.name)
# Process tmp.name, then it's automatically deleted

Cost Optimization

• Compress large files: Use gzip for CSVs to save storage costs.
  python df.to_csv('data.csv.gz', index=False, compression='gzip')
• Use Parquet for analytics: Faster and smaller than CSV.
  python df.to_parquet('data.parquet', engine='pyarrow')

Reliability & Maintainability

• Idempotency: Ensure your script can be rerun safely (e.g., don’t append duplicates).
  python if output_path.exists():
logging.warning(f"{output_path} already exists. Overwriting.")
• Atomic writes: Write to a temporary file first, then rename (prevents corruption if the script crashes).
  python temp_path = output_path.with_suffix('.tmp') df.to_csv(temp_path, index=False) temp_path.rename(output_path) # Atomic operation

Observability

• Monitor file sizes: Log the size of input/output files.
  python logging.info(f"Input file size: {input_path.stat().st_size / 1e6:.2f} MB")
• Track processing time: Use time.time() to log how long each step takes.
  python import time start = time.time() df = pd.read_csv(input_path) logging.info(f"Read took {time.time() - start:.2f} seconds")

5. ⚠️ Common Mistakes & Traps

6. ? Exam/Certification Focus

Typical Question Patterns

1. "What’s the difference between read() and readlines()?"
2. read() → Entire file as a string.

3. readlines() → List of lines (memory-heavy for large files).

4. "Why should you use with instead of open() + close()?"

5. with guarantees the file is closed, even if an error occurs.

6. "What happens if you open a file in 'w' mode and it already exists?"

7. The file is truncated (erased) immediately.

8. "How do you read a CSV file in chunks with pandas?"

9. pd.read_csv('file.csv', chunksize=1000).

10. "What’s the risk of not specifying encoding when opening a file?"

11. UnicodeDecodeError if the file contains non-ASCII characters.

Key Trap Distinctions

Scenario-Based Question

"You’re processing a 50GB CSV file. Your script crashes with MemoryError. What’s the most efficient fix?"
- ❌ Increase server RAM (expensive and not scalable).
- ✅ Use pd.read_csv(chunksize=10_000) to process in chunks.

7. ? Hands-On Challenge

Challenge

Write a script that: 1. Reads a JSON file (config.json) with this structure:
json
{
"input_path": "data.csv",
"output_path": "output.parquet",
"threshold": 100
} 2. Reads the CSV file specified in input_path.
3. Filters rows where the column value > threshold.
4. Writes the result to output_path in Parquet format.
5. Handles errors (missing file, invalid JSON, corrupt CSV).

Solution

import json
import pandas as pd
from pathlib import Path

# Read config
try:
    with open('config.json', 'r', encoding='utf-8') as f:
        config = json.load(f)
except FileNotFoundError:
    print("❌ config.json not found.")
    exit(1)
except json.JSONDecodeError:
    print("❌ config.json is not valid JSON.")
    exit(1)

# Validate paths
input_path = Path(config['input_path'])
output_path = Path(config['output_path'])

if not input_path.exists():
    print(f"❌ Input file {input_path} not found.")
    exit(1)

# Read and filter data
try:
    df = pd.read_csv(input_path)
    filtered_df = df[df['value'] > config['threshold']]
    filtered_df.to_parquet(output_path)
    print(f"✅ Success! Wrote {len(filtered_df)} rows to {output_path}.")
except Exception as e:
    print(f"❌ Error: {e}")
    exit(1)

Why it works:
- Uses with for safe file handling.
- Validates inputs before processing.
- Converts to Parquet (efficient for analytics).
- Handles errors gracefully.

8. ? Rapid-Reference Crib Sheet

File I/O Cheat Sheet

Context Manager Cheat Sheet

9. ? Where to Go Next

1. Python pathlib Docs – Modern file handling.
2. Pandas I/O Docs – Optimized file reading/writing.
3. Real Python: Reading/Writing Files – Deep dive with examples.
4. Python tempfile Docs – Secure temporary files.