TECH **Python for Data Science: Error Handling (try-except-finally, Raising Exceptions) – Zero-Fluff Study Guide**

Python for Data Science: Error Handling (try-except-finally, Raising Exceptions) – Zero-Fluff Study Guide

1. What This Is & Why It Matters

Error handling in Python for Data Science is not just about preventing crashes—it’s about building resilient pipelines that fail gracefully, log useful diagnostics, and recover automatically when things go wrong.

Why This Matters in Production

• Data pipelines break silently. A missing file, a malformed CSV, or a division by zero can corrupt downstream analytics without throwing an obvious error.
• Cloud services fail unpredictably. APIs time out, databases disconnect, and S3 buckets get throttled. If your script doesn’t handle these, you’ll lose data or waste compute costs.
• Debugging is 10x harder in production. Without proper error handling, you’ll spend hours digging through logs to figure out why a job failed at 3 AM.

Real-World Scenario

You’re building a daily ETL job that: 1. Pulls sales data from an API.
2. Cleans it with pandas.
3. Writes results to a database.

Without error handling:
- If the API returns a 500 error, your script crashes, and the database never gets updated.
- If a column name changes in the CSV, pd.read_csv() fails, and you lose a day’s worth of data.
- If the database connection drops mid-write, you corrupt the table.

With error handling:
- The script retries failed API calls.
- It logs missing columns and skips bad rows instead of crashing.
- It rolls back database transactions if something fails.

2. Core Concepts & Components

1. try-except Block

• Definition: Catches and handles exceptions (errors) that occur in a block of code.
• Production Insight: Always catch specific exceptions (e.g., FileNotFoundError) instead of a bare except:. Otherwise, you’ll mask bugs (e.g., a KeyboardInterrupt or MemoryError).

2. else Clause

• Definition: Runs if the try block succeeds (no exceptions).
• Production Insight: Use this for code that should only run if no errors occurred (e.g., committing a database transaction).

3. finally Clause

• Definition: Runs no matter what, whether an exception occurred or not.
• Production Insight: Use this for cleanup (closing files, database connections, or releasing resources). Even if your script crashes, finally ensures no leaks.

4. Raising Exceptions (raise)

• Definition: Manually trigger an exception when a condition isn’t met.
• Production Insight: Use custom exceptions (e.g., raise ValueError("Invalid date format")) to enforce data quality checks in ETL pipelines.

5. Custom Exceptions

• Definition: User-defined exception classes for domain-specific errors.
• Production Insight: Helps distinguish between different failure modes (e.g., APIRateLimitError vs. InvalidDataError).

6. try-except with Context Managers (with)

• Definition: Combines try-except with resource management (e.g., file handling).
• Production Insight: Ensures files/database connections are closed even if an error occurs.

7. Logging Exceptions

• Definition: Record errors with logging.exception() to capture stack traces.
• Production Insight: Critical for debugging production failures. Never rely on print()—logs are searchable and persistent.

8. Retry Mechanisms

• Definition: Automatically retry failed operations (e.g., API calls, database queries).
• Production Insight: Use libraries like tenacity or retry to handle transient failures (e.g., network timeouts).

3. Step-by-Step Hands-On: Building a Robust Data Pipeline

Prerequisites

• Python 3.8+
• pandas, requests, sqlalchemy installed (pip install pandas requests sqlalchemy)
• A local SQLite database (or any other DB for testing)

Task: Write a Fault-Tolerant ETL Script

We’ll build a script that: 1. Fetches JSON data from an API.
2. Cleans it with pandas.
3. Writes it to a database.
4. Handles errors gracefully (retries, logs, and skips bad data).

Step 1: Set Up Logging

import logging

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s - %(levelname)s - %(message)s",
    handlers=[logging.FileHandler("etl.log"), logging.StreamHandler()]
)

Why?
- Logs go to both a file (etl.log) and the console.
- Helps debug failures in production.

Step 2: Fetch Data from an API (with Retries)

import requests
from requests.exceptions import RequestException
from tenacity import retry, stop_after_attempt, wait_exponential

@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=4, max=10))
def fetch_data(url):
    try:
        response = requests.get(url, timeout=10)
        response.raise_for_status()  # Raises HTTPError for bad responses
        return response.json()
    except RequestException as e:
        logging.error(f"API request failed: {e}")
        raise  # Re-raise to trigger retry

Key Features:
- Retries 3 times with exponential backoff (4s, 8s, 16s).
- Logs errors before retrying.
- Raises RequestException if all retries fail.

Step 3: Clean Data with Pandas (Skip Bad Rows)

import pandas as pd

def clean_data(raw_data):
    try:
        df = pd.DataFrame(raw_data)
        # Example: Drop rows with missing 'price' column
        df = df.dropna(subset=["price"])
        return df
    except (KeyError, ValueError) as e:
        logging.error(f"Data cleaning failed: {e}")
        raise  # Re-raise to handle upstream

Why?
- Skips rows with missing critical fields (price).
- Logs errors but doesn’t crash the pipeline.

Step 4: Write to Database (with Transaction Rollback)

from sqlalchemy import create_engine, exc

def write_to_db(df, table_name):
    engine = create_engine("sqlite:///sales.db")
    try:
        with engine.begin() as conn:  # Auto-rollback on failure
            df.to_sql(table_name, conn, if_exists="append", index=False)
    except exc.SQLAlchemyError as e:
        logging.error(f"Database write failed: {e}")
        raise

Key Features:
- Uses engine.begin() for automatic transaction rollback.
- Logs database errors for debugging.

Step 5: Put It All Together (Full Pipeline)

def run_etl(api_url, table_name):
    try:
        logging.info("Starting ETL pipeline")
        raw_data = fetch_data(api_url)
        cleaned_data = clean_data(raw_data)
        write_to_db(cleaned_data, table_name)
        logging.info("ETL completed successfully")
    except Exception as e:
        logging.exception("ETL failed")  # Logs full stack trace
        raise  # Re-raise to notify monitoring systems
    finally:
        logging.info("ETL run finished (success or failure)")

# Example usage
run_etl("https://api.example.com/sales", "daily_sales")

Expected Output (Success):

2023-10-01 12:00:00 - INFO - Starting ETL pipeline
2023-10-01 12:00:01 - INFO - ETL completed successfully
2023-10-01 12:00:01 - INFO - ETL run finished (success or failure)

Expected Output (Failure):

2023-10-01 12:00:00 - INFO - Starting ETL pipeline
2023-10-01 12:00:01 - ERROR - API request failed: 500 Server Error
2023-10-01 12:00:05 - ERROR - API request failed: 500 Server Error
2023-10-01 12:00:13 - ERROR - API request failed: 500 Server Error
2023-10-01 12:00:13 - ERROR - ETL failed
Traceback (most recent call last):
  File "etl.py", line 42, in run_etl
    raw_data = fetch_data(api_url)
  File "/usr/local/lib/python3.9/site-packages/tenacity/__init__.py", line 324, in wrapped_f
    return self(f, *args, kw)
  File "/usr/local/lib/python3.9/site-packages/tenacity/__init__.py", line 404, in __call__
    do = self.iter(retry_state=retry_state)
  File "/usr/local/lib/python3.9/site-packages/tenacity/__init__.py", line 350, in iter
    raise retry_exc.reraise()
  File "/usr/local/lib/python3.9/site-packages/tenacity/__init__.py", line 193, in reraise
    raise self.last_attempt.result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 438, in result
    return self.__get_result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 390, in __get_result
    raise self._exception
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.example.com/sales
2023-10-01 12:00:13 - INFO - ETL run finished (success or failure)

4. ? Production-Ready Best Practices

Reliability & Maintainability

• Always log exceptions with logging.exception() (not print()).
• Use specific exceptions (e.g., FileNotFoundError instead of bare except:).
• Implement retries for transient failures (APIs, databases, network calls).
• Validate inputs early (e.g., check if a file exists before processing).
• Use context managers (with) for files/database connections to ensure cleanup.

Observability

• Log structured data (JSON) for easier querying in tools like ELK or Datadog.
• Include correlation IDs in logs to trace a single request across services.
• Monitor error rates (e.g., Prometheus + Grafana) to catch issues early.

Security

• Never log sensitive data (API keys, passwords, PII).
• Use environment variables for secrets (not hardcoded in scripts).
• Sanitize error messages in production (don’t expose stack traces to users).

Cost Optimization

• Fail fast—don’t waste compute resources on doomed operations.
• Use exponential backoff for retries to avoid overwhelming failing services.
• Clean up resources (e.g., close database connections) to avoid leaks.

5. ⚠️ Common Mistakes & Traps

6. ? Exam/Certification Focus

Typical Question Patterns

1. "What’s the difference between except: and except Exception:?"
2. except: catches all exceptions (including KeyboardInterrupt and SystemExit).
3. except Exception: catches all non-system-exiting exceptions.

4. Answer: Always use except Exception: (or specific exceptions) unless you have a good reason.

5. "When should you use else in a try-except block?"

6. The else block runs only if no exceptions occur.

7. Answer: Use it for code that should run on success (e.g., committing a transaction).

8. "What’s the purpose of finally?"

9. Runs no matter what, even if an exception occurs.

10. Answer: Use it for cleanup (closing files, database connections).

11. "How do you create a custom exception?"
    python
class InvalidDataError(Exception):
pass

12. Answer: Subclass Exception and raise it with raise InvalidDataError("message").

Key Trap Distinctions

• try-except vs. try-finally:
• try-except handles errors.
• try-finally ensures cleanup (but doesn’t handle errors).

• Use both together for robust error handling + cleanup.

• Logging vs. Printing:

• print() is for debugging (not production).
• logging.exception() captures stack traces (critical for debugging).

7. ? Hands-On Challenge

Challenge: Write a Fault-Tolerant CSV Reader

Write a function that: 1. Reads a CSV file.
2. Skips rows with missing values in the price column.
3. Logs errors for bad rows.
4. Returns a cleaned DataFrame.

Solution:

import pandas as pd
import logging

def read_clean_csv(file_path):
    try:
        df = pd.read_csv(file_path)
        initial_rows = len(df)
        df = df.dropna(subset=["price"])
        skipped_rows = initial_rows - len(df)
        if skipped_rows > 0:
            logging.warning(f"Skipped {skipped_rows} rows with missing 'price'")
        return df
    except FileNotFoundError:
        logging.error(f"File not found: {file_path}")
        raise
    except Exception as e:
        logging.exception(f"Failed to read CSV: {e}")
        raise

# Example usage
df = read_clean_csv("sales.csv")

Why It Works:
- Skips bad rows instead of crashing.
- Logs warnings for skipped rows.
- Raises exceptions for unrecoverable errors (e.g., missing file).

8. ? Rapid-Reference Crib Sheet

9. ? Where to Go Next

1. Python logging docs – Official guide to logging.
2. Tenacity (retry library) – Advanced retry strategies.
3. Pandas error handling – Common pandas exceptions.
4. Real Python: Exception Handling – Deep dive into Python exceptions.