TECH **Python for Data Science: Functions, Lambda Expressions, and Scope**

Python for Data Science: Functions, Lambda Expressions, and Scope

A Hyper-Practical, Zero-Fluff Study Guide

1. What This Is & Why It Matters

You’re working on a data pipeline that processes millions of rows from a CSV, applies transformations, and exports results to a database. Your teammate wrote a 500-line script with duplicated logic, hardcoded values, and no error handling. The script fails silently when the input format changes, and debugging takes hours.

Functions, lambda expressions, and scope are your tools to: - Eliminate duplication (DRY principle).
- Make code reusable (e.g., a clean_data() function used across multiple notebooks).
- Control side effects (e.g., avoiding global variables that break reproducibility).
- Write concise, readable transformations (e.g., df['column'].apply(lambda x: x.strip())).

Real-world scenario:
You inherit a Jupyter notebook where every transformation is copy-pasted. You need to refactor it into modular functions to: - Add unit tests.
- Deploy it as a serverless AWS Lambda for scheduled runs.
- Share it with your team without breaking existing workflows.

If you ignore these concepts, your code becomes: - Unmaintainable (e.g., fixing a bug in one place but missing 10 others).
- Unscalable (e.g., a 10-line script becomes 1,000 lines of spaghetti).
- Unreliable (e.g., global variables causing race conditions in parallel processing).

2. Core Concepts & Components

1. Functions (def)

• Definition: A reusable block of code that performs a specific task.
• Production insight: If you copy-paste the same logic 3+ times, refactor it into a function. Example: A preprocess_text() function used in NLP pipelines.

2. Parameters vs. Arguments

• Parameters: Variables listed in the function definition (def foo(param):).
• Arguments: Actual values passed to the function (foo("hello")).
• Production insight: Use type hints (def foo(param: str) -> int:) for better IDE support and documentation.

3. Return Values

• Definition: The output of a function (e.g., return result).
• Production insight: Always explicitly return something (even None). Silent failures are debugging nightmares.

4. Lambda Expressions (lambda)

• Definition: Anonymous, single-expression functions (e.g., lambda x: x * 2).
• Production insight: Use for short, one-off operations (e.g., df['column'].apply(lambda x: x.upper())). Avoid for complex logic—use def instead.

5. Scope (Local vs. Global)

• Local scope: Variables defined inside a function (only accessible within it).
• Global scope: Variables defined outside functions (accessible everywhere).
• Production insight: Avoid global variables—they make debugging and parallel processing a nightmare. Use function parameters instead.

6. *args and kwargs

• *args: Accepts any number of positional arguments (e.g., def foo(*args):).
• kwargs: Accepts any number of keyword arguments (e.g., def foo(kwargs):).
• Production insight: Useful for flexible functions (e.g., a plot_data() function that accepts optional styling arguments).

7. Default Arguments

• Definition: Parameters with predefined values (e.g., def foo(x=1):).
• Production insight: Mutable defaults (e.g., def foo(x=[]):) are a trap—they retain state between calls. Use None instead.

8. Closures

• Definition: A function that remembers variables from its enclosing scope (e.g., a counter function).
• Production insight: Useful for decorators (e.g., @timing to log function execution time).

9. First-Class Functions

• Definition: Functions can be passed as arguments, returned from other functions, or assigned to variables.
• Production insight: Enables higher-order functions (e.g., map(), filter(), sorted() with a key function).

10. Docstrings

• Definition: String literals that document functions (e.g., """Returns x squared.""").
• Production insight: Always include docstrings—they’re used by help(), IDEs, and tools like Sphinx for documentation.

3. Step-by-Step Hands-On: Refactoring a Data Pipeline

Prerequisites

• Python 3.8+ installed.
• A CSV file (data.csv) with columns: id, name, value, timestamp.
• Pandas installed (pip install pandas).

Task:

Refactor a messy script into modular functions with proper scope and lambda expressions.

Original Script (Messy)

import pandas as pd

df = pd.read_csv("data.csv")
df["value"] = df["value"].str.replace("$", "").astype(float)
df["timestamp"] = pd.to_datetime(df["timestamp"])
df = df[df["value"] > 0]
df["name"] = df["name"].str.strip().str.title()
df.to_csv("cleaned_data.csv", index=False)

Step 1: Identify Repeated Logic

• String cleaning (str.replace, str.strip, str.title).
• Type conversion (astype(float), pd.to_datetime).
• Filtering (df[df["value"] > 0]).

Step 2: Write Helper Functions

import pandas as pd

def clean_currency(value: str) -> float:
    """Remove $ and convert to float."""
    return float(value.replace("$", ""))

def clean_text(text: str) -> str:
    """Strip whitespace and title-case text."""
    return text.strip().title()

def filter_positive(df: pd.DataFrame, column: str) -> pd.DataFrame:
    """Filter rows where column > 0."""
    return df[df[column] > 0]

Step 3: Refactor the Main Script

def preprocess_data(input_path: str, output_path: str) -> None:
    """Clean and filter data, then save to CSV."""
    df = pd.read_csv(input_path)

    # Apply transformations
    df["value"] = df["value"].apply(clean_currency)
    df["timestamp"] = pd.to_datetime(df["timestamp"])
    df = filter_positive(df, "value")
    df["name"] = df["name"].apply(clean_text)

    # Save
    df.to_csv(output_path, index=False)

# Run the pipeline
preprocess_data("data.csv", "cleaned_data.csv")

Step 4: Add Lambda for Short Operations

# Replace clean_text with a lambda (if logic is simple)
df["name"] = df["name"].apply(lambda x: x.strip().title())

Step 5: Verify Output

head cleaned_data.csv

Expected output:

id,name,value,timestamp
1,Alice,100.0,2023-01-01
2,Bob,200.0,2023-01-02

4. ? Production-Ready Best Practices

Security

• Avoid hardcoded secrets (e.g., database passwords). Use environment variables or AWS Secrets Manager.
• Validate inputs (e.g., check df is not empty before processing).

Cost Optimization

• Use lambdas for short-lived tasks (e.g., AWS Lambda for ETL jobs).
• Cache expensive computations (e.g., @functools.lru_cache for pure functions).

Reliability & Maintainability

• Write docstrings for every function (Google style preferred).
• Use type hints (def foo(x: int) -> str:) for better IDE support.
• Keep functions small (aim for <20 lines). If it’s longer, split it.

Observability

• Log function calls (e.g., logging.info("Processing %s rows", len(df))).
• Add assertions (e.g., assert df["value"].dtype == float).

5. ⚠️ Common Mistakes & Traps

6. ? Exam/Certification Focus

Typical Question Patterns

1. Lambda vs. def:
2. "When should you use a lambda instead of a named function?"

3. Answer: For short, one-off operations (e.g., sorted(df, key=lambda x: x["value"])).

4. Scope:

5. "What’s the output of this code?"
   python
x = 10
def foo():
x = 20
print(x)
foo()
print(x)

6. Answer: 20 (local scope), then 10 (global scope).

7. Default Arguments:

8. "Why is this dangerous?"
   python
def foo(x=[]):
x.append(1)
return x

9. Answer: The list retains state between calls (e.g., foo() returns [1], then [1, 1]).

10. *args and kwargs:

11. "How would you write a function that accepts any number of keyword arguments?"
12. Answer: def foo(kwargs):.

7. ? Hands-On Challenge

Task:
Write a function that: 1. Takes a pandas DataFrame and a column name.
2. Returns a new DataFrame with rows where the column’s value is in the top 10%.
3. Use a lambda to calculate the threshold.

Solution:

def top_10_percent(df: pd.DataFrame, column: str) -> pd.DataFrame:
    threshold = df[column].quantile(0.9)
    return df[df[column].apply(lambda x: x >= threshold)]

Why it works:
- quantile(0.9) calculates the 90th percentile.
- The lambda filters rows where the value is >= the threshold.

8. ? Rapid-Reference Crib Sheet

9. ? Where to Go Next

1. Python Official Docs – Functions
2. Real Python – Lambda Expressions
3. Google Python Style Guide – Functions
4. Pandas User Guide – apply()