TECH **Python for Data Science: Working with Missing Data (isna, fillna, dropna) – Zero-Fluff Guide**

Python for Data Science: Working with Missing Data (isna, fillna, dropna) – Zero-Fluff Guide

1. What This Is & Why It Matters

Missing data is the silent killer of data pipelines. You pull a CSV from an API, load it into a DataFrame, and suddenly your model throws ValueError: Input contains NaN, infinity or a value too large for dtype('float32'). Or worse—your model trains fine, but predictions are garbage because missing values were silently imputed with zeros.

Real-world scenario:
You’re building a customer churn prediction model. Your dataset has 10% missing values in the tenure column (how long a customer has been subscribed). If you drop all rows with missing values, you lose 10% of your data—and likely bias your model (e.g., newer customers might be underrepresented). If you fill missing values with the mean, you might artificially inflate retention rates. If you ignore them, your model crashes.

Why this matters in production:
- Broken pipelines: Missing data can cause pandas operations to fail (e.g., groupby, merge, apply).
- Biased models: Imputing missing values incorrectly can skew predictions (e.g., filling NaN with 0 in a salary column).
- Regulatory risks: In healthcare or finance, missing data can violate compliance (e.g., GDPR requires explaining why data is missing).
- Costly errors: A model deployed with unhandled missing data might make bad decisions (e.g., approving loans for customers with missing credit scores).

Superpower this gives you:
You’ll be able to clean data efficiently, debug pipelines faster, and build more robust models—without losing sleep over NaN errors.

2. Core Concepts & Components

1. NaN (Not a Number)

• Definition: A special floating-point value representing missing or undefined data in pandas/numpy.
• Production insight: NaN is not the same as None (Python’s None is an object, NaN is a float). Some operations treat them differently (e.g., df.isna() catches both, but df.fillna(0) only replaces NaN).

2. isna() / isnull()

• Definition: Returns a boolean mask (True where data is missing, False otherwise). isna() and isnull() are aliases.
• Production insight: Use this to audit missing data before cleaning. Example: python df.isna().sum() # Count missing values per column
• Why it matters: If 30% of a column is missing, dropping it might be better than imputing.

3. notna() / notnull()

• Definition: Opposite of isna()—returns True where data is not missing.
• Production insight: Useful for filtering rows with complete data: python df[df['column'].notna()] # Keep rows where 'column' is not missing

4. dropna()

• Definition: Drops rows or columns containing missing values.
• Key parameters:
• axis=0 (drop rows) or axis=1 (drop columns).
• how='any' (drop if any value is missing) or how='all' (drop if all values are missing).
• thresh=N (keep rows/columns with at least N non-missing values).
• Production insight:
• Danger: df.dropna() drops all rows with any missing value by default. This can destroy your dataset.
• Better approach: Use thresh to keep rows with "enough" data (e.g., df.dropna(thresh=5) keeps rows with ≥5 non-missing values).

5. fillna()

• Definition: Replaces missing values with a specified value or method.
• Key parameters:
• value: Scalar (e.g., 0, "missing") or dict (e.g., {'column1': 0, 'column2': 'unknown'}).
• method: 'ffill' (forward fill) or 'bfill' (backward fill).
• axis: Fill along rows (axis=0) or columns (axis=1).
• Production insight:
• Never use fillna(0) blindly. Example: Filling missing age with 0 will skew averages.
• Better: Use domain knowledge (e.g., fill missing temperature with the median, not the mean).

6. interpolate()

• Definition: Fills missing values using interpolation (e.g., linear, polynomial).
• Key parameters:
• method='linear' (default), 'time' (for time-series), 'polynomial'.
• limit: Max number of consecutive NaNs to fill.
• Production insight:
• Use case: Time-series data (e.g., stock prices, sensor readings).
• Example:
  python
df['temperature'].interpolate(method='time') # Fill gaps in time-series

7. SimpleImputer (scikit-learn)

• Definition: A scikit-learn transformer for imputing missing values (e.g., mean, median, mode).
• Production insight:
• Why use it? It’s scalable (works with Pipeline) and reproducible (avoids data leakage).
• Example:
  python
from sklearn.impute import SimpleImputer
imputer = SimpleImputer(strategy='median')
df[['column']] = imputer.fit_transform(df[['column']])

8. Data Leakage

• Definition: When information from the test set "leaks" into the training set (e.g., imputing missing values using the entire dataset before splitting).
• Production insight:
• How to avoid: Fit imputers only on the training set, then transform both train and test sets.
• Example:
  python
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
imputer = SimpleImputer(strategy='median').fit(X_train)
X_train = imputer.transform(X_train)
X_test = imputer.transform(X_test) # Use same imputer, don't refit!

3. Step-by-Step Hands-On: Cleaning a Real Dataset

Prerequisites

• Python 3.8+ with pandas, numpy, and scikit-learn installed.
• A dataset with missing values (we’ll use the Titanic dataset as an example).

Task:

Clean the Titanic dataset for a survival prediction model. Handle missing values in: - Age (177 missing values) - Embarked (2 missing values) - Cabin (687 missing values)

Step 1: Load the Data

import pandas as pd
import numpy as np

# Load data
df = pd.read_csv('titanic.csv')
print(df.isna().sum())  # Audit missing values

Output:

PassengerId      0
Survived         0
Pclass           0
Name             0
Sex              0
Age            177
SibSp            0
Parch            0
Ticket           0
Fare             0
Cabin          687
Embarked         2
dtype: int64

Step 2: Drop Irrelevant Columns

• Cabin has 77% missing values—drop it (too sparse to impute).
• PassengerId, Name, Ticket are not useful for prediction.

df = df.drop(['Cabin', 'PassengerId', 'Name', 'Ticket'], axis=1)

Step 3: Impute Age (Missing 177 Values)

• Strategy: Use median age grouped by Pclass and Sex (richer passengers were older).
• Why? Median is robust to outliers (e.g., a 1-year-old in 1st class would skew the mean).

# Calculate median age by Pclass and Sex
median_age = df.groupby(['Pclass', 'Sex'])['Age'].median()
print(median_age)

Output:

Pclass  Sex
1       female    35.0
        male      40.0
2       female    28.0
        male      30.0
3       female    21.5
        male      25.0
Name: Age, dtype: float64

# Impute missing Age values
df['Age'] = df.groupby(['Pclass', 'Sex'])['Age'].apply(lambda x: x.fillna(x.median()))
print(df['Age'].isna().sum())  # Should be 0

Step 4: Impute Embarked (Missing 2 Values)

• Strategy: Use the most common port (S = Southampton).
• Why? Only 2 missing values—no need for complex imputation.

df['Embarked'] = df['Embarked'].fillna(df['Embarked'].mode()[0])
print(df['Embarked'].isna().sum())  # Should be 0

Step 5: Verify No Missing Values Remain

print(df.isna().sum())

Output:

Survived    0
Pclass      0
Sex         0
Age         0
SibSp       0
Parch       0
Fare        0
Embarked    0
dtype: int64

Step 6: Save Cleaned Data

df.to_csv('titanic_cleaned.csv', index=False)

4. ? Production-Ready Best Practices

Security & Compliance

• GDPR/CCPA: If missing data represents "erasure requests," document why values are missing (e.g., "User opted out of data collection").
• Audit logs: Log missing value handling (e.g., "Imputed Age with median by Pclass and Sex").

Cost Optimization

• Avoid dropna() on large datasets: Dropping rows can reduce training data size, increasing model error.
• Use SimpleImputer for pipelines: Faster than manual fillna() in production ML workflows.

Reliability & Maintainability

• Document imputation strategies: Add a README or data dictionary explaining how missing values were handled.
• Use thresh in dropna(): Instead of df.dropna(), use df.dropna(thresh=5) to keep rows with ≥5 non-missing values.
• Avoid fillna(0) for categorical data: Use "missing" or "unknown" instead.

Observability

• Track missing value rates: Monitor % missing in columns over time (e.g., with prometheus or datadog).
• Alert on sudden increases: If Age missingness jumps from 5% to 30%, investigate data collection issues.

5. ⚠️ Common Mistakes & Traps

6. ? Exam/Certification Focus

Typical Question Patterns

1. "Which method would you use to check for missing values?"
2. ✅ df.isna().sum()

3. ❌ df.isnull().count() (counts non-missing values)

4. "How would you drop rows where Age is missing?"

5. ✅ df.dropna(subset=['Age'])

6. ❌ df.dropna(axis=1) (drops columns, not rows)

7. "What’s the difference between fillna(0) and fillna(method='ffill')?"

8. fillna(0): Replaces all NaN with 0.

9. fillna(method='ffill'): Forward-fills NaN with the previous non-missing value.

10. "How do you avoid data leakage when imputing missing values?"

11. ✅ Fit imputer on training data, transform test data.
12. ❌ Fit imputer on the entire dataset before splitting.

Key ⚠️ Trap Distinctions

• isna() vs isnull(): They’re the same in pandas, but isna() is preferred (more explicit).
• dropna() defaults: how='any' drops rows if any value is missing. Use how='all' to drop only if all values are missing.
• fillna() order: df.fillna(0).fillna(1) fills NaN with 0 first, then remaining NaN with 1.

Scenario-Based Question

"You have a dataset with 10% missing values in Income. The distribution is right-skewed. How would you impute missing values?"
- ✅ Use median (robust to outliers).
- ❌ Use mean (skewed by high earners).
- ❌ Use 0 (biases the distribution).

7. ? Hands-On Challenge

Challenge:
You have a DataFrame df with missing values in column1 and column2. Write a function that: 1. Drops rows where both column1 and column2 are missing.
2. Fills remaining missing values in column1 with the median.
3. Fills remaining missing values in column2 with "unknown".

Solution:

def clean_missing_data(df):
    # Drop rows where both columns are missing
    df = df.dropna(subset=['column1', 'column2'], how='all')
    # Fill column1 with median
    df['column1'] = df['column1'].fillna(df['column1'].median())
    # Fill column2 with "unknown"
    df['column2'] = df['column2'].fillna("unknown")
    return df

Why it works:
- how='all' ensures we only drop rows where both columns are missing.
- fillna() with median/"unknown" handles the remaining missing values.

8. ? Rapid-Reference Crib Sheet

9. ? Where to Go Next

1. Pandas Missing Data Docs – Official guide with advanced examples.
2. Scikit-Learn Imputation – How to use SimpleImputer in ML pipelines.
3. Kaggle Titanic Tutorial – Hands-on missing data handling in a real dataset.
4. "Python for Data Analysis" (O’Reilly) – Chapter 7 covers missing data in depth.