TECH **Python for Data Science: Pipelines, ColumnTransformer, and FeatureUnion**

Python for Data Science: Pipelines, ColumnTransformer, and FeatureUnion

A Hyper-Practical, Zero-Fluff Study Guide

1. What This Is & Why It Matters

You’re building a production-grade ML pipeline for a retail client. Your raw data has: - Numerical features (e.g., price, quantity) that need scaling.
- Categorical features (e.g., product_category, region) that need one-hot encoding.
- Text features (e.g., product_description) that need TF-IDF vectorization.
- Missing values that need imputation.

If you process each feature manually, your code becomes: - Brittle: A single change in the data schema (e.g., a new column) breaks everything.
- Hard to maintain: You’ll have 10+ lines of preprocessing logic scattered across notebooks.
- Non-reproducible: Your training and inference pipelines diverge, leading to silent failures in production.

Enter Pipeline, ColumnTransformer, and FeatureUnion:
- Pipeline: Chains preprocessing and modeling steps into a single object. Ensures consistent transformations during training and inference.
- ColumnTransformer: Applies different transformations to different columns (e.g., scale numerics, encode categories). No more manual column splitting.
- FeatureUnion: Combines multiple feature extraction steps (e.g., text + numerical features) into a single output.

Why this matters in production:
1. Avoids data leakage: Ensures preprocessing (e.g., scaling) is fitted only on training data, not the entire dataset.
2. Simplifies deployment: Your entire preprocessing + model logic is a single pickle file.
3. Enables CI/CD for ML: Pipelines can be versioned, tested, and deployed like any other code.

Real-world scenario:
You inherit a legacy notebook where preprocessing is done in 50+ lines of pandas code. Your task: - Refactor it into a reusable, maintainable pipeline.
- Deploy it to AWS SageMaker or FastAPI without breaking inference.

2. Core Concepts & Components

1. Pipeline

• Definition: Chains multiple estimators (transformers + models) into a single object.
• Production insight: If you don’t use Pipeline, your inference code will silently fail when new data has missing columns or different dtypes.

2. ColumnTransformer

• Definition: Applies different transformations to different columns (e.g., StandardScaler for numerics, OneHotEncoder for categories).
• Production insight: Without it, you’ll manually split DataFrames, leading to off-by-one errors when column order changes.

3. FeatureUnion

• Definition: Combines multiple transformers into a single output (e.g., merge text features from TfidfVectorizer with numerical features).
• Production insight: Critical for multi-modal data (e.g., images + text + tabular data). Without it, you’ll end up with misaligned feature matrices.

4. SimpleImputer

• Definition: Fills missing values (e.g., with mean, median, or a constant).
• Production insight: If you impute before splitting data, you leak information from the test set into training.

5. StandardScaler / MinMaxScaler

• Definition: Scales numerical features to a standard range (e.g., mean=0, std=1).
• Production insight: If you scale before splitting, your test set will be contaminated with training statistics.

6. OneHotEncoder

• Definition: Converts categorical variables into binary columns.
• Production insight: If you don’t set handle_unknown="ignore", your pipeline will crash on new categories in production.

7. FunctionTransformer

• Definition: Wraps a custom function into a scikit-learn transformer.
• Production insight: Lets you reuse legacy pandas code (e.g., df["log_price"] = np.log(df["price"])) in a pipeline.

8. make_pipeline / make_column_transformer

• Definition: Helper functions to create Pipeline/ColumnTransformer with less boilerplate.
• Production insight: Reduces errors from manual step naming (e.g., ("scaler", StandardScaler())).

3. Step-by-Step Hands-On Section

Prerequisites

• Python 3.8+
• Libraries: scikit-learn, pandas, numpy
• Data: A CSV with mixed data types (numerical, categorical, text).

pip install scikit-learn pandas numpy

Task: Build a Production-Ready Preprocessing Pipeline

Goal: Process a retail dataset with: - Numerical features: price, quantity - Categorical features: product_category, region - Text feature: product_description - Missing values in price and region

Step 1: Load and Inspect Data

import pandas as pd

# Sample data (replace with your CSV)
data = {
    "price": [10, 20, None, 40, 50],
    "quantity": [1, 2, 3, 4, 5],
    "product_category": ["A", "B", "A", "C", "B"],
    "region": ["North", "South", None, "East", "West"],
    "product_description": ["great product", "okay product", "bad product", "awesome", "meh"]
}
df = pd.DataFrame(data)
print(df.head())

Step 2: Define Transformers

from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline

# Numerical pipeline: impute missing values + scale
numeric_features = ["price", "quantity"]
numeric_transformer = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),  # Fill missing with median
    ("scaler", StandardScaler())  # Scale to mean=0, std=1
])

# Categorical pipeline: impute missing + one-hot encode
categorical_features = ["product_category", "region"]
categorical_transformer = Pipeline([
    ("imputer", SimpleImputer(strategy="most_frequent")),  # Fill missing with mode
    ("encoder", OneHotEncoder(handle_unknown="ignore"))  # Ignore new categories in prod
])

# Text pipeline: TF-IDF vectorization
text_feature = "product_description"
text_transformer = Pipeline([
    ("tfidf", TfidfVectorizer(max_features=10))  # Limit to top 10 words
])

# Combine all transformers
preprocessor = ColumnTransformer([
    ("num", numeric_transformer, numeric_features),
    ("cat", categorical_transformer, categorical_features),
    ("text", text_transformer, text_feature)
])

Step 3: Add a Model (Optional)

from sklearn.ensemble import RandomForestClassifier

# Full pipeline: preprocessing + model
pipeline = Pipeline([
    ("preprocessor", preprocessor),
    ("classifier", RandomForestClassifier())
])

Step 4: Train and Evaluate

from sklearn.model_selection import train_test_split

# Dummy target (replace with your actual target)
y = [0, 1, 0, 1, 0]
X_train, X_test, y_train, y_test = train_test_split(df, y, test_size=0.2, random_state=42)

# Fit the pipeline
pipeline.fit(X_train, y_train)

# Score on test set
print("Test accuracy:", pipeline.score(X_test, y_test))

Step 5: Save and Deploy

import joblib

# Save the pipeline
joblib.dump(pipeline, "retail_pipeline.joblib")

# Load and predict (e.g., in a FastAPI endpoint)
loaded_pipeline = joblib.load("retail_pipeline.joblib")
new_data = pd.DataFrame({
    "price": [30],
    "quantity": [2],
    "product_category": ["A"],
    "region": ["North"],
    "product_description": ["amazing product"]
})
print("Prediction:", loaded_pipeline.predict(new_data))

Expected Output:

Test accuracy: 1.0  # (or your actual score)
Prediction: [0]    # (or your actual prediction)

4. ? Production-Ready Best Practices

Security

• Never hardcode secrets (e.g., database passwords) in pipelines. Use python-dotenv or AWS Secrets Manager.
• Sanitize text inputs (e.g., TfidfVectorizer with token_pattern=r"\b\w+\b" to avoid regex injection).

Cost Optimization

• Limit max_features in TfidfVectorizer to avoid exploding dimensionality (e.g., max_features=100).
• Use sparse=True in OneHotEncoder for high-cardinality categorical features to save memory.

Reliability & Maintainability

• Name pipeline steps explicitly (e.g., ("scaler", StandardScaler()) instead of StandardScaler()). Helps debugging.
• Use make_pipeline for quick prototyping, but switch to named steps for production.
• Version your pipelines (e.g., retail_pipeline_v1.joblib). Use MLflow or DVC for tracking.

Observability

• Log pipeline steps (e.g., print(pipeline.named_steps["preprocessor"].transformers_)).
• Monitor feature distributions (e.g., log X_train.describe()) to detect data drift.

5. ⚠️ Common Mistakes & Traps

6. ? Exam/Certification Focus

Typical Question Patterns

1. Scenario: "You have a dataset with numerical, categorical, and text features. How do you preprocess it for a scikit-learn model?"

2. Answer: Use ColumnTransformer with separate pipelines for each feature type.

3. Trap Distinction:

4. Pipeline vs FeatureUnion:

   
   
   • Pipeline: Sequential steps (e.g., impute → scale → model).
   • FeatureUnion: Parallel steps (e.g., text features + numerical features → concatenate).

5. Code Snippet Question:
   ```python
   from sklearn.pipeline import Pipeline
   from sklearn.preprocessing import StandardScaler
   from sklearn.impute import SimpleImputer

pipeline = Pipeline([
("imputer", SimpleImputer()),
("scaler", StandardScaler())
])
``
- Question: What happens if you callpipeline.fit_transform(X)whereXhas missing values?
- Answer:SimpleImputerfills missing values first, thenStandardScaler` scales the imputed data.

1. Common Scenario:
2. "You deploy a model to production, but it fails on new data with unseen categories. What’s the issue?"
3. Answer: OneHotEncoder was not configured with handle_unknown="ignore".

7. ? Hands-On Challenge (with Solution)

Challenge

You have a dataset with: - Numerical column: age (missing values) - Categorical column: gender (values: M, F, Other) - Text column: review (raw text)

Build a pipeline that: 1. Imputes missing age with the median.
2. One-hot encodes gender (ignore new categories in production).
3. Vectorizes review with TfidfVectorizer (limit to 50 features).
4. Scales all numerical features.

Solution:

from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.feature_extraction.text import TfidfVectorizer

preprocessor = ColumnTransformer([
    ("num", Pipeline([
        ("imputer", SimpleImputer(strategy="median")),
        ("scaler", StandardScaler())
    ]), ["age"]),
    ("cat", OneHotEncoder(handle_unknown="ignore"), ["gender"]),
    ("text", TfidfVectorizer(max_features=50), "review")
])

pipeline = Pipeline([("preprocessor", preprocessor)])

Why it works: - ColumnTransformer applies different steps to different columns.
- handle_unknown="ignore" prevents crashes on new gender values.
- max_features=50 limits text feature dimensionality.

8. ? Rapid-Reference Crib Sheet

9. ? Where to Go Next

1. scikit-learn Pipelines Documentation
2. ColumnTransformer User Guide
3. FeatureUnion Example
4. MLOps with Pipelines (Google Cloud)