TECH **Supervised Learning in Python: Linear/Logistic Regression, Decision Trees, Random Forest**

Supervised Learning in Python: Linear/Logistic Regression, Decision Trees, Random Forest

A Hyper-Practical, Zero-Fluff Study Guide for Data Scientists

1. What This Is & Why It Matters

You’re building a predictive model for a real-world problem—maybe classifying fraudulent transactions, predicting house prices, or forecasting customer churn. Supervised learning is your toolkit for these tasks, where you train a model on labeled data (input-output pairs) to make accurate predictions on new, unseen data.

Why this matters in production:
- If you ignore proper model selection, your predictions will be wildly off, costing your company money (e.g., misclassifying fraud = lost revenue).
- If you don’t tune hyperparameters, your model may overfit (memorizing noise instead of learning patterns) or underfit (failing to capture trends).
- If you don’t validate properly, you’ll deploy a model that works in testing but fails in the real world.

Real-world scenario:
You’re a data scientist at an e-commerce company. Your boss asks you to predict which customers will churn next month so the marketing team can intervene. You have historical data (past purchases, support tickets, demographics). You need to: 1. Choose the right model (Logistic Regression for interpretability? Random Forest for accuracy?).
2. Train it efficiently.
3. Deploy it in a way that scales (e.g., via an API).
4. Monitor performance over time.

This guide gives you the exact steps to do this—no theory fluff, just actionable Python code and best practices.

2. Core Concepts & Components

? Linear Regression

• Definition: Predicts a continuous output (e.g., house price, temperature) by fitting a straight line to data.
• Production insight: If your data has non-linear relationships, Linear Regression will fail. Always check residuals (errors) for patterns.

? Logistic Regression

• Definition: Predicts a binary outcome (e.g., "will churn: yes/no") using the logistic function to squeeze outputs between 0 and 1.
• Production insight: Works well for imbalanced datasets (e.g., fraud detection) if you adjust class weights. Otherwise, it’ll bias toward the majority class.

? Decision Trees

• Definition: Splits data into branches based on feature thresholds (e.g., "if income > $50k, go left; else, go right") to make predictions.
• Production insight: Prone to overfitting—use max_depth or min_samples_split to control complexity. Great for interpretability (you can visualize the tree).

? Random Forest

• Definition: An ensemble of Decision Trees, where each tree votes on the final prediction. Reduces overfitting by averaging multiple trees.
• Production insight: Slower to train than a single tree but more accurate. Use n_estimators=100 as a starting point. Parallelizes well (use n_jobs=-1 to speed up training).

? Train-Test Split

• Definition: Splits data into training (model learns) and testing (model is evaluated) sets.
• Production insight: Never evaluate on training data—you’ll think your model is perfect when it’s just memorizing noise. Use train_test_split from sklearn.

? Cross-Validation (CV)

• Definition: Splits data into k folds, trains on k-1 folds, and tests on the held-out fold. Repeats k times.
• Production insight: More reliable than a single train-test split—use cross_val_score to catch overfitting early.

? Hyperparameter Tuning

• Definition: Adjusting model settings (e.g., max_depth for trees, C for Logistic Regression) to improve performance.
• Production insight: GridSearchCV is brute-force but thorough. RandomizedSearchCV is faster for large parameter spaces.

? Feature Importance

• Definition: Measures how much each feature contributes to predictions (e.g., "age" is 2x more important than "income").
• Production insight: Random Forest gives feature importance out of the box. Use it to simplify models (drop low-importance features) or explain predictions to stakeholders.

? Evaluation Metrics

3. Step-by-Step Hands-On: Build, Train, and Evaluate Models

Prerequisites

• Python 3.8+ (use conda or venv for isolation).
• Libraries: pandas, numpy, scikit-learn, matplotlib, seaborn.
  bash pip install pandas numpy scikit-learn matplotlib seaborn
• Dataset: We’ll use the Titanic dataset (classification) and Boston Housing dataset (regression). Both are built into sklearn.

Step 1: Load and Explore Data

import pandas as pd
from sklearn.datasets import load_iris, load_boston

# Classification (Iris dataset)
iris = load_iris()
X_class = pd.DataFrame(iris.data, columns=iris.feature_names)
y_class = iris.target  # 0=setosa, 1=versicolor, 2=virginica

# Regression (Boston Housing dataset)
boston = load_boston()
X_reg = pd.DataFrame(boston.data, columns=boston.feature_names)
y_reg = boston.target  # Median house price in $1000s

Check for missing values:

print(X_class.isnull().sum())  # Should be 0
print(X_reg.isnull().sum())    # Should be 0

Visualize distributions:

import seaborn as sns
import matplotlib.pyplot as plt

# Classification: Pairplot to see feature relationships
sns.pairplot(pd.DataFrame(X_class).assign(target=y_class))
plt.show()

# Regression: Correlation heatmap
sns.heatmap(X_reg.corr(), annot=True, cmap="coolwarm")
plt.show()

Step 2: Preprocess Data

For classification (Logistic Regression):
- Scale features (Logistic Regression is sensitive to scale).
- Encode categorical variables (if any).

from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

# Split data
X_train, X_test, y_train, y_test = train_test_split(
    X_class, y_class, test_size=0.2, random_state=42
)

# Scale features
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

For regression (Linear Regression):
- No scaling needed (but helps with gradient descent).
- Check for outliers (e.g., using IQR).

# Remove outliers (example: cap at 99th percentile)
upper_limit = X_reg["LSTAT"].quantile(0.99)
X_reg["LSTAT"] = X_reg["LSTAT"].clip(upper=upper_limit)

Step 3: Train Models

A. Logistic Regression (Classification)

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, roc_auc_score

# Train
logreg = LogisticRegression(max_iter=1000, random_state=42)
logreg.fit(X_train_scaled, y_train)

# Predict
y_pred = logreg.predict(X_test_scaled)
y_proba = logreg.predict_proba(X_test_scaled)[:, 1]  # Probabilities for ROC-AUC

# Evaluate
print(classification_report(y_test, y_pred))
print("ROC-AUC:", roc_auc_score(y_test, y_proba))

Expected output:

              precision    recall  f1-score   support
           0       1.00      1.00      1.00        10
           1       1.00      0.90      0.95        10
           2       0.92      1.00      0.96         9
    accuracy                           0.97        29
   macro avg       0.97      0.97      0.97        29
weighted avg       0.97      0.97      0.97        29

ROC-AUC: 0.996

B. Linear Regression (Regression)

from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score

# Train
linreg = LinearRegression()
linreg.fit(X_train, y_train)

# Predict
y_pred = linreg.predict(X_test)

# Evaluate
print("RMSE:", mean_squared_error(y_test, y_pred, squared=False))
print("R²:", r2_score(y_test, y_pred))

Expected output:

RMSE: 4.93
R²: 0.67

C. Decision Tree (Classification)

from sklearn.tree import DecisionTreeClassifier

# Train
tree = DecisionTreeClassifier(max_depth=3, random_state=42)
tree.fit(X_train, y_train)

# Predict
y_pred = tree.predict(X_test)

# Evaluate
print(classification_report(y_test, y_pred))

Expected output:

              precision    recall  f1-score   support
           0       1.00      1.00      1.00        10
           1       0.90      1.00      0.95        10
           2       1.00      0.89      0.94         9
    accuracy                           0.97        29
   macro avg       0.97      0.96      0.96        29
weighted avg       0.97      0.97      0.97        29

D. Random Forest (Classification)

from sklearn.ensemble import RandomForestClassifier

# Train
rf = RandomForestClassifier(n_estimators=100, max_depth=3, random_state=42)
rf.fit(X_train, y_train)

# Predict
y_pred = rf.predict(X_test)

# Evaluate
print(classification_report(y_test, y_pred))

Expected output:

              precision    recall  f1-score   support
           0       1.00      1.00      1.00        10
           1       1.00      1.00      1.00        10
           2       1.00      1.00      1.00         9
    accuracy                           1.00        29
   macro avg       1.00      1.00      1.00        29
weighted avg       1.00      1.00      1.00        29

Step 4: Hyperparameter Tuning

Use GridSearchCV to find the best parameters.

from sklearn.model_selection import GridSearchCV

# Example: Tune Random Forest
param_grid = {
    "n_estimators": [50, 100, 200],
    "max_depth": [3, 5, None],
    "min_samples_split": [2, 5, 10]
}

grid_search = GridSearchCV(
    RandomForestClassifier(random_state=42),
    param_grid,
    cv=5,
    scoring="accuracy"
)
grid_search.fit(X_train, y_train)

print("Best params:", grid_search.best_params_)
print("Best score:", grid_search.best_score_)

Expected output:

Best params: {'max_depth': 3, 'min_samples_split': 2, 'n_estimators': 100}
Best score: 0.96

Step 5: Feature Importance

# For Random Forest
importances = rf.feature_importances_
feature_importance = pd.DataFrame({
    "Feature": X_class.columns,
    "Importance": importances
}).sort_values("Importance", ascending=False)

print(feature_importance)

Expected output:

       Feature  Importance
2  petal length      0.45
3   petal width      0.42
0  sepal length      0.08
1   sepal width      0.05

Step 6: Save and Load Model

import joblib

# Save
joblib.dump(rf, "random_forest_model.pkl")

# Load
loaded_model = joblib.load("random_forest_model.pkl")

4. ? Production-Ready Best Practices

? Security

• Never hardcode API keys or credentials in model files. Use environment variables or secret managers (e.g., AWS Secrets Manager).
• Sanitize input data to prevent adversarial attacks (e.g., someone feeding malicious data to trick your model).

? Cost Optimization

• Use n_jobs=-1 in RandomForestClassifier to parallelize training (faster = cheaper compute).
• For large datasets, use HistGradientBoostingClassifier (faster than Random Forest for big data).
• Cache intermediate results (e.g., joblib.Memory) to avoid recomputing expensive steps.

? Reliability & Maintainability

• Version your models (e.g., model_v1.pkl, model_v2.pkl). Use MLflow or DVC for tracking.
• Log hyperparameters and metrics (e.g., mlflow.log_params()).
• Write unit tests for preprocessing steps (e.g., "Does scaling work correctly?").

?️ Observability

• Monitor prediction drift (e.g., if input data distribution changes, model performance may degrade).
• Log predictions and features (e.g., to debug why a model predicted "churn" for a specific user).
• Set up alerts for:
• Sudden drops in accuracy.
• High prediction latency.
• Data quality issues (e.g., missing values).

5. ⚠️ Common Mistakes & Traps

6. ? Exam/Certification Focus

Typical Question Patterns

1. "Which model is best for interpretability?"
2. Answer: Logistic Regression or Decision Trees (you can explain coefficients or visualize the tree).

3. Trap: Random Forest is accurate but harder to interpret.

4. "How do you handle imbalanced data?"

5. Answer: Use class_weight="balanced", SMOTE, or adjust evaluation metrics (e.g., F1-score instead of accuracy).

6. Trap: Accuracy is misleading for imbalanced data.

7. "What’s the difference between max_depth and min_samples_split in Decision Trees?"

8. Answer:
   • max_depth: Limits tree depth (prevents overfitting).
   • min_samples_split: Minimum samples required to split a node (controls granularity).

9. Trap: Setting max_depth=None can lead to overfitting.

10. "When would you use Random Forest over Logistic Regression?"

11. Answer: When you need higher accuracy and can trade off interpretability. Also better for non-linear relationships.

12. Trap: Logistic Regression is faster to train.

13. "What’s the purpose of cross-validation?"

14. Answer: To get a more reliable estimate of model performance by testing on multiple splits of the data.
15. Trap: A single train-test split can be lucky/unlucky.

7. ? Hands-On Challenge (with Solution)

Challenge:
You’re given a dataset with 3 features (age, income, education) and a binary target (will_buy: 0/1). The dataset is imbalanced (90% 0, 10% 1). Train a model to predict will_buy and evaluate it using precision, recall, and F1-score.

Solution:

from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report

# Assume X, y are loaded
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Train with class_weight="balanced"
model = RandomForestClassifier(class_weight="balanced", random_state=42)
model.fit(X_train, y_train)

# Evaluate
y_pred = model.predict(X_test)
print(classification_report(y_test, y_pred))

Why it works:
- class_weight="balanced" adjusts for imbalanced data by giving more weight to the minority class.
- Random Forest handles non-linear relationships well.

8. ? Rapid-Reference Crib Sheet

Linear/Logistic Regression

• Key hyperparameters:
• C: Inverse of regularization strength (smaller = stronger regularization). ⚠️ Default=1.0.
• penalty: "l1" (Lasso) or "l2" (Ridge). ⚠️ "l1" can zero out features (feature selection).
• When to use:
• Linear: Continuous output (e.g., price prediction).
• Logistic: Binary classification (e.g., spam detection).

Decision Trees

• Key hyperparameters:
• max_depth: Limits tree depth. ⚠️ Default=None (unlimited = overfitting).
• min_samples_split: Minimum samples to split a node. ⚠️ Default=2.
• min_samples_leaf: Minimum samples in a leaf. ⚠️ Default=1.
• When to use: Interpretability > accuracy.

Random Forest

• Key hyperparameters:
• n_estimators: Number of