Data Science and Machine Learning 101: Programming and Data Engineering Data Cleaning and Preprocessing Missing Values Encoding Scaling Imbalanced Data

What This Is

Data cleaning and preprocessing turn raw, messy tables into a tidy, model‑ready matrix. It covers handling missing values, converting categorical variables to numbers, scaling numeric features, and fixing class‑imbalance so that algorithms learn the true signal instead of artefacts. In a churn‑prediction project, for example, you might impute missing “last login” dates, one‑hot encode “subscription tier”, scale “monthly spend”, and rebalance the rare “churn = 1” class before feeding the data to a Gradient‑Boosted Tree.

Key Terms & Formulas

• Missing‑Value Imputation – Replacing NaNs with a statistic (mean, median) or model prediction.
• Mean Imputation: (\tilde{x}j = \frac{1}{n}\sum) for feature (j). }^{n} x_{ij
• Median Imputation – Same as mean but uses the median; robust to outliers.
• K‑Nearest Neighbors Imputer (KNN‑Impute) – Predicts a missing value by averaging the same feature from the k most similar rows (distance on non‑missing columns).
• One‑Hot Encoding – Creates binary columns (x_{j}^{(c)} = \mathbf{1}{x_j = c}) for each category (c).
• Ordinal Encoding – Maps ordered categories to integers, e.g., {'low':0,'medium':1,'high':2}.
• Label Encoding – Assigns a unique integer to each category; use only with tree‑based models that treat integers as categorical.
• StandardScaler – (z = (x - \mu)/\sigma); centers data to mean 0, variance 1.
• MinMaxScaler – (x' = (x - x_{\min})/(x_{\max} - x_{\min})); rescales to ([0,1]).
• RobustScaler – Uses median and IQR: (x' = (x - \text{median})/\text{IQR}); safe for heavy‑tailed data.
• SMOTE (Synthetic Minority Over‑sampling Technique) – Generates new minority samples: (\tilde{x} = x_i + \lambda (x_{nn} - x_i)) where (\lambda \sim \mathcal{U}(0,1)) and (x_{nn}) is a random neighbor.
• Class Weighting – Passes class_weight='balanced' to estimators; internally multiplies loss by (w_c = \frac{n_{\text{samples}}}{n_{\text{classes}} \cdot n_c}).
• Stratified Split – train_test_split(..., stratify=y) keeps class proportions identical across folds.

Step‑by‑Step / Process Flow

1. Load & Inspect – df = pd.read_csv(...); use df.info(), df.isnull().sum() to spot missingness and dtype mismatches.
2. Handle Missing Values
3. If < 5 % missing and MCAR → simple imputation (SimpleImputer(strategy='median')).
4. If pattern‑dependent → model‑based imputation (IterativeImputer or KNN).
5. Encode Categorical Features
6. Low‑cardinality → OneHotEncoder(sparse=False).
7. High‑cardinality & tree models → OrdinalEncoder + target encoding (mean label per category).
8. Scale / Normalize Numeric Features
9. Linear models → StandardScaler.
10. Tree‑based models → skip scaling (they’re scale‑invariant).
11. Neural nets → MinMaxScaler or RobustScaler.
12. Address Imbalanced Targets
13. Split with stratify=y.
14. Choose one: SMOTE, RandomUnderSampler, or class_weight='balanced'.
15. Verify post‑sampling balance with np.bincount(y_resampled).
16. Finalize Pipeline – Wrap all steps in a sklearn.pipeline.Pipeline (or ColumnTransformer) so the same preprocessing runs on train and test data, e.g.:

preprocess = ColumnTransformer([
    ('num', Pipeline([('impute', SimpleImputer(strategy='median')),
                     ('scale', StandardScaler())]), num_cols),
    ('cat', Pipeline([('impute', SimpleImputer(strategy='most_frequent')),
                     ('oh', OneHotEncoder(handle_unknown='ignore'))]), cat_cols)
])

model = Pipeline([
    ('prep', preprocess),
    ('clf', RandomForestClassifier(class_weight='balanced'))
])

Common Mistakes

• Mistake: Imputing after train/test split.
  Correction: Fit imputer on the training set only and transform both sets; otherwise you leak information from the test set.

• Mistake: One‑hot encoding a high‑cardinality column (e.g., ZIP code) leading to thousands of sparse columns.
  Correction: Use target encoding or frequency encoding for such features, or drop them if they add little predictive power.

• Mistake: Scaling before splitting, then applying the same scaler to the test set.
  Correction: Fit scalers on training data only; Pipeline guarantees this.

• Mistake: Balancing classes with SMOTE on the whole dataset.
  Correction: Apply SMOTE only on the training fold; otherwise synthetic points can appear in the validation set, inflating performance.

• Mistake: Assuming class_weight='balanced' works for all algorithms.
  Correction: Verify the estimator supports it (e.g., LogisticRegression, RandomForest, XGBClassifier); for others, manually compute sample weights.

Data Science Interview / Practical Insights

1. “When would you prefer KNN imputation over mean imputation?” – Expect an answer about preserving multivariate relationships and handling non‑MCAR patterns.
2. “Explain the trade‑off between one‑hot and target encoding for a categorical variable with 10 000 levels.” – Look for discussion of dimensionality, over‑fitting, and leakage mitigation (use cross‑validated target encoding).
3. “How does SMOTE differ from random oversampling, and why can it still cause over‑fitting?” – Mention synthetic interpolation vs duplication, and the risk of creating borderline samples that the model memorizes.
4. “Why do tree‑based models not need feature scaling, but neural nets do?” – Highlight that trees split on order statistics (scale‑invariant) while gradient‑based nets are sensitive to feature magnitude.

Quick Check Questions

1. Scenario: Your churn dataset has 2 % churners. After training a logistic regression you see an AUC of 0.78 but precision = 0.12.
   Answer: Use SMOTE or class weighting and evaluate with Precision‑Recall curve (more informative for rare positives).

2. Scenario: You accidentally applied StandardScaler to a categorical column encoded as integers.
   Answer: The model will treat the scaled integers as continuous, corrupting the signal; fix by excluding that column from scaling or using proper encoding.

3. Scenario: After imputing missing ages with the mean, you notice the model’s residuals are biased for older customers.
   Answer: Switch to median or KNN imputation (or add a “was_missing” indicator) to capture the distributional shift.

Last‑Minute Cram Sheet (10 one‑liners)

1. ⚠️ SimpleImputer(strategy='mean') assumes data are MCAR; otherwise use IterativeImputer.
2. One‑Hot → n new columns = number of unique categories (watch cardinality!).
3. Ordinal Encoding is safe only for tree‑based models; linear models will treat the order as numeric.
4. StandardScaler: (z = (x-\mu)/\sigma); RobustScaler uses median & IQR → better for outliers.
5. SMOTE creates synthetic points along line segments between minority samples; set k_neighbors ≤ 5 to avoid noisy borders.
6. Class‑weight='balanced' computes (w_c = \frac{n_{\text{samples}}}{n_{\text{classes}} \cdot n_c}).
7. StratifiedKFold preserves class ratios in every fold – essential for imbalanced problems.
8. Pipeline → guarantees no data leakage: fit on train, transform on both train & test.
9. Target Encoding must be cross‑validated (e.g., category_encoders.TargetEncoder(cv=5)) to avoid leakage.
10. Imputation + scaling order: Impute first, then scale; scaling before imputation propagates NaNs.