Data Science and Machine Learning 101: Machine Learning Core Feature Engineering and Selection DomainDriven Features RFE Mutual Information SHAP

What This Is

Feature engineering & selection is the art & science of turning raw data into predictive variables that actually help a model learn. It blends domain knowledge (e.g., “days since last purchase”) with algorithmic tricks (Recursive Feature Elimination, Mutual Information, SHAP) to prune noisy or redundant columns, boost accuracy, and keep models interpretable.
Real‑world example: A telecom company wants to predict customer churn. Raw logs contain timestamps, call‑duration, plan codes, and network events. By engineering “average call length per week”, “ratio of prepaid vs postpaid minutes”, and then discarding features that add no information, the churn model becomes both more accurate and explainable to business stakeholders.

Key Terms & Formulas

• Domain‑Driven Feature – A variable created from business logic (e.g., tenure_months = (today - signup_date).days/30). Use when raw columns lack predictive power.
• Recursive Feature Elimination (RFE) – Wrapper method that repeatedly fits a model, removes the least‑important feature(s), and refits.
  python rfe = RFE(estimator=LogisticRegression(), n_features_to_select=10) rfe.fit(X_train, y_train)
• Mutual Information (MI) – Measures any statistical dependence between a feature X and target Y:
  [ I(X;Y)=\sum_{x,y} p(x,y)\log\frac{p(x,y)}{p(x)p(y)} ]
  High MI ⇒ strong (linear or non‑linear) relationship.
• SHAP (SHapley Additive exPlanations) – Game‑theoretic values that attribute each feature’s contribution to a single prediction.
  [ \phi_i = \sum_{S\subseteq F\setminus{i}} \frac{|S|!\,(|F|-|S|-1)!}{|F|!}\big[ f_{S\cup{i}}(x_{S\cup{i}})-f_S(x_S) \big] ]
• Variance Inflation Factor (VIF) – Detects multicollinearity:
  [ \text{VIF}_j = \frac{1}{1-R_j^2} ]
  where (R_j^2) is the R‑squared of regressing feature j on all other features. VIF > 5–10 signals redundancy.
• Embedded Feature Selection – Methods that perform selection during model training (e.g., L1‑penalized Lasso, tree‑based feature_importances_). Use when you want a single pass and built‑in regularization.
• Filter Methods – Rank features before modeling using statistics (MI, chi‑square, ANOVA F‑value). Fast, model‑agnostic, but ignore interactions.
• One‑Hot Encoding vs. Target Encoding – Convert categorical variables; target encoding replaces categories with the mean target value (with smoothing) and can reduce dimensionality for high‑cardinality columns.
• Interaction Feature – Product or ratio of two base features (e.g., price * discount). Captures non‑linear relationships without deep models.
• Curse of Dimensionality – As feature count ↑, distance metrics become less discriminative; models need exponentially more data to maintain performance.

Step‑by‑Step / Process Flow

1. Load & Inspect – df = pd.read_csv(...); call df.info(), df.describe() to spot missing values, data types, and obvious outliers.
2. Domain‑Driven Engineering – Add business‑logic columns (df['days_since_last_login'] = (today - df['last_login']).dt.days).
3. Pre‑process – Impute (SimpleImputer(strategy='median')), encode (OneHotEncoder or TargetEncoder), and scale (StandardScaler for linear models, MinMaxScaler for tree‑based).
4. Initial Filter – Compute MI (mutual_info_classif) or chi‑square; keep top k features (e.g., k=30).
5. Model‑Based Selection – Run RFE with a fast estimator (e.g., LogisticRegression) or fit a Lasso and drop coefficients ≈ 0.
6. Interpret & Refine – Use SHAP on the final model to verify that retained features make sense; drop any that consistently show near‑zero contribution or high VIF.

Common Mistakes

• Mistake: “Engineer every possible interaction and feed them all to the model.”
  Correction: Start with a small, plausible set; high‑order interactions explode dimensionality and often overfit.

• Mistake: “Rely solely on a filter method and ignore multicollinearity.”
  Correction: After filtering, compute VIF; drop one of any pair with VIF > 10 to keep coefficients stable.

• Mistake: “Scale categorical one‑hot columns with StandardScaler.”
  Correction: Scaling is unnecessary (and can harm) for binary 0/1 columns; only scale continuous numeric features.

• Mistake: “Use SHAP values to rank features globally without checking distribution.”
  Correction: SHAP is local by design; aggregate (e.g., mean absolute SHAP) across many rows before ranking.

• Mistake: “Treat missing values as a separate category in one‑hot encoding.”
  Correction: Impute or flag missingness explicitly; mixing missing with a legitimate category confuses the model.

Data Science Interview / Practical Insights

1. “When would you choose RFE over a filter method?” – Expect you to say RFE when you need a model‑aware ranking (captures interactions) and you have enough compute time; filter methods are for quick, model‑agnostic pruning.
2. “Explain the difference between Mutual Information and Pearson correlation.” – MI captures any dependency (including non‑linear), while Pearson only measures linear correlation.
3. “How do SHAP values differ from traditional feature importance?” – SHAP provides additive contributions for each prediction, obeying consistency and local accuracy; traditional importance (e.g., Gini) is a global, model‑specific heuristic.
4. “What’s the risk of target encoding on a small dataset?” – It can leak target information and cause severe over‑fitting; use K‑fold smoothing or leave‑one‑out encoding.

Quick Check Questions

1. Q: Your churn model shows high variance after adding dozens of engineered features. What should you try first?
   A: Apply a filter method (e.g., Mutual Information) to cut down to the most informative features, then re‑run RFE. Reducing dimensionality lowers variance.

2. Q: You compute SHAP values and see that feature_A has near‑zero contribution for 95 % of rows but huge spikes for a few outliers. What does this indicate?
   A: feature_A is only important for a minority of cases; consider segmenting the data or using a model that can capture rare‑event patterns rather than discarding it outright.

3. Q: After one‑hot encoding a high‑cardinality column (10 000 categories), training slows dramatically. What alternative encoding can you use?
   A: Switch to target encoding with smoothing (or hashing trick) to keep the feature space compact.

Last‑Minute Cram Sheet (10 one‑liners)

1. Domain‑driven features = business logic → often the biggest lift in Kaggle competitions.
2. RFE: estimator must expose coef_ or feature_importances_; otherwise use RFECV for automatic CV‑based selection.
3. Mutual Information works for both classification (mutual_info_classif) and regression (mutual_info_regression).
4. SHAP values sum to the model output minus the expected value (additivity).
5. VIF > 5 → start dropping; VIF > 10 is a red flag for severe multicollinearity.
6. L1 (Lasso) → drives coefficients to exactly zero; L2 (Ridge) → shrinks but never zeroes.
7. Target encoding needs smoothing: smooth = 1 / (1 + np.exp(-(count - k) / f)).
8. Interaction features should be limited to order 2 unless you have massive data; higher orders explode the feature space.
9. ⚠️ StandardScaler assumes Gaussian tails; for skewed data use PowerTransformer or RobustScaler.
10. Embedded vs. Wrapper vs. Filter: Embedded = single‑pass (Lasso, trees); Wrapper = iterative (RFE); Filter = pre‑model (MI, chi‑square).