Data Science and Machine Learning 101: Machine Learning Core Supervised Learning Classification Logistic Regression Decision Trees Random Forest SVM Evaluation Confusion Matrix Precision Recall F1 ROCAUC

What This Is

Supervised classification learns a mapping from input features X to a discrete target y (e.g., “churn / no‑churn”, “malignant / benign”). It is the work‑horse of most production pipelines because the model can be evaluated on held‑out data, tuned, and then deployed to make real‑time decisions. Example: a telecom company trains a logistic‑regression model on call‑detail records to flag customers who are likely to leave next month, allowing the retention team to intervene.

Key Terms & Formulas

• Logistic Regression – Linear model passed through the sigmoid:
  [ \hat{p}= \sigma(\mathbf{w}^\top\mathbf{x}+b)=\frac{1}{1+e^{-(\mathbf{w}^\top\mathbf{x}+b)}} ]
  Predict class = 1 if (\hat{p}>0.5). Use when relationship is roughly linear and you need interpretability.

• Cross‑Entropy Loss (Log‑Loss) – Objective minimized during training:
  [ J(\mathbf{w}) = -\frac{1}{N}\sum_{i=1}^{N}\big[y_i\log\hat{p}_i+(1-y_i)\log(1-\hat{p}_i)\big] ]

• Decision Tree – Recursive partition of feature space; each split chooses the feature/value that maximizes Information Gain (or Gini impurity reduction).

• Gini Impurity – Measure of node “purity”:
  [ G = 1-\sum_{k=1}^{K}p_k^2 ]
  where (p_k) is the proportion of class k in the node.

• Random Forest – Ensemble of M decorrelated trees (bagging + feature‑subsampling). Final prediction = majority vote (classification).

• Support Vector Machine (SVM) – Finds a hyperplane that maximizes the margin; with soft‑margin parameter C and kernel (K(\mathbf{x},\mathbf{x}')). Decision function:
  [ f(\mathbf{x}) = \text{sign}\Big(\sum_{i\in SV}\alpha_i y_i K(\mathbf{x}_i,\mathbf{x}) + b\Big) ]

• Confusion Matrix – 2×2 table for binary problems:

• Precision = TP / (TP + FP) – Fraction of predicted positives that are correct.

• Recall (Sensitivity) = TP / (TP + FN) – Fraction of actual positives that are captured.

• F1‑Score = 2·(Precision·Recall)/(Precision+Recall) – Harmonic mean; balances precision & recall.

• ROC‑AUC – Area under the Receiver Operating Characteristic curve; plots TPR vs. FPR for all thresholds.

• Stratified Train/Test Split – Guarantees class proportion is preserved in both sets (crucial for imbalanced data).

Step‑by‑Step / Process Flow

1. Load & Inspect – df = pd.read_csv('churn.csv'); df.head(), df.info().
2. Clean & Engineer – Impute missing values, encode categoricals (OneHotEncoder), create interaction features, and optionally scale numeric columns (StandardScaler).
3. Stratified Split –
   python
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, stratify=y, random_state=42)
4. Baseline Model – Fit a quick logistic regression:
   python
logreg = LogisticRegression(max_iter=500, class_weight='balanced')
logreg.fit(X_train, y_train)
5. Evaluate – Compute confusion matrix, precision/recall, and ROC‑AUC on the hold‑out set:
   python
y_pred = logreg.predict(X_test)
y_proba = logreg.predict_proba(X_test)[:,1]
print(classification_report(y_test, y_pred))
print('AUC:', roc_auc_score(y_test, y_proba))
6. Iterate / Tune – Try a RandomForest (n_estimators, max_depth) or SVM (C, kernel). Use GridSearchCV with StratifiedKFold to avoid leakage.
7. Interpret – For tree‑based models, extract feature_importances_; for logistic regression, inspect coef_ (odds ratios). Communicate business impact (e.g., “Customers with >3 support tickets have 2.3× higher churn odds”).

Common Mistakes

• Mistake: Feeding raw categorical strings directly into scikit‑learn models.
  Correction: Encode them first (One‑Hot, Ordinal, or target encoding). Unencoded strings raise a ValueError and hide useful signal.

• Mistake: Using accuracy on a heavily imbalanced dataset (e.g., 95% “no churn”).
  Correction: Report precision, recall, and AUC; consider resampling (SMOTE) or class‑weighting.

• Mistake: Forgetting to stratify the train/test split, leading to a test set with few positive cases.
  Correction: Always set stratify=y in train_test_split (or use StratifiedKFold).

• Mistake: Over‑fitting a deep decision tree and then reporting its training accuracy as the final metric.
  Correction: Prune (max_depth, min_samples_leaf) or switch to an ensemble (Random Forest) and evaluate on unseen data.

• Mistake: Interpreting ROC‑AUC as “probability of correct classification”.
  Correction: ROC‑AUC measures ranking quality; a high AUC does not guarantee good calibrated probabilities.

Data Science Interview / Practical Insights

1. “When would you prefer a linear model over a tree‑based model?” – Expect discussion of interpretability, speed, and when the decision boundary is roughly linear.
2. “Explain the bias‑variance trade‑off for Random Forest vs. a single Decision Tree.” – Random Forest reduces variance by averaging many high‑variance trees, at a modest increase in bias.
3. “How does the C hyperparameter in SVM differ from the regularization λ in logistic regression?” – C is the inverse of regularization strength; larger C → less regularization (more variance).
4. “Why might Precision‑Recall be more informative than ROC‑AUC on a fraud‑detection problem?” – When the positive class is rare, PR‑AUC focuses on the minority class performance, whereas ROC can be overly optimistic.

Quick Check Questions

1. Q: Your model’s ROC‑AUC is 0.92 but precision at the default 0.5 threshold is only 0.30 on an imbalanced dataset.
   A: Look at the Precision‑Recall curve and adjust the decision threshold; high AUC does not guarantee high precision for the minority class.

2. Q: You observe over‑fitting after increasing max_depth of a Decision Tree. What regularization can you apply?
   A: Limit max_depth, increase min_samples_leaf, or switch to a Random Forest (bagging).

3. Q: Which metric should you prioritize if the cost of a false negative (missing a churner) is much higher than a false positive?
   A: Recall (or Sensitivity) – you want to capture as many true churners as possible.

Last‑Minute Cram Sheet (10 one‑liners)

1. Logistic → sigmoid; loss = cross‑entropy.
2. Decision Tree split = max Information Gain (or min Gini impurity).
3. Random Forest = bagging + random feature sub‑sampling; reduces variance.
4. SVM margin = 2/‖w‖; C = inverse regularization (large C → hard‑margin).
5. Precision = TP/(TP+FP); Recall = TP/(TP+FN).
6. F1 balances precision & recall; useful when you need a single score for imbalanced data.
7. ROC‑AUC = probability a random positive is ranked higher than a random negative.
8. Stratified split = keep class ratios identical across train/test (critical for rare events).
9. ⚠️ Scaling matters for SVM & Logistic Regression; StandardScaler assumes Gaussian‑like features, otherwise use MinMaxScaler.
10. ⚠️ “class_weight='balanced'” in scikit‑learn automatically inverses class frequencies – a quick fix for imbalance.