Data Science and Machine Learning 101: Programming and Data Engineering Python for Data Science NumPy Pandas Scikitlearn Matplotlib Seaborn

What This Is

Python for Data Science is the toolbox that lets you turn raw data into actionable insights. It bundles NumPy for fast numeric arrays, pandas for tabular manipulation, Matplotlib/Seaborn for visual storytelling, and scikit‑learn for ready‑to‑use machine‑learning pipelines. In a churn‑prediction project, you’d load millions of customer events with NumPy, clean and aggregate them with pandas, explore patterns with Seaborn, and ship a logistic‑regression model via scikit‑learn—all in a single, reproducible Python script.

Key Terms & Formulas

• NumPy ndarray – n‑dimensional homogeneous array; operations are vectorized (e.g., X + 5 adds 5 to every element).
• Broadcasting – Implicit expansion of smaller arrays to match larger ones for element‑wise ops (e.g., shape (3, 1) + shape (1, 4) → shape (3, 4)).
• pandas DataFrame – Labeled 2‑D table; columns can hold different dtypes, enabling SQL‑like groupby, pivot, and merge.
• .loc vs .iloc – .loc selects by label, .iloc selects by integer position; mixing them is a common source of bugs.
• StandardScaler – (z = (x - \mu)/\sigma); centers data to zero mean and unit variance, crucial for algorithms that assume Gaussian‑like features (e.g., SVM, Logistic Regression).
• One‑Hot Encoding – Convert categorical variable with k levels into a binary matrix (X_{i,j} = 1) if observation i belongs to category j, else 0.
• Train‑Test Split – Randomly partition data: X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y).
• Cross‑Validation (k‑fold) – Average performance over k splits; reduces variance of the estimate. Formula: (\text{CV_score} = \frac{1}{k}\sum_{i=1}^{k} \text{score}_i).
• Logistic Regression (binary) – (p = \sigma(w^\top x + b)) where (\sigma(z)=1/(1+e^{-z})); predict class = 1 if (p > 0.5).
• ROC‑AUC – Probability that a randomly chosen positive is ranked higher than a negative; computed from the ROC curve (TPR vs. FPR).
• Bagging – Train multiple base learners on bootstrap samples and aggregate (e.g., RandomForestClassifier). Reduces variance.
• Boosting – Sequentially train learners, each focusing on previous errors (e.g., GradientBoostingClassifier). Reduces bias.

Step‑by‑Step / Process Flow

1. Load & Inspect
   python
import numpy as np, pandas as pd
df = pd.read_csv('customer_events.csv')
print(df.head(), df.describe())
2. Clean & Feature Engineer
3. Handle missing values (df.fillna(method='ffill')).
4. Create time‑based aggregates (df.groupby('customer_id')['event'].sum()), one‑hot encode categorical columns (pd.get_dummies(df['plan'])).
5. Split & Scale
   python
from sklearn.model_selection import train_test_split
X = df.drop('churn', axis=1)
y = df['churn']
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, stratify=y, random_state=42)
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler().fit(X_train)
X_train, X_test = scaler.transform(X_train), scaler.transform(X_test)
6. Baseline Model
   python
from sklearn.linear_model import LogisticRegression
model = LogisticRegression(max_iter=500, class_weight='balanced')
model.fit(X_train, y_train)
7. Evaluate & Diagnose
   python
from sklearn.metrics import classification_report, roc_auc_score
preds = model.predict(X_test)
print(classification_report(y_test, preds))
print('AUC:', roc_auc_score(y_test, model.predict_proba(X_test)[:,1]))
8. Iterate (tune, feature select, try other algorithms) – Use GridSearchCV or RandomizedSearchCV and plot learning curves with Matplotlib/Seaborn.

Common Mistakes

• Mistake: Scaling after train‑test split on the combined data.
  Correction: Fit the scaler only on the training set and apply the same transformation to the test set to avoid data leakage.

• Mistake: Ignoring class imbalance and using default accuracy.
  Correction: Use class_weight='balanced' or resampling techniques; evaluate with ROC‑AUC or Precision‑Recall instead of raw accuracy.

• Mistake: Mixing .loc and .iloc in the same indexing expression, leading to unexpected rows/columns.
  Correction: Stick to one style per operation; double‑check shapes with .shape.

• Mistake: Assuming np.mean on a DataFrame works column‑wise like pandas.
  Correction: Convert to NumPy (df.values) or use pandas methods (df.mean()) to respect column labels.

• Mistake: Over‑fitting by using the test set for hyper‑parameter tuning.
  Correction: Reserve a validation set or use nested cross‑validation; keep the final test set untouched until the very end.

Data Science Interview / Practical Insights

1. “When would you prefer MinMaxScaler over StandardScaler?” – MinMax preserves the shape of the distribution and is safe for algorithms that are sensitive to outliers (e.g., K‑Nearest Neighbors, Neural Nets).
2. “Explain the bias‑variance trade‑off in the context of Random Forest vs. Gradient Boosting.” – Random Forest reduces variance via bagging; Gradient Boosting reduces bias by sequentially correcting errors, but can overfit if depth is too high.
3. “How does pandas’ groupby differ from SQL’s GROUP BY?” – groupby returns a GroupBy object that lazily evaluates; you must call an aggregation (.sum(), .mean()) and can apply custom functions with .apply().
4. “What’s the effect of np.where vs. pandas .mask for conditional assignment?” – np.where works on NumPy arrays (fast, no index), while .mask respects DataFrame alignment and can handle mixed dtypes more safely.

Quick Check Questions

1. Scenario: Your churn model’s ROC‑AUC is 0.92 but Precision is only 0.35 on the minority class.
   Answer: Use a Precision‑Recall curve and possibly adjust the decision threshold; consider oversampling or class‑weighting.

2. Scenario: After adding a new categorical feature, model training slows dramatically.
   Answer: Apply One‑Hot Encoding only to high‑cardinality columns after frequency filtering or use Target Encoding to keep dimensionality low.

3. Scenario: You notice the training loss keeps decreasing but validation loss plateaus.
   Answer: Increase regularization (e.g., C in LogisticRegression) or add early stopping; the model is likely over‑fitting.

Last‑Minute Cram Sheet (10 one‑liners)

1. NumPy broadcasting works when trailing dimensions are equal or one of them is 1.
2. .loc → label‑based; .iloc → integer‑position‑based.
3. StandardScaler → μ = mean, σ = std; MinMaxScaler → (x‑min)/(max‑min).
4. One‑Hot creates k binary columns; beware the dummy‑variable trap (drop one column).
5. train_test_split(stratify=y) preserves class ratios in both splits.
6. LogisticRegression uses L2 by default; set penalty='l1' for sparsity.
7. ROC‑AUC is insensitive to class imbalance; PR‑AUC is more informative when positives are rare.
8. Bagging = parallel, independent learners; Boosting = sequential, error‑focused learners.
9. GridSearchCV → exhaustive; RandomizedSearchCV → faster, good for many hyperparameters.
10. ⚠️ StandardScaler assumes a roughly Gaussian distribution; if your feature is heavily skewed, apply a log or Box‑Cox transform first.