Data Science and Machine Learning 101: Foundations and Math Probability Basics Bayes Theorem Distributions Expectation Variance

What This Is

Probability basics are the mathematical foundation for reasoning about uncertainty in data‑driven problems. They let you turn raw observations into likelihoods, compute expected outcomes, and combine evidence with Bayes’ theorem. In a data‑science pipeline you’ll use them to (1) pick the right statistical model, (2) evaluate risk (e.g., “Will this customer churn?”), and (3) calibrate predictions from classifiers or recommender systems. Example: A telecom company estimates the probability a subscriber will churn next month by blending historic churn rates (prior) with recent usage spikes (likelihood) via Bayes’ rule, then uses the posterior to target retention offers.

Key Terms & Formulas

• Bayes’ Theorem – (\displaystyle P(A|B)=\frac{P(B|A)\,P(A)}{P(B)})
  (P(A|B)): posterior, (P(B|A)): likelihood, (P(A)): prior, (P(B)): evidence.

• Probability Mass Function (PMF) – For discrete (X): (\displaystyle p_X(x)=P(X=x)). Sums to 1.

• Probability Density Function (PDF) – For continuous (X): (\displaystyle f_X(x)) with (\displaystyle \int_{-\infty}^{\infty} f_X(x)dx = 1).

• Cumulative Distribution Function (CDF) – (\displaystyle F_X(x)=P(X\le x)=\int_{-\infty}^{x} f_X(t)dt).

• Expectation (Mean) – (\displaystyle \mathbb{E}[X]=\sum_x x\,p_X(x)) (discrete) or (\displaystyle \int_{-\infty}^{\infty} x\,f_X(x)dx) (continuous).

• Variance – (\displaystyle \operatorname{Var}(X)=\mathbb{E}[(X-\mu)^2]=\mathbb{E}[X^2]-\mu^2), where (\mu=\mathbb{E}[X]).

• Standard Deviation – (\sigma=\sqrt{\operatorname{Var}(X)}); measures spread in the same units as (X).

• Law of Total Probability – (\displaystyle P(B)=\sum_i P(B|A_i)P(A_i)) for a partition ({A_i}).

• Conditional Independence – (X\perp Y \mid Z) ⇔ (P(X,Y|Z)=P(X|Z)P(Y|Z)). Crucial for Naïve Bayes.

• Maximum Likelihood Estimation (MLE) – (\displaystyle \hat\theta_{\text{MLE}}=\arg\max_\theta \prod_{i=1}^n f(x_i|\theta)). Gives the parameter values that make the observed data most probable.

• Conjugate Prior – A prior distribution that yields a posterior of the same family (e.g., Beta prior → Beta posterior for Bernoulli likelihood).

Step‑by‑Step / Process Flow

1. Load & Inspect Data – df = pd.read_csv('churn.csv'); check class balance with df['churn'].value_counts(normalize=True).
2. Choose a Probabilistic Model – If the target is binary, start with Bernoulli (logistic regression) or Naïve Bayes; for counts, consider Poisson.
3. Estimate Prior & Likelihood
4. Prior: compute empirical class probability p_prior = df['churn'].mean().
5. Likelihood: fit a distribution to a feature (e.g., scipy.stats.norm.fit(df['monthly_minutes'])).
6. Apply Bayes’ Theorem – For a new customer x_new, compute posterior:
   python
from scipy.stats import norm
prior = p_prior
likelihood = norm.pdf(x_new['monthly_minutes'], *params) # PDF value
evidence = np.mean(likelihood) # approximated by averaging over training set
posterior = likelihood * prior / evidence
7. Validate – Split data (train_test_split), compute calibration curve (sklearn.calibration.calibration_curve) and AUC‑ROC. If posterior probabilities are poorly calibrated, apply Platt scaling or Isotonic regression.
8. Iterate / Deploy – Store the fitted prior/likelihood parameters, expose a REST endpoint that returns the churn probability for incoming feature vectors.

Common Mistakes

• Mistake: Treating a PDF value as a probability.
  Correction: PDFs can exceed 1; they must be integrated over an interval to yield a probability.

• Mistake: Using the sample mean as the prior without accounting for class imbalance.
  Correction: Either re‑balance (SMOTE, class weighting) or explicitly set a prior that reflects business risk (e.g., higher churn cost).

• Mistake: Assuming independence when features are correlated (Naïve Bayes).
  Correction: Verify independence with correlation matrices; if strong dependence exists, consider Tree‑augmented Naïve Bayes or a different model.

• Mistake: Forgetting the evidence term (P(B)) and normalizing posteriors manually.
  Correction: Always compute the denominator (or use a library that does it) to keep posterior probabilities summing to 1.

• Mistake: Ignoring variance when comparing two distributions (e.g., saying “means are equal → same performance”).
  Correction: Use t‑test or Welch’s test that incorporate variance, or compare confidence intervals.

Data Science Interview / Practical Insights

1. “Explain Bayes’ theorem in the context of a spam filter.” – Expect you to map prior = overall spam rate, likelihood = word‑presence probability given spam, posterior = spam probability for a new email.
2. “When is a Beta distribution a conjugate prior?” – For a Bernoulli or Binomial likelihood; the posterior remains Beta with updated α, β.
3. “How do you decide between using a Gaussian vs. a Poisson model for a count feature?” – Discuss the support (continuous vs. non‑negative integers), mean‑variance relationship (Poisson: mean = variance), and over‑dispersion.
4. “What does it mean for a model to be well‑calibrated?” – The predicted probability matches the empirical frequency; e.g., among all predictions of 0.7, ~70 % should be positive.

Quick Check Questions

1. Scenario: Your churn model outputs a posterior of 0.85 for a high‑value customer.
   Answer: The customer has an 85 % chance to churn; you would prioritize a retention offer.

2. Scenario: You fit a Gaussian to monthly minutes, but the data are heavily right‑skewed.
   Answer: The Gaussian assumption is violated; consider a log‑transform or a Gamma distribution instead.

3. Scenario: After applying Naïve Bayes, the validation AUC is 0.55, yet the training AUC is 0.99.
   Answer: The model is over‑fitting due to strong independence assumptions; try a more flexible classifier or add feature engineering.

Last‑Minute Cram Sheet (10 one‑liners)

1. Bayes: Posterior ∝ Likelihood × Prior (normalize by evidence).
2. PDF vs. PMF: PDF integrates to probability; PMF sums to probability.
3. Expectation: (\mathbb{E}[X]=\sum x p(x)) (discrete) or (\int x f(x)dx) (continuous).
4. Variance: (\operatorname{Var}(X)=\mathbb{E}[X^2]-\mathbb{E}[X]^2).
5. Law of Total Probability: Break a complex event into mutually exclusive pieces.
6. Conjugate Prior: Keeps posterior in the same family → easy updating (Beta–Bernoulli, Gamma–Poisson).
7. MLE: Maximize product of PDFs; often solved by taking log → sum of log‑likelihoods.
8. Conditional Independence: Naïve Bayes assumes (X_i \perp X_j | Y); violation hurts performance.
9. ⚠️ PDF > 1 is not an error; only the integral must be ≤ 1.
10. ⚠️ StandardScaler assumes features are roughly Gaussian; for skewed data use RobustScaler or MinMaxScaler.