Data Science and Machine Learning 101: Foundations and Math Exploratory Data Analysis EDA Univariate Bivariate Correlation Outlier Detection

What This Is

Exploratory Data Analysis (EDA) is the first‑hand, visual‑and‑statistical inspection of a dataset to uncover its shape, spot problems, and generate hypotheses before any modeling. In a churn‑prediction project, EDA tells you whether “tenure” is skewed, if “monthly charges” and “total charges” are almost duplicates, or whether a handful of customers have impossible values (e.g., negative tenure). Those insights dictate cleaning, feature engineering, and the choice of algorithms.

Key Terms & Formulas

• Univariate Distribution – Summary of a single variable (mean, median, std, histogram).
• Bivariate Relationship – Interaction between two variables (scatter plot, cross‑tab, group‑by stats).
• Pearson r = Σ[(xᵢ‑μₓ)(yᵢ‑μ_y)] / [(n‑1)σₓσ_y] – Linear correlation; values ∈ [‑1, 1].
• Spearman ρ = 1 ‑ (6 Σ dᵢ²) / (n(n²‑1)) – Rank‑based correlation; robust to monotonic but non‑linear patterns.
• Kurtosis – Measure of tail heaviness; excess kurtosis = (μ₄ / σ⁴) ‑ 3.
• IQR = Q3 ‑ Q1 – Inter‑quartile range; used for box‑plot outlier fences.
• Z‑score = (x ‑ μ) / σ – Standardized distance from the mean; |z| > 3 often flags outliers.
• Isolation Forest – Tree‑based outlier detector; isolates a point with fewer splits → higher anomaly score.
• Mahalanobis Distance = √[(x‑μ)ᵀ Σ⁻¹ (x‑μ)] – Multivariate outlier metric accounting for covariance.
• Chi‑square test of independence – χ² = Σ (O‑E)² / E; tests if two categorical variables are independent.
• Box‑Cox / Yeo‑Johnson – Power transforms to reduce skewness; λ chosen by maximum likelihood.
• pd.DataFrame.describe() – Quick numeric summary (count, mean, std, min, 25%, 50%, 75%, max).

Step‑by‑Step / Process Flow

1. Load & Inspect
   python
df = pd.read_csv('churn.csv')
print(df.shape, df.head())
df.info() # dtype, missing counts

2. Univariate Checks – For each column:

3. df[col].describe() → spot extreme min/max.
4. Plot sns.histplot(df[col], kde=True) or df[col].value_counts().plot.bar() for categoricals.

5. Compute skew/kurtosis; flag heavily skewed numeric features.

6. Bivariate Exploration – Pairwise diagnostics:

7. Numeric vs numeric: sns.scatterplot(x=col1, y=col2, data=df).
8. Numeric vs categorical: df.groupby(cat)[num].agg(['mean','std']).

9. Categorical vs categorical: pd.crosstab(df[cat1], df[cat2]) + chi2_contingency.

10. Correlation & Redundancy
    python
corr = df.select_dtypes('number').corr()
sns.heatmap(corr, cmap='coolwarm', annot=True)

11. Drop one of any pair with |r| > 0.9 (or use PCA later).

12. Outlier Detection & Treatment

13. Univariate: z = np.abs(stats.zscore(df[num])) > 3 → cap or drop.
14. Multivariate: IsolationForest().fit_predict(df[num]) → flag anomalies.

15. Decide: capping, log transform, or remove based on business impact.

16. Finalize Cleaned Data – Impute missing values (SimpleImputer(strategy='median')), encode categoricals (OneHotEncoder(handle_unknown='ignore')), and store the EDA notebook for reproducibility.

Common Mistakes

• Mistake: Relying only on mean ± 2 σ to flag outliers.
  Correction: Use robust methods (IQR, Z‑score with median, or Isolation Forest) because heavy tails distort the mean and σ.

• Mistake: Treating a high Pearson r as proof of causation.
  Correction: Remember correlation ≠ causation; always check for confounders and consider domain knowledge.

• Mistake: Dropping all rows with any missing value.
  Correction: Impute intelligently (median for skewed numeric, most_frequent for categoricals) or model‑based imputation; otherwise you lose valuable data.

• Mistake: Running a single histogram and assuming the distribution is “normal”.
  Correction: Complement histograms with Q‑Q plots (stats.probplot) and skew/kurtosis metrics; apply Box‑Cox/Yeo‑Johnson when needed.

• Mistake: Ignoring categorical correlation (e.g., using Pearson on encoded integers).
  Correction: Use Cramér’s V or chi‑square for categorical‑categorical relationships; encode only after assessing dependence.

Data Science Interview / Practical Insights

1. “Explain the difference between Pearson and Spearman correlation and when you’d choose each.” – Expect you to mention linear vs monotonic, sensitivity to outliers, and that Spearman works on ranks.
2. “How would you detect outliers in a high‑dimensional dataset where visual inspection is impossible?” – Talk about Isolation Forest, Local Outlier Factor, and Mahalanobis distance, plus the trade‑off between false positives and business cost.
3. “What’s the danger of using a correlation heatmap on one‑hot encoded variables?” – Highlight that binary variables can produce misleading high correlations due to sparse overlap; recommend using chi‑square or Cramér’s V instead.
4. “If two features have r = 0.98, should you drop one? Why or why not?” – Discuss multicollinearity, model stability, and when you might keep both (e.g., tree‑based models) versus linear models.

Quick Check Questions

1. Scenario: Your “total_charges” column is almost perfectly correlated (r = 0.99) with “monthly_charges”.
   Answer: Drop one (or combine) because the redundancy inflates variance in linear models; tree models can tolerate it.

2. Scenario: After computing Z‑scores, 5 % of rows are flagged as outliers, but they belong to a high‑value “enterprise” customer segment.
   Answer: Do not automatically drop them; investigate business relevance and consider capping or separate modeling for that segment.

3. Scenario: A scatter plot of “age” vs “spending_score” shows a curved pattern.
   Answer: Pearson r will be low; use Spearman ρ or fit a non‑linear model (e.g., polynomial regression) after confirming monotonicity.

Last‑Minute Cram Sheet (10 one‑liners)

1. ⚠️ Pearson r only captures linear relationships; use Spearman ρ for monotonic but non‑linear trends.
2. IQR‑based fences: lower = Q1 ‑ 1.5·IQR, upper = Q3 + 1.5·IQR.
3. Z‑score outlier rule: |z| > 3 (or 2.5 for small n).
4. Isolation Forest anomaly score ∈ [0,1]; > 0.5 = outlier by default.
5. Mahalanobis distance requires Σ⁻¹; regularize Σ if singular (add ε·I).
6. Chi‑square test requires expected count ≥ 5 in each cell; otherwise use Fisher’s exact test.
7. Box‑Cox works only on strictly positive data; Yeo‑Johnson handles zeros/negatives.
8. High cardinality categorical → one‑hot explosion; prefer target encoding or hashing trick.
9. Correlation heatmap → mask the diagonal to avoid visual clutter.
10. Always pair visual (histogram, box‑plot) with numeric (skew, kurtosis) checks—one alone is insufficient.