Data Science and Machine Learning 101: Machine Learning Core Unsupervised Learning KMeans Hierarchical Clustering DBSCAN PCA tSNE

Unsupervised Learning (K‑Means, Hierarchical Clustering, DBSCAN, PCA, t‑SNE)

What This Is

Unsupervised learning discovers hidden structure in data without any target labels. It is the toolbox for grouping similar observations (clustering) and compressing high‑dimensional information into a few interpretable axes (dimensionality reduction). In a real‑world pipeline you might cluster shoppers by purchase behavior to design segment‑specific promotions, or use PCA/t‑SNE to visualise millions of gene‑expression profiles before feeding them to a downstream classifier.

Key Terms & Formulas

• K‑Means Objective – Minimize within‑cluster sum of squares:
  [ J = \sum_{k=1}^{K}\sum_{x_i \in C_k}|x_i-\mu_k|_2^2 ]
  where (C_k) is cluster k and (\mu_k) its centroid.

• Silhouette Score – Measures cohesion vs. separation:
  [ s(i)=\frac{b(i)-a(i)}{\max{a(i),b(i)}} ]
  (a(i)) = avg. distance to own cluster, (b(i)) = lowest avg. distance to another cluster.

• Hierarchical Linkage – Distance between clusters for agglomerative trees:

• Single: (d(C_a,C_b)=\min_{x\in C_a,y\in C_b}|x-y|)
• Complete: (d(C_a,C_b)=\max_{x\in C_a,y\in C_b}|x-y|)

• Ward: Increase in SSE (same objective as K‑Means).

• DBSCAN Core Point – A point with at least min_samples points within radius ε: (|{x_j : |x_i-x_j|\le ε}|\ge \text{min_samples}).

• DBSCAN Cluster Assignment – All density‑connected points (core → reachable) belong to the same cluster; points not reachable become noise.

• Principal Component Analysis (PCA) – Linear projection onto eigenvectors of the covariance matrix:
  [ \Sigma = \frac{1}{n-1}X^\top X,\quad \Sigma v_k = \lambda_k v_k ]
  Projected data: (Z = X V_{(p)}) where (V_{(p)}) contains the top‑(p) eigenvectors.

• Explained Variance Ratio – Fraction of total variance captured by each component:
  [ \text{EVR}k = \frac{\lambda_k}{\sum ] }^{d}\lambda_j

• t‑SNE (t‑Distributed Stochastic Neighbor Embedding) – Converts pairwise similarities to joint probabilities and minimizes KL divergence:
  [ C = \sum_{i\neq j} P_{ij}\log\frac{P_{ij}}{Q_{ij}} ]
  (P_{ij}) = symmetrized Gaussian affinities in high‑dim space, (Q_{ij}) = Student‑t affinities in low‑dim space.

• Perplexity – Controls effective number of neighbours in t‑SNE; defined as (2^{H(P)}) where (H) is Shannon entropy of (P).

• Scikit‑Learn API – All algorithms expose fit(X), fit_predict(X), and transform(X) (for PCA/t‑SNE).

Step‑by‑Step / Process Flow

1. Load & Inspect – df = pd.read_csv(...); check missingness, data types, and basic stats.
2. Pre‑process –
3. Impute or drop missing values (SimpleImputer).
4. Scale numeric features (StandardScaler for Euclidean‑based clustering, MinMaxScaler for DBSCAN with ε in original units).
5. Encode categoricals (OneHotEncoder or OrdinalEncoder depending on distance semantics).
6. Choose Goal –
7. Clustering: decide K‑Means vs. hierarchical vs. DBSCAN based on shape, size, and noise.
8. Dimensionality reduction: apply PCA first to denoise, then t‑SNE for visualisation.
9. Fit Model –
   python
# K‑Means
km = KMeans(n_clusters=5, random_state=42).fit(X_scaled)
labels = km.labels_
# DBSCAN
db = DBSCAN(eps=0.5, min_samples=10).fit(X_scaled)
# PCA + t‑SNE
pca = PCA(n_components=0.95).fit_transform(X_scaled) # keep 95% variance
tsne = TSNE(perplexity=30, n_iter=1000).fit_transform(pca)
10. Validate & Tune –
11. Compute Silhouette or Davies‑Bouldin scores for clustering.
12. Plot explained variance curve to pick PCA components.
13. Grid‑search eps and min_samples for DBSCAN (sklearn.model_selection.ParameterGrid).
14. Interpret & Deploy –
15. Visualise clusters on the first two PCA components or t‑SNE map (sns.scatterplot).
16. Export centroids (km.cluster_centers_) or assign new points with predict.
17. Store the pipeline with joblib.dump(pipeline, 'unsup.pkl').

Common Mistakes

• Mistake: Scaling after clustering (e.g., running K‑Means on raw dollars).
  Correction: Always scale before any distance‑based algorithm; otherwise large‑scale features dominate the Euclidean distance.

• Mistake: Using K‑Means on non‑convex shapes (e.g., moons, rings).
  Correction: Switch to DBSCAN or hierarchical clustering with a linkage that captures irregular boundaries.

• Mistake: Setting n_components in PCA arbitrarily (e.g., 2) and discarding 95 % of variance.
  Correction: Inspect the cumulative explained variance plot and choose the smallest k that retains the desired variance (commonly 0.90–0.99).

• Mistake: Interpreting t‑SNE axes as meaningful coordinates.
  Correction: Treat t‑SNE as a visual tool only; distances are not preserved globally, and the embedding can change with random seed.

• Mistake: Ignoring DBSCAN’s noise points and assuming they belong to a cluster.
  Correction: Analyse noise separately—often they are outliers or rare cases worth domain review.

Data Science Interview / Practical Insights

1. “When would you prefer DBSCAN over K‑Means?” – Expect you to discuss density‑based vs. centroid‑based clustering, handling of arbitrary shapes, and robustness to outliers.
2. “Explain the bias‑variance trade‑off in PCA.” – Interviewers look for understanding that keeping too few components introduces bias (information loss), while keeping many can retain noise (high variance).
3. “How does the choice of distance metric affect hierarchical clustering?” – Be ready to compare Euclidean, Manhattan, and cosine, and how single vs. complete linkage reacts to those metrics.
4. “What are the pitfalls of using t‑SNE on very large datasets?” – Mention computational cost (O(N²)), need for early‑exaggeration, and that perplexity should be ≤ N/3.

Quick Check Questions

1. Scenario: Your customer dataset has a few high‑spending outliers that skew the K‑Means centroids.
   Answer: Use DBSCAN (or robust scaling) because it treats dense regions separately and marks extreme points as noise.

2. Scenario: After PCA you see the first component explains 30 % variance, the second 15 %, and the third 12 %. You need ≥ 85 % cumulative variance.
   Answer: Keep at least the first 7 components (30+15+12+10+9+8+5 ≈ 89 %).

3. Scenario: You run t‑SNE with perplexity = 5 on a dataset of 10 000 points and the plot looks “clumped”.
   Answer: Increase perplexity (e.g., 30–50) because low perplexity forces each point to consider too few neighbours, leading to over‑fragmented embeddings.

Last‑Minute Cram Sheet (10 one‑liners)

1. K‑Means: O(K·N·I) time; converges to local optimum → run with several n_init.
2. Silhouette > 0.5 → well‑separated clusters; < 0.25 → overlapping clusters.
3. Ward linkage = minimize SSE → produces clusters similar to K‑Means but hierarchical.
4. DBSCAN ε ≈ k‑distance plot elbow; min_samples ≈ dimensionality + 1.
5. PCA: Center data (X -= X.mean(axis=0)) before SVD; eigenvectors = right singular vectors.
6. Explained variance = eigenvalue / total variance; use explained_variance_ratio_.
7. t‑SNE: Uses Student‑t (df = 1) in low‑dim to avoid “crowding problem”.
8. Perplexity ≈ effective number of neighbours; typical 5–50.
9. StandardScaler assumes Gaussian; for heavy‑tailed data prefer RobustScaler (median & IQR).
10. ⚠️ Never evaluate clustering with accuracy; use internal metrics (Silhouette, DBI) or external labels if available.