Data Science and Machine Learning 101: Foundations and Math Linear Algebra for ML Vectors Matrices Eigenvalues SVD

What This Is

Linear algebra is the language of vectors and matrices that lets us compactly represent data, model parameters, and transformations. In ML every feature set becomes a matrix X, every weight vector w, and many algorithms (PCA, linear regression, neural‑net back‑prop) are just matrix‑vector arithmetic. For example, a recommendation engine for a streaming service stores user‑item interactions in a sparse matrix R; factorizing R with SVD uncovers latent “taste” dimensions that power personalized suggestions.

Key Terms & Formulas

• Vector ? ∈ ℝⁿ – an ordered list of n real numbers (e.g., a single data point’s features).
• Matrix X ∈ ℝ^{m×n} – m rows (samples) × n columns (features).
• Dot product ?·? = Σᵢ xᵢ yᵢ – measures similarity; used in linear models: ŷ = w·?.
• Matrix multiplication AB where A∈ℝ^{p×q}, B∈ℝ^{q×r} → AB∈ℝ^{p×r}.
• Transpose Xᵀ – flips rows ↔ columns; needed for normal equations: w = (XᵀX)⁻¹ Xᵀy.
• Eigenvalue λ & eigenvector v – satisfy Av = λv; λ tells how much v is stretched by A.
• Singular Value Decomposition (SVD): X = U Σ Vᵀ, where
• U ∈ ℝ^{m×m} (left singular vectors),
• Σ ∈ ℝ^{m×n} diagonal with singular values σ₁≥σ₂…≥0,
• V ∈ ℝ^{n×n} (right singular vectors).
• Rank‑k approximation Xₖ = Uₖ Σₖ Vₖᵀ – keep top‑k singular values; reduces dimensionality with minimal Frobenius‑norm error.
• Covariance matrix C = (1/(m‑1)) XᵀX – captures feature‑wise variance and correlation; eigen‑decomposition of C yields PCA directions.
• Pseudo‑inverse X⁺ = V Σ⁺ Uᵀ – replaces matrix inverse when X is not square or is rank‑deficient; used in ridge regression: w = (XᵀX + αI)⁻¹ Xᵀy.
• Norms – ‖?‖₂ = sqrt(Σ xᵢ²) (Euclidean), ‖?‖₁ = Σ |xᵢ| (L1); regularization penalties are added to loss functions.

Step‑by‑Step / Process Flow

1. Load & vectorize – read raw data (CSV, images, logs) into a pandas DataFrame, then convert to a NumPy matrix X (X = df.values).
2. Center & scale – subtract column means (X -= X.mean(axis=0)) and optionally divide by std (X /= X.std(axis=0)) to prepare for SVD/PCA.
3. Decompose – run SVD (U, s, VT = np.linalg.svd(X, full_matrices=False)) or eigen‑decompose the covariance (eigvals, eigvecs = np.linalg.eigh(X.T @ X)).
4. Select components – pick k based on explained variance (explained = np.cumsum(s2) / np.sum(s2)).
5. Project – obtain low‑dimensional features (X_reduced = X @ VT.T[:, :k]). Use these in downstream models (e.g., logistic regression).
6. Interpret & iterate – visualize top singular vectors (heatmaps of feature loadings) to ensure they capture meaningful patterns; adjust k or preprocessing as needed.

Common Mistakes

• Mistake: Applying SVD to non‑centered data.
  Correction: Center the matrix first; otherwise the first singular vector captures the mean rather than true variance.

• Mistake: Confusing eigenvectors of X with those of the covariance matrix.
  Correction: For PCA you need eigenvectors of XᵀX (or C), not of X itself; they are the right singular vectors V.

• Mistake: Using the full‑rank inverse (np.linalg.inv) on a singular or ill‑conditioned matrix.
  Correction: Use the pseudo‑inverse (np.linalg.pinv) or add ridge regularization (αI).

• Mistake: Dropping too many components because the scree plot looks “flat”.
  Correction: Compute cumulative explained variance and keep enough components to retain ≥90 % (or domain‑specific) variance.

• Mistake: Assuming SVD is “fast enough” for huge sparse matrices.
  Correction: Switch to truncated SVD (sklearn.utils.extmath.randomized_svd) or use sparse‑aware libraries (SciPy’s svds).

Data Science Interview / Practical Insights

1. “Explain why SVD is preferred over eigen‑decomposition for dimensionality reduction on a non‑square matrix.” – Expect you to mention numerical stability, ability to handle rectangular data, and that singular values are the square roots of eigenvalues of XᵀX.
2. “When would you use a pseudo‑inverse instead of a regular inverse?” – Look for answers about rank deficiency, over‑determined/under‑determined systems, and ridge regularization.
3. “How does the choice of norm (L1 vs L2) affect the geometry of the solution in linear regression?” – Interviewers probe sparsity (L1 → feature selection) vs shrinkage (L2 → smooth coefficients).
4. “What’s the computational complexity of a full SVD versus a truncated SVD?” – Full SVD: O(min(mn², m²n)); truncated (k components): O(mn k).

Quick Check Questions

1. Q: Your text‑classification matrix X has 100 000 documents × 50 000 terms and is extremely sparse. Which linear‑algebra tool should you use to obtain a 100‑dimensional representation?
   A: Truncated SVD (sklearn.decomposition.TruncatedSVD) because it works directly on sparse matrices and avoids forming the dense covariance.

2. Q: After fitting a linear regression with the normal equation, you get a warning about a singular matrix. What’s the quickest fix?
   A: Add a small ridge term (αI) to XᵀX before inversion, i.e., use w = (XᵀX + αI)⁻¹ Xᵀy.

3. Q: In PCA you notice the first principal component explains 70 % of variance, but the second only 5 %. Should you keep the second component?
   A: Yes, if the cumulative variance after the second component reaches your target (e.g., 90 %); otherwise you may drop it.

Last‑Minute Cram Sheet (10 one‑liners)

1. Vector dot product = similarity: np.dot(a, b).
2. Matrix‑vector multiply: y = X @ w (fast O(mn) operation).
3. Normal equation: w = np.linalg.inv(X.T @ X) @ X.T @ y.
4. Ridge regression: w = np.linalg.inv(X.T @ X + α*np.eye(n)) @ X.T @ y.
5. SVD: U, s, VT = np.linalg.svd(X, full_matrices=False).
6. Truncated SVD (sparse): svd = TruncatedSVD(n_components=k); X_red = svd.fit_transform(X).
7. Explained variance: explained = np.cumsum(s2) / np.sum(s2).
8. Pseudo‑inverse: X_pinv = np.linalg.pinv(X).
9. Eigen‑decomposition for PCA: eigvals, eigvecs = np.linalg.eigh(X.T @ X).
10. ⚠️ Pitfall: Centering is mandatory for PCA/SVD; forgetting it makes the first component capture the mean, not true variance.