Business Analytics 101: Machine Learning Fundamentals Dimensionality Reduction PCA tSNE Factor Analysis

What This Is

Dimensionality reduction is a family of techniques used to simplify complex data by reducing the number of features or variables while retaining the most important information. This matters in business analytics because high-dimensional data can be difficult to visualize, analyze, and interpret. For example, consider a retail company with 100 product features, such as price, color, material, and brand. Using dimensionality reduction, the company can reduce the number of features to 5-10, making it easier to identify customer segments, predict sales, and optimize marketing campaigns.

Key Formulas & Metrics

• Principal Component Analysis (PCA)
• Eigenvalue (λ) = (1/n) Σ(xᵢ − μ)² – measures the amount of variance explained by each principal component.
• Eigenvalue (λ) > 1 – indicates that the principal component explains more variance than the original feature.
• Cumulative Proportion of Variance Explained (CPVE) = Σ(λᵢ / Σλ) – measures the proportion of variance explained by the top k principal components.
• t-Distributed Stochastic Neighbor Embedding (t-SNE)
• Kullback-Leibler Divergence (KL) = Σ(p(x) log(p(x) / q(x))) – measures the difference between the probability distributions of the original and embedded data.
• Perplexity (P) = exp(-KL / n) – measures the average perplexity of the embedded data.
• Factor Analysis
• Common Factor (F) = Σ(aᵢ xᵢ) – measures the common variance among the original features.
• Specific Factor (S) = xᵢ - Σ(aᵢ F) – measures the unique variance of each original feature.
• Factor Loading (a) = Σ(F xᵢ) / Σ(F²) – measures the correlation between each original feature and the common factor.

Step-by-Step Procedure

1. Data Preparation: Select the relevant features and data points, and check for missing values and outliers.
2. Scaling: Scale the data to have zero mean and unit variance, if necessary.
3. Choosing the Number of Components: Select the number of components to retain based on the eigenvalues, cumulative proportion of variance explained, or cross-validation.
4. Applying the Dimensionality Reduction Technique: Use PCA, t-SNE, or Factor Analysis to reduce the dimensionality of the data.
5. Interpreting the Results: Visualize the reduced data and identify patterns, clusters, or correlations.
6. Model Evaluation: Evaluate the performance of the reduced data using metrics such as accuracy, precision, or R².

Common Mistakes

• Mistake: Confusing correlation with causation.
• Correction: Correlation does not imply causation. Use techniques such as regression analysis or experimentation to establish causality.
• Mistake: Misinterpreting p-values.
• Correction: A small p-value indicates that the observed effect is statistically significant, but not necessarily practically significant. Consider the effect size and context.
• Mistake: Using the wrong error metric for a business problem.
• Correction: Choose an error metric that aligns with the business objective, such as mean absolute error (MAE) for forecasting or precision for classification.

Software / Tool Tips

• Python with scikit-learn: Use the PCA, TSNE, and FactorAnalysis classes to apply dimensionality reduction techniques.
• R: Use the prcomp, Rtsne, and factanal functions to apply dimensionality reduction techniques.
• Tableau: Use the Dimensional Modeling feature to reduce the dimensionality of data and create visualizations.

Quick Practice Problem

A retail company wants to segment its customers based on their purchasing behavior. The company has 10 features, such as age, income, and purchase frequency. Using PCA, the company reduces the dimensionality to 3 features. What does this mean?

Answer: The company has reduced the dimensionality of the data to 3 features, which captures the most important information about the customers' purchasing behavior.

Last-Minute Cram Sheet

1. PCA: Reduces dimensionality by retaining the top k principal components, which explain the most variance.
2. t-SNE: Reduces dimensionality by preserving the local structure of the data.
3. Factor Analysis: Reduces dimensionality by retaining the common factors, which explain the variance among the original features.
4. Eigenvalue: Measures the amount of variance explained by each principal component.
5. Cumulative Proportion of Variance Explained (CPVE): Measures the proportion of variance explained by the top k principal components.
6. Kullback-Leibler Divergence (KL): Measures the difference between the probability distributions of the original and embedded data.
7. Perplexity (P): Measures the average perplexity of the embedded data.
8. Common Factor (F): Measures the common variance among the original features.
9. Specific Factor (S): Measures the unique variance of each original feature.
10. Factor Loading (a): Measures the correlation between each original feature and the common factor.