Business Analytics 101: Exploratory Data Analysis Correlation Analysis Pearson Spearman Covariance Scatterplot Matrices

What This Is

Correlation analysis is a statistical technique used to measure the strength and direction of the linear relationship between two continuous variables. It's essential in business analytics as it helps identify relationships between variables, detect patterns, and make predictions. For instance, a retail company might use correlation analysis to understand the relationship between sales and advertising expenses to optimize their marketing budget.

Key Formulas & Metrics

• Pearson Correlation Coefficient (ρ) = Σ[(xi - x̄)(yi - ȳ)] / (√[Σ(xi - x̄)²] * √[Σ(yi - ȳ)²]) – measures the strength and direction of the linear relationship between two continuous variables.
• Spearman Rank Correlation Coefficient (ρs) = 1 - (6 * Σ(d²)) / (n² - 1) – measures the strength and direction of the monotonic relationship between two continuous variables.
• Covariance (σxy) = Σ[(xi - x̄)(yi - ȳ)] / (n - 1) – measures the average product of deviations from the mean for two continuous variables.
• Scatterplot Matrix – a visual representation of the correlation between multiple pairs of variables.
• Coefficient of Determination (R²) = 1 - (SS_res / SS_tot) – measures the proportion of variance explained by a regression model.
• Coefficient of Variation (CV) = σ / μ – measures the relative variability of a dataset.
• Standard Deviation (σ) = √[Σ(xi - x̄)² / (n - 1)] – measures the spread of a dataset.
• Mean Absolute Deviation (MAD) = (1/n) Σ|xi - x̄| – measures the average absolute deviation from the mean.
• Interquartile Range (IQR) = Q3 - Q1 – measures the spread of a dataset, excluding outliers.

Step-by-Step Procedure

1. Choose the correlation coefficient: Select the appropriate correlation coefficient based on the type of data (Pearson for continuous data, Spearman for ordinal or ranked data).
2. Check assumptions: Verify that the data meets the assumptions of the chosen correlation coefficient (e.g., normality, linearity).
3. Compute the correlation coefficient: Use the formula to calculate the correlation coefficient.
4. Interpret the results: Determine the strength and direction of the relationship between the variables.
5. Visualize the data: Create a scatterplot to visualize the relationship between the variables.
6. Consider multiple correlation coefficients: Compute multiple correlation coefficients to compare the relationships between multiple pairs of variables.

Common Mistakes

• Mistake: Confusing correlation with causation.
• Correction: Correlation does not imply causation; a third variable may be influencing the relationship.
• Mistake: Misinterpreting p-values.
• Correction: A small p-value indicates that the observed relationship is unlikely to occur by chance, but it does not necessarily imply causation.
• Mistake: Using the wrong error metric for a business problem.
• Correction: Choose an error metric that aligns with the business objective (e.g., mean absolute error for forecasting).

Software / Tool Tips

• Python with pandas and scikit-learn: Use the corr() function to compute the correlation coefficient and scatter_matrix() to create a scatterplot matrix.
• R: Use the cor() function to compute the correlation coefficient and pairs() to create a scatterplot matrix.
• Excel: Use the CORREL() function to compute the correlation coefficient and Scatter Plot to create a scatterplot.

Quick Practice Problem

A company wants to understand the relationship between sales and advertising expenses. The data shows a correlation coefficient of 0.8. What does this mean?

Answer: The correlation coefficient of 0.8 indicates a strong positive linear relationship between sales and advertising expenses.

Last-Minute Cram Sheet

• Pearson correlation coefficient measures the strength and direction of the linear relationship between two continuous variables.
• Spearman rank correlation coefficient measures the strength and direction of the monotonic relationship between two continuous variables.
• Covariance measures the average product of deviations from the mean for two continuous variables.
• Coefficient of determination (R²) measures the proportion of variance explained by a regression model.
• Coefficient of variation (CV) measures the relative variability of a dataset.
• Standard deviation (σ) measures the spread of a dataset.
• Mean absolute deviation (MAD) measures the average absolute deviation from the mean.
• Interquartile range (IQR) measures the spread of a dataset, excluding outliers.
• ⚠️ p-value is NOT the probability that H₀ is true – it’s the probability of observing the data (or more extreme) if H₀ is true.
• ⚠️ Correlation does not imply causation.