Business Analytics 101: Regression Analysis Simple Linear Regression Model Y β0 β1X ε Least Squares R² Interpretation

What This Is

Simple Linear Regression (SLR) is a statistical method used to model the relationship between a dependent variable (Y) and one independent variable (X). It's a fundamental technique in business analytics, helping organizations understand how changes in one variable affect another. For instance, a retail company might use SLR to forecast sales based on advertising spend, or a bank to detect credit card fraud based on transaction patterns.

Key Formulas & Metrics

• Y = β0 + β1X + ε – the linear regression equation, where Y is the dependent variable, X is the independent variable, β0 is the intercept, β1 is the slope, and ε is the error term.
• β1 = Σ[(xi - x̄)(yi - ȳ)] / Σ(xi - x̄)² – the slope coefficient, calculated as the covariance between X and Y divided by the variance of X.
• β0 = ȳ - β1x̄ – the intercept, calculated as the mean of Y minus the product of the slope and the mean of X.
• R² = 1 - (SS_res / SS_tot) – the coefficient of determination, measuring the proportion of variance in Y explained by X.
• MAE = (1/n) Σ|yᵢ - ŷᵢ| – the mean absolute error, calculated as the average absolute difference between observed and predicted values.
• MSE = (1/n) Σ(yᵢ - ŷᵢ)² – the mean squared error, calculated as the average squared difference between observed and predicted values.
• p-value = P(T > |t|) – the probability of observing a test statistic at least as extreme as the one observed, assuming the null hypothesis is true.
• t-statistic = (β1 - β10) / (sβ1 / √n) – the test statistic for the slope coefficient, calculated as the difference between the estimated and hypothesized slope divided by the standard error.

Step-by-Step Procedure

1. Formulate the research question: Identify the dependent and independent variables, and the research question to be answered.
2. Collect and preprocess data: Gather data on the variables of interest, and perform any necessary data cleaning and transformation.
3. Check assumptions: Verify that the data meet the assumptions of linear regression, including linearity, independence, homoscedasticity, normality, and no multicollinearity.
4. Estimate the model: Use a suitable estimation method (e.g., ordinary least squares) to estimate the model parameters.
5. Evaluate the model: Assess the model's goodness of fit, using metrics such as R², MAE, and MSE.
6. Interpret the results: Examine the estimated coefficients, their standard errors, and the p-values to draw conclusions about the relationship between the variables.

Common Mistakes

• Mistake: Confusing correlation with causation.
• Correction: Correlation does not imply causation; a third variable may be driving the relationship between the variables.
• Mistake: Misinterpreting p-values.
• Correction: A small p-value indicates that the observed effect is statistically significant, but not necessarily practically significant.
• Mistake: Using the wrong error metric for a business problem.
• Correction: Choose an error metric that aligns with the business objective, such as MAE for forecasting or MSE for prediction.

Software / Tool Tips

• Python with pandas and scikit-learn: Use the LinearRegression class from scikit-learn to estimate the model, and the mean_absolute_error function to evaluate the model's performance.
• R: Use the lm function to estimate the model, and the summary function to evaluate the model's performance.
• Excel: Use the LINEST function to estimate the model, and the AVERAGEIF function to evaluate the model's performance.

Quick Practice Problem

A company wants to forecast sales based on advertising spend. The data shows a strong positive relationship between the two variables. What does an R² of 0.85 mean?

Answer: An R² of 0.85 means that 85% of the variance in sales is explained by the advertising spend.

Last-Minute Cram Sheet

1. Y = β0 + β1X + ε: the linear regression equation.
2. β1 = Σ[(xi - x̄)(yi - ȳ)] / Σ(xi - x̄)²: the slope coefficient.
3. β0 = ȳ - β1x̄: the intercept.
4. R² = 1 - (SS_res / SS_tot): the coefficient of determination.
5. MAE = (1/n) Σ|yᵢ - ŷᵢ|: the mean absolute error.
6. MSE = (1/n) Σ(yᵢ - ŷᵢ)²: the mean squared error.
7. p-value = P(T > |t|): the probability of observing a test statistic at least as extreme as the one observed.
8. t-statistic = (β1 - β10) / (sβ1 / √n): the test statistic for the slope coefficient.
9. ⚠️ p-value is NOT the probability that H₀ is true – it’s the probability of observing the data (or more extreme) if H₀ is true.
10. ⚠️ Correlation does not imply causation.