Business Analytics 101: Regression Analysis Multiple Linear Regression Model Assumptions Linearity Independence Homoscedasticity Normality Multicollinearity Adjusted R² Ftest

What This Is

Multiple Linear Regression (MLR) is a statistical method used to model the relationship between a dependent variable (target) and one or more independent variables (predictors). It's essential in business analytics for tasks like forecasting sales, segmenting customers, detecting fraud, and understanding the impact of multiple factors on a business outcome. For instance, a retail company might use MLR to analyze the relationship between sales, advertising spend, and weather to predict future sales.

Key Formulas & Metrics

• Multiple Linear Regression Equation: y = β₀ + β₁x₁ + β₂x₂ + … + βₙxₙ + ε – the linear combination of predictors to predict the target variable.
• y: dependent variable (target)
• β₀: intercept or constant term
• β₁, β₂, …, βₙ: coefficients or slopes for each predictor
• x₁, x₂, …, xₙ: independent variables (predictors)
• ε: error term or residual
• Coefficient of Determination (R²): R² = 1 − (SS_res / SS_tot) – proportion of variance explained by the regression model.
• SS_res: sum of squared residuals
• SS_tot: total sum of squares
• Adjusted R²: Adjusted R² = 1 − ((n - 1) / (n - k - 1)) * (1 - R²) – adjusted proportion of variance explained by the regression model.
• n: sample size
• k: number of predictors
• F-statistic: F = (MSR / MSE) – ratio of mean squared regression to mean squared error.
• MSR: mean squared regression
• MSE: mean squared error
• p-value: p-value = P(T ≥ |t|) – probability of observing the data (or more extreme) if the null hypothesis is true.
• T: test statistic
• t: t-statistic
• Mean Absolute Error (MAE): MAE = (1/n) Σ|yᵢ − ŷᵢ| – average absolute forecast error.
• yᵢ: actual value
• ŷᵢ: predicted value
• Root Mean Squared Error (RMSE): RMSE = √((1/n) Σ(yᵢ − ŷᵢ)²) – square root of the average squared forecast error.

Step-by-Step Procedure

1. Formulate the research question: Clearly define the problem and the target variable.
2. Collect and preprocess the data: Gather relevant data, handle missing values, and transform variables as necessary.
3. Select the predictors: Choose the independent variables to include in the model.
4. Build the model: Use a linear regression algorithm to estimate the coefficients and intercept.
5. Evaluate the model: Assess the model's performance using metrics like R², MAE, and RMSE.
6. Interpret the results: Examine the coefficients, p-values, and confidence intervals to understand the relationships between the predictors and the target variable.

Common Mistakes

• Mistake: Confusing correlation with causation.
• Correction: Establish a clear causal relationship between the predictors and the target variable before drawing conclusions.
• Mistake: Misinterpreting p-values.
• Correction: Understand that a low p-value indicates statistical significance, but not necessarily practical significance.
• Mistake: Using the wrong error metric for a business problem.
• Correction: Choose an error metric that aligns with the business objective, such as MAE for forecasting or RMSE for prediction.

Software / Tool Tips

• Python with pandas and scikit-learn: Use the LinearRegression class from scikit-learn to build and evaluate a linear regression model.
• R: Utilize the lm() function to fit a linear model and the summary() function to obtain the model summary.
• Excel: Employ the LINEST() function to perform linear regression and the RSQ() function to calculate R².

Quick Practice Problem

A company wants to predict sales based on advertising spend and weather. Given the following data:

What does an R² of 0.85 mean?

Answer: The model explains 85% of the variance in sales.

Last-Minute Cram Sheet

1. Multiple Linear Regression (MLR) Equation: y = β₀ + β₁x₁ + β₂x₂ + … + βₙxₙ + ε
2. Coefficient of Determination (R²): R² = 1 − (SS_res / SS_tot)
3. Adjusted R²: Adjusted R² = 1 − ((n - 1) / (n - k - 1)) * (1 - R²)
4. F-statistic: F = (MSR / MSE)
5. p-value: p-value = P(T ≥ |t|)
6. Mean Absolute Error (MAE): MAE = (1/n) Σ|yᵢ − ŷᵢ|
7. Root Mean Squared Error (RMSE): RMSE = √((1/n) Σ(yᵢ − ŷᵢ)²)
8. ⚠️ p-value is NOT the probability that H₀ is true – it’s the probability of observing the data (or more extreme) if H₀ is true
9. ⚠️ Correlation does NOT imply causation
10. ⚠️ R² is a measure of goodness of fit, not a measure of predictive power