Intro to Business Statistics: Correlation and Regression Multicollinearity VIF Stepwise Selection

What This Is

Multicollinearity and stepwise selection are statistical techniques used to identify and manage redundant variables in a regression model. A retail chain wants to know if average daily sales exceed $10,000, but they have multiple variables such as store size, location, and advertising budget. If these variables are highly correlated, it can lead to inaccurate predictions and unreliable results. Multicollinearity helps identify these issues, while stepwise selection helps select the most important variables for the model.

Key Formulas & Symbols

• VIF (Variance Inflation Factor) = 1 / (1 - R²) where R² = coefficient of determination.
• R² = (SSR / SST) * 100 where SSR = sum of squares regression, SST = sum of squares total.
• SSR = Σ (ŷ - ȳ)² where ŷ = predicted value, ȳ = mean of y.
• SST = Σ (y - ȳ)² where y = actual value.
• F-statistic = (MSR / MSError) where MSR = mean square regression, MSError = mean square error.
• F-statistic = (R² / (1 - R²)) * ((n - 2) / (n - k - 1)) where n = sample size, k = number of predictors.
• p-value = F-statistic * (1 - F-distribution) + F-distribution where F-distribution is the F-distribution with (k, n - k - 1) degrees of freedom.
• Stepwise selection criteria: F-to-enter = 2.0, F-to-remove = 3.84 (for α = 0.05 and 5% significance level).
• R² change: the change in R² when a variable is added or removed from the model.

Step-by-Step Procedure

1. Check for multicollinearity: calculate VIF for each predictor variable.
2. Determine the threshold: set a threshold for VIF (e.g., 5 or 10).
3. Select variables: choose variables with VIF below the threshold.
4. Perform stepwise selection: use a stepwise selection algorithm to add or remove variables based on the F-statistic and p-value.
5. Evaluate the model: check the R², F-statistic, and p-value to ensure the model is reliable.
6. Interpret results: use the selected variables to make predictions or explain the relationship between the variables.

Common Mistakes

• Mistake: Using VIF without setting a threshold.
• Correction: Set a threshold for VIF (e.g., 5 or 10) to determine which variables to include in the model.
• Mistake: Misinterpreting the F-statistic as a measure of the strength of the relationship between the variables.
• Correction: The F-statistic is a test statistic used to determine the significance of the relationship between the variables.
• Mistake: Failing to check for multicollinearity before performing stepwise selection.
• Correction: Check for multicollinearity before performing stepwise selection to ensure the model is reliable.

Quick Practice Problems

1. A retail chain wants to know if the average daily sales exceed $10,000. They have three predictor variables: store size, location, and advertising budget. The VIF for store size is 5, location is 10, and advertising budget is 3. What is the threshold for VIF?
   Answer: 5 (or 10, depending on the threshold set)
   Explanation: The threshold for VIF is set to determine which variables to include in the model.

2. A company wants to know if the average daily sales exceed $10,000. They have three predictor variables: store size, location, and advertising budget. The F-statistic is 3.2, and the p-value is 0.01. What is the conclusion?
   Answer: Reject the null hypothesis
   Explanation: The p-value is less than α (0.05), so we reject the null hypothesis.

3. A company wants to know if the average daily sales exceed $10,000. They have three predictor variables: store size, location, and advertising budget. The R² change is 0.05 when a variable is added to the model. What is the conclusion?
   Answer: The variable is significant
   Explanation: The R² change is greater than 0.01 (a common threshold), so the variable is significant.

Last-Minute Cram Sheet

• ⚠️ p-value is NOT the probability that H₀ is true – it’s the probability of observing the data (or more extreme) if H₀ is true.
• VIF = 1 / (1 - R²)
• R² = (SSR / SST) * 100
• SSR = Σ (ŷ - ȳ)²
• SST = Σ (y - ȳ)²
• F-statistic = (MSR / MSError)
• F-statistic = (R² / (1 - R²)) * ((n - 2) / (n - k - 1))
• p-value = F-statistic * (1 - F-distribution) + F-distribution
• Stepwise selection criteria: F-to-enter = 2.0, F-to-remove = 3.84 (for α = 0.05 and 5% significance level)
• R² change: the change in R² when a variable is added or removed from the model
• ⚠️ F-statistic is a test statistic, not a measure of the strength of the relationship between the variables.