Business Analytics 101: Regression Analysis Variable Selection Forward Backward Stepwise AIC BIC CrossValidation

What This Is

Variable selection is a crucial step in building predictive models, where we choose the most relevant variables to include in our model. This matters in business analytics because it directly affects the accuracy and interpretability of our models. For example, consider a retail company trying to predict sales using historical data. If we include irrelevant variables, such as the number of rainy days, our model may not accurately capture the underlying drivers of sales.

Key Formulas & Metrics

• R² = 1 − (SS_res / SS_tot) – proportion of variance explained by the regression model.
  • SS_res: sum of squared residuals
  • SS_tot: total sum of squares
• Mean Absolute Error (MAE) = (1/n) Σ|yᵢ − ŷᵢ| – average absolute forecast error.
  • yᵢ: actual value
  • ŷᵢ: predicted value
  • n: number of observations
• Akaike Information Criterion (AIC) = 2k - 2ln(L) – measure of model fit and complexity.
  • k: number of parameters
  • L: likelihood function
• Bayesian Information Criterion (BIC) = kln(n) - 2ln(L) – measure of model fit and complexity.
  • k: number of parameters
  • n: number of observations
• Cross-Validation (CV) = (1/n) Σ(yᵢ - ŷᵢ)^2 – measure of model performance on unseen data.
  • yᵢ: actual value
  • ŷᵢ: predicted value
  • n: number of observations
• Adjusted R² = 1 - (n-1)/(n-k-1) * (1-R²) – adjusted proportion of variance explained by the regression model.
  • n: number of observations
  • k: number of parameters
• F-statistic = (SS_model / k) / (SS_res / (n-k-1)) – test statistic for model significance.
  • SS_model: sum of squares due to regression
  • k: number of parameters
  • n: number of observations

Step-by-Step Procedure

1. Select candidate variables: Choose a set of variables that are likely to be relevant to the problem.
2. Split data: Split the data into training and testing sets.
3. Fit models: Fit a set of models using different combinations of candidate variables.
4. Evaluate models: Evaluate each model using metrics such as R², MAE, AIC, and BIC.
5. Select best model: Choose the model with the best balance of fit and complexity.
6. Refine model: Refine the selected model by removing irrelevant variables and improving its fit.

Common Mistakes

1. Mistake: Confusing correlation with causation.
   • Correction: Correlation does not imply causation. Use techniques such as regression analysis to establish causality.
2. Mistake: Misinterpreting p-values.
   • Correction: A small p-value does not necessarily mean that the null hypothesis is false. Use other metrics such as R² and MAE to evaluate model performance.
3. Mistake: Using the wrong error metric for a business problem.
   • Correction: Choose an error metric that aligns with the business problem, such as MAE for forecasting or classification accuracy for classification problems.

Software / Tool Tips

• Python with pandas/scikit-learn: Use the pandas library to load and manipulate data, and the scikit-learn library to fit and evaluate models.
• R: Use the dplyr library to load and manipulate data, and the caret library to fit and evaluate models.
• Excel: Use the Analysis ToolPak add-in to fit and evaluate models.
• Tableau: Use the Data Analysis tab to fit and evaluate models.

Quick Practice Problem

Problem: Compute the moving average forecast for period 4 given data: 100, 110, 105.
Answer: 108.33 (average of 100, 110, and 105).
Explanation: The moving average forecast is a simple method for forecasting future values based on past values.

Last-Minute Cram Sheet

1. R² measures the proportion of variance explained by the regression model.
2. MAE measures the average absolute forecast error.
3. AIC and BIC measure model fit and complexity.
4. Cross-validation measures model performance on unseen data.
5. Adjusted R² adjusts for the number of parameters in the model.
6. F-statistic tests the significance of the model.
7. Correlation does not imply causation.
8. A small p-value does not necessarily mean that the null hypothesis is false.
9. Choose an error metric that aligns with the business problem.
10. Use techniques such as regression analysis to establish causality.