Business Analytics 101: Machine Learning Fundamentals Overfitting vs Underfitting BiasVariance Tradeoff

What This Is

Overfitting vs Underfitting (Bias-Variance Tradeoff) is a fundamental concept in business analytics that deals with the tension between model complexity and accuracy. In simple terms, it's about finding the right balance between fitting the noise in the data and capturing the underlying patterns. Imagine you're trying to predict sales for a new product launch. If your model is too simple, it might not capture the nuances of the market, leading to underfitting. On the other hand, if your model is too complex, it might fit the noise in the data, leading to overfitting. The goal is to find the sweet spot where your model generalizes well to new, unseen data.

Key Formulas & Metrics

• Mean Absolute Error (MAE) = (1/n) Σ|yᵢ − ŷᵢ|: average absolute forecast error, where yᵢ is the actual value and ŷᵢ is the predicted value.
• R² = 1 − (SS_res / SS_tot): proportion of variance explained by the regression model, where SS_res is the sum of squared residuals and SS_tot is the total sum of squares.
• Mean Squared Error (MSE) = (1/n) Σ(yᵢ − ŷᵢ)²: average squared forecast error.
• Root Mean Squared Error (RMSE) = √(MSE): square root of the average squared forecast error.
• Akaike Information Criterion (AIC) = n * ln(MSE) + 2 * k: measure of model fit, where n is the sample size, MSE is the mean squared error, and k is the number of parameters.
• Bayesian Information Criterion (BIC) = n * ln(MSE) + k * ln(n): measure of model fit, where n is the sample size, MSE is the mean squared error, and k is the number of parameters.
• Cross-Validation Error = (1/n) Σ(yᵢ − ŷᵢ)²: average squared forecast error, where yᵢ is the actual value and ŷᵢ is the predicted value from the cross-validation model.
• Coefficient of Determination (R²) = 1 − (SS_res / SS_tot): proportion of variance explained by the regression model.
• Adjusted R² = 1 − ((n-1)/(n-k-1)) * (1-R²): adjusted proportion of variance explained by the regression model.

Step-by-Step Procedure

1. Split your data: Divide your data into training and testing sets (e.g., 80% for training and 20% for testing).
2. Choose a model: Select a model that is suitable for your problem (e.g., linear regression, decision trees, random forests).
3. Train the model: Train the model on the training data using various metrics (e.g., MAE, MSE, R²).
4. Evaluate the model: Evaluate the model on the testing data using various metrics (e.g., MAE, MSE, R²).
5. Compare models: Compare the performance of different models using metrics such as AIC, BIC, and cross-validation error.
6. Tune hyperparameters: Tune the hyperparameters of the model to improve its performance.

Common Mistakes

• Mistake: Confusing correlation with causation.
  • Correction: Correlation does not imply causation. Make sure to establish a causal relationship between variables before drawing conclusions.
• Mistake: Misinterpreting p-values.
  • Correction: A p-value is the probability of observing the data (or more extreme) if the null hypothesis is true. It does not indicate the probability that the null hypothesis is true.
• Mistake: Using the wrong error metric for a business problem.
  • Correction: Choose an error metric that is relevant to the business problem (e.g., MAE for forecasting, R² for regression).

Software / Tool Tips

• Python with pandas/scikit-learn: Use the train_test_split function to split your data, and the LinearRegression class to train a linear regression model.
• R: Use the train function to split your data, and the lm function to train a linear regression model.
• Excel: Use the Data Analysis tool to split your data, and the Regression tool to train a linear regression model.

Quick Practice Problem

Problem: Compute the moving average forecast for period 4 given data: 100, 110, 105, 108.

Answer: 107.5

Explanation: The moving average forecast is the average of the last few values in the data. In this case, the moving average forecast for period 4 is the average of the last 3 values: (110 + 105 + 108) / 3 = 107.5.

Last-Minute Cram Sheet

• Overfitting: when a model is too complex and fits the noise in the data.
• Underfitting: when a model is too simple and fails to capture the underlying patterns in the data.
• Bias-Variance Tradeoff: the tension between model complexity and accuracy.
• MAE: average absolute forecast error.
• MSE: average squared forecast error.
• R²: proportion of variance explained by the regression model.
• AIC: measure of model fit.
• BIC: measure of model fit.
• Cross-Validation Error: average squared forecast error.
• Coefficient of Determination (R²): proportion of variance explained by the regression model.
• Adjusted R²: adjusted proportion of variance explained by the regression model.
• p-value: probability of observing the data (or more extreme) if the null hypothesis is true.
• Hyperparameters: parameters that are set before training a model.
• Cross-validation: a technique for evaluating a model's performance on unseen data.