Business Analytics 101: Machine Learning Fundamentals CrossValidation kfold Leaveoneout

What This Is

Cross-validation is a technique used to evaluate the performance of a machine learning model by training and testing it on different subsets of the data. This is crucial in business analytics as it helps to prevent overfitting, which occurs when a model is too complex and only fits the noise in the training data. For instance, a company wants to predict customer churn using historical data. They build a model using 80% of the data and test it on the remaining 20%. However, this model may not perform well on new, unseen data. Cross-validation helps to ensure that the model generalizes well to new data.

Key Formulas & Metrics

• Mean Absolute Error (MAE) = (1/n) Σ|yᵢ − ŷᵢ|: The average absolute difference between actual and predicted values. Here, yᵢ is the actual value, ŷᵢ is the predicted value, and n is the number of observations.
• R² = 1 − (SS_res / SS_tot): The proportion of variance in the dependent variable explained by the regression model. SS_res is the sum of squared residuals, and SS_tot is the total sum of squares.
• Cross-Validation Error = (1/k) ΣEᵢ: The average error across k folds. Here, Eᵢ is the error on the ith fold.
• k-Fold Cross-Validation: A technique where the data is divided into k subsets or folds. The model is trained on k-1 folds and tested on the remaining fold. This process is repeated k times, and the average error is calculated.
• Leave-One-Out Cross-Validation (LOOCV): A special case of k-fold cross-validation where k is equal to the number of observations. The model is trained on all observations except one, and the error is calculated on the left-out observation.
• Training Error = (1/n) Σ|yᵢ − ŷᵢ|: The average absolute difference between actual and predicted values on the training data.
• Test Error = (1/n) Σ|yᵢ − ŷᵢ|: The average absolute difference between actual and predicted values on the test data.

Step-by-Step Procedure

1. Split the data: Divide the data into k subsets or folds.
2. Train the model: Train the model on k-1 folds.
3. Test the model: Test the model on the remaining fold.
4. Calculate the error: Calculate the error on the test fold.
5. Repeat the process: Repeat steps 2-4 k times.
6. Calculate the average error: Calculate the average error across k folds.

Common Mistakes

• Mistake: Confusing cross-validation with bootstrapping.
• Correction: Cross-validation involves splitting the data into subsets, while bootstrapping involves resampling the data with replacement.
• Mistake: Using the same data for training and testing.
• Correction: This can lead to overfitting and biased estimates of the model's performance.
• Mistake: Not using enough folds in k-fold cross-validation.
• Correction: Using too few folds can lead to overfitting and biased estimates of the model's performance.

Software / Tool Tips

• Python with scikit-learn: Use the KFold class to perform k-fold cross-validation.
• R: Use the cv.glm function to perform cross-validation.
• Excel: Use the SUBTOTAL function to perform cross-validation.

Quick Practice Problem

A company wants to predict customer churn using historical data. They use k-fold cross-validation with k=5. The average error across the 5 folds is 0.12. What does this mean?

Answer: The model's average error is 0.12, which means that on average, the model predicts customer churn 12% of the time.

Last-Minute Cram Sheet

• Cross-validation: A technique to evaluate the performance of a machine learning model by training and testing it on different subsets of the data.
• k-Fold Cross-Validation: A technique where the data is divided into k subsets or folds.
• Leave-One-Out Cross-Validation (LOOCV): A special case of k-fold cross-validation where k is equal to the number of observations.
• Training Error: The average absolute difference between actual and predicted values on the training data.
• Test Error: The average absolute difference between actual and predicted values on the test data.
• Mean Absolute Error (MAE): The average absolute difference between actual and predicted values.
• R²: The proportion of variance in the dependent variable explained by the regression model.
• ⚠️ p-value is NOT the probability that H₀ is true – it’s the probability of observing the data (or more extreme) if H₀ is true.