Business Analytics 101: Forecasting Time Series Forecast Accuracy MAE MSE RMSE MAPE MASE

What This Is

Forecast accuracy is a crucial concept in business analytics, as it helps organizations make informed decisions by evaluating the reliability of their predictions. Imagine a retail company that wants to forecast sales for the upcoming holiday season. By accurately predicting sales, the company can optimize inventory levels, manage supply chains, and make strategic decisions about pricing and marketing. In this study guide, we'll explore key metrics for evaluating forecast accuracy, including Mean Absolute Error (MAE), Mean Squared Error (MSE), Root Mean Squared Error (RMSE), Mean Absolute Percentage Error (MAPE), and Mean Absolute Scaled Error (MASE).

Key Formulas & Metrics

• Mean Absolute Error (MAE) = (1/n) Σ|yᵢ − ŷᵢ|: Average absolute forecast error, where yᵢ is the actual value and ŷᵢ is the forecasted value.
• Mean Squared Error (MSE) = (1/n) Σ(yᵢ − ŷᵢ)²: Average squared forecast error, where yᵢ is the actual value and ŷᵢ is the forecasted value.
• Root Mean Squared Error (RMSE) = √(MSE): Square root of the average squared forecast error, providing a measure of the forecast error in the same units as the data.
• Mean Absolute Percentage Error (MAPE) = (1/n) Σ|yᵢ − ŷᵢ| / yᵢ × 100: Average absolute percentage forecast error, where yᵢ is the actual value and ŷᵢ is the forecasted value.
• Mean Absolute Scaled Error (MASE) = (1/n) Σ|yᵢ − ŷᵢ| / (1/n) Σ|yᵢ − yᵢ-1|: Average absolute forecast error relative to the average absolute error of a naive forecast (e.g., last period's value).

Step-by-Step Procedure

1. Collect historical data: Gather relevant data for the time series you want to forecast, including actual values and any relevant external factors.
2. Split data into training and testing sets: Divide the data into two sets: one for training the model (e.g., 80% of the data) and another for testing the model (e.g., 20% of the data).
3. Choose a forecasting model: Select a suitable model based on the data characteristics and business problem, such as ARIMA, exponential smoothing, or machine learning algorithms.
4. Train and evaluate the model: Train the model on the training data and evaluate its performance on the testing data using the chosen metrics (e.g., MAE, MSE, RMSE).
5. Interpret results and refine the model: Analyze the results to understand the strengths and weaknesses of the model and refine it as needed to improve forecast accuracy.

Common Mistakes

• Mistake: Confusing correlation with causation.
• Correction: Establish a clear causal relationship between variables before drawing conclusions.
• Mistake: Misinterpreting p-values.
• Correction: Understand that p-values indicate the probability of observing the data (or more extreme) if the null hypothesis is true, not the probability that the null hypothesis is true.
• Mistake: Using the wrong error metric for a business problem.
• Correction: Choose the most relevant metric based on the business context and data characteristics (e.g., MAPE for percentage-based data).

Software / Tool Tips

• Python with pandas and scikit-learn: Use the mean_absolute_error function from scikit-learn to compute MAE, and the mean_squared_error function to compute MSE and RMSE.
• R: Use the MAE function from the Metrics package to compute MAE, and the RMSE function to compute RMSE.
• Excel: Use the AVERAGEIF function to compute MAE, and the POWER function to compute RMSE.

Quick Practice Problem

Problem: Compute the MAE for a forecasted value of 120 and an actual value of 110.

Answer: 10

Explanation: MAE = |120 - 110| = 10

Last-Minute Cram Sheet

• MAE = (1/n) Σ|yᵢ − ŷᵢ|: Average absolute forecast error.
• MSE = (1/n) Σ(yᵢ − ŷᵢ)²: Average squared forecast error.
• RMSE = √(MSE): Square root of the average squared forecast error.
• MAPE = (1/n) Σ|yᵢ − ŷᵢ| / yᵢ × 100: Average absolute percentage forecast error.
• MASE = (1/n) Σ|yᵢ − ŷᵢ| / (1/n) Σ|yᵢ − yᵢ-1|: Average absolute forecast error relative to the average absolute error of a naive forecast.
• ⚠️ p-value is NOT the probability that H₀ is true – it’s the probability of observing the data (or more extreme) if H₀ is true.