Business Analytics 101: Data Management and Preparation Data Collection Primary vs Secondary Sampling Web Scraping APIs

What This Is

Data collection is the process of gathering and acquiring data from various sources to support business decision-making. It's essential in business analytics as it enables organizations to make informed decisions, identify trends, and optimize operations. For instance, a retail company might collect data on customer purchases to forecast sales, segment customers, and detect potential fraud.

Key Formulas & Metrics

• Sampling Fraction (SF) = n / N – proportion of population sampled.
• n: sample size
• N: population size

• Interpretation: measures the proportion of the population included in the sample.

• Stratified Sampling Ratio (SSR) = nᵢ / Nᵢ – proportion of population in each stratum sampled.

• nᵢ: sample size in stratum i
• Nᵢ: population size in stratum i

• Interpretation: measures the proportion of each stratum included in the sample.

• Mean Absolute Error (MAE) = (1/n) Σ|yᵢ − ŷᵢ| – average absolute forecast error.

• yᵢ: actual value
• ŷᵢ: forecasted value

• Interpretation: measures the average difference between actual and forecasted values.

• Root Mean Squared Error (RMSE) = √[(1/n) Σ(yᵢ − ŷᵢ)²] – square root of average squared forecast error.

• yᵢ: actual value
• ŷᵢ: forecasted value

• Interpretation: measures the square root of the average squared difference between actual and forecasted values.

• R² = 1 − (SS_res / SS_tot) – proportion of variance explained by the regression model.

• SS_res: residual sum of squares
• SS_tot: total sum of squares

• Interpretation: measures the proportion of variance in the dependent variable explained by the independent variable(s).

• Coefficient of Determination (R²) = 1 − (Σ(yᵢ − ŷᵢ)² / Σ(yᵢ − ȳ)²) – proportion of variance explained by the regression model.

• yᵢ: actual value
• ŷᵢ: forecasted value
• ȳ: mean of actual values

• Interpretation: measures the proportion of variance in the dependent variable explained by the independent variable(s).

• Confidence Interval (CI) = ȳ ± (Z * (σ / √n)) – range of values within which the true population parameter is likely to lie.

• ȳ: sample mean
• Z: Z-score corresponding to desired confidence level
• σ: population standard deviation
• n: sample size
• Interpretation: provides a range of values within which the true population parameter is likely to lie.

Step-by-Step Procedure

1. Define the problem: Clearly articulate the business problem or question you're trying to answer.
2. Determine the data requirements: Identify the type and amount of data needed to answer the problem.
3. Choose a data collection method: Select a primary or secondary data collection method, such as surveys, experiments, or web scraping.
4. Design the data collection instrument: Create a survey, experiment, or web scraping script to collect the required data.
5. Collect the data: Execute the data collection instrument and gather the required data.
6. Clean and preprocess the data: Remove errors, handle missing values, and transform the data into a suitable format for analysis.

Common Mistakes

• Mistake: Confusing correlation with causation.
• Correction: Establish a clear cause-and-effect relationship between variables before drawing conclusions.
• Mistake: Misinterpreting p-values.
• Correction: Understand that p-values indicate the probability of observing the data (or more extreme) if the null hypothesis is true, not the probability that the null hypothesis is true.
• Mistake: Using the wrong error metric for a business problem.
• Correction: Choose an error metric that aligns with the business problem, such as mean absolute error (MAE) for forecasting or root mean squared error (RMSE) for regression.

Software / Tool Tips

• Python with pandas and scikit-learn: Use the pandas library to handle and manipulate data, and the scikit-learn library to implement machine learning algorithms.
• R: Utilize the dplyr and tidyr packages for data manipulation and the caret package for machine learning.
• Excel: Employ the POWERPivot add-in for data modeling and the Solver add-in for optimization.
• Tableau: Use the Data Source feature to connect to various data sources and the Analysis feature to create visualizations and perform data analysis.

Quick Practice Problem

Problem: Compute the moving average forecast for period 4 given data: 100, 110, 105, 115.
Answer: (100 + 110 + 105 + 115) / 4 = 110 Explanation: The moving average forecast is calculated by taking the average of the last four values.

Last-Minute Cram Sheet

1. Primary data: collected directly from the source, e.g., surveys, experiments.
2. Secondary data: collected from existing sources, e.g., databases, literature.
3. Sampling: selecting a subset of the population to represent the whole.
4. Stratified sampling: dividing the population into subgroups and sampling each subgroup.
5. Mean Absolute Error (MAE): average absolute difference between actual and forecasted values.
6. Root Mean Squared Error (RMSE): square root of average squared difference between actual and forecasted values.
7. R²: proportion of variance explained by the regression model.
8. Confidence Interval (CI): range of values within which the true population parameter is likely to lie.
9. ⚠️ p-value is NOT the probability that H₀ is true – it’s the probability of observing the data (or more extreme) if H₀ is true.
10. ⚠️ Correlation does NOT imply causation.