Introductory Statistics: Regression Correlation Regression Assumptions Linearity Independence Normal Residuals Equal Variance

What Is This?

Regression Assumptions are the conditions that must be met for a regression model to be valid and reliable. These assumptions include linearity, independence, normal residuals, and equal variance. This topic appears in exams to test your understanding of the theoretical underpinnings of regression analysis and your ability to apply these assumptions in practical scenarios.

Why It Matters

Regression assumptions are tested in statistics exams, data science certifications, and job interviews for roles like data analyst, data scientist, and statistician. They appear frequently and can carry significant marks, typically 10-20% of the total score. This topic tests your ability to critically evaluate regression models and ensure their reliability.

Core Concepts

1. Linearity: The relationship between the independent variables (predictors) and the dependent variable (response) should be linear.
2. Independence: The residuals (errors) should be independent of each other.
3. Normal Residuals: The residuals should be normally distributed.
4. Equal Variance (Homoscedasticity): The variance of the residuals should be constant across all levels of the independent variables.

Prerequisites

1. Basic Statistics: Understanding of mean, variance, and standard deviation.
2. Regression Basics: Knowledge of simple and multiple linear regression.
3. Probability Distributions: Familiarity with normal distribution.

The Rule-Book (How It Works)

Primary Rule

For a regression model to be valid, it must satisfy the following assumptions: 1. Linearity: The relationship between predictors and the response variable is linear.
2. Independence: Residuals are independent.
3. Normal Residuals: Residuals are normally distributed.
4. Equal Variance: Residuals have constant variance.

Sub-rules and Edge Cases

• Non-linearity: If the relationship is not linear, transformations or polynomial terms may be needed.
• Autocorrelation: If residuals are not independent (e.g., time series data), special techniques like autoregressive models are required.
• Heteroscedasticity: If residuals have non-constant variance, weighted least squares or robust standard errors may be used.

Visual Pattern

Imagine a scatter plot of residuals vs. fitted values: - Linearity: Points should be randomly scattered.
- Independence: No pattern or structure.
- Normal Residuals: Histogram of residuals should resemble a bell curve.
- Equal Variance: Points should form a horizontal band.

Exam / Job / Audit Weighting

• Frequency: High
• Difficulty Rating: Intermediate
• Question Type: Multiple choice, short answer, data interpretation

Difficulty Level

Intermediate

Must-Know Rules, Formulas, Standards, or Principles

1. Linearity: Check scatter plots of residuals vs. fitted values.
2. Independence: Use Durbin-Watson test for autocorrelation.
3. Normal Residuals: Use Q-Q plots or Shapiro-Wilk test.
4. Equal Variance: Use Breusch-Pagan test.

Worked Examples (Step-by-Step)

Easy

Question: You are given a regression model with the following residual plot. Identify the assumption that is likely violated.

Reasoning: 1. Observe the residual plot.
2. Notice the funnel shape indicating increasing variance.
3. Conclude that the assumption of equal variance is violated.

Answer: Equal Variance

Medium

Question: Given the residuals from a regression model, you perform a Shapiro-Wilk test and get a p-value of 0.02. What can you conclude?

Reasoning: 1. Recall that a p-value less than 0.05 indicates a significant result.
2. The Shapiro-Wilk test checks for normality.
3. Conclude that the residuals are not normally distributed.

Answer: Residuals are not normally distributed.

Hard

Question: You are analyzing time series data and notice that the residuals from your regression model show a pattern. What assumption is violated, and what technique can you use to address it?

Reasoning: 1. Identify the pattern in residuals indicating autocorrelation.
2. Recall that residuals should be independent.
3. Use an autoregressive model to account for the dependence.

Answer: Independence; use an autoregressive model.

Common Exam Traps & Mistakes

1. Mistake: Assuming linearity without checking.
2. Wrong Answer: The model is linear.

3. Correct Approach: Check scatter plots of residuals vs. fitted values.

4. Mistake: Ignoring autocorrelation in time series data.

5. Wrong Answer: Residuals are independent.

6. Correct Approach: Use Durbin-Watson test.

7. Mistake: Assuming normal residuals without testing.

8. Wrong Answer: Residuals are normally distributed.

9. Correct Approach: Use Q-Q plots or Shapiro-Wilk test.

10. Mistake: Overlooking heteroscedasticity.

11. Wrong Answer: Residuals have constant variance.
12. Correct Approach: Use Breusch-Pagan test.

Shortcut Strategies & Exam Hacks

• Memory Aid: Remember "LINER" (Linearity, Independence, Normal Residuals, Equal Variance).
• Elimination Strategy: If a residual plot shows a pattern, eliminate options suggesting independence and equal variance.
• Pattern Recognition: Funnel shape in residual plots indicates heteroscedasticity.

Question-Type Taxonomy

1. Multiple Choice: Identify the violated assumption from a residual plot.
2. Example: Which assumption is violated in the given residual plot?

3. Favored by: Statistics exams, data science certifications.

4. Short Answer: Explain the impact of a violated assumption.

5. Example: What happens if the assumption of normal residuals is violated?

6. Favored by: Job interviews, practical exams.

7. Data Interpretation: Analyze given data and identify assumptions.

8. Example: Given the residuals, which assumption is likely violated?
9. Favored by: Advanced statistics exams, data science roles.

Practice Set (MCQs)

Question 1

Question: Which assumption is violated if the residuals form a funnel shape in a residual plot?

Options: A. Linearity B. Independence C. Normal Residuals D. Equal Variance

Correct Answer: D. Equal Variance

Explanation: A funnel shape indicates increasing variance, violating the assumption of equal variance.

Why the Distractors Are Tempting: - A: Linearity issues typically show a curve, not a funnel.
- B: Independence issues show patterns or structures.
- C: Normal residuals issues show non-normal distributions in Q-Q plots.

Question 2

Question: You perform a Durbin-Watson test and get a statistic of 0.5. What can you conclude?

Options: A. Residuals are independent.
B. Residuals are not independent.
C. Residuals are normally distributed.
D. Residuals have constant variance.

Correct Answer: B. Residuals are not independent.

Explanation: A Durbin-Watson statistic significantly less than 2 indicates autocorrelation, violating the independence assumption.

Why the Distractors Are Tempting: - A: Independence would be indicated by a statistic close to 2.
- C: Normality is checked by Shapiro-Wilk test.
- D: Constant variance is checked by Breusch-Pagan test.

Question 3

Question: Which test is used to check for normal residuals?

Options: A. Durbin-Watson test B. Shapiro-Wilk test C. Breusch-Pagan test D. t-test

Correct Answer: B. Shapiro-Wilk test

Explanation: The Shapiro-Wilk test is specifically used to check for the normality of residuals.

Why the Distractors Are Tempting: - A: Durbin-Watson test checks for autocorrelation.
- C: Breusch-Pagan test checks for heteroscedasticity.
- D: t-test checks for differences in means.

Question 4

Question: If the residuals are not normally distributed, which assumption is violated?

Options: A. Linearity B. Independence C. Normal Residuals D. Equal Variance

Correct Answer: C. Normal Residuals

Explanation: Non-normal residuals violate the assumption of normal residuals.

Why the Distractors Are Tempting: - A: Linearity issues show non-linear patterns.
- B: Independence issues show autocorrelation.
- D: Equal variance issues show heteroscedasticity.

Question 5

Question: Which assumption is violated if the residuals show a pattern over time?

Options: A. Linearity B. Independence C. Normal Residuals D. Equal Variance

Correct Answer: B. Independence

Explanation: A pattern over time indicates autocorrelation, violating the independence assumption.

Why the Distractors Are Tempting: - A: Linearity issues show non-linear relationships.
- C: Normal residuals issues show non-normal distributions.
- D: Equal variance issues show heteroscedasticity.

30-Second Cheat Sheet

• Linearity: Check scatter plots of residuals vs. fitted values.
• Independence: Use Durbin-Watson test; no pattern in residuals.
• Normal Residuals: Use Shapiro-Wilk test; Q-Q plots should be linear.
• Equal Variance: Use Breusch-Pagan test; residual plot should be a horizontal band.
• Memory Aid: "LINER" (Linearity, Independence, Normal Residuals, Equal Variance).

Learning Path

1. Beginner Foundation: Review basic statistics and regression concepts.
2. Core Rules: Understand the four regression assumptions.
3. Practice: Solve practice problems and interpret residual plots.
4. Timed Drills: Practice identifying assumptions under time constraints.
5. Mock Tests: Take full-length mock exams to simulate test conditions.

Related Topics

1. Multiple Linear Regression: Understanding how multiple predictors affect the response variable.
2. Model Diagnostics: Techniques for evaluating the performance and validity of regression models.
3. Transformations: Methods for transforming data to meet regression assumptions.