CAIA Level II: Methods and Models — Multivariate Empirical Methods and Performance Persistence

CAIA Level II: Methods and Models — Multivariate Empirical Methods and Performance Persistence

What Is It?

1. What is this topic?
   Statistical techniques (e.g., regression, factor models) used to analyze multiple variables simultaneously, assessing hedge fund performance persistence and risk-adjusted returns.
2. How is it tested, applied, or used?
   Tested via calculations, interpretation of regression outputs, and real-world applications in due diligence, portfolio construction, and risk management.

Why Does the Exam Ask This?

CAIA tests this to evaluate your ability to: - Interpret multivariate models (e.g., Fama-French, Carhart) for performance attribution. - Assess persistence in hedge fund returns (alpha decay, skill vs. luck). - Apply empirical methods to due diligence, manager selection, and risk decomposition. - Critique model limitations (survivorship bias, data mining, structural breaks).

What Do I Need to Know First?

1. Linear regression (OLS, R², t-stats, multicollinearity).
2. Factor models (CAPM, Fama-French 3/5-factor, Carhart 4-factor).
3. Performance metrics (Sharpe ratio, Sortino ratio, alpha, beta).
4. Hypothesis testing (p-values, confidence intervals, Type I/II errors).
5. Survivorship bias and its impact on empirical studies.

Topic Snapshot

This topic bridges quantitative methods and alternative investments, focusing on how multivariate models explain hedge fund returns and persistence. It’s critical for: - Manager selection (identifying skilled vs. lucky funds). - Risk decomposition (separating alpha from factor exposures). - Regulatory/compliance (validating performance claims in marketing materials).

Exam / Job / Audit Weighting

• Frequency: High (appears in ~15-20% of Level II questions).
• Difficulty Rating: Advanced (requires synthesis of stats, finance, and judgment).
• Question Type:
• Exam: Calculation-based (regression outputs, factor loadings), interpretation (persistence tests), and scenario analysis (due diligence cases).
• Real-World: Due diligence reports, risk attribution, compliance audits.

Difficulty Level

Advanced

Must-Know Rules, Formulas, Standards, or Principles

1. Carhart 4-Factor Model (Regression Equation)
   [
   R_{i,t} - R_{f,t} = \alpha_i + \beta_{i,M}(R_{M,t} - R_{f,t}) + \beta_{i,SMB}SMB_t + \beta_{i,HML}HML_t + \beta_{i,MOM}MOM_t + \epsilon_{i,t}
   ]

2. Key: Alpha (α) measures skill; persistence is tested via rolling-window regressions.

3. Persistence Tests (Spearman Rank Correlation)

4. Rule: Rank funds by past performance, then correlate ranks across periods.

5. Threshold: |ρ| > 0.3 suggests persistence; < 0.1 suggests luck.

6. Survivorship Bias Adjustment

7. Principle: Exclude defunct funds from datasets to avoid overestimating returns.
8. Standard: Use TASS, HFR, or BarclayHedge databases (include graveyard funds).

Misconceptions

1. "High R² means the model is good."
2. Reality: R² measures fit, not predictive power or economic significance.
3. "Alpha persistence = manager skill."
4. Reality: May reflect illiquidity premiums, stale pricing, or structural factors.
5. "Factor models capture all risks."
6. Reality: Omit tail risk, operational risk, and dynamic strategies (e.g., CTAs).
7. "Regression coefficients are stable over time."
8. Reality: Factor loadings shift with market regimes (e.g., β spikes in crises).
9. "All hedge funds exhibit persistence."
10. Reality: Only certain strategies (e.g., equity market-neutral) show weak persistence.

Common Mistakes

1. Ignoring multicollinearity → Inflated t-stats, misleading factor significance.
2. Using raw returns instead of excess returns → Biases alpha estimates.
3. Overlooking survivorship bias → Overstates performance by 2-4% annually.
4. Misinterpreting p-values → Confusing statistical significance with economic relevance.
5. Assuming linear relationships → Hedge funds often have nonlinear exposures (e.g., options).

The Common Trap

Confusing statistical significance with economic significance. - Trap: A factor with p < 0.05 may explain <1% of returns (e.g., a tiny HML loading). - Fix: Check economic magnitude (e.g., a 0.1 beta to SMB adds ~0.5% annual return).

Terms to Remember

1. Alpha decay – Erosion of outperformance over time (e.g., due to capacity constraints).
2. Factor mimicking portfolio – Synthetic portfolio replicating a factor (e.g., SMB = small-cap minus large-cap).
3. Rolling-window regression – Repeated regressions over sub-periods to test persistence.
4. Data mining – Overfitting models to historical data (Type I error inflation).
5. Structural break – Sudden change in factor relationships (e.g., 2008 crisis).

Step-by-Step Process

1. Select the Model

• For equity-like funds: Carhart 4-factor (Mkt, SMB, HML, MOM).
• For macro/CTAs: Fung-Hsieh 7-factor (trend, volatility, credit, etc.).
• For private equity: Public market equivalent (PME) or IRR decomposition.

2. Clean the Data

• Step 1: Merge live + graveyard funds (avoid survivorship bias).
• Step 2: Winsorize outliers (e.g., cap returns at 99th percentile).
• Step 3: Convert to excess returns (subtract risk-free rate).

3. Run the Regression

• Step 1: Estimate coefficients (α, βs) via OLS.
• Step 2: Check diagnostics:
• Multicollinearity: VIF < 5 (for each factor).
• Heteroskedasticity: White test (p > 0.05 = no issue).
• Autocorrelation: Durbin-Watson ~2 (no serial correlation).

4. Test for Persistence

• Step 1: Rank funds by past α (e.g., Year 1).
• Step 2: Correlate ranks with future α (e.g., Year 2).
• Step 3: Interpret:
• ρ > 0.3: Weak persistence.
• ρ < 0.1: No persistence (luck-driven).

5. Validate Economic Significance

• Step 1: Calculate annualized α (e.g., 0.2% monthly α → 2.4% annual).
• Step 2: Compare to fees (e.g., 2% management + 20% performance).
• Step 3: Stress-test (e.g., 2008 crisis: does α hold?).

6. Document Limitations

• Step 1: Note biases (survivorship, backfill, stale pricing).
• Step 2: Acknowledge omitted risks (liquidity, tail events).
• Step 3: Disclose model assumptions (linearity, stationarity).

Exam Answer Builder

1-Mark Question (Single-Best-Answer MCQ)

What it tests: Recall of persistence metrics. Example: "Which test is most commonly used to assess performance persistence in hedge funds?" Options: A) Sharpe ratio B) Spearman rank correlation C) Sortino ratio D) Maximum drawdown Correct Answer: B Key Tip: Persistence = rank correlation; Sharpe/Sortino measure risk-adjusted returns, not persistence.

3-Mark Question (Calculation + Interpretation)

What it tests: Regression output interpretation. Example: "A Carhart 4-factor regression yields the following for Fund X over 5 years: - α = 0.3% (p = 0.04) - β_Mkt = 0.8 (p = 0.00) - β_SMB = 0.2 (p = 0.12) - R² = 0.75

1. Is the fund’s alpha statistically significant? 2. What does the SMB loading imply? 3. Is the model a good fit? Justify."

Key Tips: 1. Statistical significance: p < 0.05 → yes (but check economic magnitude: 0.3% monthly = 3.6% annual). 2. SMB loading: Positive = small-cap tilt (but p = 0.12 → not statistically significant). 3. Model fit: R² = 0.75 is high, but check for overfitting (e.g., too many factors for the sample size).

5-Mark Question (Scenario-Based)

What it tests: Application to due diligence. Example: "You are evaluating a hedge fund with the following Carhart regression results (3-year rolling windows): - Year 1: α = 0.5% (p = 0.01), R² = 0.80 - Year 2: α = 0.2% (p = 0.20), R² = 0.75 - Year 3: α = -0.1% (p = 0.60), R² = 0.60

1. What does the trend in alpha suggest? 2. How would you adjust your due diligence process? 3. What risks might this fund face?"

Key Tips: 1. Alpha trend: Decaying → skill erosion or capacity constraints. 2. Due diligence: Dig into:
- Capacity: AUM growth vs. strategy liquidity.
- Style drift: Check factor loadings over time.
- Survivorship bias: Confirm dataset includes dead funds. 3. Risks: Liquidity mismatch, crowding, or structural breaks (e.g., post-2008 factor shifts).

This vs That

Time-Saver Hack

Eliminate "noise factors" quickly: - If a factor’s p-value > 0.1, drop it (unless economically justified). - If VIF > 5, combine or remove correlated factors (e.g., merge SMB and HML if both are high).

Mini Scenarios

1. Basic Scenario

"A fund’s Carhart regression shows α = 0.4% (p = 0.03) and β_MOM = 0.5 (p = 0.00)." What to notice: - Alpha: Statistically significant but small (0.4% monthly = 4.8% annual). - MOM loading: High → fund likely follows a momentum strategy (check for crowding risk).

2. Applied Scenario

"A due diligence report claims Fund Y has ‘persistent alpha’ based on a 10-year Spearman ρ = 0.4. However, the dataset excludes funds that closed before 2015." What to notice: - Survivorship bias: Excluding dead funds inflates persistence estimates. - Action: Request graveyard data or discount the claim.

3. Tricky Scenario

"Fund Z’s regression shows α = 0.6% (p = 0.01) in 2018-2020 but α = -0.2% (p = 0.30) in 2021-2023. The manager argues this is due to ‘market regime shifts.’" What to notice: - Structural break: Factor relationships changed (e.g., MOM crashed post-2020). - Due diligence: Test for: - Style drift (e.g., new strategy). - Capacity issues (e.g., AUM growth). - Data mining (e.g., cherry-picking periods).

Diagnostic MCQ Bank

Easy

Question: "Which of the following is a key limitation of using the Carhart 4-factor model for hedge funds?" Options: A) It ignores market risk. B) It cannot measure alpha. C) It assumes linear factor exposures. D) It requires daily returns. Correct Answer: C Explanation: - Why right: Hedge funds often have nonlinear exposures (e.g., options, dynamic strategies). - Trap option: A (Carhart includes market risk via Mkt factor).

Medium

Question: "A fund’s rolling-window regression shows α declining from 0.5% to 0.1% over 3 years. What is the most likely explanation?" Options: A) Increased market efficiency. B) Survivorship bias in the dataset. C) Capacity constraints in the strategy. D) Data mining in the model. Correct Answer: C Explanation: - Why right: Capacity constraints (e.g., AUM growth) erode alpha over time. - Trap option: B (survivorship bias inflates alpha, not deflates it).

Hard

Question: "You run a Carhart regression for a long/short equity fund and find: - α = 0.3% (p = 0.02) - β_Mkt = 0.4 (p = 0.00) - β_SMB = 0.1 (p = 0.50) - β_HML = 0.2 (p = 0.10) - β_MOM = 0.3 (p = 0.01)

Which action is most justified?" Options: A) Drop SMB and HML from the model. B) Conclude the fund has persistent skill. C) Increase the sample size to improve p-values. D) Switch to a Fung-Hsieh 7-factor model. Correct Answer: A Explanation: - Why right: SMB and HML are statistically insignificant (p > 0.1) and add noise. - Trap option: D (Fung-Hsieh is for macro/CTAs, not equity funds).

Real-World Patterns

1. Due Diligence Reports
2. Pattern: Managers highlight statistically significant α but omit economic magnitude (e.g., 0.1% monthly α = 1.2% annual).

3. Red Flag: α decays over time → capacity constraints or strategy crowding.

4. Risk Attribution

5. Pattern: Funds with high MOM loadings outperform in trending markets but crash in reversals (e.g., 2022).

6. Action: Stress-test factor exposures under different regimes.

7. Compliance Audits

8. Pattern: GIPS-compliant firms must disclose survivorship bias adjustments.
9. Audit Focus: Verify graveyard fund inclusion and backfill exclusions.

30-Second Cheat Sheet

1. Carhart 4-factor: Mkt, SMB, HML, MOM → isolates α.
2. Persistence test: Spearman ρ > 0.3 = weak persistence.
3. Survivorship bias: Exclude dead funds → overstates returns by 2-4%.
4. Economic significance: 0.1% monthly α = 1.2% annual (check vs. fees).
5. Structural breaks: Factor loadings shift post-crises (e.g., 2008, 2020).

Related Concepts

1. Fung-Hsieh 7-Factor Model (for macro/CTAs).
2. Performance Decomposition (alpha vs. beta vs. luck).
3. Liquidity and Stale Pricing (impact on persistence tests).

Verified Source List

1. CAIA Association –