Linear regression

Let’s talk about a basic regression model:

yi=β0+β1x1i+⋯+βp−1xp−1,i+eiy=Xβ+e estimated by ordinary least squares:

ˆβ=(X′X)−1X′y.

• Classical inference methods assume homoskedasticity: Var(ei)=σ2
• But there are many settings where the we would rather allow for heteroskedasticity of an unknown form: Var(ei)=σ2i
• One way to do inference in this setting is by using heteroskedasticity-consistent covariance matrix estimators (HCCMEs; Eicker, 1967; Huber, 1967; White, 1980).

HCCMEs

• Variance of the OLS estimator: Var(c′ˆβ)=1nc′M(1nn∑i=1σ2ixix′i)Mc,M=(1nX′X)−1

• The original HCCME (sandwich) estimator: VHC0=1nc′M(1nn∑i=1ˆe2ixix′i)Mc

• HC0 is asymptotically consistent, but biased in small samples; hypothesis tests based on HC0 can have poor coverage in small samples.
• Features of design matrix (especially leverage) influence bias and coverage.

Potential small-sample improvements

• Modified sandwich estimators (MacKinnon & White, 1985; Davidson & MacKinnon, 1993): VHC0=1nc′M(1nn∑i=1ωiˆe2ixix′i)Mc where HC1:ωi=n/(n−p)HC2:ωi=(1−hii)−1HC3:ωi=(1−hii)−2

• Long and Ervin (2000) conducted a comprehensive simulation study on hypothesis test coverage with HCCMEs, recommended HC3 as default.

• Subsequently (Cribari-Neto et al., 2004, 2007, 2011): HC4:ωi=(1−hii)−δi,δi=min{hiin/p,4}HC4m:ωi=(1−hii)−δi,δi=min{hiin/p,1}+min{hiin/p,1.5}HC5:ωi=(1−hii)−δi,δi=min{hiin/p,max{4,0.7h(n)(n)n/p}}

Other potential small-sample improvements

Approximations for the reference distribution of test statistic:

• Satterthwaite approximation (Lipsitz, Ibrahim, & Parzen, 1999)
• Edgeworth approximation (Kauermann & Carroll, 2001)
• Saddlepoint approximation (McCaffrey & Bell, 2006)

• All three approximations developed assuming a homoskedastic “working model” for the error structure.

• These approaches have been largely ignored. Never compared to HC* variants or to each other.

Simulations

• Simple regression with one predictor: yi=β0+β1xi+σiei
• Predictor variable xi∼χ2 with varying degrees of skewness
• Errors ei∼N(0,1), t5, or χ25
• Skedasticity function σi=exp(ζxi), ζ=0.00,0.02,0.04,...,0.20
• sample sizes of n=25,50,100
• Target Type-I error rates of α=.05,.01,.005
• Rejection rates estimated from 50000 replications

Size of HC* variants, α=.05

Size of HC* variants, α=.01

Size of HC* variants, α=.005

Size of selected tests, α=.05

Size of selected tests, α=.01

Size of selected tests, α=.005

Findings

1. Currently recommended test HC3 does not adequately control type-I error rate.
2. At the α=.05 level, HC4 maintains most accurate rejection rates of all tests considered.
3. At smaller α levels, Satterthwaite and Edgeworth approximations out-perform HC3 and HC4.

Discussion

• In ongoing work, we are examining the relative power of tests (after size adjustment)
• Generality of findings requires further investigation (covariate distribution, skedasticity function, number of predictors)
• Distributional approximations warrant wider consideration because they can be generalized to more complex models.

Degree of heteroskedasticity