Why Linear Regression Assumes its Assumptions

1Why Regression, and Why Matrix Notation

Linear regression is one of the most widely used tools in causal inference because it provides a simple way to estimate the relationship between a treatment and an outcome while adjusting for other relevant variables. When used carefully, regression can help isolate the effect of a variable of interest by controlling for confounding factors, making it easier to compare otherwise similar observations. However, regression alone does not prove causality, and its causal interpretation depends on the assumptions being made.

These assumptions can feel abstract when stated in words, so the next step is to build intuition for them using the matrix notation of linear regression. Writing regression in matrix form helps show more clearly how coefficients are constructed, what variation is being used to estimate them, and why certain conditions are needed for those coefficients to have a causal interpretation. This perspective connects the algebra of regression to the causal assumptions that make regression estimates meaningful.

2Ordinary Regression Formula

For observation i, a standard two-feature linear regression model is

Y_i = β₀ + β₁X_1i + β₂X_2i + u_i.

For any candidate coefficients b₀, b₁, b₂, the predicted value is

Ŷ_i = b₀ + b₁X_1i + b₂X_2i.

OLS selects the particular candidate values that minimize the squared residuals; we write those chosen estimates as β₀, β₁, and β₂. The fitted value Ŷ_i and the residual below use these chosen estimates, not arbitrary candidates.

The residual for observation i is then

e_i = Y_i − Ŷ_i.

The terms mean:

Y_i is the observed outcome for row i.
X_1i and X_2i are the two feature values for row i.
β₀, β₁, and β₂ are the population coefficients in the target model.
u_i is the unobserved error term for row i.
e_i is the observed residual after fitting the model.

3Regression’s Two Layers

Before the detailed derivation, here is the main picture. Linear regression has two layers.

First, there is the linear algebra layer. OLS chooses the coefficient vector β that minimizes squared residuals (or Sum of Squared Errors — SSE in the formulas below). In the formulas below:

y is an n×1 outcome vector: each row is the observed outcome for one observation.
X is an n×3 design matrix in the two-feature model with an intercept: rows are observations and columns are the intercept, X₁, and X₂.
β is a 3×1 vector: its rows are the coefficients on the intercept, X₁, and X₂.

For a candidate coefficient vector β, the deviation vector y − Xβ is n×1, with one entry per observation; the residual vector e = y − Xβ is this quantity evaluated at the fitted β. The squared-error objective minimized by OLS is

SSE(β) = (y − Xβ)′(y − Xβ).

At the minimizing β this equals e′e = ∑_i e_i², the sum of squared residuals.

The key matrix condition is full rank: X′X must be invertible for the coefficient vector to be uniquely computed using

β = (X′X)⁻¹ X′y.

Here X′X is a 3×3 matrix whose rows and columns correspond to the same three model terms: intercept, X₁, and X₂.

Second, there is the statistical and causal layer. Exogeneity, correct adjustment, no omitted confounding, appropriate dependence structure, and valid standard errors determine whether the coefficients can be interpreted causally and whether inference is trustworthy. The variance formula makes this especially clear:

Var(β | X) = (X′X)⁻¹ X′ Ω X (X′X)⁻¹.

Here Ω = Var(u | X) is an n×n error covariance matrix. Its rows and columns correspond to observations. Diagonal entries are error variances for individual observations. Off-diagonal entries are covariances between errors from different observations.

If the off-diagonal elements of Ω are zero, the errors are uncorrelated across observations. If all diagonal elements of Ω are the same, the errors are homoskedastic. If both conditions hold, then Ω = σ²I_n and Var(β | X) = σ²(X′X)⁻¹. The identity matrix I_n is n×n: its rows and columns correspond to observations; it has ones on the diagonal and zeros off the diagonal.

So the matrix intuition is important, but only one common assumption is literally about matrix invertibility. Most causal assumptions are about whether the unobserved error term u is related to the regressors, not about whether OLS can mechanically compute a coefficient.

4What the Algebra Can See — and What Breaks When It Can’t

The OLS derivation touches the error u only through its first two conditional moments: the mean E[u | X] and the covariance Ω = Var(u | X). An assumption is visible in the algebra exactly when it can be written with those two objects (and the design matrix X). Anything that needs more — the full shape of the errors, or knowledge of the causal process behind the data — is invisible to it.

Visible in the derivation

These enter the formulas directly, so the algebra shows you the mechanism of what goes wrong — and, for most of them, how to fix it. Exogeneity is the partial exception: it appears in the derivation, but because u is unobserved the algebra cannot warn you it has failed and offers no repair — see its entry below.

No perfect multicollinearity (full rank)

In the algebra: the condition that lets you form (X′X)⁻¹ at all.

When violated.

X′X is singular and the inverse does not exist, so there is no unique coefficient vector — infinitely many β produce the identical fitted values. Software falls back to a generalized inverse or silently drops a column; either way the individual coefficients are meaningless and only estimable combinations survive. If x₂ = 2x₁, only b₁ + 2b₂ is identified, never the two separately.

Coefficients: not identified · Std. errors: undefined · Causal reading: nothing to interpret.

Sufficient variation / no near-collinearity

In the algebra: the independent variation left in each column, Var(x̃_k), after the others are partialled out.

When violated.

The inverse still exists, so OLS runs, but X′X is ill-conditioned. Coefficients stay unbiased yet turn fragile and imprecise: since Var(β_k) = σ²/Σx̃_k², shrinking a regressor’s independent variation inflates its variance. A tiny data change can swing an estimate wildly — in a deliberately ill-conditioned toy design, nudging one outcome from 17.0 to 17.1 can move a coefficient from about 4.0 to 7.1 — even while the overall fit stays excellent. Weak overlap pushes estimates onto extrapolation.

Coefficients: unbiased but unstable · Std. errors: inflated · Causal reading: weak, rests on little support.

Exogeneity, E[u | X] = 0

In the algebra: the cross-product X′u in β = β + (X′X)⁻¹X′u.

When violated.

OLS is still perfectly computable — this is not an invertibility problem — but X′u no longer averages to zero, so β is biased and inconsistent, centered on the wrong value. Confounding is the most familiar source — an unobserved driver of the outcome correlated with the regressors — but the same failure arises from measurement error in a regressor (which attenuates its coefficient toward zero), reverse causality or simultaneity (the outcome also drives a regressor), and sample selection (conditioning on how units entered the data). More data does not rescue any of these; it just pins down the biased number more tightly. A process built from y = 1 + 2x₁ + … can return 3 for a coefficient whose truth is 2.

Note the asymmetry with the other visible assumptions: because u is unobserved, you cannot detect this failure from the data, and there is no algebraic repair. The fixes — instrumental variables, randomization, or a better adjustment set — all live in design, not in the matrices. Exogeneity is “visible” only in that E[u | X] is one of the two moments the derivation uses.

Coefficients: biased · Std. errors: valid formula around the wrong center · Causal reading: broken.

Homoskedasticity

In the algebra: equal diagonal entries of Ω in the sandwich variance.

When violated.

Heteroskedasticity leaves the coefficients untouched and still unbiased, but the classical σ²(X′X)⁻¹ is the wrong variance — standard errors, t-statistics, p-values, and confidence intervals are invalid. Whether they come out too small or too large depends on how the error variance lines up with high-leverage points; the common case is understated standard errors, so you over-reject, but the bias can run the other way. OLS is also no longer the most efficient linear estimator. The repair is heteroskedasticity-robust (HC) standard errors; the point estimates never move.

Coefficients: fine · Std. errors: wrong — use robust · Causal reading: fine.

No autocorrelation / independence

In the algebra: zero off-diagonal entries of Ω.

When violated.

Correlated errors — repeated rows from the same user, clustered groups, serial dependence over time — again leave the coefficients unbiased under exogeneity, but the sandwich variance no longer collapses to the classical form. Standard errors are wrong, usually too small when the correlation is positive, so significance is overstated. Fix the inference with clustered or HAC (Newey–West) standard errors, or model the dependence directly with panel or GLS methods — the latter re-estimate the coefficients, not just the standard errors. Inference is the casualty, not the estimate.

Coefficients: fine · Std. errors: wrong — cluster / HAC · Causal reading: fine.

Invisible to the derivation

The computation runs happily whether or not these hold; the coefficient can be precise and significant and still be meaningless. Nothing in the matrices warns you.

Normality of errors

Outside the algebra: the full shape of u, beyond its first two moments — it never enters the derivation of β or its variance.

When violated.

For typical sample sizes, essentially nothing: β and its variance are unaffected, and the central limit theorem keeps the usual t and F inference approximately valid. The only strict casualty is exact small-sample inference — with few observations the finite-sample t and F distributions are no longer exact, so those p-values and intervals become approximate. Heavy tails or outliers can also cost efficiency, since OLS is sensitive to extreme points.

Coefficients: fine · Std. errors: fine in large n · Causal reading: fine · only exact small-sample tests affected.

Correct functional form

Outside the algebra: whether the columns of X can represent the true relationship — the derivation takes them as given.

When violated.

OLS still returns the best linear fit within the columns you supplied, but it can be the best answer to the wrong question: the slope becomes a misleading summary and predictions extrapolate poorly. Worse, the omitted curvature or interaction is dumped into u and typically correlates with X, so misspecification quietly resurfaces as an exogeneity failure and biases the coefficients. Adding a squared term or interaction — literally new columns — can, in a constructed example, take a fit from R² = 0.33 to R² = 0.9995.

Coefficients: wrong shape, often biased · Std. errors: can mislead · Causal reading: broken.

Correct adjustment set / no omitted confounding

Outside the algebra: which variables to control for — a fact about the causal process, not the matrices.

When violated.

The regression computes a precise, significant coefficient that is simply the wrong causal quantity. Leaving a confounder out biases the estimate; adding a bad control is just as dangerous — conditioning on a post-treatment variable or a collider can create bias where there was none, and controlling a mediator strips out part of the very effect you want. A tight t-statistic offers no protection. The fix lives in design (a DAG and a deliberate adjustment set), not in the estimator.

Coefficients: non-causal / biased · Std. errors: fine · Causal reading: broken.

Consistency & no interference (SUTVA)

Outside the algebra: each row’s outcome is its own potential outcome, and one unit’s treatment does not change another’s — a design fact with no matrix counterpart.

When violated.

The regression runs and looks clean, but a single treatment coefficient blurs together direct effects, spillovers, and exposure effects, so “the effect of x” is ill-defined. This is common in marketplaces and social networks, where a promotion shown to one user spills onto competitors, sellers, or friends. The number is mechanically estimable but causally ambiguous; you either model the interference explicitly or design around it, for example with cluster-level randomization.

Coefficients: a mix of effects · Std. errors: can be affected · Causal reading: ambiguous.

The whole list at a glance

Assumption	Visible?	Effect on coefficients	Effect on standard errors	Effect on causal reading
No perfect multicollinearity	visible	not identified	undefined	n/a
Sufficient variation / overlap	visible	unbiased, unstable	inflated	weak
Exogeneity, E[u \| X] = 0	visible*	biased	valid formula, wrong center	broken
Homoskedasticity	visible	fine	wrong — use robust	fine
No autocorrelation / independence	visible	fine	wrong — cluster / HAC	fine
Normality of errors	invisible	fine	fine in large n	fine
Correct functional form	invisible	wrong shape / biased	can mislead	broken
Correct adjustment set	invisible	non-causal	fine	broken
Consistency / no interference	invisible	mixes effects	can be affected	ambiguous

*Exogeneity enters the algebra through the two error moments, but a violation cannot be detected from the data and has no algebraic fix — it is “visible” only in that limited sense.

Appendix

One Picture for Specification, Overlap, and Multicollinearity

The Frisch–Waugh–Lovell theorem turns three separate‑looking assumptions into a single matrix quantity. Each coefficient in a multiple regression equals a simple regression slope on the part of that regressor left over after the other columns are partialled out:

β_k = Cov(y, x̃_k) / Var(x̃_k)

where x̃_k is the residual from regressing column k on all the other columns (Angrist & Pischke’s “regression anatomy,” eq. 3.1.3, after Frisch & Waugh, 1933). Read from this one formula:

Holding controls fixed is not a metaphor — it is the residualization x̃_k. Sufficient variation and overlap is the requirement that Var(x̃_k) — the independent variation left in the regressor — is not near zero. And near‑multicollinearity is exactly the case where the other columns explain x_k so well that Var(x̃_k) collapses. Because the coefficient’s own variance is σ² divided by that leftover variation, a coefficient with little independent variation is both fragile and imprecise. The same denominator that identifies a coefficient also sets how well you can estimate it. This orthogonalization is also the engine behind modern debiasing methods (Facure, Causal Inference in Python, ch. 4).

Homoskedasticity, No Autocorrelation, and the Gauss–Markov Theorem

The two “inference-only” assumptions in Section 4 — homoskedasticity and no autocorrelation — are exactly the pair that makes the error covariance spherical, Ω = σ²I_n. That is the condition of the Gauss–Markov theorem: given exogeneity, full rank, and spherical errors, OLS is the best linear unbiased estimator (BLUE) — it has the smallest variance among all linear unbiased estimators. Neither assumption is needed for OLS to be unbiased, which is why violating them leaves the point estimates intact and damages only their efficiency and the standard-error formula.

When either fails, a generalized least squares (GLS) estimator that reweights by Ω⁻¹ restores efficiency; when it is not worth modeling Ω, robust or clustered standard errors keep inference valid while leaving the OLS point estimates unchanged. Adding normality of the errors on top of Gauss–Markov upgrades OLS to best among all unbiased estimators, not merely the linear ones, and makes the t and F distributions exact in small samples — which is why normality sits in the “invisible” group: it sharpens the guarantees but never enters the computation of β or its variance.