§6.4

Panel Data: Fixed and Random Effects

Across stores, customers, and markets, business units differ permanently in ways that are extremely hard to measure: neighborhood income, store layout, manager skill, local competitor density. If we pool data across units and run one regression, the differences across units pollute the relationship we wanted to estimate within units. The fix is structural: track each unit over time, and use that within-unit variation to identify the effect. The technique is called fixed effects, and it is the workhorse identification strategy of empirical business analysis.

This article builds the within-unit-vs-across-unit distinction visually first, then derives the demeaning transformation that makes fixed effects mechanically equivalent to a simple regression on within-unit deviations, walks through the alternative — random effects — and the practical rule for choosing between them, and ends with a data case on store fixed effects in soup pricing.

The Executive Question: Are We Comparing Stores to Stores, or to Themselves?

A naive pooled regression of latte volume on price across 100 stores returns a positive slope: stores charging more sell more lattes. The recommendation looks like "raise prices."

The audit reveals the obvious story. Suburban stores charge $3.80 and sell 8,000 lattes a week; urban stores charge $3.00 and sell 5,000. The pooled comparison is between two different kinds of stores — and the across-store demographic gap dominates the within-store price response that any sane pricing decision would actually depend on.

The executive question is not "what is the correlation between price and volume in the dataset" but:

What happens to the same store's volume when that store changes its own price?

That is the question fixed effects answers, by construction.

Visualizing the Difference

Figure 1 shows the same conceptual picture three times. Three stores with different baseline levels (the intercepts) move modestly in response to a lever (the within-unit slopes). Looking across stores, the levels dominate the picture. Looking within each store, the slopes are similar and modest.

Fixed effects use the variation within each unit, not across

Cross-store comparison mixes level differences (intercepts) with the lever's effect. Store fixed effects subtract each store's own mean (dashed) and identify the slope from within-store wiggles only.

Figure 1. Cross-unit comparison mixes stable level differences (intercepts) with the lever's effect. Store fixed effects subtract each store's own mean (dashed) and identify the slope from within-store wiggles only.

A pooled regression on this data fits one line through all the points and is dragged by the level gaps. A store-fixed-effects regression fits one slope to all the points after subtracting each store's mean from each store's points — and that slope is the within-store response.

Comparison	What the variation comes from	What's absorbed	What's not
Pooled OLS	Across all stores and weeks	Nothing	Stable store differences + common time shocks
Entity (store) fixed effects	Within a store, across weeks	All stable store-level differences	Common time shocks (e.g. holidays)
Two-way fixed effects (store + week)	Within a store, against the common-week baseline	Stable store differences + week-level shocks	Time-varying, store-specific shocks

The two-way fixed effects (TWFE) specification — store and week — is the standard starting point for any panel pricing analysis.

The Method: Demeaning

Let units be indexed by $i \in \{1, \dots, N\}$ and time periods by $t \in \{1, \dots, T\}$ . The TWFE model is

Two-way fixed effects (TWFE)

Y_{it} = \beta_1 X_{it} + \beta_2 W_{it} + \alpha_i + \gamma_t + u_{it}

where $\alpha_i$ is an intercept for each unit, $\gamma_t$ is an intercept for each time period, $X_{it}$ is the treatment, and $W_{it}$ are time-varying controls.

For thousands of units, estimating thousands of dummy intercepts is infeasible by brute force. The within transformation sidesteps the problem. Define each unit's mean over time:

\bar{Z}_i \;=\; \frac{1}{T}\sum_{t=1}^{T} Z_{it}

and subtract it from every observation of that variable:

\tilde{Z}_{it} \;=\; Z_{it} - \bar{Z}_i

Apply that transformation to both sides of the model. Because $\alpha_i$ is constant over time within unit $i$ , its mean is itself, and demeaning makes it vanish:

Demeaned regression

\tilde{Y}_{it} \;=\; \beta_1 \tilde{X}_{it} + \beta_2 \tilde{W}_{it} + \tilde{u}_{it}

The coefficient $\beta_1$ in this demeaned regression is exactly the fixed-effects coefficient from the original specification. Two consequences are worth memorizing:

Every time-invariant unit characteristic — observed or unobserved — has a demeaned value of exactly zero. Square footage, ZIP-code income, store age, manager identity: all of it is absorbed. You do not have to measure stable confounders. You only have to assume they are stable.
All of the identifying variation comes from changes within a unit. If a store never changes its price across the panel, that store contributes nothing to the price coefficient — its within-unit price variation is zero.

Centering, Live

The pooled-vs-centered comparison in Figure 2 lets you switch between the two views. The pooled scatter shows the misleading positive slope. The centered scatter slides each store's points to the origin and reveals the true within-store relationship.

How Store Fixed Effects Isolate Within-Store Price Sensitivity

Pooled raw comparisons (Confounded by neighborhood demographics, creating a false positive slope)

Store A (Suburban) Store B (Urban)

What to notice: Suburban Store A operates in a high-income area with high baseline demand ($V_A = 8.16k$) and has higher average pricing ($P_A = $2.30$). Urban Store B operates in a lower-income area with lower demand ($V_B = 5.86k$) and lower average prices ($P_B = $1.50$). If we naively pool them, the regression compares across stores, producing a false positive slope (+2.88) which suggests that raising prices increases demand.

Figure 2. Pooled and centered views of the same panel data. In the pooled view the across-store demographic gap fits a positive slope; in the centered view, with each store's mean subtracted, the true within-store negative slope emerges.

The centered view is what the fixed-effects regression "sees." That visual lens — pull every unit's mean to the origin and look at the residual variation — is the most useful mental model for reading any fixed-effects coefficient.

Concept check

Three questions spanning what regression isolates, what identification adds, and what fixed effects buy.

1.
The Frisch–Waugh–Lovell view of multiple regression says that the coefficient on $X_1$ is the slope of…
2.
Which of the following best captures the difference between identification and estimation?
3.
A pooled regression of volume on price returns a positive slope. The same regression with store fixed effects returns a negative slope. Which interpretation is most defensible?

The Alternative: Random Effects

Fixed effects buys its identification guarantee by brute force: it throws away all across-unit information and estimates the treatment effect from within-unit movement alone. That guarantee is not free. It costs precision — the across-unit variation it discards is variance a more efficient estimator could have used — and it costs the ability to say anything at all about a variable that never changes within a unit. Store square footage, urban-vs-suburban format, whether the store has a drive-through: fixed effects demeans every one of them to exactly zero. Ask a fixed-effects model "does format explain lower price sensitivity?" and it cannot answer, by construction.

Random effects (RE) is the model that tries to buy back both of those losses. Instead of estimating a separate intercept $\alpha_i$ for each store — a fixed, unit-specific number to be absorbed — RE treats $\alpha_i$ as a random draw from a distribution, typically $\alpha_i \sim \mathcal{N}(0, \sigma_\alpha^2)$ , uncorrelated with the regressors:

Random effects (RE)

Y_{it} = \beta_1 X_{it} + \beta_2 W_{it} + \delta S_i + \alpha_i + u_{it}, \qquad \operatorname{Cov}(\alpha_i, X_{it}) = 0

Because the store intercept is now treated as random noise rather than something to be swept away, RE can keep a time-invariant store characteristic $S_i$ (format, footprint, drive-through) directly in the equation and estimate $\delta$ . And because RE uses a weighted blend of the within-store variation and the across-store variation — rather than discarding the latter entirely — its standard errors on $\beta_1$ are smaller than fixed effects' whenever the underlying assumption is actually true.

That "whenever" is the whole story. The RE assumption — that the store-level intercept is statistically unrelated to price — is precisely the assumption fixed effects was invented to avoid needing. If unmeasured store characteristics (local competition, neighborhood income, manager pricing habits) are correlated with the store's own pricing behavior, $\operatorname{Cov}(\alpha_i, X_{it}) \neq 0$ , and RE's efficiency gain comes at the cost of the same omitted-variable bias a pooled cross-store regression has. RE is not a free upgrade over FE — it is a bet that a specific, checkable assumption holds.

The practical decision rule

A manager does not need the asymptotic theory to make this call sensibly. Three questions do most of the work:

Is there a plausible story where unmeasured unit differences correlate with the regressor of interest? In pricing, staffing, and most operational panels, the answer is almost always yes — stores with wealthier catchment areas tend to both price differently and perform differently for reasons that have nothing to do with the lever being tested. When that story is plausible, fixed effects is the safe default.
Do you need the coefficient on a time-invariant variable? If the deliverable is "how much does drive-through format change customer response," fixed effects cannot answer that question at all, since format never varies within a store. Random effects (or a hybrid/Mundlak approach) is the only way to get that number — but only if the uncorrelatedness assumption is defensible for this specific variable.
How many time periods do you have, relative to the number of units? Fixed effects estimates a separate intercept per unit, which is only supportable when there are enough repeated observations per unit to make within-unit demeaning informative. With very short panels (few periods per unit) relative to a large number of units, RE's borrowing of across-unit information becomes more attractive, provided the correlation concern in point 1 is genuinely weak.

	Fixed effects	Random effects
Unit intercept	Estimated freely per unit; absorbed via demeaning	Modeled as a random draw, assumed uncorrelated with regressors
Identifying assumption	None on the unit effect — robust to any stable confounder	Unit effect uncorrelated with regressors (strong, often implausible in pricing)
Time-invariant regressors	Cannot be estimated (demeaned to zero)	Can be estimated
Efficiency	Lower (discards across-unit variation)	Higher, if the uncorrelatedness assumption holds
Safe default when...	Unit-level differences plausibly correlate with the treatment (most business panels)	Few periods per unit, a time-invariant variable is the target, or correlation is genuinely implausible

Checking the choice: the Hausman intuition

There is a formal Hausman test for this, but the intuition a manager needs is simpler than the derivation: run both models and compare the coefficient on the variable you care about.

If the FE and RE estimates are close, RE's uncorrelatedness assumption was not doing much work in this dataset — the extra efficiency is a legitimate bonus, and reporting the RE (or pooled) estimate with its tighter confidence interval is defensible.
If the FE and RE estimates diverge substantially, that divergence is the evidence that RE's assumption is violated — the unit effects are correlated with the regressor, RE is biased, and fixed effects is the number to trust, even though its standard errors are wider.

The question to bring to a review meeting is not "which model has smaller standard errors" but "does it matter which one I use, and if it does, why would I expect the random-effects assumption to hold here?" A large FE-vs-RE gap that nobody can explain is itself a finding — it says the unit-level story (demographics, competition, management) is doing real work in the data.

For the Progresso panel, this is not a hypothetical. Store fixed effects were adopted earlier in this article precisely because unmeasured store-level demographics and local competitive intensity plausibly correlate with a store's own pricing pattern — high-income, low-competition stores both price differently and sell differently for reasons that have nothing to do with the price coefficient we want. That is exactly the condition under which random effects would return a biased answer: the "efficiency" it offers would be efficiently estimating the wrong number. Fixed effects being less efficient but assumption-free is not a compromise in this setting — it is the correct call, and it is why the chapter reached for it in the first place.

Data Case: Store Fixed Effects in the Progresso Panel

We climb the same regression ladder built in the previous article — bivariate, then with seasonality, then with competitor price, then with regional dummies, then with store fixed effects. The final rung is the within-store estimate that the previous article promised.

The elasticity estimate changes as the comparison gets cleaner

88,409 store-months across 2,042 stores. Coefficient is on log(Progresso price).

Figure 3. The Progresso regression ladder, ending in store fixed effects. The store-FE specification explains a much larger share of variance because it absorbs the stable across-store differences. The coefficient on price settles near −2.23.

The step from Model 4 (regional dummies) to Model 5 (store fixed effects) is the smallest movement on the ladder but the most meaningful from an identification standpoint. Regional dummies absorb broad geographic differences. Store fixed effects absorb everything stable at the store level — neighborhood income, local competition, shelf placement, manager habits, parking. The coefficient moves modestly because regional dummies already captured a fair amount of the variation, but the reason we trust the estimate now is structural: there is no remaining stable store-level confounder it could absorb.

Reading the headline: a 1% increase in Progresso price within a given store, in a given month, is associated with roughly a 2.23% decrease in unit volume, after controlling for that store's stable characteristics, competitor pricing, and seasonal demand.

A pricing recommendation built on the naive −3.21 would have overstated price sensitivity by nearly a factor of one and a half, with predictable consequences — over-discounting to chase imagined volume, margin erosion, no actual lift. The fixed-effects estimate is the safe basis for pricing not because the number is smaller but because the comparison is fair.

Net-net

Fixed effects are the cheapest, highest-value identification step a corporate analytics team has access to. Three rules:

Default to two-way fixed effects (entity + time) for any panel pricing or rollout analysis. The cost is one line of code; the benefit is the absorption of every stable unit difference and every common time shock.
Read the FE coefficient as a within-unit response. Always state the conditioning explicitly: "within a store, across weeks, holding ___ constant."
Reach for random effects only when you have a specific reason to — a time-invariant variable you must estimate, a short panel, or a genuine argument that unit effects are uncorrelated with the treatment — and check that reason against a fixed-effects run of the same model. A large gap between the two is a red flag, not a nuisance to average away.
After absorbing stable unit and time effects, name the time-varying, unit-specific shocks that could still confound the treatment. That residual list is the heart of a professional identification memo for a panel design.

The next chapter takes one further step: when fixed effects are not enough — when there is a specific event we want to evaluate — we move to designs that build the counterfactual from the comparison group's trajectory.