Matching the unit of analysis to the shock: Amazon HQ2, Arlington and Long Island City

How to read this note

The note works through one investment decision with one main method and a set of checks. It starts with the investment committee's argument for holding a Long Island City parcel, shows how to build a fair stand-in for what home values would have done without Amazon's announcement, and applies it to the two ZIP codes where Amazon announced sites. The checks test the result a second way, at metropolitan level, with rents, with neighboring ZIPs, under alternative setups and against event dates. A section on statistical power explains why the New York test is weak, and the last sections take the committee's five claims apart and say what the committee could and could not conclude.

Terms are explained where they first appear and again in the glossary. The text reports gaps in percent. Figures and tables report them in log points (100 times the natural log of a ratio), which are close to percent for gaps this small.

What the analysis found

After Amazon's announcement, home values in Crystal City moved above a matched comparison and home values in Long Island City did not. From November 2018 to February 2020 the typical home value in Crystal City (ZIP 22202) averaged 3.7 percent above its synthetic control, a weighted blend of other Washington-area ZIPs built to track 22202 closely before the announcement. Among 22202 and 304 placebo ZIPs (ZIPs that received no announcement), 22202 has the largest gap relative to its pre-announcement fit. It ranks 1 of 305, with a p-value of 0.003.

Two cautions apply. The Crystal City gap opens slowly. It averaged 0.5 percent in the first three months and reached 6.9 percent by February 2020, after several later milestones. And the three neighboring Arlington ZIPs show an average gap of 2.5 percent, so the Crystal City estimate bundles the announcement with wider-area movement and later events.

In Long Island City (ZIP 11101) the early gap is +0.01 percent, so values did not jump when the news came. The gap turns negative in the value dated 2019-01, a month before the cancellation of 2019-02-14, and averages -3.7 percent from November 2018 to February 2020. Placebo ZIPs in New York produce a gap that large about 15 percent of the time, so the gap is inside the range of chance. The smoothed condo series shows an early gap of +1.9 percent that fades to +0.3 percent by February 2020, also inside that range.

The metropolitan-level comparison finds nothing for either site. Against the 12 other finalist metropolitan areas, the Washington metro's typical home value moved -0.2 percent on average and the New York metro's -1.1 percent, and both are inside the range of chance. Washington's null sits next to a ZIP-level result that ranks first of 305, because a change in one ZIP is diluted in a metropolitan average.

The rent (ZORI) fits are poor, with a typical monthly miss before the announcement of 1.42, 1.69 and 2.65 log points for the three rent ZIPs. The rent results are reported and not relied on.

The committee's argument for holding the parcel rests on five claims, and each fails for a different reason: a before-and-after change with no comparison, a premium assumed for a ZIP whose raw series shows none, a loss dated from a smoothed series, a unit of analysis too large for the shock, and a co-movement the data do not show. The section on the five claims gives the number that exposes each. All results are reported as computed, including nulls and poor fits.

What you should be able to do after this note

1. Explain why a before-and-after change in one ZIP says little, and build a comparison group that shows what the ZIP would have done without the news.
2. Read a synthetic control, a placebo test and a p-value well enough to explain each to a colleague.
3. Match the unit of analysis to the size of the shock, and say why a metropolitan average cannot see a change confined to one neighborhood.
4. Decide whether a test could have seen an effect of the plausible size before treating a null result as evidence of no effect.
5. Say what monthly, smoothed and restated home-value indexes can and cannot date.
6. Turn an imprecise estimate into a sell or hold recommendation, fix the date, comparison, outcome and test before looking, and name the evidence that would change the recommendation.

The decision and the argument for holding

An investment committee holds a Long Island City development parcel, bought in December 2018 weeks after Amazon named the neighborhood for half of its second headquarters. In early March 2020 it must decide whether to sell. A memo argues for holding in five steps (Table 2). Each step rests on a number that is correct as arithmetic. The rest of the note asks whether the number supports the step.

Three ways to build the comparison

The memo compares each ZIP with itself a year earlier. A fair comparison asks what this ZIP's home values would have done without the announcement. That path is the counterfactual. It cannot be observed, so it is replaced by a stand-in built from a comparison group, ZIPs that the announcement did not reach. The stand-in is trustworthy only if it moved like the treated ZIP before the announcement. This section builds three stand-ins for each of the two treated ZIPs, Crystal City (22202, in Arlington, Virginia) and Long Island City (11101, in Queens), and grades each on that test.

All three use the ZIP's typical home value (Zillow's ZHVI, raw series) turned into an index equal to 100 times the natural log of the value, set to zero at its average over the twelve months from November 2017 to October 2018. The gap in a month is the ZIP's index minus the stand-in's index. Because 100 times a log difference is close to a percent difference, a gap of +3.7 means the ZIP sits about 3.7 percent above its stand-in. Table 3 defines the summaries used from here on.

The first stand-in, the pool average, is the unweighted mean of every ZIP in the same metropolitan area with a complete series (304 ZIPs in Washington, 815 in New York). ZIPs the announcement could have reached are left out. These are the ring and co-treatment ZIPs described later and one low-confidence ZIP. The pool average is easy to explain and fits poorly. Before the announcement its typical monthly miss is 3.85 log points for 22202 and 6.54 for 11101.

The second stand-in, the nearest ten, averages only the ten pool ZIPs whose paths from 2014-01 to 2018-10 stayed closest to the treated ZIP's, as measured by the root mean squared difference between the two paths. The typical monthly miss before the announcement falls to 0.26 and 2.13.

The third stand-in, the synthetic control, blends 40 pool ZIPs (the donors, screened as the nearest to the treated ZIP) with weights that are never negative and add to one. The weights are chosen so that before the announcement the blend moves as closely as possible like the treated ZIP. Its typical monthly miss is 0.15 and 0.84.

For Crystal City the average gap is positive under all three stand-ins, between +3.7 and +4.2 percent. For Long Island City it is negative under all three but varies more, from -6.4 to -3.5 percent. A stand-in that did not track the ZIP before the announcement cannot be trusted after it, and the spread follows the fit error. Ranking each average gap against placebo ZIPs (last column of Table 4; the ranking is explained in the next section) puts Crystal City 7th or 8th of 305 under every stand-in and Long Island City between 135th and 170th of 816. The Washington result survives the choice of comparison, and the New York result is a null under all three.

What the ZIP-level comparison shows

Synthetic controls for Crystal City and Long Island City

The main analysis gives each treated ZIP its own synthetic control. The blend draws on the 40 pool ZIPs nearest to the treated ZIP on its 2014-01 to 2018-10 path (58 months), taken from its metropolitan area's pool of ZIPs with complete data (304 in Washington, 815 in New York), and its weights are chosen to make the squared monthly gap over those months as small as possible. The fit error is 0.15 log points for 22202 and 0.84 for 11101. Neither is above the 90th percentile of the fit errors of its placebo ZIPs (it sits at percentile 16 for 22202 and 82 for 11101), so neither fit is flagged as poor.

Figure 2 shows each ZIP with its synthetic control. Crystal City and its stand-in coincide until 2018 and separate afterwards; Long Island City and its stand-in stay close until early 2019. Figure 3 plots the difference between the two lines and compares it with the gaps that placebo ZIPs produce.

Placebo tests: how large a gap chance produces

A gap of a few percent could still be chance, because home values wander and no blend tracks a ZIP perfectly. A placebo test measures how large a gap chance produces. Pretend that each of the other ZIPs in the metropolitan area had received the announcement, give each its own synthetic control, and compute the same gap. The 304 Washington placebo gaps (and 815 for New York) show what the method produces when nothing special happened. The treated ZIP is then ranked among them. A rank of 1 of 305 means the biggest of 305, and the p-value is the rank divided by the number of ZIPs, 0.003 in that case.

Two statistics can be ranked. The ratio test divides a ZIP's typical monthly miss after the announcement by its miss before; it is the main test in the design file, and the size test is an additional ranking. A ZIP that was tracked tightly before the announcement gets a large ratio from a modest gap. The size test ranks the average gap itself, ignoring sign, and is the stricter reading for a ZIP with a tight fit. Table 5 gives both.

Crystal City ranks 1 of 305 on the ratio test (p = 0.003; 0.004 after dropping placebo ZIPs whose fit error before the announcement is more than 5 times Crystal City's own). On the size test it ranks 10 of 305 (p = 0.033). Its early gap is not unusual on the size test, which fits a gap that builds slowly. Dropping any one donor with weight of at least 0.01 moves the average gap only between +3.6 and +4.1 percent.

Long Island City ranks 295 of 816 on the ratio test (p = 0.362), and the size-test p-values for its average gap and give-back are 0.152 and 0.092. Its give-back is -4.6 percent. After the cancellation the gap moved down, and the fall is inside the range of chance. None of the Long Island City p-values is below the usual 0.05 threshold.

Which ZIPs make up each blend

Although 40 donors are available, the weights concentrate on a few. The largest single weight is 0.26 for 22202 and 0.52 for 11101. Counting each ZIP by the square of its weight, the blend is equivalent to 6.2 equally weighted ZIPs for 22202 and 3.2 for 11101 (the effective number of donors). Table 6 lists the largest weights.

Checks on the ZIP-level result

Event study: a second method gives the same answer

The first check changes the method. An event study follows the treated ZIP month by month, from 2016-11 to 2020-02, and estimates how far it moved relative to the simple average of the comparison ZIPs in its metropolitan area, with October 2018 as zero. "Stacked" means each treated ZIP gets its own block of data (the ZIP plus the comparison ZIPs of its metropolitan area), with all dates counted from the announcement. Both treated ZIPs share one announcement date, so no ZIP that has already been treated serves as a comparison.

For Crystal City the average coefficient after the announcement is +4.7 percent (p = 0.003), the same sign and a similar size to the synthetic-control average gap. For Long Island City it is -5.2 percent (p = 0.148), with a give-back of -7.2 percent (p = 0.069). The coefficients for the months before the announcement have root mean square 1.32 (22202) and 2.80 (11101) log points, with placebo p-values of 0.452 and 0.496, so neither pre-announcement trend stands out.

The conventional standard-error intervals in the replication output are narrower than the placebo bands. With one treated ZIP per block they are not reliable, so the placebo bands are the measure of uncertainty here.

Metropolitan level: a null that cannot see a one-ZIP change

The unit of analysis is the thing being compared. In the ZIP-level comparison it is a ZIP. In the metropolitan-level comparison it is a whole metropolitan area. The typical home value of the Washington (or New York) metro is compared with the average of the other finalist metros, again from October 2018, using the same event-study and synthetic-control methods and the same placebo logic with metros in place of ZIPs. The raw comparison has 12 comparison metros because Los Angeles is missing a month; the smoothed comparison has 13.

Neither metro moves. The average event-study coefficients are -0.2 percent for Washington (p = 0.846) and -1.1 percent for New York (p = 0.462), and the synthetic-control gaps are -2.5 and -1.0 percent. Against 80 large non-finalist metros the New York coefficient is -1.3 percent (p = 0.346), the same picture. With only 12 comparison metros, the smallest p-value any placebo ranking can reach is 0.077, so this check cannot produce strong evidence in either direction.

The Washington result shows why the unit matters. The same metropolitan area contains a ZIP whose gap ranks 1 of 305, yet its metropolitan coefficient is indistinguishable from zero. The arithmetic is simple. If Crystal City sits 3.7 percent above where it would otherwise be and no other ZIP moves, the unweighted average of the 320 ZIPs in the Washington metropolitan area moves by 0.012 percent. That is the by-count dilution; the Zillow files give no housing units, so the share of the housing stock is not computed. A change in one neighborhood is invisible in a metropolitan average, and a metropolitan null is therefore weak evidence of no neighborhood effect.

Rents: the fits are too poor to use

Rent (Zillow's observed rent index, ZORI) is a second outcome. All three rent fits are poor. The typical monthly miss before the announcement is 1.42, 1.69 and 2.65 log points for 22202, 11101 and 11109, each above the 90th percentile of the placebo ZIPs' misses. For comparison, the home-value fits miss by 0.15 and 0.84. For 22202 the average rent gap is -0.8 percent with a ratio p of 1.000.

The two Long Island City ZIPs show positive rent gaps (average gap +3.7 percent for 11101 and +3.3 for 11109) and positive give-backs (+2.3 and +4.0), so the rent gap does not reverse after the cancellation. The two p-values for 11101 disagree (0.386 on the ratio test against 0.018 by size) because a poor fit before the announcement inflates the denominator of the ratio. Anchoring on October 2018 alone instead of the twelve-month average moves the 11101 rent gap from +3.7 to +1.0 percent. ZIP 11109 has no home-value series, so rents are the only outcome available for the waterfront part of the site. ZORI is smoothed and not seasonally adjusted. The rent results carry little weight and should not be used to time anything.

Nearby ZIPs: Arlington's neighbors move with Crystal City

The announcement could have reached ZIPs next to the sites, so those ZIPs are kept out of every comparison pool and are estimated as treated ZIPs of their own, against the same placebo set as the treated ZIP of their metropolitan area. Ring ZIPs are the neighbors the announcement could plausibly have reached: 22201, 22204 and 22206 around Crystal City and 7 ZIPs in Queens and Brooklyn around Long Island City. Co-treatment ZIPs are two Alexandria ZIPs tied to Virginia's package through the Virginia Tech Innovation Campus: 22301, and 22305, which was named as the campus site only from 2019-06-10.

The Arlington ring ZIPs open gaps like Crystal City's. Their unweighted average gap is +2.5 percent, which is 68 percent of Crystal City's average gap, and ZIP 22301 shows +2.2 percent. A gap of this sign and size next to 22202 means that the 22202 result does not isolate the HQ2 footprint. It is consistent with a spillover from the announcement, a county-wide Arlington trend, or a comparison pool that does not match northern Virginia. The three Arlington ring ZIPs rank high on the ratio test (p = 0.003, 0.007 and 0.007) but not on the size test, where all three p-values are above 0.05, so the two rankings disagree for them.

The 7 Long Island City ring ZIPs do not move together. Their average gap is -1.0 percent, individual values run from negative to positive, and none has a size-test p-value below 0.10. ZIP 11104 has an early gap of +2.8 percent but an average gap of +1.1, so its early gap is not sustained. ZIP 22314 is a low-confidence ZIP that is kept out of the comparison pool and not estimated; returning it to the pool leaves the 22202 average gap unchanged within optimizer tolerance (see the sensitivity table).

Sensitivity: the sign survives nearly every alternative setup

The design file lists alternative setups for the ZIP-level comparison: different donor sets (20, 80 or all pool ZIPs, or urban-core donors only), a different base month, a fit window that starts in 2016, shorter post-announcement windows, ring ZIPs allowed as donors, and different versions of the home-value series. Across the 11 setups that keep the raw series and the full post-announcement window, Crystal City's average gap runs from +3.1 percent (full pool) to +4.4 percent (pre-period from 2016-01). Long Island City's runs from -5.6 percent (shortlist month) to -3.0 percent (full pool).

Counting the changed series (condo homes, smoothed all-homes) and the shortened windows but not the placebo in time, the average gap keeps the sign of the base estimate in 15 of 15 setups for 22202 and 12 of 13 for 11101. The exception for 11101 is the condo series, whose average gap is +1.2 percent. The placebo in time uses a fake announcement date of 2016-11, when nothing happened, and gives fake average gaps of -0.4 percent for 22202 and -1.7 percent for 11101. Of the 141 estimation fits, 4 did not converge; they carry asterisks in the table and are listed in Table 16.

Timing: Crystal City's gap builds after the announcement window

The Crystal City gap is not an announcement-window jump. The early gap is +0.5 percent, 15 percent of the average gap in log points, and the gap in 2020-02 is 4.2 times its value in 2019-02. Ending the post-announcement window at 2019-02 leaves 22 percent of the average gap (+0.8 percent; ratio p 0.013, size p 0.252). Most of the gap accumulates in a period that holds the County Board vote (2019-03-16), the Virginia Tech site move (2019-06-10), the Phase 1 site plan (2019-12-14) and the start of demolition (2020-01-22). The data do not say which of these, if any, moved values.

A second timing check moves the treatment month back to the January 2018 shortlist and measures the gap over 2018-01 to 2018-10, before the November 2018 reports. It is +0.0 percent for 22202 and -1.0 percent for 11101, so neither ZIP shows a positive run-up of meaningful size.

For Long Island City the raw gap is +0.4 percent in 2018-11 and +0.7 in 2018-12, then turns negative in the January 2019 value (-1.0) and reaches -3.0 in 2019-02. The 2019-02 value straddles the cancellation, which falls on day 14 of a 28-day month. Monthly data cannot separate a news channel (values fell before the cancellation) from an index-timing channel (Zillow's January estimate reflects later information).

Zillow's smoothed series is slower. Across 1,123 ZIPs, smoothed monthly changes line up best with raw changes 1 month later. For 11101 the smoothed all-homes gap is +0.1 percent in 2019-01 and -1.4 in 2019-02. The smoothed condo gap rises to +2.6 percent in 2019-01 and is +0.3 by 2020-02, which looks like a premium given back. It ranks 269 of 816 (ratio p 0.854), and its average gap ranges from -1.7 to +1.2 percent (-1.74 to +1.15 log points) when any one donor is dropped.

A study of prices near the sites finds earlier and larger premia

Chen, Wilkoff and Yoshida [1] study the same episode with housing prices near the sites. Their abstract reports a premium of 4.9 percent near the Virginia headquarters months before the decision, and a premium for New York that reaches 17.5 percent before the decision and disappears on cancellation, with no significant effects in the other finalist cities. A premium here means higher prices than comparable homes elsewhere. These two figures are literature values taken from the abstract, not results of this analysis.

The ZIP-level numbers here differ. The mean gap over 2018-01 to 2018-10, before the November 2018 reports, is +0.0 percent for 22202 and -1.0 percent for 11101. The raw early gap for 11101 is +0.01 percent and the condo early gap is +1.9 percent, the largest positive announcement-window figure for 11101 in this analysis. The give-back for 11101 is -4.6 percent in the raw series and -0.9 in the condo series, with size-test p-values of 0.092 and 0.752. For 22202 the first three months average +0.5 percent. The ZIP-level data show no anticipation premium in either ZIP, and no raw announcement-window premium in Long Island City.

The two can differ for reasons that this analysis cannot sort out, and the following are possibilities, not findings. Prices near a site can respond while a ZIP-wide, model-based index barely moves. Of the 109 sampled points of the Long Island City lots, 69 fall in 11101 and 40 in 11109, which has no ZHVI, and ZHVI describes homes in the 35th to 65th percentile band of a whole ZIP.

Zillow's smoothed series spreads and lags a change by about 1 month, which can blur a short window. And the comparison groups differ, because here the comparison is a weighted set of ZIPs matched on the pre-announcement path, which can differ from the comparison in a study of prices near the sites. A question for discussion is which design could have detected a premium confined to nearby blocks.

A Crystal City-sized gain would not have stood out in the New York test

A test is informative only if it could have found what it is looking for. Statisticians call this power. A New York p-value of 0.362 would be evidence of no effect only if a real gain of Crystal City's size would have produced a small p-value. It would not have, because the placebo gaps that set the benchmark are much more spread out in New York than in Washington.

Before 2018-10 the width of the middle half of New York placebo gaps (the interquartile range) averages 0.59 log points; after it, 3.08, a factor of 5.2. Washington's goes from 0.38 to 1.34. In 2020-02 the middle 90 percent of New York placebo gaps spans -6.9 to +7.7 percent, 2.4 times the width of Washington's span. The monthly 11101 gap is below the placebo 5th percentile in 4 months (2019-05 to 2019-08) and is back inside the 90 percent band by 2020-02.

The size test makes the same point. The 95th percentile of the placebo average gap, ignoring sign, is 6.28 log points in New York and 2.94 in Washington. A gap the size of Crystal City's average gap (3.7 percent) would be matched or exceeded in size by 130 of 815 New York placebo ZIPs (16 percent) and by 9 of 304 Washington placebo ZIPs (3 percent). A gain of Crystal City's size would not have produced a significant result in Long Island City, so the p-value says little about whether one occurred.

Two further features weaken the New York test. The 11101 fit is looser than the Crystal City fit (fit error 0.84 against 0.15) and rests on 3.2 effective donors. And three-month changes in 11101 are noisy. Relative to the middle New York ZIP, their standard deviation before the announcement is 2.41 log points, at percentile 93 of the pool. The metropolitan-level check is weaker still, because a change confined to one ZIP is diluted in the average and 12 comparison metros cannot produce a p-value below 0.077.

The five claims and the number that exposes each

Each of the five claims in the memo fails on a different point. Table 14 lists them; the subsections give the arithmetic.

Claim 1: a before-and-after change with no comparison

The 10.0 percent compares Crystal City with itself a year earlier, from 2018-10 to 2019-10. Over the same twelve months its synthetic control rose 4.1 percent and the median of the 304 other Washington-area comparison ZIPs rose 3.2 percent, so comparable ZIPs were rising too. In log terms the synthetic control accounts for 42 percent of the ZIP's rise. What is left after the comparison is 5.7 percent against the synthetic control and 6.6 percent against the median ZIP.

That remainder is not the announcement alone. The ring ZIPs average +2.5 percent, and the County Board vote of 2019-03-16 and the Virginia Tech site move of 2019-06-10 fall inside the 2018-10 to 2019-10 window, which the claim that nothing else changed ignores. The base month is not the problem. The mean gap over 2018-01 to 2018-10 is +0.0 percent, so 22202 shows no run-up before the announcement.

Claim 2: a premium assumed for Long Island City

The raw series for 11101 shows no announcement-window gain. The early gap is +0.01 percent against a placebo range of -1.9 to +2.2 percent (5th to 95th percentile), with a size-test p-value of 0.994. The ZIP stood +0.5 and +0.8 percent from 2018-10 in 2018-11 and 2018-12, against -0.3 and -0.6 percent for the median New York ZIP.

From 2018-10 to 2019-10 the ZIP changed by -4.7 percent against the +10.0 percent that the claimed premium implies, and its synthetic control changed by -0.3 percent. The claim is also inconsistent with itself. A premium of about 10 percent removed by the cancellation would return the ZIP to its October 2018 level, and the ZIP is 6.6 percent below it. The same terms of 25,000 jobs and $2.5 billion do not guarantee the same response. Crystal City's gap reached +1.6 percent three months in and +6.9 percent after fifteen, and the milestones behind that growth did not exist for Long Island City.

Claim 3: timing read from a smoothed series

The memo dates the loss to the cancellation using the smoothed condo index. The raw series for 11101 falls 2.5 percent from December 2018 to January 2019, in a value dated before the cancellation. The smoothed condo index falls 0.2 percent over the same month and the smoothed all-homes index 0.4 percent, because across 1,123 ZIPs the smoothed series follows raw changes 1 month late.

The raw fall of 2.50 log points is large against the one-month standard deviation of 0.64 log points for New York ZIPs relative to their metropolitan middle, so it is not noise alone. Monthly data cannot say whether it reflects news before 2019-02-14 or the way Zillow's estimates are timed. They also cannot separate the press reports of 2018-11-02 to 2018-11-05 from the official announcement of 2018-11-13, which all fall between the 2018-10-31 and 2018-11-30 observations.

Claim 4: the wrong unit of analysis

The memo's comparison is the New York metropolitan coefficient, -1.1 percent (p = 0.462). The same check gives -0.2 percent for Washington (p = 0.846), where the ZIP-level result for 22202 ranks 1 of 305. A metropolitan null cannot separate no change from a change in a small part of the metropolitan area. ZIP 11101 is one of 838 New York-metro ZIPs, 0.12 percent by count. With 12 comparison metros the smallest p-value the placebo ranking can give is 0.077.

The claim also answers a different question from the investor's. The parcel's neighbors are the ZIP and its 7 ring ZIPs, not the metropolitan area.

Claim 5: Long Island City does not follow the metropolitan area

Both rose from 2014-01 to 2018-10, ZIP 11101 by 45.3 percent and the metropolitan area by 25.3 percent, so 11101 grew 1.8 times as fast and ranks at percentile 91 of New York-metro ZIPs for growth over 2014-01 to 2017-12. Their changes do not move together. Over the months to 2018-10 the correlation of 12-month log changes between 11101 and the metropolitan area is -0.56 (46 overlapping windows, so far fewer independent observations) and the correlation of monthly changes is -0.20. Since 2018-10 the ZIP is -6.6 percent and the metropolitan area +2.4 percent.

The synthetic control matches 11101's own path, not the metropolitan average, and puts the average gap at -3.7 percent. The data do not show the ZIP following the metropolitan area, so they give no support to the claim that it will recover with it.

What an investor could and could not conclude

An investor could conclude that the typical home value in 11101 shows no announcement-window gain over its synthetic control (early gap +0.01 percent), and that the gap after January 2019 is negative (-3.7 percent on average) but no larger than gaps that placebo ZIPs show in 15 percent of cases (size p = 0.152; ratio p = 0.362). The smoothed condo series shows an early gain of +1.9 percent that fades, which fits a premium given back, and it also lies inside the placebo range (ratio p = 0.854).

An investor could not conclude that Long Island City gave back an HQ2 premium, and could not conclude that it did not. The test lacks the power to see a gain of Crystal City's size in a single New York ZIP. The placebo range for the early gap runs from -1.9 to +2.2 percent. The metropolitan check shows nothing at metropolitan scale, which is the expected result for a shock this size. Of the two treated ZIPs only Arlington's beats its placebos, and it cannot be separated from a wider Arlington trend or from later milestones.

A sell or hold decision on the parcel should not rest on these estimates alone. The home-value index describes existing homes, while the parcel's value depends on what can be built on it and at what cost; the estimates bear on the first, not the second. Evidence that would change the recommendation includes the parcel's own appraisals and comparable land trades, a transaction-level analysis of the ZIP and its ring, and rents or leasing activity for the site's neighbors.

What these data cannot show

ZHVI in small ZIPs is mostly model output. Zillow's size rank is 4,267 for 22202 and 1,464 for 11101, so these series reflect Zillow's model estimates for homes that did not sell, not observed sales. History is restated every month, so the values here are the 2026-10-01 estimate of 2014 to 2020, not what an investor saw at the time.

ZIP boundaries do not follow the sites. Of 109 sampled points in the announced Long Island City lots, 69 fall in 11101 and 40 in 11109, which has no ZHVI. The condo share of each ZIP is unknown. The all-homes value sits at position 0.55 between the condo and single-family values in 22202 and 0.07 in 11101, but that is a position between two values, not a share. The window ends with the U.S. national emergency of 2020-03-13 [2], so no estimate here says what the cancellation did to values after the pandemic began, or what Arlington's later construction did.

Evidence on individual homes near the sites is outside these data; the section on prices near the sites sets one study of such prices [1] next to the ZIP-level estimates. Greenstone, Hornbeck and Moretti compare counties that won a large plant with the runners-up for the same plant [3]; the analogue here would be the sites Amazon considered last, which the Zillow files do not identify. Abadie, Diamond and Hainmueller [4] and Abadie [5] set out the synthetic control method and its inference by placebo.

How the fits were audited

Each synthetic control is found by an optimizer (sequential least squares programming) that searches for the best weights, which must be non-negative and add to one. An optimizer can stop short of the best answer, so every fit records whether it converged. Across the Washington and New York fits, 501 of the 13,406 placebo fits (3.7%) did not converge. A fit that did not converge is reported and flagged, not replaced. The share is higher for condo series, which have fewer donors and thinner pools.

The two main fits, for 22202 and 11101, both converged. A second, independent solver (non-negative least squares) reproduces them, with a largest difference in the average gap of -2.3e-06 log points for 22202 and -1.4e-04 for 11101. The design (dates, comparison pools, outcome series, tests and the list of alternative setups) was written to a design file and fingerprinted before any gap was computed; the fingerprint and the other audit details are in the methods note in the appendix.

References

1. Chen, Y., Wilkoff, S. and Yoshida, J. (2024). Amazon is coming to town: sequential information revelation in the housing market. Real Estate Economics 52(2), 277-323. https://doi.org/10.1111/1540-6229.12457
2. White House (2020-03-13). Proclamation on declaring a national emergency concerning the novel coronavirus disease (COVID-19) outbreak. https://trumpwhitehouse.archives.gov/presidential-actions/proclamation-declaring-national-emergency-concerning-novel-coronavirus-disease-covid-19-outbreak/
3. Greenstone, M., Hornbeck, R. and Moretti, E. (2010). Identifying agglomeration spillovers: evidence from winners and losers of large plant openings. Journal of Political Economy 118(3), 536-598. https://doi.org/10.1086/653714
4. Abadie, A., Diamond, A. and Hainmueller, J. (2010). Synthetic control methods for comparative case studies: estimating the effect of California's tobacco control program. Journal of the American Statistical Association 105(490), 493-505. https://www.jstor.org/stable/29747059
5. Abadie, A. (2021). Using synthetic controls: feasibility, data requirements, and methodological aspects. Journal of Economic Literature 59(2), 391-425. https://www.aeaweb.org/articles?id=10.1257/jel.20191450
6. Zillow Research. Housing data (ZHVI and ZORI downloads). https://www.zillow.com/research/data/
7. Zillow Research. Zillow Home Value Index methodology, 2019 revision: getting under the hood. https://www.zillow.com/research/zhvi-methodology-2019-deep-26226/
8. Zillow Research. Methodology: Zillow Observed Rent Index (ZORI). https://www.zillow.com/research/methodology-zori-repeat-rent-27092/
9. Amazon (2017-09-07). Amazon opens search for Amazon HQ2, a second headquarters city in North America. Press release. https://press.aboutamazon.com/2017/9/amazon-opens-search-for-amazon-hq2-a-second-headquarters-city-in-north-america
10. Amazon (2018-01-18). Amazon announces candidates for HQ2. Press release. https://press.aboutamazon.com/news-releases/news-release-details/amazon-announces-candidates-hq2/
11. Amazon (2018-11-13). Amazon selects New York City and Northern Virginia for new headquarters. Company news. https://www.aboutamazon.com/news/company-news/amazon-selects-new-york-city-and-northern-virginia-for-new-headquarters
12. Amazon (2019-02-14). Update on plans for New York City headquarters. Company news. https://www.aboutamazon.com/news/company-news/update-on-plans-for-new-york-city-headquarters