This post does a nice job summarizing a recent meta-analysis of lockdown studies.
If you’re not familiar with meta-analyses, they are a study of studies: researchers gather a group of studies an area, like lockdowns, and statistically analyze the collective results. This helps us understand what the field really knows, rather than relying on that one study that was trumpeted by the media.
One the most shocking findings is that of the over 1,000 lockdown studies they examined, only 34 used a statistical approach, called diff-in-diff, that comes close to estimating a causal effect!
They screened over 18,000 studies, most of which weren’t related to the narrow lockdown efficacy question. 1,048 studies remained, where most were excluded for not answering the two core eligibility questions:
Does the study measure the effect of lockdowns on mortality?
Does the study use an empirical diff-in-diff approach?
Of the 117 studies that remain, the authors exclude 83 that were duplicates, used modeling, or synthetic controls. Structural-break studies weren’t enough, the authors argue, “as the effect of lockdowns in these studies might contain time-dependent shifts, such as seasonality.”
34 studies thus make it into their analysis, and they are divided into three segments: mortality impacts associated with the stringency of Covid policies (following the much-publicized Oxford metric); Shelter-in-Place studies; and studies that target specific non-pharmaceutical interventions.
Diff-in-diff (short for difference-in-difference) is widely used by economists and evaluators to study phenomena when you can’t randomize treatment. The key idea is that we can follow two groups of units, like cities, states or countries, over time, before and after some of the units institute a lockdown. We can compare the before and after change for the treated units to the before and after change for the untreated units – if the change is similar, then the treatment likely did not have an effect.
Their findings?
The authors are pretty severe in their final conclusions. Lockdowns didn’t meaningfully reduce Covid-19 mortalities: “the effect is little to none.”
