The empirical evaluation of policy effects typically relies on cross-regional (or group) variation in the time of policy implementation. In that context, an ideal scenario is one in which two regions that would be otherwise identical in terms of an outcome variable of interest differ in that a “treated” region implements a policy at time tp and a “control” region does not. In that setting, where absent of policy these two regions are identical, the difference in the outcome variable that emerges across regions after tp is the effect of policy. In the BSE Working Paper No. 1209, “Evaluating the Effectiveness of Policies Against a Pandemic,” Christian Alemán, Christopher Busch, Alexander Ludwig and Raül Santaeulàlia-Llopis propose a new methodology where the standardly used variation in the time of policy implementation is not necessary to evaluate the effectiveness of policy. The new methodology proposed by the authors exploits cross-regional stage variation of the outcome variable of interest to identify policy effects even in instances where the policy is implemented at the same time across all regions.
Applying this new empirical approach to the Spanish nationwide stay-home policy against Covid-19, the authors estimate that this policy saved 15.9% of lives relative to the number of deaths that would have occurred without policy intervention. In addition, the authors show that early implementation in terms of the epidemic stages is critical: A one-week delay reduced the effectiveness of the stay-home policy to less than one fourth.
Uncovering stage variation across regions
The novel key insight of the proposed approach is that stage variation of the outcome variable of interest can exist across regions even at the same calendar time. For example, in the context of an epidemic with the outcome variable of interest being the flow of deaths, there exists cross-regional variation in the stage of the epidemic: namely, at a given point in time there is heterogeneity in how far regions moved along an epidemic path in terms of the flow of deaths. This is illustrated in Panel (a) of Figure 1, which shows a stylized epidemic path in terms of the daily flow of deaths associated with an epidemic. Imagine that two regions go through the same path. If the epidemic in region C starts earlier than in region T, then at the same calendar time t the epidemic is in a more advanced stage in region C.
Identification of policy effects: constructing the counterfactual epidemic dynamics
In reality, there is not only heterogeneity in the calendar date at which an epidemic starts. Precisely, epidemics can differ across regions in several dimensions—such as the speed of disease diffusion and the magnitude (or overall death toll). Panel (b) of Figure 1 illustrates the time-path of the epidemic of two example regions, where policy is effective at time tp in both, but the epidemic in region C starts earlier, evolves faster and has a larger magnitude than in region T.
The proposed method consists of normalizing the time, speed and magnitude of the epidemic paths across regions. In particular, the normalization ensures that all regions share the same normalized epidemic path before the effective policy date tp. This is illustrated in panel (c) of Figure 1. In this manner, the normalization unveils an interval D(s)—the orange shaded area in Panel (c)—within which the epidemic path in region T is affected by the policy but the epidemic path of region C is not. Accordingly, on this interval, the path of C describes the counterfactual path for T —i.e., the flow of deaths region T would have experienced had the policy reform not been implemented—and the difference between the paths across the two regions can thus be used to estimate the policy effect.
Application to the COVID-19 epidemic in Spain
In their application of the methodology to the Covid-19 epidemic in Spain, the authors estimate the effects of the stay-home policies, the ‘lockdown’ of Spain, which was enacted on March 14 and which lasted until May 2, when the strictest measures were lifted. The date of policy reform tp accordingly corresponds to March 14. On the basis of reports from the Instituto de Salud Carlos III stating that it takes on average between twelve to twenty-three days from infection to death, the authors choose March 26 as the effective policy date tp +τ where τ = 12 is an effectiveness lag. Likewise, since the strictest measures were lifted on May 2, taking the same effectiveness lag of twelve days into consideration the policy was effective until May 14. To summarize, the period of analysis covers March 26 until May 14.
The authors base their analysis on regional daily deaths data provided by the Ministry of Health. The first step to implement the novel method is to identify a leading region (C). The authors use Madrid as the counterfactual region for the epidemic dynamics in all other regions (region C), and they construct one treatment region T, the Rest of Spain. The use of Madrid as leading region is corroborated by the normalization procedure implemented in the paper. In particular, the normalization applied to the epidemic dynamics of Madrid and the Rest of Spain uncovers an overlap interval D(s) of seven days during which the epidemic dynamics in the Rest of Spain are subject to the policy, while the dynamics in Madrid are not (yet). This means that the last week before the policy treatment in Madrid is informative about how the (normalized) epidemic path would have looked in the rest of Spain during the first week of the policy—if the policy had not been implemented. Thus, the distance between the time path of the flow of deaths in Madrid and the normalized time path in the rest of Spain can be used to measure the effect of policy on the flow of death. The authors use this procedure to obtain an estimate of the percentage and the number of lives saved, that is, the difference between the counterfactual deaths that would have occurred without policy intervention and the actual number of deaths.
Their estimate of the percentage of lives saved on this overlap interval is 18.7%, which translates into 1,074 lives in the rest of Spain. Extrapolating this estimate of the percentage effect of the reduction of deaths from the seven days overlap interval until the end of the observation period on May 14 results in an estimate of 3,786 lives saved in the rest of Spain.
Notes: Andalucía (AND), Aragón (ARA), Asturias (AST), Baleares (BAL), Canarias (CAN), Cantabria (CNT), Castilla-La Mancha (CLM), Castilla y Leon (CLL), Catalunya (CAT), Ceuta (CEU), Valencia (VAL), Extremadura (EXT), Galicia (GAL), Madrid (MAD), Melilla (MEL), Murcia (MUR), Navarra (NAV), País Vasco (PVC), La Rioja (RIO). Three subgroups (grey circles): Early-stage implementers (G1); Mid-stage implementers (G2); and Late-stage implementers (G3). The size of the yellow and grey circles is the stock of deaths per thousand inhabitants accumulated during the overlap period.
How much does the stage of the epidemic matter for the effectiveness of policy?
The authors find that regions in which the stay-home policy is implemented in earlier stages of the epidemic benefit the most in terms of the percentage of lives saved. This is summarized in Figure 2, which shows the region-specific effects of the policy by stage in the respective region. The size of the region-specific circles reflects the population size in the respective region. Figure 2 also shows the flow of deaths in Madrid to illustrate the dynamics in the respective regions at the stage of policy implementation. For instance, in terms of their normalized epidemic dynamics Aragon and the Baleares looked like Madrid around March 16 and the flows of deaths in those regions were still convexly increasing whereas the flow of deaths in Valencia looked like the one in Madrid on March 24 and was already close to the peak when the policy became effective. As a further illustration of the importance of the stage of policy implementation Figure 2 also shows three aggregate regions: early-stage (G1), mid-stage (G2) and late-stage (G3) policy implementers. The authors’ results become quite intuitive with this figure: the earlier the stage at which the policy becomes effective in reducing the flow of deaths, the higher is its percentage impact on the flow of deaths because without policy intervention at earlier stages the epidemic dynamics feature a steeper increase in the flow of deaths. In particular, the authors estimate that at early stages (G1) the policy is four times more effective than at lager stages (G3).
Finally, the authors extrapolate the estimates to compute the number of lives saved in Spain. To this purpose they use the above variation to estimate the policy effect in Madrid, yielding 4.1% or 237 lives saved. Adding this to the number of lives saved in the rest of Spain gives a total of 4,029, or 15.9%, lives saved for Spain.
Concluding discussion
The authors develop a novel empirical approach to evaluate the effectiveness of policy. The approach uncovers heterogeneity across regions in the stage of the underlying dynamics of an outcome variable of interest at any calendar time. The policy effect is identified by explicitly using the uncovered stage variation at the time of policy implementation. The proposed methodology is therefore not restricted to the existence of variation at the time of policy implementation.
The authors apply their novel method to estimate the effects of the strict stay-home policies enacted during the first wave of the Covid-19 pandemic in Spain during March 14 and May 2. They show that this policy saved about 4,029 lives. There is considerable variation in this aggregate effect across disaggregated regions of Spain and the effectiveness of policy in reducing the flow of deaths declines rapidly across epidemic stages—dropping by three fourths in the matter of a week.
