Difference between revisions of "Quasi-Experimental Methods"

Jump to: navigation, search
(20 intermediate revisions by 5 users not shown)
Line 1: Line 1:
== Introduction ==
Quasi-experimental methods are research designs that that aim to identify the impact of a particular intervention, program or event (a "treatment") by comparing treated units (households, groups, villages, schools, firms, etc.) to control units. While quasi-experimental methods use a control group, they differ from [[Experimental Methods | experimental methods]] in that they do not use [[Randomization in Stata | randomization]] to select the control group. Quasi-experimental methods are useful for estimating the impact of a program or event for which it is not [[Research Ethics | ethically]] or logistically feasible to randomize. This page outlines common types of quasi-experimental methods.


Like [[Experimental Methods]], quasi-experimental research methods are intended to satisfy the conditions where a researcher can construct a model where exposure to a treatment effect is uncorrelated with other characteristics. Unlike in experimental research methods, the investigator does not have direct control over the exposure, and instead uses "natural experiments" such as [[Regression Discontinuity]] designs to identify existing circumstances where random assignment of some treatment took place.
==Read First==
*Common examples of quasi-experimental methods include [[Difference-in-Differences | difference-in-differences]], [[Regression Discontinuity | regression discontinuity design]], [[Instrumental Variables | instrumental variables]] and [[Matching | matching]]. For more details on each type, please visit their respective pages.
*In general, quasi-experimental methods require larger sample sizes and more assumptions than experimental methods in order to provide valid and unbiased estimates of program impacts.


== The Power of Quasi-Experimental Methods ==
== Overview ==
Like experimental methods, quasi-experimental methods aim to estimate program effects free of confoundedness, reverse causality or simultaneous causality. While quasi-experimental methods use a counterfactual, they differ from [[Experimental Methods | experimental methods]] in that they do not [[Randomization in Stata | randomize]] treatment assignment. Instead, quasi-experimental methods exploit existing circumstances in which treatment assignment has a sufficient element of randomness, as in [[Regression Discontinuity | regression discontinuity design]] or event studies; or simulate an experimental counterfactual by constructing a control group as similar as possible to the treatment group, as in [[Propensity Score Matching|propensity score matching]]. Other examples of quasi-experimental methods include [[Instrumental Variables | instrumental variables]] and [[Difference-in-Differences | difference-in-differences]].


Like experimental methods, quasi-experimental methods aim to solve the problems of [[confoundedness]] and [[reverse or simultaneous causality]] in identifying the marginal effect of a change in condition. Unlike [[experimental methods]], quasi-experiments tend to be based on policy changes or natural events that were not actually randomly assigned, but occurred in such a way that they can be considered "[[as good as randomly assigned]]" relative to some of the outcomes under consideration. Two examples illustrate these differences well.
==Assumptions and Limitations==
In general, quasi-experimental methods require larger samples than experimental methods. Further, for quasi-experimental methods to provide valid and unbiased estimates of program impacts, researchers must make more assumptions about the control group than in experimental methods. For example, difference-in-differences relies on the equal trends assumption (see [[Difference-in-Differences]] for more details), while matching assumes identical unobserved characteristics between the treatment and control groups.


First, consider a time-based shock that can potentially apply to all people participating in a given market, such as joining a job guarantee program. [[Regression Discontinuity]] and [[Event Study]] designs are natural choices in such a situation, but comparing across time (ie, before and after the join date) at the individual level is inappropriate, since individuals may have joined the program due to other factors that would have affected their income, such as job market seasonality, business failure, or family changes. This type of [[reverse causality]] confounds the effect of the estimate because the true [[counterfactual]] cannot be estimated.
== Additional Resources ==
* Robert Michael's slides on [http://www.indiana.edu/~educy520/sec6342/week_05/quasi_designs_2up.pdf Quasi-Experimental Designs]
*Gertler et al.’s [https://siteresources.worldbank.org/EXTHDOFFICE/Resources/5485726-1295455628620/Impact_Evaluation_in_Practice.pdf Impact Evaluation in Practice]


Instead, it is necessary to use a population-level cutoff such as the implementation of the program in an area [[differences-in-differences]] or [[staged rollout]] designs, or, if only one area is implementing the policy, [[synthetic control]] methods or the assumption that events occurring on either side of the policy cutoff could not manipulate on which side of the cutoff they occurred (the [[running variable]]).
[[Category: Quasi-Experimental Methods ]]
 
Second, consider a natural disaster that has a various impact across space, such as an earthquake with a random fault/epicenter activation. Assuming that there was no way for households to foresee the timing or location of the earthquake, and that there was no serious [[differential attrition]] on critical dimensions, it can be used as a natural experiment on outcomes later in life for affected individuals. However, for the input component to be truly [[exogenous]], something like the distance to the faultline should be used as the treatment variable - even though estimates of local magnitude may be available.
 
This is because the local magnitude may be correlated with other varying characteristics (such as soil quality or hilliness) that also contribute to the outcomes (such as family income) and therefore re-introduce the confounding problem by including a non-random component that is correlated to a key characteristic. Distance from a randomly placed disaster, though a less precise measure of "exposure", is critically uncorrelated with almost any imaginable outcome under the correct assumption (unlike, for example, distance to a volcano, or to a tsunami-affected shoreline).

Revision as of 07:03, 12 February 2020

Quasi-experimental methods are research designs that that aim to identify the impact of a particular intervention, program or event (a "treatment") by comparing treated units (households, groups, villages, schools, firms, etc.) to control units. While quasi-experimental methods use a control group, they differ from experimental methods in that they do not use randomization to select the control group. Quasi-experimental methods are useful for estimating the impact of a program or event for which it is not ethically or logistically feasible to randomize. This page outlines common types of quasi-experimental methods.

Read First

Overview

Like experimental methods, quasi-experimental methods aim to estimate program effects free of confoundedness, reverse causality or simultaneous causality. While quasi-experimental methods use a counterfactual, they differ from experimental methods in that they do not randomize treatment assignment. Instead, quasi-experimental methods exploit existing circumstances in which treatment assignment has a sufficient element of randomness, as in regression discontinuity design or event studies; or simulate an experimental counterfactual by constructing a control group as similar as possible to the treatment group, as in propensity score matching. Other examples of quasi-experimental methods include instrumental variables and difference-in-differences.

Assumptions and Limitations

In general, quasi-experimental methods require larger samples than experimental methods. Further, for quasi-experimental methods to provide valid and unbiased estimates of program impacts, researchers must make more assumptions about the control group than in experimental methods. For example, difference-in-differences relies on the equal trends assumption (see Difference-in-Differences for more details), while matching assumes identical unobserved characteristics between the treatment and control groups.

Additional Resources