Experimental Methods

Introduction

Experimental methods are research designs in which the investigator explicitly and intentionally induces exogenous variation in the uptake of the program to be evaluated. Experimental methods, such as Randomized Control Trials, are typically considered the gold standard design for impact evaluation, since by construction the takeup of the treatment is uncorrelated with other characteristics of the treated population. Under these conditions, it is always possible for the analyst to construct a regression model in which the estimate of the treatment effect is unbiased.

The Power of Experimental Methods

Specifically, non-experimental methods leave the researcher with a dataset where an unknown combination of factors are involved in the true data-generating process of the outcome variable of interest. Estimating marginal effects of any of these factors on the outcome itself (such as schooling on earnings, for example), leaves open two key avenues for biased estimates.

First, the estimate may be confounded, in the sense that it masks an effect produced reality by another, correlated variable. For example, schooling may improves the quality of job offers via network exposure, but the actual education adds no value. In this case the result remains "correct" in the sense that those who got more schooling got higher earnings, but "incorrect" in the sense that the estimate is not the marginal value of education.

Second, the direction of causality may be reversed or simultaneous. For example, individuals who are highly motivated may choose to complete more years of schooling as well as being more competent at work in general; or those who are highly motivated by financial returns in the workplace may choose more schooling because of that motivation.

Experimental variation solves these problems by imposing a known variation on the study population. This guarantees that the intervention effect is not confounded (since it is not correlated with any external variable) and that causality is identified, since selection into the randomization is not possible. However, this leads to natural concerns about the structure of differential takeup and attrition in a randomization setting which must be addressed in every sample where noncompliance is a possibility.