A pre-analysis plan (PAP) lays out how the researcher will analyze data, at the design stage of an impact evaluation. The objective of a PAP is to prevent data mining and specification searching.
While most economics journals do not currently require PAPs as a condition for publication, researchers may choose to produce a PAP prior to data analysis to: (i) increase the credibility of their findings; and (ii) help researchers finetune their analysis strategy.
While PAPs provide the benefit of potentially reducing the prevalence of spurious results, this comes at the cost of tying researcher hands more formally to ex ante analysis plans that may limit the potential of exploratory learning. Benjamin Olken provides a summary of the costs and benefits associated with fully pre-specifying the analysis for a development economics RCT . He notes that "forcing all papers to be fully pre-specified from start to end would likely results in simpler papers, which could potentially lose some of the nuance of current work", but that "in many contexts, pre-specification of one (or a few) key primary outcome variables, statistical specifications, and control variables offers a number of advantages".
The Berkeley Institute for Transparency in Social Sciences prepared a template for * Pre-Analysis Plans, which provides the overall structure and guidance on what details to include (.doc and .tex formats available).
We recommend also consulting this Pre-analysis plan checklist from the Development Impact Blog.
You can find 13 examples of pre-analysis plans at the JPAL Hypothesis Registry.
Back to Parent
This article is part of the topic Research Ethics
- Olken, Benjamin A.. 2015. "Promises and Perils of Pre-analysis Plans." Journal of Economic Perspectives, 29(3): 61-80.