Pre-Analysis Plan
A pre-analysis plan (PAP) is a document produced at the design stage of an impact evaluation that sets out in advance how the researcher will analyze data. While the main objective of a PAP is to prevent data mining and specification searching, it can also help the researcher think through questionnaire design and, once data is collected, make data analysis much quicker and easier. This page will briefly summarize the pros and cons of a PAP, outline its elements, and provide additional resources.
Read First
- PAPs increase credibility of research findings and help researchers to fine tune their analysis strategy.
- PAPs may tie researcher hands to ex ante analysis plans and limit opportunities of exploratory learning.
Overview
While most economics journals do not currently require PAPs as a condition for publication, researchers may choose to produce a PAP prior to data analysis to: (i) increase the credibility of their findings, and (ii) fine tune their analysis strategy.
While PAPs may reduce the prevalence of spurious results, they threaten to formally tie researcher hands to ex ante analysis plans that may limit the potential of exploratory learning. In Benjamin Olken’s paper on the costs and benefits associated with fully pre-specifying the analysis for a development economics RCT, he notes that "forcing all papers to be fully pre-specified from start to end would likely results in simpler papers, which could potentially lose some of the nuance of current work," though "in many contexts, pre-specification of one (or a few) key primary outcome variables, statistical specifications, and control variables offers a number of advantages."
Elements
In A Pre-Analysis Plan Checklist, David McKenzie outlines the elements to include in a PAP: a description of the sample to be used in the study, key data sources, hypotheses to be tested throughout the causal chain, a specification of how variables will be constructed and how the treatment effect equation will be estimated, a plan for dealing with multiple outcomes and multiple hypothesis testing, procedures to address survey attrition, methods to deal with outcomes with limited variation, and, if you are going to be testing a model, the model. Alejandro Ganimian’s pre-analysis plan template further lays out the elements to include in a PAP. It is available in .doc and .tex versions here.
Back to Parent
This article is part of the topic Research Ethics
Additional Resources
- You can find 13 examples of pre-analysis plans at the JPAL Hypothesis Registry.
- Visit The Berkeley Initiative for Transparency in the Social Sciences (BITSS) dedicated page for pre-analysis plans.
- Previously mentioned on this page, and worth repeating: David McKenzie's pre-analysis plan checklist, and Benjamin Olken's Promises and Perils of Pre-Analysis Plans.
- This Berkeley Initiative for Transparency in the Social Sciences (BITSS) module discusses the advantages of registration and explores the debate over how useful pre-analysis plans are..