Survey Pilot

What is a survey pilot?

• A field test of the survey instrument and all survey protocols
• Piloting is not just about the questionnaire: test all protocols and learn about relevant logistics
• Piloting is done before Enumerator Training. It is not the same as the field practice all enumerators do at the end of Training!! It typically involves significant changes to the survey instrument and/or protocols – which should never be made after enumerator training!
• A complete survey pilot includes 3 stages (see Figure 1)

Figure 1: A Complete Survey Pilot Includes 3 Stages

Stage 1 - Pre-Pilot	Stage 2 - Content-focused Pilot	Stage 3 - Data-focused Pilot
Answer broad questions about survey design and context through qualitative interviews and focus group discussions	Refine overall order and structure, wording of specific questions, and translations. Check completeness of answer choice options, response variance, survey length.	Validate programming, export a sample dataset, check dataset structure and completeness, test all data quality checks
Early, printable draft, and/or qualitative instruments	A translated, printable, complete draft	A translated, programmed, final draft
Pen-and-Paper	Pen-and-Paper	Tablet/Phone-based

Are all 3 stages of piloting necessary for every survey?

• Not necessarily. If your survey is …
  • … a brand new survey instrument? 
    • --> always start with Stage 1 – Pre-Pilot
  • … an adaptation of a well-designed questionnaire from reliable source in the same country?
    • --> may start with Stage 2 – Content-focused pilot
  • … an adaption of a survey instrument from a previous data collection for the same project?
    • If significant revisions or additions
      • --> start with Stage 2
    • If no major changes
      • --> skip to Stage 3 – Data -focused pilot

Why do a pen-and-paper pilot for a CAPI survey?

• Paper questionnaires are a more flexible way to record answers and qualitative observations
  • Record open-ended responses (critical for a pilot) more quickly / easily
  • Draw lines and arrows between questions to suggest restructuring
  • Record their observations and feedback in the margins
  • Make notes of questionnaire wording or translation problems directly in the text
• A good pilot will provide significant inputs to questionnaire design, and result in significant changes to content and structure. Changing programming is time-consuming and can create bugs (e.g. if order of questions shifts and all skip codes need to be re-programmed)

What is the timeline for a survey pilot?

• Piloting should start 4-6 months before survey launch. Indicative times are shown in Figure 2. This will vary depending on the complexity of the survey instruments. The earlier you start the better!
  • Remember: pilot needs to be done in time for final questionnaire to get ethical approval
• Pre-pilot happens before the questionnaire is translated to the local language(s). Good translation is time-consuming (for details see Guidelines for Questionnaire Translation). Making major changes after translation is costly and often leads to version control problems between different languages.

Figure 2: Fitting Piloting into the Overall Survey Timeline

Pre-Pilot	1-2 weeks
Questionnaire Design	4 weeks
Questionnaire Translation	2-3 weeks
Content-focused pilot (& revisions)	2-3 weeks
IRB approval	{context dependent, concurrent to programming & data pilot}
Questionnaire Programming	4-6 weeks
Data-focused Pilot	2-3 weeks
Enumerator Training	2 weeks

Beyond the questionnaire, what should be tested during the pilot?

• All Field Protocols should be tested in Stage 2 – Content-focused pilot. Revise as needed, and then pilot again during Stage 3 – Data-focused pilot. All protocols should be finalized before enumerator training.
• Interview scheduling
  • Is there a particular time of day when respondents are more / less likely to be available?
  • What is the most effective way to communicate with respondents and schedule appointments?
• What infrastructure will field teams have access to?
  • Electricity? If blackouts are frequent, are there generators (and fuel)?
  • Will there be any internet access?
  • How is the coverage of mobile phone networks?
• Does the sampling protocol work well in the field? (for more details on developing the protocol, see Guidelines for Impact Evaluation Sampling)
  • What challenges are enumerators likely to face? What clarifications need to be made?
  • If working from an existing sampling frame, is it up to date?
  • Is the replacement strategy effective?
  • If doing a listing, how much time should be built into the field plan? How will you ensure that no sampling units (e.g. households, firms, traders) are skipped?
• Geo-data: Will you use the tablet or separate GPS?
  • If GPS unit, test protocol for saving with filenames that include unique ID linked to respondent.
  • How long does it take to lock in a signal? How accurate is the data point?