Survey Pilot

Guidelines for a Survey Pilot

What is a survey pilot?

• A field test of the survey instrument and all survey protocols
• Piloting is not just about the questionnaire: test all protocols and learn about relevant logistics
• Piloting is done before Enumerator Training. It is not the same as the field practice all enumerators do at the end of Training!! It typically involves significant changes to the survey instrument and/or protocols – which should never be made after enumerator training!
• A complete survey pilot includes 3 stages (see Figure 1)

Stage 1 - Pre-Pilot	Stage 2 - Content-focused Pilot	Stage 3 - Data-focused Pilot
Answer broad questions about survey design and context through qualitative interviews and focus group discussions	Refine overall order and structure, wording of specific questions, and translations. Check completeness of answer choice options, response variance, survey length.	validate programming, export a sample dataset, check dataset structure and completeness, test all data quality checks
Early, printable draft, and/or qualitative instruments	A translated, printable, complete draft	A translated, programmed, final draft
Pen-and-Paper	Pen-and-Paper	Tablet/Phone-based

• Are all 3 stages of piloting necessary for every survey? Not necessarily. If your survey is …
  • … a brand new survey instrument?  always start with Stage 1 – Pre-Pilot
  • … an adaptation of a well-designed questionnaire from reliable source in the same country?
    • --> may start with Stage 2 – Content-focused pilot
  • … an adaption of a survey instrument from a previous data collection for the same project?
    • If significant revisions or additions
      • -->  start with Stage 2
    • If no major changes
      • --> skip to Stage 3 – Data -focused pilot

• Early stage pilots should be paper-based, even for CAPI surveys
  • Paper questionnaires are a more flexible way to record answers and qualitative observations
    • Record open-ended responses (critical for a pilot) more quickly / easily
    • Draw lines and arrows between questions to suggest restructuring
    • Record their observations and feedback in the margins
    • Make notes of questionnaire wording or translation problems directly in the text
  • A good pilot will provide significant inputs to questionnaire design, and result in significant changes to content and structure. Changing programming is time-consuming and can create bugs (e.g. if order of questions shifts and all skip codes need to be re-programmed)