Difference between revisions of "Survey Pilot"

Jump to: navigation, search
Line 1: Line 1:


= Guidelines for a Survey Pilot =
= Guidelines for a Survey Pilot =
== What is a survey pilot? ==
= What is a survey pilot? =


* A field test of the survey instrument and all survey protocols  
* A field test of the survey instrument and all survey protocols  
* Piloting is not just about the questionnaire: test all protocols and learn about relevant logistics
* '''Piloting is not just about the questionnaire''': test all protocols and learn about relevant logistics
* Piloting is done before Enumerator Training. It is not the same as the field practice all enumerators do at the end of Training!! It typically involves significant changes to the survey instrument and/or protocols – which should never be made after enumerator training!  
* '''Piloting is done before Enumerator Training'''. It is not the same as the field practice all enumerators do at the end of Training!! It typically involves significant changes to the survey instrument and/or protocols – which should never be made after enumerator training!  
* A complete survey pilot includes 3 stages (see Figure 1)
* A complete survey pilot includes '''3 stages''' (see Figure 1)


{| class="wikitable"
{| class="wikitable" style="margin-left: auto; margin-right: auto; border: none;"
|+ Figure 1: A Complete Survey Pilot Includes 3 Stages
|
|-
|-
! Stage 1 - Pre-Pilot
! Stage 1 - Pre-Pilot
Line 15: Line 17:
|-
|-
| Answer broad questions about survey design and context through qualitative interviews and focus group discussions
| Answer broad questions about survey design and context through qualitative interviews and focus group discussions
| Refine overall order and structure, wording of specific questions, and translations. Check completeness of answer choice options, response variance, survey length.  
| Refine overall order and structure, wording of specific questions, and translations.  
Check completeness of answer choice options, response variance, survey length.  
| Validate programming, export a sample dataset, check dataset structure and completeness, test all data quality checks
| Validate programming, export a sample dataset, check dataset structure and completeness, test all data quality checks
|-
|-
Line 27: Line 30:
|}
|}


* Are all 3 stages of piloting necessary for every survey? Not necessarily. If your survey is …  
== Are all 3 stages of piloting necessary for every survey? ==
* Not necessarily. If your survey is …  
** … a brand new survey instrument?   
** … a brand new survey instrument?   
*** always start with Stage 1 – Pre-Pilot
*** --> always start with Stage 1 – Pre-Pilot
** … an adaptation of a well-designed questionnaire from reliable source in the same country?
** … an adaptation of a well-designed questionnaire from reliable source in the same country?
*** --> may start with Stage 2 – Content-focused pilot
*** --> may start with Stage 2 – Content-focused pilot
Line 38: Line 42:
**** --> skip to Stage 3 – Data -focused pilot
**** --> skip to Stage 3 – Data -focused pilot


* Early stage pilots should be paper-based, even for CAPI surveys
== Why do a pen-and-paper pilot for a CAPI survey? ==
** Paper questionnaires are a more flexible way to record answers and qualitative observations
* Paper questionnaires are a more flexible way to record answers and qualitative observations
*** Record open-ended responses (critical for a pilot) more quickly / easily
** Record open-ended responses (critical for a pilot) more quickly / easily
*** Draw lines and arrows between questions to suggest restructuring
** Draw lines and arrows between questions to suggest restructuring
*** Record their observations and feedback in the margins
** Record their observations and feedback in the margins
*** Make notes of questionnaire wording or translation problems directly in the text
** Make notes of questionnaire wording or translation problems directly in the text
** A good pilot will provide significant inputs to questionnaire design, and result in significant changes to content and structure. Changing programming is time-consuming and can create bugs (e.g. if order of questions shifts and all skip codes need to be re-programmed)
* A good pilot will provide significant inputs to questionnaire design, and result in significant changes to content and structure. Changing programming is time-consuming and can create bugs (e.g. if order of questions shifts and all skip codes need to be re-programmed)
 
 
= What is the timeline for a survey pilot? =
* Piloting should start 4-6 months before survey launch. Indicative times are shown in Figure 2. This will vary depending on the complexity of the survey instruments. The earlier you start the better!
** Remember: pilot needs to be done in time for final questionnaire to get ethical approval
* Pre-pilot happens before the questionnaire is translated to the local language(s). Good translation is time-consuming (for details see Guidelines for Questionnaire Translation). Making major changes after translation is costly and often leads to version control problems between different languages.
 
{| class="wikitable" style="margin-left: auto; margin-right: auto; border: none;"
|+ Figure 2: Fitting Piloting into the Overall Survey Timeline
|
 
|-
| Pre-Pilot
| 1-2 weeks
|-
| Questionnaire Design
| 4 weeks
|-
| Questionnaire Translation
| 2-3 weeks
|-
| Content-focused pilot (& revisions)
| 2-3 weeks
|-
| IRB approval
| {context dependent, concurrent to programming & data pilot}
|-
| Questionnaire Programming
| 4-6 weeks
|-
| Data-focused Pilot
| 2-3 weeks
|-
| Enumerator Training
| 2 weeks
|}
 
 
= Beyond the questionnaire, what should be tested during the pilot? =
* All Field Protocols should be tested in Stage 2 – Content-focused pilot. Revise as needed, and then pilot again during Stage 3 – Data-focused pilot. All protocols should be finalized before enumerator training.
* Interview scheduling
** Is there a particular time of day when respondents are more / less likely to be available?
** What is the most effective way to communicate with respondents and schedule appointments?
* What infrastructure will field teams have access to?
** Electricity? If blackouts are frequent, are there generators (and fuel)?
** Will there be any internet access?
** How is the coverage of mobile phone networks? 
* Does the sampling protocol work well in the field? (for more details on developing the protocol, see Guidelines for Impact Evaluation Sampling)
** What challenges are enumerators likely to face? What clarifications need to be made?
** If working from an existing sampling frame, is it up to date?
** Is the replacement strategy effective?
** If doing a listing, how much time should be built into the field plan? How will you ensure that no sampling units (e.g. households, firms, traders) are skipped?
* Geo-data: Will you use the tablet or separate GPS?
** If GPS unit, test protocol for saving with filenames that include unique ID linked to respondent.
** How long does it take to lock in a signal? How accurate is the data point?

Revision as of 16:57, 6 January 2017

Guidelines for a Survey Pilot

What is a survey pilot?

  • A field test of the survey instrument and all survey protocols
  • Piloting is not just about the questionnaire: test all protocols and learn about relevant logistics
  • Piloting is done before Enumerator Training. It is not the same as the field practice all enumerators do at the end of Training!! It typically involves significant changes to the survey instrument and/or protocols – which should never be made after enumerator training!
  • A complete survey pilot includes 3 stages (see Figure 1)
Figure 1: A Complete Survey Pilot Includes 3 Stages
Stage 1 - Pre-Pilot Stage 2 - Content-focused Pilot Stage 3 - Data-focused Pilot
Answer broad questions about survey design and context through qualitative interviews and focus group discussions Refine overall order and structure, wording of specific questions, and translations.

Check completeness of answer choice options, response variance, survey length.

Validate programming, export a sample dataset, check dataset structure and completeness, test all data quality checks
Early, printable draft, and/or qualitative instruments A translated, printable, complete draft A translated, programmed, final draft
Pen-and-Paper Pen-and-Paper Tablet/Phone-based

Are all 3 stages of piloting necessary for every survey?

  • Not necessarily. If your survey is …
    • … a brand new survey instrument? 
      • --> always start with Stage 1 – Pre-Pilot
    • … an adaptation of a well-designed questionnaire from reliable source in the same country?
      • --> may start with Stage 2 – Content-focused pilot
    • … an adaption of a survey instrument from a previous data collection for the same project?
      • If significant revisions or additions
        • --> start with Stage 2
      • If no major changes
        • --> skip to Stage 3 – Data -focused pilot

Why do a pen-and-paper pilot for a CAPI survey?

  • Paper questionnaires are a more flexible way to record answers and qualitative observations
    • Record open-ended responses (critical for a pilot) more quickly / easily
    • Draw lines and arrows between questions to suggest restructuring
    • Record their observations and feedback in the margins
    • Make notes of questionnaire wording or translation problems directly in the text
  • A good pilot will provide significant inputs to questionnaire design, and result in significant changes to content and structure. Changing programming is time-consuming and can create bugs (e.g. if order of questions shifts and all skip codes need to be re-programmed)


What is the timeline for a survey pilot?

  • Piloting should start 4-6 months before survey launch. Indicative times are shown in Figure 2. This will vary depending on the complexity of the survey instruments. The earlier you start the better!
    • Remember: pilot needs to be done in time for final questionnaire to get ethical approval
  • Pre-pilot happens before the questionnaire is translated to the local language(s). Good translation is time-consuming (for details see Guidelines for Questionnaire Translation). Making major changes after translation is costly and often leads to version control problems between different languages.
Figure 2: Fitting Piloting into the Overall Survey Timeline
Pre-Pilot 1-2 weeks
Questionnaire Design 4 weeks
Questionnaire Translation 2-3 weeks
Content-focused pilot (& revisions) 2-3 weeks
IRB approval {context dependent, concurrent to programming & data pilot}
Questionnaire Programming 4-6 weeks
Data-focused Pilot 2-3 weeks
Enumerator Training 2 weeks


Beyond the questionnaire, what should be tested during the pilot?

  • All Field Protocols should be tested in Stage 2 – Content-focused pilot. Revise as needed, and then pilot again during Stage 3 – Data-focused pilot. All protocols should be finalized before enumerator training.
  • Interview scheduling
    • Is there a particular time of day when respondents are more / less likely to be available?
    • What is the most effective way to communicate with respondents and schedule appointments?
  • What infrastructure will field teams have access to?
    • Electricity? If blackouts are frequent, are there generators (and fuel)?
    • Will there be any internet access?
    • How is the coverage of mobile phone networks?
  • Does the sampling protocol work well in the field? (for more details on developing the protocol, see Guidelines for Impact Evaluation Sampling)
    • What challenges are enumerators likely to face? What clarifications need to be made?
    • If working from an existing sampling frame, is it up to date?
    • Is the replacement strategy effective?
    • If doing a listing, how much time should be built into the field plan? How will you ensure that no sampling units (e.g. households, firms, traders) are skipped?
  • Geo-data: Will you use the tablet or separate GPS?
    • If GPS unit, test protocol for saving with filenames that include unique ID linked to respondent.
    • How long does it take to lock in a signal? How accurate is the data point?