Difference between revisions of "Survey Pilot"

Jump to: navigation, search
Line 38: Line 38:
|}
|}


'''Are all 3 stages necessary for every survey?'''
However, note that all three stages may not be necessary for every survey. The research team has to determine which stage to begin from on a case-to-case basis.
<br/>Not necessarily.  
* If the research team is using a brand new survey instrument, then they must start with stage 1, the '''pre-pilot.'''
*If your survey is a brand new survey instrument, always start with Stage 1, '''Pre-Pilot'''.
* If the survey instrument is an adaptation of (based on) a different instrument which was used for a previous data collection:
*If your survey is an adaptation of a well-designed questionnaire from a reliable source in the same country, then you may start with Stage 2, '''Content-focused pilot'''. <br/>If your survey is an adaptation of a survey instrument from a previous data collection for the same project, but significant revisions or additions have been made, then again, start with Stage 2, '''Content-focused piloting'''.  
*#If it was used for another project in the same region (province/country), and if the survey instrument does not have any design issues and was shared by a reliable source, then the research team can start with stage 2, the '''content-focused pilot'''.
*Only in cases where you are piloting a follow-up survey with no major changes from the baseline is it appropriate to skip directly to Stage 3, '''Data-focused pilot'''.
*#If it was used for a previous data collection for the same project, but the research team had to make significant revisions or additions, then too the research team can start with stage 2, the '''content-focused pilot'''.
* If the research team is '''piloting''' a '''follow-up''' survey, and there are no major changes from the '''baseline''' (initial) survey, only in such a case, the research team may skip directly to stage 3, the '''data-focused pilot'''.


== Pen-and-paper pilots for CAPI ==  
== Pen-and-paper pilots for CAPI ==  

Revision as of 18:13, 8 April 2020

Survey pilot refers to the process of carrying out interviews and tests on different components of a survey, including content and protocols. A good pilot provides the research team with important feedback before they start the process of data collection. This feedback can help the research team review and improve instrument design, translations, as well as survey protocols related to interview scheduling, sampling, and geo data.

Read First

Stages of a Survey Pilot

A complete survey pilot is conducted over three stages- pre-pilot, content-focused pilot, and data-focused pilot. The table below discusses these stages in more detail.

Stage 1 : Pre-Pilot Stage 2 : Content-focused Pilot Stage 3 : Data-focused Pilot
Objective Answer broad questions about qualitative factors like survey design and the context for conducting the study. Discuss these with the concerned teams. Refine the order and wording of specific questions, the overall structure of the questionnaire, and translations.

Check that the answer choices are complete, that is, they cover all possibilities.

Check how long it takes to answer the questions. Also check how answers differ between respondents (response variance)

Check if the programmed instrument displays questions in the correct order, and follows the correct patterns (such as a group of questions which will always appear together, or will repeat).

Load a sample data set (based on pilot interviews). Check if the data set has any missing fields. Perform all data quality checks, like back checks and high frequency checks.

Status of survey instrument Early, printable version of the draft, and notes for further discussion. A translated, printable draft. Ready to be programmed. A translated, programmed, final draft. Ready for starting data collection.
Mode Pen-and-paper Pen-and-paper Tablet/phone

However, note that all three stages may not be necessary for every survey. The research team has to determine which stage to begin from on a case-to-case basis.

  • If the research team is using a brand new survey instrument, then they must start with stage 1, the pre-pilot.
  • If the survey instrument is an adaptation of (based on) a different instrument which was used for a previous data collection:
    1. If it was used for another project in the same region (province/country), and if the survey instrument does not have any design issues and was shared by a reliable source, then the research team can start with stage 2, the content-focused pilot.
    2. If it was used for a previous data collection for the same project, but the research team had to make significant revisions or additions, then too the research team can start with stage 2, the content-focused pilot.
  • If the research team is piloting a follow-up survey, and there are no major changes from the baseline (initial) survey, only in such a case, the research team may skip directly to stage 3, the data-focused pilot.

Pen-and-paper pilots for CAPI

Pen-and-paper pilots are recommended even for CAPI surveys, due to their flexibility in recording answers and qualitative observations. Pen-and-paper pilots are especially useful for:

  • Recording open-ended responses (critical for a pilot) more quickly / easily.
  • Drawing lines and arrows between questions to suggest restructuring.
  • Recording observations and feedback in the margins.
  • Making notes of questionnaire wording or translation problems directly in the text.

Timelines

Piloting should start 4-6 months before survey launch, the sooner the better. Do not confuse the pilot with field testing during the enumerator training. It typically involves making significant changes to the survey instrument and/or protocols – and should therefore always be made before enumerator training starts. See timeline of survey pilot for details.

Purpose

A comprehensive pilot should test questionnaire content, questionnaire programming , and all survey protocols.

Participants

Typically, survey pilots are done before the survey firm is on-board. The Field Coordinator plays a central role. Ideally other research team members (e.g. the Principal Investigator) should also participate. For details, see survey pilot participants.

Structure

The success of a pilot often depends on logistics. See guidelines on effectively structuring a survey pilot. You should develop a clear pilot protocol for each stage and share with the research team for approval in advance.

Guidelines

  • Throughout the questionnaire design process, and discussion with the research team, take notes of what needs to be part of the pilot.
  • Take careful notes during the pilot. These deliberations and clarifications will be an important part of the enumerator manual, and discussions at training sessions.
  • Do a data-focused pilot for your back-check questionnaire.
  • Hire a local "mobilizer" to coordinate with respondents.
  • Mobilizers explain the purpose behind conducting the survey and facilitate the process of obtaining consent, particularly in urban areas or settings in which people are particularly busy. This can improve the outcomes of piloting, for instance, by reducing down-time between surveys.(This may not always be consistent with piloting sampling protocols)

DIME Analytics has also created the following checklists to assist researchers and enumerators in preparing for, and implementing a pilot:

Related Pages

This article is part of the topic Primary Data Collection

Additional Resources

LSMS e-Learning course on questionnaire design & piloting: http://lsms.adeptanalytics.org/course/Home_eng.html [Requires Flash Plug-in]