Dimewiki

Survey Pilot

Jump to: navigation, search

Survey pilot refers to the process of carrying out interviews and tests on different components of a survey, including content and protocols. A good pilot provides the research team with important feedback before they start the process of data collection. This feedback can help the research team review and improve instrument design, translations, as well as survey protocols related to interview scheduling, sampling, and geo data.

Read First

Stages of a Survey Pilot

A complete survey pilot is conducted over three stages- pre-pilot, content-focused pilot, and data-focused pilot. The table below discusses these stages in more detail.

Stage 1 : Pre-Pilot Stage 2 : Content-focused Pilot Stage 3 : Data-focused Pilot
Objective Answer broad questions about qualitative factors like survey design and the context for conducting the study. Discuss these with the concerned teams. Refine the order and wording of specific questions, the overall structure of the questionnaire, and translations.

Check that the answer choices are complete, that is, they cover all possibilities.

Check how long it takes to answer the questions. Also check how answers differ between respondents (response variance)

Check if the programmed instrument displays questions in the correct order, and follows the correct patterns (such as a group of questions which will always appear together, or will repeat).

Load a sample data set (based on pilot interviews). Check if the data set has any missing fields. Perform all data quality checks, like back checks and high frequency checks.

Status of survey instrument Early, printable version of the draft, and notes for further discussion. A translated, printable draft. Ready to be programmed. A translated, programmed, final draft. Ready for starting data collection.
Mode Pen-and-paper Pen-and-paper Tablet/phone

However, note that all three stages may not be necessary for every survey. The research team has to determine which stage to begin from on a case-to-case basis.

  • If the research team is using a brand new survey instrument, then they must start with stage 1, the pre-pilot.
  • If the survey instrument is an adaptation of (based on) a different instrument which was used for a previous data collection:
    1. If it was used for another project in the same region (province/country), and if the survey instrument does not have any design issues and was shared by a reliable source, then the research team can start with stage 2, the content-focused pilot.
    2. If it was used for a previous data collection for the same project, but the research team had to make significant revisions or additions, then too the research team can start with stage 2, the content-focused pilot.
  • If the research team is piloting a follow-up survey, and there are no major changes from the baseline (initial) survey, only in such a case, the research team may skip directly to stage 3, the data-focused pilot.

Pen-and-Paper Pilots

Pen-and-paper pilots provide members of the research team and interviewers flexibility when it comes to recording answers and qualitative observations. Even in the case of CAPI and CATI, pen-and-paper pilots are useful for the following tasks:

  • Open-ended responses. These responses are critical for a survey pilot because they allow the research team to make the list of choices for a question more detailed. Pen-and-paper pilots allow the interviewers to make note of these.
  • Restructuring. Pen-and-paper pilots allow the interviewers to draw lines and arrows between questions, which makes it easier to restructure the instrument.
  • Observations. Pen-and-paper pilots allow interviewers to record observations and feedback of interviewers in the margins. This allows the research team to take note of issues with wording or translations.

Timelines

Piloting should start 4-6 months before survey launch, the sooner the better. Do not confuse the pilot with field testing during the enumerator training. It typically involves making significant changes to the survey instrument and/or protocols – and should therefore always be made before enumerator training starts. See timeline of survey pilot for details.

Purpose

A comprehensive pilot should test questionnaire content, questionnaire programming , and all survey protocols.

Participants

Typically, survey pilots are done before the survey firm is on-board. The Field Coordinator plays a central role. Ideally other research team members (e.g. the Principal Investigator) should also participate. For details, see survey pilot participants.

Structure

The success of a pilot often depends on logistics. See guidelines on effectively structuring a survey pilot. You should develop a clear pilot protocol for each stage and share with the research team for approval in advance.

Guidelines

  • Throughout the questionnaire design process, and discussion with the research team, take notes of what needs to be part of the pilot.
  • Take careful notes during the pilot. These deliberations and clarifications will be an important part of the enumerator manual, and discussions at training sessions.
  • Do a data-focused pilot for your back-check questionnaire.
  • Hire a local "mobilizer" to coordinate with respondents.
  • Mobilizers explain the purpose behind conducting the survey and facilitate the process of obtaining consent, particularly in urban areas or settings in which people are particularly busy. This can improve the outcomes of piloting, for instance, by reducing down-time between surveys.(This may not always be consistent with piloting sampling protocols)

DIME Analytics has also created the following checklists to assist researchers and enumerators in preparing for, and implementing a pilot:

Related Pages

This article is part of the topic Primary Data Collection

Additional Resources

LSMS e-Learning course on questionnaire design & piloting: http://lsms.adeptanalytics.org/course/Home_eng.html [Requires Flash Plug-in]

Retrieved from "https://dimewiki.worldbank.org/index.php?title=Survey_Pilot&oldid=6011"