Survey Pilot
Survey pilot refers to the process of carrying out interviews and tests on different components of a survey, including content and protocols. A good pilot provides the research team with important feedback before they start the process of data collection. This feedback can help the research team review and improve instrument design, translations, as well as survey protocols related to interview scheduling, sampling, and geo data.
Read First
- A pilot can be conducted for both field surveys and remote surveys.
- Participants of a survey pilot include the research team, interviewers, and respondents.
- Note that a pilot is different from the practice survey that all enumerators do at the end of their training.
- Conduct the pilot before enumerator training.
- Conduct the pilot before moving on to questionnaire programming. Repeated programming is time-consuming and can create bugs, by shifting the order of questions for example.
Stages of a Survey Pilot
A complete survey pilot is conducted over three stages- pre-pilot, content-focused pilot, and data-focused pilot. The table below discusses these stages in more detail.
| Stage 1 : Pre-Pilot | Stage 2 : Content-focused Pilot | Stage 3 : Data-focused Pilot | |
|---|---|---|---|
| Objective | Answer broad questions about qualitative factors like survey design and the context for conducting the study. Discuss these with the concerned teams. | Refine the order and wording of specific questions, the overall structure of the questionnaire, and translations.
Check that the answer choices are complete, that is, they cover all possibilities. Check how long it takes to answer the questions. Also check how answers differ between respondents (response variance) |
Check if the programmed instrument displays questions in the correct order, and follows the correct patterns (such as a group of questions which will always appear together, or will repeat).
Load a sample data set (based on pilot interviews). Check if the data set has any missing fields. Perform all data quality checks, like back checks and high frequency checks. |
| Status of survey instrument | Early, printable version of the draft, and notes for further discussion. | A translated, printable draft. Ready to be programmed. | A translated, programmed, final draft. Ready for starting data collection. |
| Mode | Pen-and-paper | Pen-and-paper | Tablet/phone |
However, note that all three stages may not be necessary for every survey. The research team has to determine which stage to begin from on a case-to-case basis.
- If the research team is using a brand new survey instrument, then they must start with stage 1, the pre-pilot.
- If the survey instrument is an adaptation of (based on) a different instrument which was used for a previous data collection for another project in the same region (province or country), the research team can start with stage 2, the content-focused pilot. However, they must make sure that the survey instrument does not have any design issues and was shared by a reliable source.
- If the survey instrument is an adaptation of an instrument used for a previous data collection for the same project, but the research team had to make significant revisions or additions, even then the research team should start with stage 2, the content-focused pilot.
- Only if the research team is piloting a follow-up survey, and there are no major changes from the baseline (initial) survey, then in this case the research team may skip directly to stage 3, the data-focused pilot.
Pen-and-Paper Pilots
Pen-and-paper pilots provide members of the research team and interviewers flexibility when it comes to recording answers and qualitative observations. Even in the case of computer-assisted personal interviews (CAPI) and phone surveys (CATI), pen-and-paper pilots are useful for the following tasks:
- Open-ended responses. These responses are critical for a survey pilot because they allow the research team to make the list of choices for a question more detailed. Pen-and-paper pilots allow the interviewers to make note of these.
- Restructuring. Pen-and-paper pilots allow the interviewers to draw lines and arrows between questions, which makes it easier to restructure the instrument.
- Observations. Pen-and-paper pilots allow interviewers to record observations and feedback of interviewers in the margins. This allows the research team to take note of issues, such as the wording of questions, or their translations.
Timelines
Piloting should start 4-6 months before survey launch. In fact, starting a pilot early allows the research team to use the feedback from each stage to improve the survey instrument and survey protocols. This makes the process of implementing a survey more efficient. Therefore, the research team must draft a detailed timeline for the pilot to allow enough time for discussions and revisions. Do not confuse the pilot with field testing (practice interviews) which is conducted at the end of enumerator training.
Participants
Typically, survey pilots are done before the survey firm is on-board. The Field Coordinator plays a central role. Ideally other research team members (e.g. the Principal Investigator) should also participate. For details, see survey pilot participants.
Structure
The success of a pilot often depends on logistics. See guidelines on effectively structuring a survey pilot. You should develop a clear pilot protocol for each stage and share with the research team for approval in advance.
Guidelines
- Throughout the questionnaire design process, and discussion with the research team, take notes of what needs to be part of the pilot.
- Take careful notes during the pilot. These deliberations and clarifications will be an important part of the enumerator manual, and discussions at training sessions.
- Do a data-focused pilot for your back-check questionnaire.
- Hire a local "mobilizer" to coordinate with respondents.
- Mobilizers explain the purpose behind conducting the survey and facilitate the process of obtaining consent, particularly in urban areas or settings in which people are particularly busy. This can improve the outcomes of piloting, for instance, by reducing down-time between surveys.(This may not always be consistent with piloting sampling protocols)
DIME Analytics has also created the following checklists to assist researchers and enumerators in preparing for, and implementing a pilot:
- Checklist: Preparing for a survey pilot
- Checklist: Refining questionnaire content
- Checklist: Refining questionnaire data
Related Pages
This article is part of the topic Primary Data Collection
Additional Resources
- DIME Analytics’ guidelines on piloting surveys
LSMS e-Learning course on questionnaire design & piloting: http://lsms.adeptanalytics.org/course/Home_eng.html [Requires Flash Plug-in]