Survey Pilot

Survey pilot is the process of carrying out interviews and tests on different components of a survey, including content and protocols. A good pilot provides the research team with important feedback before they start the process of data collection. This feedback can help the research team review and improve instrument design, translations, as well as survey protocols related to interview scheduling, sampling, and geo data.

Read First

• A pilot can be conducted for both field surveys and remote surveys.
• Participants of a survey pilot include the research team, interviewers, and respondents.
• Note that a survey pilot is different from the field testing (practice interviews) that enumerators conduct at the end of enumerator training.
• Conduct the pilot before moving on to enumerator training.
• Conduct the pilot before programming the questionnaire. Repeated programming is time-consuming and can create bugs (errors), for instance, by disturbing the order of questions.

Stages of a Survey Pilot

A complete survey pilot is conducted over three stages - pre-pilot, content-focused pilot, and data-focused pilot. The table below discusses these stages in more detail.

	Stage 1: Pre-pilot	Stage 2: Content-focused pilot	Stage 3: Data-focused pilot
Objective	Answer broad questions about qualitative factors like survey design and the context for conducting the study. Discuss these with the concerned teams.	Refine the order and wording of specific questions, the overall structure of the questionnaire, and translations. Check that the answer choices are complete, that is, they cover all possibilities. Check how long it takes to answer the questions. Also check how answers differ between respondents (response variance)	Check if the programmed instrument displays questions in the correct order, and follows the correct patterns (such as a group of questions which will always appear together, or will repeat). Load a sample data set (based on pilot interviews). Check if the data set has any missing fields. Perform all data quality checks, like back checks and high frequency checks.
Status of survey instrument	Early, printable version of the draft, and notes for further discussion.	A translated, printable draft. Ready to be programmed.	A translated, programmed, final draft. Ready for starting data collection.
Mode	Pen-and-paper	Pen-and-paper	Tablet/phone

However, note that all three stages may not be necessary for every survey. The research team has to determine which stage to begin from on a case-to-case basis.

• If the research team is using a brand new survey instrument, then they must start with Stage 1, the pre-pilot.
• If the survey instrument is an adaptation of (based on) a different instrument which was used for a previous data collection for another project in the same region (province or country), the research team can start with Stage 2, the content-focused pilot. However, they must make sure that the survey instrument does not have any design issues and was shared by a reliable source.
• If the survey instrument is an adaptation of an instrument used for a previous data collection for the same project, but the research team had to make significant revisions or additions, even then the research team should start with Stage 2, the content-focused pilot.
• Only if the research team is piloting a follow-up survey, and there are no major changes from the baseline (or first round) survey, then in this case the research team may skip directly to Stage 3, the data-focused pilot.

Pen-and-Paper Pilots

Pen-and-paper pilots provide members of the research team and interviewers flexibility when it comes to recording answers and qualitative observations. They are useful in conducting pilots not only for pen-and-paper personal interviews (PAPI), but also for computer-assisted personal interviews (CAPI) and phone surveys (CATI). Pen-and-paper pilots can help with the following:

• Open-ended responses. These responses are critical for a survey pilot because they allow the research team to make the list of choices for a question more detailed. Pen-and-paper pilots allow the interviewers to make note of these.
• Restructuring. Pen-and-paper pilots allow the interviewers to draw lines and arrows between questions, which makes it easier to restructure the instrument.
• Observations. Pen-and-paper pilots allow interviewers to record observations and feedback of interviewers in the margins. This allows the research team to take note of issues, such as the wording of questions, or their translations.

Timeline

Piloting should start 4-6 months before survey launch. In fact, starting a pilot early allows the research team to use the feedback from each stage to improve the survey instrument and survey protocols. This makes the process of implementing a survey more efficient. Therefore, the research team must draft a detailed timeline for the pilot to allow enough time for discussions and revisions. Do not confuse the pilot with field testing (practice interviews) which is conducted at the end of enumerator training.

Structure

In order to conduct a successful survey pilot, the research team must decide on the structure of the survey pilot. This involves agreeing upon the logistics for the following aspects of the pilot:

• Length of the pilot. The pilot should ideally go on until the research team agrees that there are no changes left to be made.
• Approvals. Obtain all necessary approvals, for example, budget approval from the organization funding the study. Sometimes the research team must obtain an IRB approval even for a pilot.
• Revisions. Set aside time for feedback sessions and revisions. The research team should make sure everyone involved in the survey pilot gets a chance to express their concerns. Therefore, it is a good practice to draft guidelines on issues like the frequency of review sessions.
• Training. As part of enumerator training, set aside time specifically for discussing best practices for conducting a survey pilot.

Participants

Typically, a survey pilot is conducted before a survey firm is brought on-board. Each of the survey pilot participants plays an important role in the implementation of a successful pilot.

• Respondents. They should be as close to the target population as possible. None of the respondents from a pilot should be in the sample for the actual survey.
• Interviewers. They should be fluent in all local languages of the study area and the language of the research team. The size of the team of interviewers depends on the stage of pilot.
• Field coordinator (FC). The field coordinators (FCs) play a central role as part of the research team. They are responsible for updating the research team on all monitoring and evaluation (M&E) activities in the field.
• Principal investigator (PI). The principal investigators (PIs) oversee the overall functioning of the impact evaluation (study). They should brief the FCs regularly during the pilot.
• Other research team members.Ideally, other members of the impact evaluation team such as the impact evaluation coordinator, research manager, programmers, and translators should also participate.

Guidelines

Finally, it is important to keep the following best practices in mind while planning and conducting a pilot:

• Throughout the process of designing the questionnaire, discuss with other members of the research team, and take notes of what needs to be part of the pilot.
• Take careful notes of the discussions and clarifications that come up during the pilot. These will be an important part of the enumerator manual which is used during enumerator training.
• Use the data-focused pilot to perform back checks.
• Hire a local mobilizer to coordinate with respondents. Mobilizers explain the purpose behind conducting the survey, and facilitate the process of obtaining consent. This is particularly helpful in urban areas, or in cases where respondents are busy. This can improve the outcomes of piloting, for instance, by reducing gap between surveys.

DIME Analytics has also created the following checklists to assist researchers and enumerators in preparing for, and implementing a pilot:

• Checklist: Preparing for a survey pilot
• Checklist: Refining questionnaire content
• Checklist: Refining questionnaire data

Related Pages

Click here for pages that link to this topic.

Additional Resources

• DIME Analytics (World Bank), Survey pilot guidelines
• DIME Analytics (World Bank), Survey instrument design and pilot