Structuring a Survey Pilot

Structuring of a survey pilot involves agreeing upon the logistics (practical aspects) of a survey pilot, including duration, approvals, review sessions, and training. In order to conduct a successful pilot, the impact evaluation team (or research team) must discuss each of these aspects in detail to ensure that the survey pilot is conducted smoothly. A good pilot helps to improve the quality of the data collection process and the survey protocols.

Read First

• Survey pilot involves carrying out interviews and tests on different components of a survey, including content and protocols.
• The research team should discuss the structure with all participants of a pilot, and resolve any issues that might come up.
• While finalizing the structure, keep the timeline of the survey pilot in mind.
• A pilot should be conducted before enumerator training to avoid enumerator effects, which arise when different enumerators ask the same question differently to different respondents.

Duration of the Pilot

It is important for the members of research team to agree on the expected time the entire process will take from start to finish. They should also keep a buffer (extra time) in case some issues come up during the survey pilot. The average duration of the pilot will depend on:

• Extent of changes. If a lot of changes are made in the instrument design, translations or protocols, the pilot will take longer to complete. Ideally, the pilot process ends when everyone in the research team agrees that there are no more changes to be made.
• Printing facilities. In case of pen-and-paper pilots, availability of printing facilities in the area where the pilot is being conducted will also affect the duration.
• Pilot location. The location of the area where the pilot is being conducted will also affect duration. For example, if the location of the pilot area is far from home base (the location of the research team), the duration of the pilot will increase. In such cases, it is better for the field teams to stay in the pilot area, and conduct the pilot on consecutive days to complete the piloting on time.

Approvals

Check what approvals are needed. Sometimes local IRB approval is required even for a pilot. A letter of support from the relevant Ministry or implementing partner always helps.

• Consider obtaining IRB approval before the pilot (when an IRB is needed). When you are developing new instruments, there are some opportunities for publication even at the piloting level. The PIs will know when this is a possibility. This requires further planning but it can be well worth it.

Review Sessions

for feedback and revision

• Plan time for group feedback and discussion sessions at the end of each day.
  • Make sure all voices heard at feedback sessions (especially if age/ gender/ ethnicity differences)
  • Finish fieldwork early so there is time to debrief. You will get better feedback if the team is not tired and hungry!

• Plan sufficient time to make revisions each evening, and pilot again the next day
  • If logistically feasible, best to pilot every other day (otherwise all-nighters are common)
  • For pen-and-paper pilots, make sure you will have access to a printer to make and share revisions in real time.
    • Depending on context, using research budget to purchase a printer the FC can travel with may be necessary/ cost-effective.

• Be aware of the need for careful version control: If the survey is in a language the FC doesn't understand, it can be tricky to keep track of daily changes in both local language and English version.
  • Best to work with an assistant who speaks the local language to make the edits in that version of the survey while the FC makes them in the English version.
  • Ideally this is someone other than an enumerator (they should go rest so they're fresh for the pilot on the following day, and end of the day edits can go well into the night if piloting needs to happen the next morning)

Training

Length of training depends completely on complexity of instrument and survey protocols. 1-day minimum.

Interviewers must be familiar with the instrument and the objectives of the pilot by the end

Interviewers will have useful insights and feedback on the survey instrument at the training itself. Plan time to incorporate their feedback/ make revisions before starting the actual pilot

Build in a minimum of 1 day between training and the start of the piloting

For data-focused pilots, essential to have interviewers do mock interviews with each other to familiarize themselves with programming, and catch any bugs missed in office tests.

Follow-up surveys

• If you will be ‘pre-loading’ data during the survey (e.g. from a baseline survey), you will need to simulate this during the pilot.
  • If you have reduced the sample size from previous rounds, you can use non-sampled households for whom baseline data exists.
  • Otherwise you may need to do ‘pre-interviews’ one day ahead to collect basic indicators for pre-loading. This is logistically challenging, but very worthwhile!

Related Pages

Click here for pages that link to this topic.

Additional Resources

• DIME Analytics (World Bank), Survey guidelines
• DIME Analytics (World Bank), Survey design and pilot