Difference between revisions of "Survey Pilot"

Jump to: navigation, search
 
(59 intermediate revisions by 2 users not shown)
Line 1: Line 1:
'''Survey pilot''' refers to the process of carrying out interviews and tests on different components of a survey, including [[Piloting Survey Content|content]] and [[Piloting Survey Protocols|protocols]]. A good '''pilot''' provides the research team with important feedback before they start the process of [[Primary Data Collection|data collection]]. This feedback can help the [[Impact Evaluation Team|research team]] review and improve [[Questionnaire Design|instrument design]], [[Questionnaire Translation|translations]], as well as [[Survey Protocols|survey protocols]] related to [[Piloting Survey Protocols#Interview scheduling|interview scheduling]], [[Sampling|sampling]], and [[Geo Spatial Data|geo data]].
'''Survey pilot''' is the process of carrying out interviews and tests on different components of a survey, including [[Checklist: Content-focused Pilot|content]] and [[Checklist: Piloting Survey Protocols|protocols]]. A good '''pilot''' provides the research team with important feedback before they start the process of [[Primary Data Collection|primary data collection]]. This feedback can help the [[Impact Evaluation Team|research team]] review and improve the [[Questionnaire Design| design of the questionnaire]], [[Questionnaire Translation|translated questionnaires]], as well as [[Survey Protocols|survey protocols]] related to scheduling interviews, [[Sampling|sampling]], and [[Geo Spatial Data|geo data]].
== Read First ==
== Read First ==
* A '''pilot''' can be conducted for both [[Field Surveys|field surveys]] and [[Remote Surveys|remote surveys]].
* A comprehensive '''survey pilot''' includes 3 stages: a pre-pilot, a content-focused pilot, and a data-focused pilot.
* The '''pre-pilot''' is usually done during the process of [[Questionnaire Design|designing a questionnaire]], to refine the design, and wording of questions.
* The '''content-focused pilot''' is conducted after [[Questionnaire Design|designing a questionnaire]], and sharing it with field and sector experts for their comments and inputs.
* Finally, the '''data-focused pilot''' allows the [[Impact Evaluation Team|research team]] to review the data collected during the pilot, and identify possible errors in the [[Questionnaire Programming|programmed questionnaire]].
* A '''survey pilot''' is different from the '''field testing''' (practice interviews) that enumerators conduct at the end of [[Enumerator Training|enumerator training]]. The pilot should be complete before enumerator training begins. 
* [[Survey Pilot Participants|Participants of a survey pilot]] include the [[Impact Evaluation Team|research team]], '''interviewers''', and '''respondents'''.
* [[Survey Pilot Participants|Participants of a survey pilot]] include the [[Impact Evaluation Team|research team]], '''interviewers''', and '''respondents'''.
* Note that a '''survey pilot is different from the practice survey that all enumerators do at the end of their training.
* Conduct the first two stages of the pilot before [[Questionnaire Programming|programming the questionnaire]]. Repeated programming is time-consuming and can create '''bugs''' (errors), for instance, by disturbing the order of questions.
* Conduct the pilot before [[Enumerator Training |enumerator training]].
* Conduct the pilot before moving on to [[Questionnaire Programming|questionnaire programming]]. Repeated programming is time-consuming and can create bugs, by shifting the order of questions for example.


== Stages of a Survey Pilot ==
== Stages of a Survey Pilot ==
A complete '''survey pilot''' is conducted over three stages- '''pre-pilot''', [[Piloting Survey Content|content-focused pilot]], and '''data-focused pilot'''. The table below discusses these stages in more detail.
A complete '''survey pilot''' is conducted over three stages - '''pre-pilot''', [[Checklist:_Content-focused_Pilot|content-focused pilot]], and [[Checklist:_Data-focused_Pilot|data-focused pilot]]. The table below discusses these stages in more detail.


{| class="wikitable" style="margin-left: auto; margin-right: auto; border: none;"
{| class="wikitable"  
|-
|-
! style="height:50px; width:150px; text-align:center;" |
! style="height:50px; width:150px; text-align:center;" |
! style="height:50px; width:250px; text-align:center;" | Stage 1 : Pre-Pilot
! style="height:50px; width:320px; text-align:center;" | Stage 1: Pre-pilot
! style="height:50px; width:250px; text-align:center;" | Stage 2 : Content-focused Pilot
! style="height:50px; width:320px; text-align:center;" | Stage 2: Content-focused pilot
! style="height:50px; width:250px; text-align:center;" | Stage 3 : Data-focused Pilot
! style="height:50px; width:320px; text-align:center;" | Stage 3: Data-focused pilot
|-
|-
| style="text-align:center;" | '''Objective'''
| style="text-align:center;" | '''Objective'''
| Answer broad questions about '''qualitative factors''' like [[Questionnaire Design|survey design]] and the context for conducting the study. Discuss these with the concerned teams.
| style="vertical-align:top;"| Answer broad questions about <br>'''qualitative factors''' like [[Questionnaire Design|questionnaire design]] and the context for <br> conducting the study. Discuss these <br>with the concerned teams.
| Refine the order and wording of specific questions, the overall structure of the questionnaire, and [[Questionnaire Translation|translations]].  
 
Learn more about how people think to <br>see what structure and flow makes <br> sense.
For example, do people think <br>about their input use at plot-level? <br>By crop? Overall?
 
It helps to think of the '''pre-pilot''' as a part of the process of [[Questionnaire Design|questionnaire design]]. <br>For example, the '''pre-pilot''' can answer <br>specific questions such as - important sources of income, relevant types of shocks, local food groups, and so on.
 
In cases where the [[Impact Evaluation Team|research team]] is <br>using a pre-existing questionnaire, they <br>can skip this step. However, a '''pre-pilot''' <br>is absolutely essential if the '''research <br>team''' is designing a questionnaire from scratch, or if the questionnaire asks questions about issues that are <br>difficult to measure.
| style="vertical-align:top;"| Refine the order and wording of specific questions, the overall structure of the questionnaire, and [[Questionnaire Translation|translations]].  
 
 
 
Check that the answer [[SurveyCTO Choice Lists|choices]] are '''comprehensive''', that is, they cover all possibilities.
 
 
 
Flag any sensitive questions.
 
 
 
 


Check that the answer choices are '''complete''', that is, they cover all possibilities.


Check how long it takes to answer the questions. Also check how answers differ between respondents ('''response variance''')
Check how long it takes to answer the questions. Also check how answers differ between respondents ('''response variance''')
| Check if the [[Questionnaire Programming|programmed instrument]] displays questions in the correct order, and follows the correct patterns (such as a group of questions which will always appear together, or will repeat).
| style="vertical-align:top;"| Check if the [[Questionnaire Programming|programmed instrument]] displays questions in the correct order, and follows the correct patterns (such as a group of questions which will always appear together, or will repeat).


Load a sample data set (based on '''pilot interviews'''). Check if the data set has any missing fields. Perform all data quality checks, like [[Back Checks|back checks]] and [[Monitoring Data Quality#High Frequency Checks|high frequency checks]].
Load a sample data set (based on '''pilot interviews'''). Check if the data set has any missing fields. Perform all data quality checks, like [[Back Checks|back checks]] and [[Monitoring Data Quality#High Frequency Checks|high frequency checks]].
|-
|-
| style="height:100px;text-align:center;" | '''Status of survey instrument'''
| style="height:80px;text-align:center;" | '''Status of survey instrument'''
| Early, printable version of the draft, and notes for further discussion.
| style="vertical-align:center;" | Early, printable version of the draft, and notes for further discussion.
| A translated, printable draft. Ready to be '''programmed'''.
| style="vertical-align:center;" | A translated, printable draft. Ready to be '''programmed'''.
| A translated, programmed, final draft. Ready for starting data collection.
| style="vertical-align:center;" | A translated, programmed, final draft. Ready for starting data collection.
|-
|-
| style="height:100px;text-align:center;" | '''Mode'''
| style="height:80px;text-align:center;" | '''Mode'''
| style="text-align:center;"| Pen-and-paper
| style="text-align:center;"| Pen-and-paper
| style="text-align:center;"| Pen-and-paper
| style="text-align:center;"| Pen-and-paper
| style="text-align:center;"| Tablet/phone
| style="text-align:center;"| Electronic (phone/tablet)
|}
|}


However, note that all three stages may not be necessary for every survey. The research team has to determine which stage to begin from on a case-to-case basis.
However, note that all three stages may not be necessary for every survey. The research team has to determine which stage to begin from on a case-to-case basis.
* If the research team is using a brand new survey instrument, then they must start with stage 1, the '''pre-pilot.'''
* If the research team is using a brand new survey instrument, then they must start with Stage 1, the '''pre-pilot.'''
* If the survey instrument is an adaptation of (based on) a different instrument which was used for a previous data collection for another project in the same region (province or country), the research team can start with stage 2, the '''content-focused pilot'''. However, they must make sure that the survey instrument does not have any design issues and was shared by a reliable source.
* If the survey instrument is an adaptation of (based on) a different instrument which was used for a previous data collection for another project in the same region (province or country), the research team can start with Stage 2, the '''content-focused pilot'''. However, they must make sure that the survey instrument does not have any design issues and was shared by a reliable source.
* If the survey instrument is an adaptation of an instrument used for a previous data collection for the same project, but the research team had to make significant revisions or additions, even then the research team should start with stage 2, the '''content-focused pilot'''.
* If the survey instrument is an adaptation of an instrument used for a previous data collection for the same project, but the research team had to make significant revisions or additions, even then the research team should start with Stage 2, the '''content-focused pilot'''.
* Only if the research team is '''piloting''' a '''follow-up''' survey, and there are no major changes from the '''baseline''' (initial) survey, then in this case the research team may skip directly to stage 3, the '''data-focused pilot'''.
* Only if the research team is '''piloting''' a '''follow-up''' survey, and there are no major changes from the '''baseline''' (or first round) survey, then in this case the research team may skip directly to Stage 3, the '''data-focused pilot'''.
 
== Pilot on paper first ==
The first two stages of the survey pilot (pre-pilot and content-focused pilot) are best done on pen-and-paper is done on paper, regardless of the planned survey mode ([[Pen-and-Paper_Personal_Interviews_(PAPI)|pen-and-paper personal interviews (PAPI)]],
[[Computer-Assisted_Personal_Interviews_(CAPI)|computer-assisted personal interviews (CAPI)]], [[Remote_Surveys|phone surveys (CATI)]]). A pen-and-paper pilot allows members of the research team and interviewers flexibility when it comes to recording answers and '''qualitative''' observations.  


== Pen-and-Paper Pilots ==
Piloting on paper is particularly helpful for the following:
'''Pen-and-paper pilots''' provide members of the research team and interviewers flexibility when it comes to recording answers and '''qualitative''' observations. They are useful in conducting pilots not only for [[Pen-and-Paper_Personal_Interviews_(PAPI)|pen-and-paper personal interviews (PAPI)]], but also for [[Computer-Assisted Personal Interviews (CAPI)|computer-assisted personal interviews (CAPI)]] and [[Remote Surveys#Phone Surveys (CATI)|phone surveys (CATI)]]. Pen-and-paper pilots can help with the following:
* '''Open-ended responses.''' pre-pilot responses help the research team to make the list of choices for a question more detailed and comprehensive. Pen-and-paper pilots allow the interviewers to make note of these.
* '''Open-ended responses.''' These responses are critical for a '''survey pilot''' because they allow the research team to make the list of choices for a question more detailed. Pen-and-paper pilots allow the interviewers to make note of these.
* '''Restructuring.''' Pen-and-paper pilots allow the interviewers to draw lines and arrows between questions, which makes it easier to restructure the '''instrument'''.
* '''Restructuring.''' Pen-and-paper pilots allow the interviewers to draw lines and arrows between questions, which makes it easier to restructure the '''instrument'''.
* '''Observations.''' Pen-and-paper pilots allow interviewers to record observations and feedback of interviewers in the margins. This allows the research team to take note of issues, such as the wording of questions, or their [[Questionnaire Translation|translations]].
* '''Observations.''' Pen-and-paper pilots allow interviewers to record observations and feedback of interviewers in the margins. This allows the research team to take note of issues, such as the wording or flow of questions.


== Timeline ==
== Timeline ==
'''Piloting''' should start 4-6 months before [[Field Surveys#Survey Launch|survey launch]]. In fact, starting a pilot early allows the research team to use the feedback from each stage to improve the [[Piloting Survey Content|survey instrument]] and [[Piloting Survey Protocols| survey protocols]]. This makes the process of implementing a survey more efficient. Therefore, the research team must draft a detailed [[Timeline of Survey Pilot|timeline for the pilot]] to allow enough time for discussions and revisions. Do not confuse the pilot with '''field testing''' (practice interviews) which is conducted at the end of [[Enumerator_Training |enumerator training]].
In order to conduct a successful [[Survey Pilot|survey pilot]], the [[Impact Evaluation Team|impact evaluation team]] must draft a detailed '''timeline for a survey pilot'''. The '''timeline''' should allow enough time for discussions and revisions about aspects like [[Questionnaire Design|instrument design]], [[Questionnaire Translation|translation]], and [[Survey Protocols|protocols]]. '''Piloting''' should ideally start 4-6 months before [[Field Surveys#Survey Launch|survey launch]] since this allows the research team to use the feedback from each [[Survey Pilot#Stages of a Survey Pilot|stage of a survey pilot]] to improve the survey content and [[Checklist: Piloting Survey Protocols|protocols]].
 
The '''timeline''' is an important part of the process of [[Structuring a Survey Pilot|structuring a survey pilot]]. The actual [[Structuring a Survey Pilot#Duration of the Pilot|duration of the pilot]] will depend on factors like number of changes made, location of the study area, and availability of equipment for '''field teams'''. Based on '''best practices''', the following are the steps involved in conducting a [[Survey Pilot|survey pilot]] (from start to finish), and the recommended time for each step.
{| class="wikitable" style="margin-left: auto; margin-right: auto; border: none;"
!style="width:350px; text-align:center;"| Task
!style="width:350px; text-align:center;"| Estimated Time
|-
|style="text-align:center;"| [[Survey Pilot#Stages of a Survey Pilot|Pre-pilot]]
|style="text-align:center;"| 1-2 weeks
|-
|style="text-align:center;"| [[Questionnaire Design|Questionnaire design]]
|style="text-align:center;"| 4 weeks
|-
|style="text-align:center;"| [[Checklist: Content-focused Pilot|Content-focused pilot]]
|style="text-align:center;"| 2-3 weeks
|-
|style="text-align:center;"| [[Structuring a Survey Pilot#Review Sessions|Review sessions]]
|style="text-align:center;"| 1-2 weeks
|-
|style="text-align:center;"| [[Questionnaire Translation|Questionnaire translation]]
|style="text-align:center;"| 2-3 weeks
|-
|style="text-align:center;"| [[IRB Approval|IRB approval]]
|style="text-align:center;"| Depends on the complexity of the survey. Applying for [[IRB Approval|approvals]], [[Questionnaire Programming|programming]], and the [[Survey Pilot#Stages of a Survey Pilot|data-focused pilot]] - should all happen at the same time.
|-
|style="text-align:center;"| [[Questionnaire Programming|Questionnaire programming]]
|style="text-align:center;"| 4-6 weeks
|-
|style="text-align:center;"| [[Survey Pilot#Stages of a Survey Pilot|Data-focused pilot]]
|style="text-align:center;"| 2-3 weeks
|-
|style="text-align:center;"| [[Enumerator Training|Enumerator training]]
|style="text-align:center;"| 2 weeks
|}
 
Do not confuse the pilot with '''field testing''' (practice interviews) which is conducted at the end of [[Enumerator_Training |enumerator training]]!


== Structure ==
== Structure ==
In order to conduct a successful '''survey pilot''', the research team must decide on the [[Structuring a Survey Pilot|structure of the survey pilot]]. This involves agreeing upon the '''logistics''' for the following aspects of the pilot:
The process of '''structuring a survey pilot''' involves agreeing upon the '''logistics''' (or practical aspects) of a [[Survey Pilot|survey pilot]], including '''duration''', '''approvals''', '''review sessions''', and '''training.''' In order to conduct a successful pilot, the [[Impact Evaluation Team|impact evaluation team]] (or research team) must discuss each of these aspects in detail to ensure that the '''survey pilot''' is conducted smoothly. A good pilot helps to improve the quality of the [[Primary Data Collection|data collection process]] and the [[Survey Protocols|survey protocols]]. Go to [[Structuring a Survey Pilot|Structuring a Survey Pilot]] for a detailed discussion of recommended protocols.
* '''Length of the pilot.''' The pilot should ideally go on until the research team agrees that there are no changes left to be made.
* '''Approvals.''' Obtain all necessary approvals, for example, [[Survey Budget|budget]] approval from the organization funding the study. Sometimes the research team must obtain an [[IRB Approval|IRB approval]] even for a pilot.
* '''Revisions.''' Set aside time for feedback sessions and revisions. The research team should make sure everyone involved in the '''survey pilot''' gets a chance to express their concerns. Therefore, it is a good practice to draft guidelines on issues like the '''frequency''' of review sessions.
* '''Training.''' As part of [[Enumerator Training|enumerator training]], set aside time specifically for discussing best practices for conducting a '''survey pilot'''.


== Participants ==
== Participants ==
Typically, a '''survey pilot''' is conducted before a [[Survey Firm|survey firm]] is brought on-board. Each of the [[Survey Pilot Participants|survey pilot participants]] plays an important role in the implementation of a successful pilot.  
Typically, a '''survey pilot''' is conducted before a [[Survey Firm|survey firm]] is brought on-board. Each of the pilot participants plays an important role in the implementation of a successful pilot. Typical participants include respondents, interviewiers, field coordinator, principal investigator, and other research team members. For a detailed discussion of the role of each participant, and how to select appropriate respondents and interviewers for a pilot, go to the article on [[Survey Pilot Participants| Survey Pilot Participants]].
* '''Respondents.''' They should be as close to the '''target population''' as possible. None of the respondents from a pilot should be in the sample for the actual survey.
* '''Interviewers.''' They should be fluent in all local languages of the study area and the language of the research team. The size of the team of interviewers depends on the stage of pilot.
* '''Field coordinator (FC).''' The [[Impact_Evaluation_Team#Field_Coordinator|field coordinators (FCs)]] play a central role as part of the research team. They are responsible for updating the research team on all monitoring and evaluation '''(M&E)''' activities in the field.
* '''Principal investigator (PI).''' The [[Impact_Evaluation_Team#Principal_Investigator|principal investigators (PIs)]] oversee the overall functioning of the '''impact evaluation''' (study). They should brief the FCs regularly during the pilot.
* '''Other research team members.'''Ideally, other members of the [[Impact Evaluation Team|impact evaluation team]] such as the [[Impact Evaluation Team#Impact Evaluation Coordinator|impact evaluation coordinator]], research manager, programmers, and translators should also participate.


== Guidelines ==
== Guidelines ==
Finally, it is important to keep the following '''best practices''' in mind while planning and conducting a pilot:
Finally, it is important to keep the following '''best practices''' in mind while planning and conducting a pilot:
* Throughout the process of [[Questionnaire Design|designing the questionnaire]], discuss with other members of the research team, and take notes of what needs to be part of the pilot.
* Throughout the process of [[Questionnaire Design|designing the questionnaire]], take notes of what needs to be explored or tested during the pilot.
* Encourage interviewers to probe and follow-up much more than they would in a typical interview
* Encourage respondents to think out-loud, to understand how they are coming up with their answer, and ask enumerators to record these notes.
* In some cultures, it will be appropriate and useful to ask respondents their feedback at the end of the interview
* Take careful notes of the discussions and clarifications that come up during the pilot. These will be an important part of the [[Enumerator Training#Enumerator Manual|enumerator manual]] which is used during [[Enumerator Training|enumerator training]].
* Take careful notes of the discussions and clarifications that come up during the pilot. These will be an important part of the [[Enumerator Training#Enumerator Manual|enumerator manual]] which is used during [[Enumerator Training|enumerator training]].
* Use the '''data-focused pilot''' to perform [[Back Checks|back checks]].  
* It is useful for research team members to sit in on pilot interviews even if they do not speak the local language. Take note of: questions that take longer than expected; any moments at which the respondent seems frustrated, confused, or uncomfortable; any questions that generate significant discussion; and the overall flow of modules and survey instrument.
* Hire a local '''mobilizer''' to coordinate with respondents. '''Mobilizers''' explain the purpose behind conducting the survey, and facilitate the process of obtaining consent. This is particularly helpful in urban areas, or in cases where respondents are busy. This can improve the outcomes of piloting, for instance, by reducing '''down-time''' (gap) between surveys.(This may not always be consistent with piloting sampling protocols)
* Use the '''data-focused pilot''' to test [[Back Checks|back check]] templates.  
* Hire a local '''mobilizer''' to coordinate with respondents. '''Mobilizers''' explain the purpose behind conducting the survey, and facilitate the process of obtaining consent. This is particularly helpful in urban areas, or in cases where respondents are busy. This can improve the outcomes of piloting, for instance, by reducing gap between surveys.


'''DIME Analytics''' has also created the following checklists to assist researchers and enumerators in preparing for, and implementing a pilot:
[https://www.worldbank.org/en/research/dime/data-and-analytics DIME Analytics] has also created the following checklists to assist researchers and enumerators in preparing for, and implementing a pilot:
*[[Preparing_for_the_survey_checklist|Checklist: Preparing for a survey pilot]]
*[[Preparing_for_the_survey_checklist|Checklist: Preparing for a survey pilot]]
*[[Checklist: Refine the Questionnaire (Content)|Checklist: Refining questionnaire content]]
*[[Checklist: Content-focused Pilot|Checklist: Refining questionnaire content]]
*[[Checklist: Refine the Questionnaire (Data)|Checklist: Refining questionnaire data]]
*[[Checklist: Data-focused Pilot|Checklist: Refining questionnaire data]]
 
== Related Pages ==
== Related Pages ==
[[Special:WhatLinksHere/Survey_Pilot|Click here for pages that link to this topic.]]
[[Special:WhatLinksHere/Survey_Pilot|Click here for pages that link to this topic.]]


== Additional Resources ==  
== Additional Resources ==  
* DIME Analytics (World Bank), [https://github.com/worldbank/DIME-Resources/blob/master/survey-guidelines.pdf Guidelines on piloting surveys]
* DIME Analytics (World Bank), [https://osf.io/bpdgr Design and Pilot a Survey]
* DIME Analytics (World Bank), [https://github.com/worldbank/DIME-Resources/blob/master/survey-instruments.pdf Survey instrument design and pilot]
[[Category: Research Design]]
 
[[Category: Primary Data Collection]]
[[Category: Primary Data Collection]] [[Category: Questionnaire Design]]

Latest revision as of 18:58, 28 June 2023

Survey pilot is the process of carrying out interviews and tests on different components of a survey, including content and protocols. A good pilot provides the research team with important feedback before they start the process of primary data collection. This feedback can help the research team review and improve the design of the questionnaire, translated questionnaires, as well as survey protocols related to scheduling interviews, sampling, and geo data.

Read First

  • A comprehensive survey pilot includes 3 stages: a pre-pilot, a content-focused pilot, and a data-focused pilot.
  • The pre-pilot is usually done during the process of designing a questionnaire, to refine the design, and wording of questions.
  • The content-focused pilot is conducted after designing a questionnaire, and sharing it with field and sector experts for their comments and inputs.
  • Finally, the data-focused pilot allows the research team to review the data collected during the pilot, and identify possible errors in the programmed questionnaire.
  • A survey pilot is different from the field testing (practice interviews) that enumerators conduct at the end of enumerator training. The pilot should be complete before enumerator training begins.
  • Participants of a survey pilot include the research team, interviewers, and respondents.
  • Conduct the first two stages of the pilot before programming the questionnaire. Repeated programming is time-consuming and can create bugs (errors), for instance, by disturbing the order of questions.

Stages of a Survey Pilot

A complete survey pilot is conducted over three stages - pre-pilot, content-focused pilot, and data-focused pilot. The table below discusses these stages in more detail.

Stage 1: Pre-pilot Stage 2: Content-focused pilot Stage 3: Data-focused pilot
Objective Answer broad questions about
qualitative factors like questionnaire design and the context for
conducting the study. Discuss these
with the concerned teams.

Learn more about how people think to
see what structure and flow makes
sense. For example, do people think
about their input use at plot-level?
By crop? Overall?

It helps to think of the pre-pilot as a part of the process of questionnaire design.
For example, the pre-pilot can answer
specific questions such as - important sources of income, relevant types of shocks, local food groups, and so on.

In cases where the research team is
using a pre-existing questionnaire, they
can skip this step. However, a pre-pilot
is absolutely essential if the research
team
is designing a questionnaire from scratch, or if the questionnaire asks questions about issues that are
difficult to measure.

Refine the order and wording of specific questions, the overall structure of the questionnaire, and translations.


Check that the answer choices are comprehensive, that is, they cover all possibilities.


Flag any sensitive questions.




Check how long it takes to answer the questions. Also check how answers differ between respondents (response variance)

Check if the programmed instrument displays questions in the correct order, and follows the correct patterns (such as a group of questions which will always appear together, or will repeat).

Load a sample data set (based on pilot interviews). Check if the data set has any missing fields. Perform all data quality checks, like back checks and high frequency checks.

Status of survey instrument Early, printable version of the draft, and notes for further discussion. A translated, printable draft. Ready to be programmed. A translated, programmed, final draft. Ready for starting data collection.
Mode Pen-and-paper Pen-and-paper Electronic (phone/tablet)

However, note that all three stages may not be necessary for every survey. The research team has to determine which stage to begin from on a case-to-case basis.

  • If the research team is using a brand new survey instrument, then they must start with Stage 1, the pre-pilot.
  • If the survey instrument is an adaptation of (based on) a different instrument which was used for a previous data collection for another project in the same region (province or country), the research team can start with Stage 2, the content-focused pilot. However, they must make sure that the survey instrument does not have any design issues and was shared by a reliable source.
  • If the survey instrument is an adaptation of an instrument used for a previous data collection for the same project, but the research team had to make significant revisions or additions, even then the research team should start with Stage 2, the content-focused pilot.
  • Only if the research team is piloting a follow-up survey, and there are no major changes from the baseline (or first round) survey, then in this case the research team may skip directly to Stage 3, the data-focused pilot.

Pilot on paper first

The first two stages of the survey pilot (pre-pilot and content-focused pilot) are best done on pen-and-paper is done on paper, regardless of the planned survey mode (pen-and-paper personal interviews (PAPI), computer-assisted personal interviews (CAPI), phone surveys (CATI)). A pen-and-paper pilot allows members of the research team and interviewers flexibility when it comes to recording answers and qualitative observations.

Piloting on paper is particularly helpful for the following:

  • Open-ended responses. pre-pilot responses help the research team to make the list of choices for a question more detailed and comprehensive. Pen-and-paper pilots allow the interviewers to make note of these.
  • Restructuring. Pen-and-paper pilots allow the interviewers to draw lines and arrows between questions, which makes it easier to restructure the instrument.
  • Observations. Pen-and-paper pilots allow interviewers to record observations and feedback of interviewers in the margins. This allows the research team to take note of issues, such as the wording or flow of questions.

Timeline

In order to conduct a successful survey pilot, the impact evaluation team must draft a detailed timeline for a survey pilot. The timeline should allow enough time for discussions and revisions about aspects like instrument design, translation, and protocols. Piloting should ideally start 4-6 months before survey launch since this allows the research team to use the feedback from each stage of a survey pilot to improve the survey content and protocols.

The timeline is an important part of the process of structuring a survey pilot. The actual duration of the pilot will depend on factors like number of changes made, location of the study area, and availability of equipment for field teams. Based on best practices, the following are the steps involved in conducting a survey pilot (from start to finish), and the recommended time for each step.

Task Estimated Time
Pre-pilot 1-2 weeks
Questionnaire design 4 weeks
Content-focused pilot 2-3 weeks
Review sessions 1-2 weeks
Questionnaire translation 2-3 weeks
IRB approval Depends on the complexity of the survey. Applying for approvals, programming, and the data-focused pilot - should all happen at the same time.
Questionnaire programming 4-6 weeks
Data-focused pilot 2-3 weeks
Enumerator training 2 weeks

Do not confuse the pilot with field testing (practice interviews) which is conducted at the end of enumerator training!

Structure

The process of structuring a survey pilot involves agreeing upon the logistics (or practical aspects) of a survey pilot, including duration, approvals, review sessions, and training. In order to conduct a successful pilot, the impact evaluation team (or research team) must discuss each of these aspects in detail to ensure that the survey pilot is conducted smoothly. A good pilot helps to improve the quality of the data collection process and the survey protocols. Go to Structuring a Survey Pilot for a detailed discussion of recommended protocols.

Participants

Typically, a survey pilot is conducted before a survey firm is brought on-board. Each of the pilot participants plays an important role in the implementation of a successful pilot. Typical participants include respondents, interviewiers, field coordinator, principal investigator, and other research team members. For a detailed discussion of the role of each participant, and how to select appropriate respondents and interviewers for a pilot, go to the article on Survey Pilot Participants.

Guidelines

Finally, it is important to keep the following best practices in mind while planning and conducting a pilot:

  • Throughout the process of designing the questionnaire, take notes of what needs to be explored or tested during the pilot.
  • Encourage interviewers to probe and follow-up much more than they would in a typical interview
  • Encourage respondents to think out-loud, to understand how they are coming up with their answer, and ask enumerators to record these notes.
  • In some cultures, it will be appropriate and useful to ask respondents their feedback at the end of the interview
  • Take careful notes of the discussions and clarifications that come up during the pilot. These will be an important part of the enumerator manual which is used during enumerator training.
  • It is useful for research team members to sit in on pilot interviews even if they do not speak the local language. Take note of: questions that take longer than expected; any moments at which the respondent seems frustrated, confused, or uncomfortable; any questions that generate significant discussion; and the overall flow of modules and survey instrument.
  • Use the data-focused pilot to test back check templates.
  • Hire a local mobilizer to coordinate with respondents. Mobilizers explain the purpose behind conducting the survey, and facilitate the process of obtaining consent. This is particularly helpful in urban areas, or in cases where respondents are busy. This can improve the outcomes of piloting, for instance, by reducing gap between surveys.

DIME Analytics has also created the following checklists to assist researchers and enumerators in preparing for, and implementing a pilot:

Related Pages

Click here for pages that link to this topic.

Additional Resources