Dimewiki

Difference between revisions of "Structuring a Survey Pilot"

Jump to: navigation, search
 
(37 intermediate revisions by the same user not shown)
Line 1: Line 1:
In order to conduct a successful [[Survey Pilot|survey pilot]], the [[Impact Evaluation Team|research team]] must discuss the '''structuring of a survey pilot.'''. This involves agreeing upon the '''logistics''' (practical considerations) like '''duration''', '''approvals''', '''review sessions''', and '''training.''' It is important to discuss each of these aspects in detail to ensure that the '''survey pilot''' helps improve the quality of the [[Primary Data Collection|data collection process]] and the [[Survey Protocols|survey protocols]].
The process of '''structuring a survey pilot''' involves agreeing upon the '''logistics''' (or practical aspects) of a [[Survey Pilot|survey pilot]], including '''duration''', '''approvals''', '''review sessions''', and '''training.''' In order to conduct a successful pilot, the [[Impact Evaluation Team|impact evaluation team]] (or research team) must discuss each of these aspects in detail to ensure that the '''survey pilot''' is conducted smoothly. A good pilot helps to improve the quality of the [[Primary Data Collection|data collection process]] and the [[Survey Protocols|survey protocols]].
== Read First ==
== Read First ==
* [[Survey Pilot|Survey pilot]] involves carrying out interviews and tests on different components of a survey, including [[Piloting Survey Content|content]] and [[Piloting Survey Protocols|protocols]].
* [[Survey Pilot|Survey pilot]] involves carrying out interviews and tests on different components of a survey, including [[Piloting Survey Content|content]] and [[Checklist: Piloting Survey Protocols|protocols]].
* A pilot should be conducted before [[Enumerator Training|enumerator training]].
* The research team should discuss the structure with all [[Survey Pilot Participants|participants of a pilot]], and resolve any issues that might come up.
* The research team should discuss the '''structure''' with all [[Participants of a Survey Pilot|participants of the pilot]].
* While finalizing the structure, keep the [[Timeline of Survey Pilot|timeline of the survey pilot]] in mind.
* A pilot should be conducted before [[Enumerator Training|enumerator training]] to avoid '''enumerator effects''', which arise when different enumerators ask the same question differently to different respondents.
 
== Duration of the Pilot ==  
== Duration of the Pilot ==  
* Pilot until there are no more substantial changes to be made. Field plan will depend on:  
It is important for the members of [[Impact Evaluation Team|research team]] to agree on the expected time the entire process will take from start to finish. They should also keep extra '''buffer''' time in case some issues come up during the [[Survey Pilot|survey pilot]]. The average duration of the pilot will depend on:
** Extent of changes to be made  
* '''Extent of changes.''' If a lot of changes are made in the [[Questionnaire Design|instrument design]], [[Questionnaire Translation|translations]] or [[Survey Protocols|protocols]], the pilot will take longer to complete. Ideally, the pilot process ends when everyone in the research team agrees that there are no more changes to be made.
** Availability of printing facilities (if paper pilot) in the area where you are piloting.  
* '''Printing facilities.''' In case of [[Survey Pilot#Pen-and-Paper Pilots|pen-and-paper pilots]], availability of printing facilities in the area where the pilot is being conducted will also affect the duration.
** Where the pilot location is (if far from home-base, and teams are staying in the area, breaking for a full day may not be practical / cost-effective)
* '''Pilot location.''' The location of the area where the pilot is being conducted will also affect duration. For example, if the location of the pilot area is far from '''home base''' (the location of the research team), the duration of the pilot will increase. In such cases, it is better for the '''field teams''' to stay in the pilot area, and conduct the pilot on consecutive days to complete the '''piloting''' on time.
 
== Approvals ==
== Approvals ==
Check what approvals are needed. Sometimes local IRB approval is required even for a pilot. A letter of support from the relevant Ministry or implementing partner always helps.  
Even before starting the [[Survey Pilot|survey pilot]], the [[Impact Evaluation Team|research team]] must ensure that all '''approvals''' have been obtained. These include approvals for:
* Consider obtaining [[IRB Approval | IRB approval]] before the pilot (when an IRB is needed). When you are developing new instruments, there are some opportunities for publication even at the piloting level. The PIs will know when this is a possibility. This requires further planning but it can be well worth it.
* '''Budget.''' The organizing funding the study has to approve the [[Survey Budget|survey budget]]. This includes approving the budget for the pilot, including expenses for transport, equipment, communication, and other '''operational costs'''.
* '''Institutional review boards (IRBs).''' In some cases, the research team will need to obtain an [[IRB Approval|IRB approval]] even for the pilot stage. It is useful to consult the [[Impact Evaluation Team#Principal Investigators (PIs)|principal investigators (PIs)]], since they will know when this might need to be done for a pilot. Consider a case where the research team wants to publish certain aspects of the study at the '''piloting''' stage itself. Since the research team is handling [[Personally Identifiable Information (PII)|personally identifiable information (PII)]], they will need an '''IRB approval''' for the '''survey pilot''' as well.
* '''Government agencies.''' While an approval from government agencies is not always necessary for a pilot, a '''letter of support (LOS)''' from the relevant ministry or implementing agency can help conduct the pilot smoothly.
 
== Review Sessions ==
== Review Sessions ==
for feedback and revision
It is important to hold '''review sessions''' regularly during the [[Survey Pilot|survey pilot]]. These sessions allow the [[Impact Evaluation Team|research team]] to discuss feedback from the early [[Survey Pilot#Stages of a Survey Pilot|stages of a pilot]], and use it to improve components like [[Questionnaire Design|instrument design]], [[Questionnaire Translation|translations]], and [[Survey Protocols|survey protocols]]. Keep in mind the following guidelines for the review sessions:
* Plan time for group feedback and discussion sessions at the end of each day.
* '''Schedule.''' Schedule the review sessions frequently. Ideally, the research team should conduct a session with '''field teams''' at the end of each day to resolve any issues they might face. Do not schedule these sessions very late in the day. The sessions will be more useful if the field teams are not tired and hungry. If feasible, give a gap of at least one day between two rounds of '''piloting''' to allow everyone to recover.
** Make sure all voices heard at feedback sessions (especially if age/ gender/ ethnicity differences)
* '''Listen.''' Listen to what everyone has to say. Every opinion or feedback can help improve the overall quality of the [[Primary Data Collection|data collection process]]. This is particularly important when the '''field teams''' vary in terms of age, gender, ethnicity and so on.
** Finish fieldwork early so there is time to debrief. You will get better feedback if the team is not tired and hungry!
* '''Revise.''' Ideally, the research team should make the revisions before the next round of '''piloting''', so that the changes can be tested in the next round. This helps get an idea of whether the changes were useful or not.
* '''Logistics.''' In the case of [[Survey Pilot#Pen-and-Paper Pilots|pen-and-paper pilots]], the research team should have access to printers to share the revisions immediately. It might be a good idea to purchasing a portable printer so that the [[Impact Evaluation Team#Field Coordinators (FCs)|field coordinators (FCs)]] can carry it to the area of the pilot.
* '''Version control.''' It allows the research team to keep track of all changes that are made throughout the review process. For example, if the [[Field Surveys|survey]] is in French, while the '''field coordinator (FC)''' is only fluent in English, keeping track of the revisions can be tricky. In this case, the research team can hire an assistant who speaks French. Then the hired assistant can edit the French version, while the '''field coordinator (FC)''' can edit the English version. Ideally this assistant should be someone other than an enumerator to allow the enumerator enough time to rest and prepare for the next round of '''piloting.'''


* Plan sufficient time to make revisions each evening, and pilot again the next day
== Training ==
** If logistically feasible, best to pilot every other day (otherwise all-nighters are common)
As part of the [[Enumerator Training|enumerator training]], it is important to familiarize enumerators with the objectives for conducting the [[Survey Pilot|survey pilot]]. Keep in mind the following guidelines for this step:
** For pen-and-paper pilots, make sure you will have access to a printer to make and share revisions in real time.  
* '''Length of training.''' This depends entirely on the complexity of the [[Field Surveys|survey instrument]] and the related [[Survey Protocols|protocols]]. It should at least take one full day.
*** Depending on context, using research budget to purchase a printer the FC can travel with may be necessary/ cost-effective.  
* '''Feedback.''' Interviewers can provide useful feedback on the '''survey instrument''' and the '''protocols'''. The research team should take this feedback into account to improve the training outcomes.
* '''Gap.''' There should be a gap of at least 1 day between the end of training and the start of '''piloting'''.
* '''Mock interviews.''' In case of [[Survey Pilot#Stages of a Survey Pilot|data-focused pilots]], the research team should ensure that the interviewers conduct mock interviews. This will allow them to become familiar with the process and also help to catch any '''bugs''' (errors) in [[Questionnaire Programming|programmed]] instrument.


* Be aware of the need for careful '''version control''': If the survey is in a language the FC doesn't understand, it can be tricky to keep track of daily changes in both local language and English version.
== Follow-up Surveys ==
** Best to work with an assistant who speaks the local language to make the edits in that version of the survey while the FC makes them in the English version.  
It is a good idea to '''pilot''' even in case of '''follow-up''' surveys which are conducted after a '''baseline''' (or first round) survey. If the [[Impact Evaluation Team|research team]] has reduced the sample size from previous rounds, they can use '''pre-loaded''' data of respondents who were not part of the actual baseline survey, but for whom data is available because of the pilot rounds. However, if there is no pre-loaded data available, the '''field team''' can conduct '''pre-interviews''' before the actual pilot to collect data for '''pre-loading'''. While this can be challenging in practice, it is useful for improving the quality of the actual follow-up survey.
** Ideally this is someone other than an enumerator (they should go rest so they're fresh for the pilot on the following day, and end of the day edits can go well into the night if piloting needs to happen the next morning)


== Training ==
<table><tr><td>
* Length of training depends completely on complexity of instrument and survey protocols. 1-day minimum.</td></tr>
<tr><td>
* Interviewers must be familiar with the instrument and the objectives of the pilot by the end </td></tr>
<tr><td>
* Interviewers will have useful insights and feedback on the survey instrument at the training itself. Plan time to incorporate their feedback/ make revisions before starting the actual pilot</td></tr>
<tr><td>
* Build in a minimum of 1 day between training and the start of the piloting</td></tr>
<tr><td>
* For data-focused pilots, essential to have interviewers do mock interviews with each other to familiarize themselves with programming, and catch any bugs missed in office tests.</td></tr></table>
== Follow-up surveys ==
* If you will be ‘pre-loading’ data during the survey (e.g. from a baseline survey), you will need to simulate this during the pilot.
** If you have reduced the sample size from previous rounds, you can use non-sampled households for whom baseline data exists.
** Otherwise you may need to do ‘pre-interviews’ one day ahead to collect basic indicators for pre-loading. This is logistically challenging, but very worthwhile!
== Related Pages ==
== Related Pages ==
This article is part of the topic [[Survey Pilot]]
[[Special:WhatLinksHere/Structuring_a_Survey_Pilot|Click here for pages that link to this topic.]]
 
== Additional Resources ==
== Additional Resources ==
*DIME Analytics’ guidelines on [https://github.com/worldbank/DIME-Resources/blob/master/survey-guidelines.pdf piloting surveys]
* DIME Analytics (World Bank), [https://osf.io/63uv9/ Survey guidelines]
*DIME Analytics’ guidelines on [https://github.com/worldbank/DIME-Resources/blob/master/survey-instruments.pdf survey design and pilot]
* DIME Analytics (World Bank), [https://osf.io/rqb5m/ Survey instrument design and pilot]
 
* DIME Analytics (World Bank), [https://osf.io/av5yw/ Survey budgeting]
* DIME Analytics (World Bank), [https://osf.io/a4q27/ Survey training]
[[Category: Research Design]]
[[Category: Primary Data Collection]]
[[Category: Primary Data Collection]]

Latest revision as of 17:17, 6 May 2021

The process of structuring a survey pilot involves agreeing upon the logistics (or practical aspects) of a survey pilot, including duration, approvals, review sessions, and training. In order to conduct a successful pilot, the impact evaluation team (or research team) must discuss each of these aspects in detail to ensure that the survey pilot is conducted smoothly. A good pilot helps to improve the quality of the data collection process and the survey protocols.

Read First

  • Survey pilot involves carrying out interviews and tests on different components of a survey, including content and protocols.
  • The research team should discuss the structure with all participants of a pilot, and resolve any issues that might come up.
  • While finalizing the structure, keep the timeline of the survey pilot in mind.
  • A pilot should be conducted before enumerator training to avoid enumerator effects, which arise when different enumerators ask the same question differently to different respondents.

Duration of the Pilot

It is important for the members of research team to agree on the expected time the entire process will take from start to finish. They should also keep extra buffer time in case some issues come up during the survey pilot. The average duration of the pilot will depend on:

  • Extent of changes. If a lot of changes are made in the instrument design, translations or protocols, the pilot will take longer to complete. Ideally, the pilot process ends when everyone in the research team agrees that there are no more changes to be made.
  • Printing facilities. In case of pen-and-paper pilots, availability of printing facilities in the area where the pilot is being conducted will also affect the duration.
  • Pilot location. The location of the area where the pilot is being conducted will also affect duration. For example, if the location of the pilot area is far from home base (the location of the research team), the duration of the pilot will increase. In such cases, it is better for the field teams to stay in the pilot area, and conduct the pilot on consecutive days to complete the piloting on time.

Approvals

Even before starting the survey pilot, the research team must ensure that all approvals have been obtained. These include approvals for:

  • Budget. The organizing funding the study has to approve the survey budget. This includes approving the budget for the pilot, including expenses for transport, equipment, communication, and other operational costs.
  • Institutional review boards (IRBs). In some cases, the research team will need to obtain an IRB approval even for the pilot stage. It is useful to consult the principal investigators (PIs), since they will know when this might need to be done for a pilot. Consider a case where the research team wants to publish certain aspects of the study at the piloting stage itself. Since the research team is handling personally identifiable information (PII), they will need an IRB approval for the survey pilot as well.
  • Government agencies. While an approval from government agencies is not always necessary for a pilot, a letter of support (LOS) from the relevant ministry or implementing agency can help conduct the pilot smoothly.

Review Sessions

It is important to hold review sessions regularly during the survey pilot. These sessions allow the research team to discuss feedback from the early stages of a pilot, and use it to improve components like instrument design, translations, and survey protocols. Keep in mind the following guidelines for the review sessions:

  • Schedule. Schedule the review sessions frequently. Ideally, the research team should conduct a session with field teams at the end of each day to resolve any issues they might face. Do not schedule these sessions very late in the day. The sessions will be more useful if the field teams are not tired and hungry. If feasible, give a gap of at least one day between two rounds of piloting to allow everyone to recover.
  • Listen. Listen to what everyone has to say. Every opinion or feedback can help improve the overall quality of the data collection process. This is particularly important when the field teams vary in terms of age, gender, ethnicity and so on.
  • Revise. Ideally, the research team should make the revisions before the next round of piloting, so that the changes can be tested in the next round. This helps get an idea of whether the changes were useful or not.
  • Logistics. In the case of pen-and-paper pilots, the research team should have access to printers to share the revisions immediately. It might be a good idea to purchasing a portable printer so that the field coordinators (FCs) can carry it to the area of the pilot.
  • Version control. It allows the research team to keep track of all changes that are made throughout the review process. For example, if the survey is in French, while the field coordinator (FC) is only fluent in English, keeping track of the revisions can be tricky. In this case, the research team can hire an assistant who speaks French. Then the hired assistant can edit the French version, while the field coordinator (FC) can edit the English version. Ideally this assistant should be someone other than an enumerator to allow the enumerator enough time to rest and prepare for the next round of piloting.

Training

As part of the enumerator training, it is important to familiarize enumerators with the objectives for conducting the survey pilot. Keep in mind the following guidelines for this step:

  • Length of training. This depends entirely on the complexity of the survey instrument and the related protocols. It should at least take one full day.
  • Feedback. Interviewers can provide useful feedback on the survey instrument and the protocols. The research team should take this feedback into account to improve the training outcomes.
  • Gap. There should be a gap of at least 1 day between the end of training and the start of piloting.
  • Mock interviews. In case of data-focused pilots, the research team should ensure that the interviewers conduct mock interviews. This will allow them to become familiar with the process and also help to catch any bugs (errors) in programmed instrument.

Follow-up Surveys

It is a good idea to pilot even in case of follow-up surveys which are conducted after a baseline (or first round) survey. If the research team has reduced the sample size from previous rounds, they can use pre-loaded data of respondents who were not part of the actual baseline survey, but for whom data is available because of the pilot rounds. However, if there is no pre-loaded data available, the field team can conduct pre-interviews before the actual pilot to collect data for pre-loading. While this can be challenging in practice, it is useful for improving the quality of the actual follow-up survey.

Related Pages

Click here for pages that link to this topic.

Additional Resources

Retrieved from "https://dimewiki.worldbank.org/index.php?title=Structuring_a_Survey_Pilot&oldid=8195"