Difference between revisions of "Survey Pilot"
| Line 1: | Line 1: | ||
The | The '''Survey Pilot''' is a field-test of the questionnaire(s) and all survey protocols. | ||
== Read First ==<onlyinclude> | == Read First ==<onlyinclude> | ||
*'''The Pilot is conducted ''before'' [[Enumerator_Training | emunerator training]]''' | *'''The Pilot is conducted ''before'' [[Enumerator_Training | emunerator training]]'''. It is different from the field practice that all enumerators do at the end of training. | ||
*'''Piloting is not just about the questionnaire''' | *'''Piloting is not just about the questionnaire'''. It is also an opportunity to test survey protocols (e.g. respondent selection, replacement, geo-data collection) and learn about relevant logistics. | ||
</onlyinclude> | </onlyinclude> | ||
== Guidelines for Survey Pilot == | == Guidelines for Survey Pilot == | ||
=== Stages of a Survey Pilot === | === Stages of a Survey Pilot === | ||
A complete survey pilot | A complete survey pilot has '''3 key stages''': | ||
{| class="wikitable" style="margin-left: auto; margin-right: auto; border: none;" | {| class="wikitable" style="margin-left: auto; margin-right: auto; border: none;" | ||
|+ Figure 1: A Complete Survey Pilot Includes 3 Stages | |+ Figure 1: A Complete Survey Pilot Includes 3 Stages | ||
|- | |- | ||
! | |||
! Stage 1 - Pre-Pilot | ! Stage 1 - Pre-Pilot | ||
! Stage 2 - Content-focused Pilot | ! Stage 2 - Content-focused Pilot | ||
! Stage 3 - Data-focused Pilot | ! Stage 3 - Data-focused Pilot | ||
|- | |- | ||
| '''Objective''' | |||
| Answer broad questions about survey design and context through qualitative interviews and focus group discussions. | | Answer broad questions about survey design and context through qualitative interviews and focus group discussions. | ||
| Refine overall order and structure, wording of specific questions, and translations. | | Refine overall order and structure, wording of specific questions, and translations. | ||
Check completeness of answer choice options, response variance, survey length. | Check completeness of answer-choice options, response variance, survey length. | ||
| Validate programming, export a sample dataset, check dataset structure and completeness, | | Validate programming, export a sample dataset, check dataset structure and completeness, perform all data quality checks | ||
|- | |- | ||
| '''Output''' | |||
| Early, printable draft, and/or qualitative instruments | | Early, printable draft, and/or qualitative instruments | ||
| A translated, printable, complete draft | | A translated, printable, complete draft | ||
| A translated, programmed, final draft | | A translated, programmed, '''final''' draft | ||
|- | |- | ||
| '''Conducted via''' | |||
| Pen-and-Paper | | Pen-and-Paper | ||
| Pen-and-Paper | | Pen-and-Paper | ||
| Tablet/Phone | | Tablet/Phone | ||
|} | |} | ||
'''Are all 3 stages | '''Are all 3 stages necessary for every survey?''' | ||
<br/>Not necessarily. | <br/>Not necessarily. | ||
*If your survey is a brand new survey instrument, always start with Stage 1, '''Pre-Pilot'''. | |||
*If your survey is an adaptation of a well-designed questionnaire from a reliable source in the same country, then you may start with Stage 2, '''Content-focused pilot'''. <br/>If your survey is an adaptation of a survey instrument from a previous data collection for the same project, but significant revisions or additions have been made, then again, start with Stage 2, '''Content-focused piloting'''. | |||
*Only in cases where you are piloting a follow-up survey with no major changes from the baseline is it appropriate to skip directly to Stage 3, '''Data-focused pilot'''. | |||
=== Why do a pen-and-paper pilot for a CAPI survey? === | === Why do a pen-and-paper pilot for a CAPI survey? === | ||
| Line 42: | Line 49: | ||
* Making notes of questionnaire wording or translation problems directly in the text. | * Making notes of questionnaire wording or translation problems directly in the text. | ||
A good pilot will provide significant inputs to questionnaire design, and lead to significant changes in content and structure. | A good pilot will provide significant inputs to questionnaire design, and lead to significant changes in content and structure. Repeated programming in CAPI software is time-consuming and can create bugs (e.g. if order of questions shifts and all skip codes need to be re-programmed), so it is best done '''''after''''' the questionnaire has been finalized. | ||
=== What is the timeline for a survey pilot? === | === What is the timeline for a survey pilot? === | ||
Piloting should start 4-6 months before survey launch, the sooner the better. Do not confuse the pilot with field testing during the [[Enumerator_Training | enumerator training]]. It typically involves significant changes to the survey instrument and/or protocols – | Piloting should start 4-6 months before survey launch, the sooner the better. Do not confuse the pilot with field testing during the [[Enumerator_Training | enumerator training]]. It typically involves making significant changes to the survey instrument and/or protocols – and should therefore always be made before enumerator training starts. See [[Timeline of Survey Pilot|timeline of survey pilot]] for details. | ||
=== What should be tested during the pilot? === | === What should be tested during the pilot? === | ||
A comprehensive pilot should test [[Piloting Survey Content | questionnaire content]], [[Questionnaire Programming|questionnaire programming ]], and [[Piloting Survey Protocols|all survey protocols.]] | |||
=== Who should be involved in a survey pilot? === | === Who should be involved in a survey pilot? === | ||
Typically, survey pilots are done before the survey firm is on-board. The [[Field Coordinator]] plays a central role. Ideally other research team members (e.g. the [[Principal Investigator]]) | Typically, survey pilots are done before the survey firm is on-board. The [[Field Coordinator]] plays a central role. Ideally other research team members (e.g. the [[Principal Investigator]]) should also participate. For details, see [[Survey Pilot Participants|survey pilot participants]]. | ||
=== How | === How should a survey pilot be structured? === | ||
The success of a pilot often depends on logistics. See guidelines on effectively [[Structuring a Survey Pilot]]. You should develop a clear | The success of a pilot often depends on logistics. See guidelines on effectively [[Structuring a Survey Pilot|structuring a survey pilot]]. You should develop a clear '''pilot protocol''' for each stage and share with the research team for approval in advance. | ||
=== Tips and Reminders === | === Tips and Reminders === | ||
* Throughout the questionnaire design process, and discussion with the research team, take notes of what needs to be | * Throughout the questionnaire design process, and discussion with the research team, take notes of what needs to be part of the pilot. | ||
* Take careful notes during the pilot | * Take careful notes during the pilot. These deliberations and clarifications will be an important part of the enumerator manual, and discussions at training sessions. | ||
* Do a data-focused pilot for your | * Do a data-focused pilot for your back-check questionnaire. | ||
* | * Hire a local "mobilizer" to coordinate with respondents. '''Mobilizers''' explain the purpose behind conducting the survey and facilitate the process of obtaining consent, particularly in urban areas or settings in which people are particularly busy. This can improve the outcomes of piloting, for instance, by reducing down-time between surveys.(This may not always be consistent with piloting sampling protocols) | ||
== Checklists == | == DIME's Checklists for Effective Piloting== | ||
We recommend using the [[Preparing_for_the_survey_checklist|Checklist: Preparing for a Survey Pilot]], [[Checklist: Refine the Questionnaire (Content)]] and [[Checklist: Refine the Questionnaire (Data)]] as you prepare for and implement a | We recommend using the [[Preparing_for_the_survey_checklist|Checklist: Preparing for a Survey Pilot]], [[Checklist: Refine the Questionnaire (Content)]] and [[Checklist: Refine the Questionnaire (Data)]] as you prepare for, and implement a survey pilot. | ||
== Back to Parent == | == Back to Parent == | ||
Revision as of 22:31, 16 March 2020
The Survey Pilot is a field-test of the questionnaire(s) and all survey protocols.
Read First
- The Pilot is conducted before emunerator training. It is different from the field practice that all enumerators do at the end of training.
- Piloting is not just about the questionnaire. It is also an opportunity to test survey protocols (e.g. respondent selection, replacement, geo-data collection) and learn about relevant logistics.
Guidelines for Survey Pilot
Stages of a Survey Pilot
A complete survey pilot has 3 key stages:
| Stage 1 - Pre-Pilot | Stage 2 - Content-focused Pilot | Stage 3 - Data-focused Pilot | |
|---|---|---|---|
| Objective | Answer broad questions about survey design and context through qualitative interviews and focus group discussions. | Refine overall order and structure, wording of specific questions, and translations.
Check completeness of answer-choice options, response variance, survey length. |
Validate programming, export a sample dataset, check dataset structure and completeness, perform all data quality checks |
| Output | Early, printable draft, and/or qualitative instruments | A translated, printable, complete draft | A translated, programmed, final draft |
| Conducted via | Pen-and-Paper | Pen-and-Paper | Tablet/Phone |
Are all 3 stages necessary for every survey?
Not necessarily.
- If your survey is a brand new survey instrument, always start with Stage 1, Pre-Pilot.
- If your survey is an adaptation of a well-designed questionnaire from a reliable source in the same country, then you may start with Stage 2, Content-focused pilot.
If your survey is an adaptation of a survey instrument from a previous data collection for the same project, but significant revisions or additions have been made, then again, start with Stage 2, Content-focused piloting. - Only in cases where you are piloting a follow-up survey with no major changes from the baseline is it appropriate to skip directly to Stage 3, Data-focused pilot.
Why do a pen-and-paper pilot for a CAPI survey?
Pen-and-paper pilots are recommended even for CAPI surveys, due to their flexibility in recording answers and qualitative observations. Pen-and-paper pilots are especially useful for:
- Recording open-ended responses (critical for a pilot) more quickly / easily.
- Drawing lines and arrows between questions to suggest restructuring.
- Recording observations and feedback in the margins.
- Making notes of questionnaire wording or translation problems directly in the text.
A good pilot will provide significant inputs to questionnaire design, and lead to significant changes in content and structure. Repeated programming in CAPI software is time-consuming and can create bugs (e.g. if order of questions shifts and all skip codes need to be re-programmed), so it is best done after the questionnaire has been finalized.
What is the timeline for a survey pilot?
Piloting should start 4-6 months before survey launch, the sooner the better. Do not confuse the pilot with field testing during the enumerator training. It typically involves making significant changes to the survey instrument and/or protocols – and should therefore always be made before enumerator training starts. See timeline of survey pilot for details.
What should be tested during the pilot?
A comprehensive pilot should test questionnaire content, questionnaire programming , and all survey protocols.
Who should be involved in a survey pilot?
Typically, survey pilots are done before the survey firm is on-board. The Field Coordinator plays a central role. Ideally other research team members (e.g. the Principal Investigator) should also participate. For details, see survey pilot participants.
How should a survey pilot be structured?
The success of a pilot often depends on logistics. See guidelines on effectively structuring a survey pilot. You should develop a clear pilot protocol for each stage and share with the research team for approval in advance.
Tips and Reminders
- Throughout the questionnaire design process, and discussion with the research team, take notes of what needs to be part of the pilot.
- Take careful notes during the pilot. These deliberations and clarifications will be an important part of the enumerator manual, and discussions at training sessions.
- Do a data-focused pilot for your back-check questionnaire.
- Hire a local "mobilizer" to coordinate with respondents. Mobilizers explain the purpose behind conducting the survey and facilitate the process of obtaining consent, particularly in urban areas or settings in which people are particularly busy. This can improve the outcomes of piloting, for instance, by reducing down-time between surveys.(This may not always be consistent with piloting sampling protocols)
DIME's Checklists for Effective Piloting
We recommend using the Checklist: Preparing for a Survey Pilot, Checklist: Refine the Questionnaire (Content) and Checklist: Refine the Questionnaire (Data) as you prepare for, and implement a survey pilot.
Back to Parent
This article is part of the topic Primary Data Collection
Additional Resources
- DIME Analytics’ guidelines on piloting surveys
LSMS e-Learning course on questionnaire design & piloting: http://lsms.adeptanalytics.org/course/Home_eng.html [Requires Flash Plug-in]