Difference between revisions of "Survey Pilot"
Maria jones (talk | contribs) |
Maria jones (talk | contribs) |
||
| Line 58: | Line 58: | ||
== Who should be involved in a survey pilot? == | == Who should be involved in a survey pilot? == | ||
Typically, survey pilots are done before the survey firm is on board. The DIME Field Coordinator plays a central role. Ideally other research team members (e.g. the Principal Investigator) will participate. For details, see [[Pilot Participants]] | Typically, survey pilots are done before the survey firm is on board. The DIME Field Coordinator plays a central role. Ideally other research team members (e.g. the Principal Investigator) will participate. For details, see [[Survey Pilot Participants]] | ||
== How to structure a survey pilot? == | == How to structure a survey pilot? == | ||
Revision as of 17:03, 17 January 2017
The survey pilot is a field test of the questionnaire(s) and all survey protocols.
READ FIRST
The Pilot is done before Enumerator Training. It is not the same as the field practice all enumerators do at the end of Training. Piloting is not just about the questionnaire: it is also an opportunity to test survey protocols (e.g. respondent selection, replacement, geo-data collection) and learn about relevant logistics.
GUIDELINES FOR SURVEY PILOT
Stages of a Survey Pilot
A complete survey pilot includes 3 stages:
| Stage 1 - Pre-Pilot | Stage 2 - Content-focused Pilot | Stage 3 - Data-focused Pilot |
|---|---|---|
| Answer broad questions about survey design and context through qualitative interviews and focus group discussions | Refine overall order and structure, wording of specific questions, and translations.
Check completeness of answer choice options, response variance, survey length. |
Validate programming, export a sample dataset, check dataset structure and completeness, test all data quality checks |
| Early, printable draft, and/or qualitative instruments | A translated, printable, complete draft | A translated, programmed, final draft |
| Pen-and-Paper | Pen-and-Paper | Tablet/Phone-based |
Are all 3 stages of piloting necessary for every survey?
All three stages of piloting are not necessarily required for every survey. If your survey is a brand new survey instrument, then it is necessary to start with Stage 1 - Pre-Pilot. If your survey is an adaptation of a well-designed questionnaire from a reliable source in the same country, then you may start the survey pilot with Stage 2 – Content focused pilot.
If your survey is an adaptation of a survey instrument from a previous data collection for the same project, and if significant revisions or additions have been made then start with Stage 2 – Content focused piloting. Otherwise in cases where you have not done any major changes to the adaptation, skip directly to Stage 3 i.e. Data-focused pilot.
Why do a pen-and-paper pilot for a CAPI survey?
Pen-and-paper pilots hold an advantage over CAPI surveys due to their flexibility in recording answers and qualitative observations. Pen-and-paper pilots are especially useful to:
- Record open-ended responses (critical for a pilot) more quickly / easily
- Draw lines and arrows between questions to suggest restructuring
- Record their observations and feedback in the margins
- Make notes of questionnaire wording or translation problems directly in the text
A good pilot will provide significant inputs to questionnaire design, and result in significant changes to content and structure. Changing programming in CAPI software is time-consuming and can create bugs (e.g. if order of questions shifts and all skip codes need to be re-programmed).
What is the timeline for a survey pilot?
Piloting should start 4-6 months before survey launch. Do not confuse the Pilot with field testing during the enumerator training. It typically involves significant changes to the survey instrument and/or protocols – which should always be made before enumerator training starts. See Timeline of Survey Pilot for details The earlier you start the better!
What should be tested during the pilot?
The pilot should test: questionnaire content Piloting Survey Content, questionnaire programming Questionnaire Programming and all survey protocols Piloting Survey Protocols
Who should be involved in a survey pilot?
Typically, survey pilots are done before the survey firm is on board. The DIME Field Coordinator plays a central role. Ideally other research team members (e.g. the Principal Investigator) will participate. For details, see Survey Pilot Participants
How to structure a survey pilot?
The success of a pilot often depends on logistics. See guidelines on effectively Structuring a Survey Pilot. You should develop a clear Pilot Protocol for each stage and share with the research team for approval in advance.
Tips and Reminders
- Throughout the questionnaire design process, and discussion with the research team, take notes of what needs to be piloted
- Take careful notes during the pilot, the discussions and clarifications will be important to include in the enumerator manual and to discuss at training
- Do a data-focused pilot for your backcheck questionnaire
- Hiring a local "mobilizer" to coordinate with respondents can facilitate piloting, particularly in urban areas or settings in which people are particularly busy. They explain the purpose of the survey and get consent, reducing down time between surveys. (though do note this may not be consistent with piloting sampling protocols)
CHECKLISTS FOR FIELD COORDINATORS
Field coordinators can also use the Preparing for the Survey and Refining the Questionnaire checklists to make sure that they have not forgotten anything during a survey pilot.
ADDITIONAL RESOURCES
Wiki Links
Timeline of Survey Pilot for details , Piloting Survey Content, Piloting Survey Protocols, Pilot Participants, Structuring a Survey Pilot Preparing for the Survey, Refining the Questionnaire
External Resources
LSMS e-Learning course on questionnaire design & piloting: http://lsms.adeptanalytics.org/course/Home_eng.html