Difference between revisions of "Survey Pilot Participants"

Jump to: navigation, search
 
(18 intermediate revisions by 4 users not shown)
Line 1: Line 1:
<onlyinclude>The [[Field Coordinator]] plays a central role. Ideally other members of the research team will also participate (particularly important for pre-pilots and content-focused pilots). Staff from the relevant Government Ministry or implementing partner should be included as much as feasible (depending on availability and direct engagement). Typically, it is most productive for all pilot participants (field coordinator, PIs, local partners) to accompany skilled interviewers and observe and take notes, rather than conducting the interviews themselves.  
Typically, conducting a [[Survey Pilot|survey pilot]] requires several stages of planning and discussions. Apart from the [[Impact Evaluation Team|impact evaluation team]], survey pilot participants include interviewers, respondents, and even local government agencies. For instance, during the actual [[Field Surveys|field data collection]], such as in a [[Computer-Assisted Personal Interviews (CAPI)|computer-assisted personal interview (CAPI)]], it is the interviewer who reads out the questions to the respondent. But in the piloting stages before the actual '''survey''', the [[Field Coordinator|field coordinators (FCs)]] and the [[Impact Evaluation Team#Principal Investigators (PIs)|principal investigators (PIs)]] oversee the overall [[Preparing for Field Data Collection|preparation]] and finalize the [[Survey Protocols|survey protocols]].
</onlyinclude>
 
== Read First ==
* Each of the pilot participants has a specific role during each of the three stages.
* Respondents should be as close to the target population as possible, but none of the [[Sampling|sample]] for the actual [[Survey Pilot|survey]] should be included in the pilot.
* Interviewers should be fluent in all local languages of the study area and the language of the [[Impact Evaluation Team|research team]].
* [[Impact_Evaluation_Team#Field_Coordinator|Field coordinators (FCs)]] play a central role in organizing, managing, and supervising the pilot.
* [[Impact Evaluation Team#Principal Investigators (PIs)|Principal investigator (PI)]] should be briefed regularly during the pilot if not participating in person.
* Other research team members such as the [[Impact Evaluation Team#Impact Evaluation Coordinator|impact evaluation coordinator]], research manager, programmers, and translators should also participate in pilot discussions.
 
== Participant Roles ==
{| class="wikitable" style="margin-left: auto; margin-right: auto; border: none;"
{| class="wikitable" style="margin-left: auto; margin-right: auto; border: none;"
|+ Pilot Participants and Characteristics
|-
|-
|-
|-
! style="text-align:center;"|
!
!
!
! style="height:50px; width:250px; text-align:center;"| Stage 1 - Pre-Pilot
! Stage 1 - Pre-Pilot
! style="height:50px; width:250px; text-align:center;"| Stage 2 - Content-focused Pilot
! Stage 2 - Content-focused Pilot
! style="height:50px; width:250px; text-align:center;"| Stage 3 - Data-focused Pilot
! Stage 3 - Data-focused Pilot
|-
|-
| '''Respondents'''
| '''Respondents'''
| ''Never use any respondents in the final sample. Neighboring villages / schools / firms are good options.''
|  
| Respondents for the pilot should be as similar as possible to respondents for the actual survey in terms of age, gender, education, and socioeconomic (income) status.
 
Neighboring villages, schools, or firms are good options for selecting respondents for the pilot.
 
Never use any of these respondents in the final sample.  
| style="vertical-align:top;"| Test the protocols for [[Sampling|sampling]] in this stage. Use these protocols to select respondents for the pilot.
 
 
 
This helps to find out if the '''sampling frame''' (list from which respondents are selected) needs to be revised. It also helps to reduce [[Selection Bias|selection bias]].  
| style="vertical-align:top;"| Re-test your '''sampling protocols'''.
 
 


| Respondents should be as similar as possible to actual respondents.
 
Include variations in age, gender, education, and income / socioeconomic status.
 
| Test sampling protocol by using it to select respondents.
Include revisions based on the previous round of '''piloting'''.
Helps troubleshoot (e.g. find out sampling frame is outdated) and reduces bias in selection.
| Test your sampling protocol, including any revisions based on previous piloting stage
|-
|-
| rowspan=5 | '''Interviewers'''
| rowspan=4 | '''Enumerators (or interviewers)'''
| ''Language''
|style="height:100px;"| '''Language'''
| Fully fluent (read/write/ speak) in local language and language of research team
| Fully '''fluent''' (can read, write and/or speak) in the local language(s) and the language of research team.
| Fully fluent (speaking and reading) in local language and language of research team
| Fully fluent in the local language(s) and the language of research team.
| Fully fluent (speaking) in the local language and language of research team
| Fully fluent in the local language(s) and the language of research team.
|-
|-
| ''Experience''
| '''Required experience'''
| Experience as qualitative enumerator
|style="vertical-align:top;"| '''Qualitative''' experience as an enumerator. For instance, experience in identifying issues in wording and structure of questions.
Sectoral knowledge  
 
|Experience as quantitative enumerator
'''Sectoral''' knowledge. For instance, knowledge about agricultural patterns in a survey that aims to collect data on farm output.
Sectoral knowledge
| style="vertical-align:top;"| '''Quantitative''' experience as an enumerator. For instance, experience in entering data electronically.
|Experience as quantitative enumerator
 
Experience with tablets
 
Sectoral knowledge.
| style="vertical-align:top;"|Quantitative experience as an enumerator.
 
 
 
Experience working with phones or tablets.
|-
|-
| ''Background''
| '''Background'''
| Diversity in age / race / religion / gender / etc
| style="vertical-align:top;"| The pool of interviewers should be diverse in terms of age, race, religion, gender etc.
| Be mindful of (and learn about) cultural requirements (should women interview women? Can women interview men?)
| style="vertical-align:top;"| Interviewers should be mindful of (and learn about) cultural considerations. For instance, if women can interview men, or if men can interview women in the study area.
| Interviewer pool should reflect decisions regarding composition of final survey team, based on previous stage
| style="vertical-align:top;"| The background of the team of interviewers should be similar to that of the final survey team, which is decided based on previous stages of '''piloting'''.
|-
|-
| ''Size of Team''
| '''Size of the team'''
| Very small (~2). Always accompanied by research team member.
|style="vertical-align:top;"| Very small - not more than 2.  
| Small (~2-4). Ideally each enumerator accompanied by research team member or other observer.
 
| Larger (~4-8). The more interviews the better for debugging and data checks.
A member of the research team should always accompany the enumerators.
|style="vertical-align:top;"| Small - between 2 and 4.  
 
Ideally, a member of the research team should monitor each enumerator.
|style="vertical-align:top;"| Large - between 4 and 8.
 
More enumerators help simplify the process of [[Monitoring Data Quality|monitoring data quality]] and '''debugging''' (identifying errors) in the [[Questionnaire Programming|coded instrument]].
|-
| '''Survey firm'''
|
| Not part of this stage.
 
This stage is typically conducted before survey firm is on board.
| Not part of this stage.
 
This stage is typically conducted before survey firm is on board.
| Part of this stage.  
 
In fact, the survey firm should lead the pilot in this round.  
|-
|-
| ''Who?''
| '''Members of the research team'''
| Typically done before survey firm is on board, using local research assistants (STCs hired through the WB or a local PI), local government staff, or university students
|
| Typically done before survey firm is on board, using local research assistants (STCs hired through the WB or a local PI), local government staff, or university students
|style="vertical-align:top;"| The [[Impact Evaluation Team#Field Coordinators (FCs)|field coordinators (FCs)]]. Ideally they should participate in each stage.
| Best case scenario: survey firm leads. Staff who will be field supervisors for the upcoming survey act as interviewers. They gain experience with the instrument, which also helps for enumerator training.  
 
The [[Impact Evaluation Team#Principal Investigators (PIs)|principal investigators (PIs)]] should participate directly. They should also conduct daily discussions to '''debrief''' (exchange notes with) the enumerators.
|style="vertical-align:top;"| The field coordinators (FCs).
 
 
The [[Impact Evaluation Team#Impact Evaluation Coordinator|impact evaluation coordinator]] (or research manager) should participate directly. 
|style="vertical-align:top;"| The field coordinators (FCs).
 
 
The '''programmer''' should participate directly (to debug the instrument in real time)
 
Members of the research team who will be '''field supervisors''' in the actual survey should act as interviewers in this stage. This gives them experience with the instrument, which helps them conduct [[Enumerator Training|enumerator training]].
|-
|-
| '''Research Team'''
| '''Others'''
| ''Field Coordinator participates in all stages''
|  
| PI should participate directly. If not possible, plan for daily debriefs and discussions.  
| Local [[Impact Evaluation Team#Research Assistants (RAs|research assistants (RAs)]]. Includes short-term consultants (STCs) hired through the World Bank or through a local principal investigator (PI).
| Research Manager or Impact Evaluation Coordinator should participate directly.
 
| Programmer participates directly (to adapt / debug in real time)
Local government staff.
 
University students.
| Local research assistants (RAs).
 
 
 
 
 
Local government staff.
 
University students.
| Local research assistants (RAs).
 
 
 
 
 
Local government staff.
 
University students.
|}
|}


== Back to Parent ==
== Guidelines ==
Finally, the pilot participants should keep certain best practices in mind while planning and conducting a [[Survey Pilot|survey pilot]]. These can improve the outcomes of the pilot, for instance, by reducing the down-time (gap) between '''surveys'''.
* Throughout the process of planning the pilot, discuss with other members of the [[Impact Evaluation Team|research team]], and take note of what needs to be part of the pilot.
* Take careful notes of the clarifications that come up during the [[Structuring a Survey Pilot#Review Sessions|review sessions]]. These will be an important part of the [[Enumerator Training#Enumerator Manual|enumerator manual]] which supervisors use to train '''enumerators.
* Hire a local mobilizer to coordinate with respondents. Mobilizers explain the purpose behind conducting the '''survey''' to the respondents, and facilitate the process of obtaining consent. This is particularly helpful in urban areas, or in cases where respondents are busy.


This article is a part of the topic [[Survey Pilot]].
== Related Pages ==
[[Special:WhatLinksHere/Survey_Pilot_Participants|Click here for pages that link to this topic.]]


== Additional Resources ==
== Additional Resources ==
Please add here related articles, including a brief description and link.  
* DIME Analytics (World Bank), [https://osf.io/wmpvh/ Team roles]
 
* DIME Analytics (World Bank), [https://osf.io/63uv9/ Guidelines on piloting surveys]
* DIME Analytics (World Bank), [https://osf.io/rqb5m/ Survey instrument design and pilot]
* DIME Analytics (World Bank), [https://osf.io/6rvkj/ Survey preparing]
* DIME Analytics (World Bank), [https://osf.io/ek9ab/ Monitoring survey content]
* DIME Analytics (World Bank), [https://osf.io/4q6kb/ Monitoring data quality]
[[Category: Survey Pilot]]
[[Category: Survey Pilot]]

Latest revision as of 18:50, 14 August 2023

Typically, conducting a survey pilot requires several stages of planning and discussions. Apart from the impact evaluation team, survey pilot participants include interviewers, respondents, and even local government agencies. For instance, during the actual field data collection, such as in a computer-assisted personal interview (CAPI), it is the interviewer who reads out the questions to the respondent. But in the piloting stages before the actual survey, the field coordinators (FCs) and the principal investigators (PIs) oversee the overall preparation and finalize the survey protocols.

Read First

  • Each of the pilot participants has a specific role during each of the three stages.
  • Respondents should be as close to the target population as possible, but none of the sample for the actual survey should be included in the pilot.
  • Interviewers should be fluent in all local languages of the study area and the language of the research team.
  • Field coordinators (FCs) play a central role in organizing, managing, and supervising the pilot.
  • Principal investigator (PI) should be briefed regularly during the pilot if not participating in person.
  • Other research team members such as the impact evaluation coordinator, research manager, programmers, and translators should also participate in pilot discussions.

Participant Roles

Stage 1 - Pre-Pilot Stage 2 - Content-focused Pilot Stage 3 - Data-focused Pilot
Respondents Respondents for the pilot should be as similar as possible to respondents for the actual survey in terms of age, gender, education, and socioeconomic (income) status.

Neighboring villages, schools, or firms are good options for selecting respondents for the pilot.

Never use any of these respondents in the final sample.

Test the protocols for sampling in this stage. Use these protocols to select respondents for the pilot.


This helps to find out if the sampling frame (list from which respondents are selected) needs to be revised. It also helps to reduce selection bias.

Re-test your sampling protocols.



Include revisions based on the previous round of piloting.

Enumerators (or interviewers) Language Fully fluent (can read, write and/or speak) in the local language(s) and the language of research team. Fully fluent in the local language(s) and the language of research team. Fully fluent in the local language(s) and the language of research team.
Required experience Qualitative experience as an enumerator. For instance, experience in identifying issues in wording and structure of questions.

Sectoral knowledge. For instance, knowledge about agricultural patterns in a survey that aims to collect data on farm output.

Quantitative experience as an enumerator. For instance, experience in entering data electronically.


Sectoral knowledge.

Quantitative experience as an enumerator.


Experience working with phones or tablets.

Background The pool of interviewers should be diverse in terms of age, race, religion, gender etc. Interviewers should be mindful of (and learn about) cultural considerations. For instance, if women can interview men, or if men can interview women in the study area. The background of the team of interviewers should be similar to that of the final survey team, which is decided based on previous stages of piloting.
Size of the team Very small - not more than 2.

A member of the research team should always accompany the enumerators.

Small - between 2 and 4.

Ideally, a member of the research team should monitor each enumerator.

Large - between 4 and 8.

More enumerators help simplify the process of monitoring data quality and debugging (identifying errors) in the coded instrument.

Survey firm Not part of this stage.

This stage is typically conducted before survey firm is on board.

Not part of this stage.

This stage is typically conducted before survey firm is on board.

Part of this stage.

In fact, the survey firm should lead the pilot in this round.

Members of the research team The field coordinators (FCs). Ideally they should participate in each stage.

The principal investigators (PIs) should participate directly. They should also conduct daily discussions to debrief (exchange notes with) the enumerators.

The field coordinators (FCs).


The impact evaluation coordinator (or research manager) should participate directly.

The field coordinators (FCs).


The programmer should participate directly (to debug the instrument in real time)

Members of the research team who will be field supervisors in the actual survey should act as interviewers in this stage. This gives them experience with the instrument, which helps them conduct enumerator training.

Others Local research assistants (RAs). Includes short-term consultants (STCs) hired through the World Bank or through a local principal investigator (PI).

Local government staff.

University students.

Local research assistants (RAs).



Local government staff.

University students.

Local research assistants (RAs).



Local government staff.

University students.

Guidelines

Finally, the pilot participants should keep certain best practices in mind while planning and conducting a survey pilot. These can improve the outcomes of the pilot, for instance, by reducing the down-time (gap) between surveys.

  • Throughout the process of planning the pilot, discuss with other members of the research team, and take note of what needs to be part of the pilot.
  • Take careful notes of the clarifications that come up during the review sessions. These will be an important part of the enumerator manual which supervisors use to train enumerators.
  • Hire a local mobilizer to coordinate with respondents. Mobilizers explain the purpose behind conducting the survey to the respondents, and facilitate the process of obtaining consent. This is particularly helpful in urban areas, or in cases where respondents are busy.

Related Pages

Click here for pages that link to this topic.

Additional Resources