https://dimewiki.worldbank.org/api.php?action=feedcontributions&user=Maria+ruth+jones&feedformat=atomDimewiki - User contributions [en]2024-03-28T10:34:47ZUser contributionsMediaWiki 1.37.2https://dimewiki.worldbank.org/index.php?title=Primary_Data_Collection&diff=8446Primary Data Collection2022-05-06T02:36:31Z<p>Maria ruth jones: </p>
<hr />
<div>== Data Quality Assurance Plan ==<br />
The [[Impact Evaluation Team|research team]] must draft a [[Data Quality Assurance Plan|data quality assurance plan]], and share it with everyone in the '''research team''', as well as the [[Survey Firm|survey firm]] before starting with data collection. A '''data quality assurance plan''' considers everything that could go wrong ahead of time, and makes a plan to resolve these issues. Some of the issues that can affect [[Monitoring Data Quality|data quality]] include errors in [[Questionnaire Programming|programming]] or [[Questionnaire Translation|translation]], '''attrition''' (or dropping out of respondents during a [[Field Surveys|survey]], and faulty '''tablets''' used during [[Computer-Assisted Personal Interviews (CAPI)|computer-assisted personal interviews (CAPI)]], among others. A comprehensive '''data quality assurance plan''' has 3 major components for each of the following stages - '''before''', '''during''', and '''after''' data collection.<br />
=== Before data collection ===<br />
Before data collection, the '''research team''' can include the following in the '''data quality assurance plan''':<br />
* '''Survey design and programming.''' Make sure the [[Questionnaire Design|instrument design and structure]] are in line with the context of the study. Use the [[Survey Pilot|pilot]] to review and revise the '''instrument'''. Hire a professional translator to perform the [[Questionnaire Translation|translation]]. Check the [[Questionnaire Programming|programmed instrument]] for bugs, and make sure all '''skip patterns''' and '''repeat groups''' work properly.<br />
<br />
* '''Enumerator training.''' [[Enumerator Training|Train enumerators]] and conduct regular feedback sessions with them to refine the [[Checklist: Content-focused Pilot|survey content]] and [[Checklist:_Piloting_Survey_Protocols|protocols]]. Wherever possible, conduct [[Survey Pilot#Pen-and-Paper Pilots|pen-and-paper pilots]], since in that case it is easier for '''enumerators''' to write down the issues they are facing. Make sure enumerators conduct several practice interviews before the actual [[Field Surveys|fieldwork]] starts.<br />
<br />
=== During data collection ===<br />
During data collection, the '''research team''' can include the following in the '''data quality assurance plan''':<br />
* '''Communication and reporting.''' Clear communication is important to ensure that both '''enumerators''' and '''respondents''' are able to understand the questions in the survey instrument. It also allows [[Impact Evaluation Team#Field Coordinators (FCs)|field coordinators]] to regularly discuss issues faces by '''enumerators'''. For instance, enumerators may face issues like faulty equipment or connectivity issues, which can affect the quality of data.<br />
<br />
* '''Spot checks and field monitoring.''' '''Field coordinators (FCs)''' and '''supervisors''' should monitor the performance of enumerators through unannounced spot checks. Spot checkers accompany enumerators and observe the interview, taking detailed notes to provide feedback and troubleshooting tips. The objective is to assess whether the enumerator understands they survey instrument and is following all survey protocols. There should be a clear list of parameters that spotecheckers should use to judge performance. They should share feedback after (not during) the interviews. Spotcheckers should fill in a '''tracking sheet''' or a form that records observations about each enumerator who works under a supervisor. Ideally some spot checks are done by people who are independent of the data collection firm, especially in the first weeks of data collection, but supervisors should also conduct spot checks for the duration of the survey. <br />
<br />
* '''Minimize attrition.''' There are several reasons for '''attrition''' of respondents. For instance, it is possible that the respondent moved away from the location of the study, or refuses to participate. It is important to first identify the reason for '''attrition'''. Generally, attrition rates of more than 5% are considered poor, and the research team must try to resolve these issues. High attrition rates can affect the quality of data and introduce bias in the results of a study.<br />
<br />
* '''Back checks and real-time checks.''' At the same time, it is also important to constantly [[Monitoring Data Quality|monitor quality]] of every new round of data shared by the '''field teams'''. In a [[Back Checks|back check]], an independent survey team asks some selected questions to a randomly selected subset of respondents again to validate the answers. <br />
<br />
* ''' High Frequency checks.''' While the survey is ongoing, the research team should scrutinize the data in real-time through automated [[Monitoring_Data_Quality#High_Frequency_Checks|high frequency checks]].<br />
<br />
=== After data collection ===<br />
After data collection ends, the [[Survey Firm|survey firm]] usually provides a final '''field report'''. This report can be used to improve data quality in the last stage of the data collection process. It can provide qualitative information to the '''research team''' about everything that could not been captured by the survey instrument, such as :<br />
* '''Issues in understanding.''' Sometimes respondents do not understand a question and answer randomly. This information is especially important for the research team if a study or experiment only shows marginal impact.<br />
<br />
* '''Limited option choices.''' Sometimes respondents may convey that the option choices for particular questions were not comprehensive, which can also affect the quality of data. <br />
<br />
* '''Other feedback.''' It also allows the research team to understand issues like size and structure of the communities that were part of the [[Sampling|sample]]. Such information is often useful to weight each group within a sample differently, which can improve accuracy of results.</div>Maria ruth joneshttps://dimewiki.worldbank.org/index.php?title=Informed_Consent&diff=8444Informed Consent2022-05-02T20:15:06Z<p>Maria ruth jones: </p>
<hr />
<div>Before involving any '''human subject''' in a research study, the [[Impact Evaluation Team|research team]] must obtain '''informed consent''' from each individual. '''Informed consent''' means that the '''research team''' must clearly mention all possible risks and benefits to an individual from participating in a study, either in a [[Survey Pilot|pilot]] or as a [[Survey Pilot Participants|respondent]] in the actual study. This is done through an informed consent module conducted at the start of a survey interview. Based on this information, the individuals then must agree to participate without any use of force, fraud, or illegitimate incentives beyond those that are approved for use in research. Another aspect of '''informed consent''' is that the '''human subjects''' can withdraw their consent at any point during the study. The form for obtaining '''informed consent''' is one of the first elements of [[Questionnaire Design|drafting survey instruments]] (or questionnaires).<br />
<br />
==Read First==<br />
* '''Informed consent''' is a critical component of [[Research Ethics | ethical research]].<br />
* The [[Impact Evaluation Team|research team]] should allocate a dedicated session to '''informed consent'''during [[Enumerator Training|enumerator training]].<br />
* All '''human subjects''' must provide '''informed consent''' before any research activity begins. A '''human subject''' is any individual about whom the '''research team''' collects data through an intervention, or interaction with the individual.<br />
* Informed consent can be collected orally, in writing, or through SMS. [https://www.worldbank.org/en/research/dime/data-and-analytics DIME Analytics] has created the following '''boilerplate''' [https://github.com/worldbank/dime-standards/tree/master/dime-research-standards/pillar-1-research-ethics/research-ethics-resources/informed-consent-templates templates for informed consent].<br />
<br />
== Informed Consent and Ethics ==<br />
Informed consent is essential to [[Research Ethics | ethical research]], which upholds the principles of autonomy, beneficence, and justice:<br />
<br />
*'''Autonomy''' means that each person should be given the respect, time, and opportunity necessary to make his or her own decisions, with adequate information and without undue pressure. It requires that vulnerable populations such as children and prisoners be protected.<br />
*'''Beneficence''' is the duty to protect subjects from harm and to ensure that they have equal opportunities to experience possible benefits of participation.<br />
*'''Justice''' requires to treat all subjects equitably and to distribute benefits and burdens fairly.<br />
<br />
The following sections explain how to implement informed consent in a manner that properly upholds these three principles.<br />
<br />
==Protocols==<br />
The process of informed consent is guided by a project-specific consent form with which all subjects of the study are presented. In a standard household survey, the enumerator will read or present the consent form to the subject. The consent form must advise subjects of any information or adverse event that might affect their willingness to participate. The consent form must be presented in a language that the research subject can easily understand. Subjects must then be given the opportunity to discuss any questions or concerns before consenting. <br />
<br />
The research team must give subjects sufficient time to consider whether they want to participate. If subjects choose to consent, they provide their signature. Oral consent (i.e. no signature) might be [[IRB Approval | approved by IRB]] in certain cases. For minors, researchers should obtain the minor’s informed consent as well as and the permission of a parent/guardian. In these cases, the language of the informed consent process should be age-appropriate. <br />
<br />
The importance of informed consent cannot be overstated; for this reason, it is advisable to allocate at least 3 hours during the [[Enumerator Training | enumerator training]] to discuss its content and to practice it in pairs and in front of the class.<br />
<br />
==Elements of Informed Consent==<br />
<br />
The informed consent should provide the subject with information on all of the following points below. Note that the IRB may permit a waiver of one or more of its elements upon request. <br />
<br />
*Purpose of the research, including what the study is about and why the subject was selected<br />
*Participation and withdrawal information, stating that participation is voluntary and the subject can refuse to answer any questions and/or withdraw at any time without a penalty or loss of benefits to which s/he is otherwise entitled<br />
*A description of the study procedures in chronological order (i.e. for surveys, the topics covered, the timing, and the duration)<br />
*Identification of any procedures that are experimental<br />
*Expected duration of the subject’s participation<br />
*Risks and benefits, including a description of any reasonably foreseeable risks, discomforts, inconveniences, and how these will be managed; as well as a description of any benefits to subjects and to society or lack thereof – if the subject will not benefit from participation and the research only helps the public at large, this must be clearly stated<br />
*Payment for participation, including a mention of any payment or gift that will be provided to compensate subjects for their time, provided that the payment or gift is not coercive in amount or method of distribution<br />
*Alternative to participation<br />
*Confidentiality, including a mention of anyone who will have access to the data and a brief description of the procedures for maintaining confidentiality of records identifying the subject <br />
*Identification of investigators, including names and contact details of Field Coordinator, survey firm staff, and of Principal Investigators<br />
*Explanation of whom to contact for answers to pertinent questions about the research and research subjects' rights<br />
*Name of the sponsor(s), when appropriate<br />
*Rights of research subjects, including contact details of the IRB (national IRB, if obtained, or others)<br />
*If consent given, a signature of subject, including a statement that all described procedures have been understood and that any question has been answered, and that the respondent agrees to participate in this study<br />
<br />
The consent should not contain any exculpatory language through which subjects waive any of their legal rights or release the research team from liability for negligence.<br />
<br />
== Related Pages ==<br />
[[Special:WhatLinksHere/Informed_Consent|Click here for pages that link to this topic.]]<br />
<br />
== Additional Resources ==<br />
* Berk Özler (World Bank), [https://blogs.worldbank.org/impactevaluations/research-adolescents-issues-surrounding-consent Research with adolescents: Issues surrounding consent]<br />
* Berk Özler (World Bank), [http://blogs.worldbank.org/impactevaluations/taking-ethics-seriously-response-1 Taking Ethics Seriously: Response #1]<br />
* Harvard, [https://economics.harvard.edu/files/economics/files/sample_interview_consent_form_3-29-2013.doc Sample interview consent form from the Department of Economics] <br />
* ICPSR (University of Michigan), [https://www.icpsr.umich.edu/icpsrweb/content/datamanagement/confidentiality/conf-language.html Recommended Informed Consent Language for Data Sharing]<br />
* London School of Economics, [https://info.lse.ac.uk/staff/services/Policies-and-procedures/Assets/Documents/infCon.pdf Guidelines on informed consent]<br />
* Martin Ravallion (World Bank), [https://blogs.worldbank.org/impactevaluations/taking-ethical-validity-seriously Taking Ethical Validity Seriously]<br />
* United States Department of Health & Human Services (HHS), [https://www.hhs.gov/ohrp/regulations-and-policy/guidance/checklists/index.html Informed Consent Checklist]<br />
<br />
[[Category: Research Ethics]]</div>Maria ruth joneshttps://dimewiki.worldbank.org/index.php?title=Research_Ethics&diff=8443Research Ethics2022-05-02T19:01:23Z<p>Maria ruth jones: </p>
<hr />
<div>[[Impact Evaluation Team|Impact evaluation teams]] (or '''research teams''') often work with, or have access to, datasets that contain sensitive or [[Personally Identifiable Information (PII)|personal identifiable information (PII)]] on individuals. They can either have [[Primary Data Collection|direct access]] (for instance, through a [[Field Surveys|field survey]]), or [[Secondary Data Sources|indirect access]] to this data (for instance, in the form of '''call data records (CDR)'''). In both cases, it is important to ensure that '''research teams''' act '''ethically'''. Broadly, '''ethical research''' means that research teams must [[IRB Approval|obtain necessary approvals]], [[Protecting Human Research Subjects|protect human subjects]], [[Informed Consent|obtain informed consent]], and [[De-identification|ensure confidentiality]]. This also means that the research team should clearly think about, and be transparent about [[Data Ownership|ownership]] of the data they use. '''Research ethics''' ensure that that steps taken to reach the outcomes of a study are just as important as the outcomes themselves, and help improve the validity of results of a study. Note that Research Ethics and [[Data Security|data security]] go hand-in-hand.<br />
<br />
== Read First ==<br />
* The [https://github.com/worldbank/dime-standards/tree/master/dime-research-standards/pillar-1-research-ethics DIME Research Ethics Guidelines] discuss key guidelines for researchers affiliated with [https://www.worldbank.org/en/research/dime DIME].<br />
* A '''human subject''' is a living individual about whom the [[Impact Evaluation Team|research team]] obtains [[Personally Identifiable Information (PII)|personal identifiable information (PII)]], either directly or indirectly.<br />
* The idea of '''research ethics''' has the following aspects - [[IRB Approval|ethics approvals]], [[Protecting Human Research Subjects|protecting human subjects]], [[Informed Consent|informed consent]], and [[De-identification|confidentiality]].<br />
* Each of these components is important to ensure that '''research teams''' stick to certain ethical standards while conducting research.<br />
* '''Research ethics''', along with other pillars of research - [[Pre-Registration|transparency]], [[Reproducible Research|reproducibility]], [[Data Security|data security]], and [[Publishing Data|data publication]] - ensures greater validity for the results of a research study.<br />
<br />
== Protecting Human Subjects == <br />
Members of the [[Impact Evaluation Team|research team]] must ensure that they [[Protecting Human Research Subjects|protect human research subjects]] and their rights, including the '''right to privacy'''. In this context, all living individuals whose sensitive or [[Personally Identifiable Information (PII)|personally identifiable information (PII)]] is contained in the datasets being used in the study are considered '''human subjects'''. Therefore, '''ethical research''' requires that all members of the '''research team''' who handle '''personally identifiable information (PII)''' must have up-to-date '''human research subjects certification'''. Two organizations that offer courses to certify research team members are:<br />
* [https://phrp.nihtraining.com/users/login.php Protecting Human Research Participants (PHRP)]<br />
* [https://about.citiprogram.org/en/series/human-subjects-research-hsr/ Collaborative Institutional Training Initiative (CITI)]<br />
<br />
'''Note''': It does not matter whether the research team has [[Primary Data Collection|direct]], or [[Secondary Data Sources|indirect]] access to the data. If the dataset being used in the study contains '''personally identifiable information (PII)''' about individuals, then research team must protect the '''right to privacy''' of every such individual. Right to privacy includes ensuring confidentiality, privacy, and anonymity of study participants.<br />
<br />
== Ethics Approvals == <br />
'''Institutional review boards (IRBs)''' are organizations that review and monitor research studies to [[Protecting Human Research Subjects|protect the rights]] of '''human subjects'''. The [[Impact Evaluation Team|research team]] must obtain [[IRB Approval|IRB approvals]] for studies that use [[Personally Identifiable Information (PII)|personally identifiable information]]. This is a powerful tool to promote '''ethical research''' because '''IRBs''' can deny research teams the right to use data if they do not follow proper guidelines while handling the data. <br />
<br />
'''Note''': Keep the following points in mind with respect to '''ethics approvals''':<br />
* It does not matter whether the research team collected the data [[Primary Data Collection|directly]], or [[Secondary Data Sources|indirectly]]. For example, consider a study that uses mobile phone location data to assess effectiveness of stay-at-home orders during COVID-19. In this case, the research team does not directly collect this data, but gets access to it through the telecom service provider. However, since this dataset contains sensitive information like names, phone numbers, and location of individuals, the research team must obtain an '''IRB approval'''.<br />
<br />
* In addition to '''IRB approvals''', the research team should also obtain approvals from local institutions in the location of the study. This will ensure that the study complies with local regulations, and does not violate any laws in that area, particular with respect to the '''right to privacy'''.<br />
<br />
== Informed Consent ==<br />
Before collecting personal data from any individual involved in a research study, the [[Impact Evaluation Team|research team]] must obtain [[Informed Consent|informed consent]] from each individual. This means that the research team must clearly mention all possible risks and benefits from participating in a study, either for the [[Survey Pilot|survey pilot]] or as a [[Survey Pilot Participants|respondent]] in the actual survey. This is done through an informed consent module conducted at the start of a survey interview.<br />
<br />
'''Note''': Keep the following points in mind regarding '''informed consent''':<br />
* The '''human subjects''' must be able to refuse participation in a study at any point during the study.<br />
<br />
* [https://www.worldbank.org/en/research/dime/data-and-analytics DIME Analytics] has created templates for [https://github.com/worldbank/dime-standards/blob/master/dime-research-standards/pillar-1-research-ethics/research-ethics-resources/informed-consent-templates/informed-consent-template-written.md written], [https://github.com/worldbank/dime-standards/blob/master/dime-research-standards/pillar-1-research-ethics/research-ethics-resources/informed-consent-templates/informed-consent-template-verbal.md oral], and [https://github.com/worldbank/dime-standards/blob/master/dime-research-standards/pillar-1-research-ethics/research-ethics-resources/informed-consent-templates/informed-consent-template-sms.md SMS] consent. Templates adapted to include COVID-19 risks are also available on the same [https://github.com/worldbank/dime-standards/tree/master/dime-research-standards/pillar-1-research-ethics/research-ethics-resources/informed-consent-templates repository].<br />
<br />
* The mechanism of '''informed consent''' applies only to cases where the research team itself is [[Primary Data Collection|collecting data]], such as through a [[Field Surveys|field survey]], because in such cases the research team is directly interacting with the participants.<br />
<br />
* However, in some cases, there is no opportunity for the research team to obtain '''informed consent'''. For example, consider cases where the research team only has [[Secondary Data Sources|indirect]] access to data, such as through '''call data records (CDR)''', or [[Administrative and Monitoring Data|administrative data]]. This does not mean that the '''right to privacy''' of individuals is not important in such a scenario. It simply means that the research team must use other mechanisms, such as ensuring '''confidentiality''' of sensitive information, which is discussed in the next section.<br />
<br />
== Confidentiality ==<br />
The [[Impact Evaluation Team|research team]] must also ensure complete '''anonymity''' and '''confidentiality'''. This means that the identity of study participants should remain hidden, and sensitive information of individuals should never be shared with anyone outside the research team. Research team members should be held personally liable for any actions that result in disclosure of sensitive and [[Personally Identifiable Information (PII)|personal]] data.<br />
<br />
'''Note''': Keep the following points in mind regarding '''confidentiality''' and '''anonymity''' of study participants:<br />
<br />
* All direct identifiers must be removed from working data sets as early in the research process as possible. This process is called [[De-identification|de-identification]].<br />
<br />
* '''De-identification''' must be done even in cases where the research team collects data [[Secondary Data Sources|indirectly]], for instance, in the case of [[Administrative and Monitoring Data|administrative data]].<br />
<br />
* At the time of '''de-identifying''' data, the research team must store an [[Encryption|encrypted]] version of the original raw data which still contains the '''personal identifiers'''.<br />
<br />
* Only those members of the research team who are listed by the [[IRB Approval|institutional review board (IRB)]] should have access to the '''encrypted''' versions of confidential information.<br />
<br />
* All research assistants '''(RAs)''', field coordinators '''(FCs)''', and other consultants or interns affiliated with the research team must sign '''non-disclosure agreements (NDAs)'''. These agreements should clearly state that they will not share any sensitive information with anyone outside the research team. For example, the [https://github.com/worldbank/dime-standards/blob/master/dime-research-standards/pillar-4-data-security/data-security-resources/dime-data-nda-mou.md DIME NDA] is an example of such an agreement.<br />
<br />
== Other Aspects of Ethical Research == <br />
=== Transparency ===<br />
One of the most common concerns in research is the possibility of [[Impact Evaluation Team|research teams]] manipulating the results of a study before publishing it in a journal. This is called '''publication bias''', and can waste precious time and money. In order to make results of a study more '''transparent''', the '''research team''' can [[Pre-Registration | pre-register]] their study with a '''registry'''. Some of the common registries are:<br />
* The [https://www.aeaweb.org/ American Economic Association (AEA)] hosts a [https://www.socialscienceregistry.org/ trial registry] specifically for [[Randomized Control Trials|randomized control trials]]. <br />
<br />
* The [https://www.3ieimpact.org/ International Initiative for Impact Evaluation (3ie)] hosts the [https://ridie.3ieimpact.org/ Registry for International Development Impact Evaluations (RIDIE)] for both, [[Experimental Methods|experimental]] and [[Quasi-Experimental Methods|quasi-experimental research]] in developing countries. <br />
<br />
* The [https://osf.io/institutions/cos/?gclid=Cj0KCQjw-_j1BRDkARIsAJcfmTFQUYhsJDTOljYCOKEsRzjNJo5QQHO_SSgFCglP5wxcF_l0zImzEaoaAvAuEALw_wcB Open Science Framework] offers a [https://osf.io/prereg/ pre-registration platform], and also houses its own [https://osf.io/registries registry].<br />
<br />
However, while ensuring '''transparency''', research teams must not forget concerns about privacy of participants. Therefore, the research team must make sure that they do not disclose or publish [[Personally Identifiable Information (PII)|personally identifiable information (PII)]] or confidential data, and should [[Encryption|encrypt]] all such data as soon as they receive it from the '''field teams''', or an external organisation.<br />
<br />
=== Reproducibility ===<br />
[[Reproducible Research|Reproducible research]] is the system of [[Data Documentation|documenting]] and [[Publishing Data|publishing]] results of an '''impact evaluation'''. At the very least, '''reproducibility''' allows other researchers to [[Data Analysis|analyze]] the same data to get the same results as the original study, which strengthens the conclusions of the original study. It is important to push researchers towards publishing '''reproducible research''' because the path to research findings is just as important as the findings themselves.<br />
<br />
[https://www.worldbank.org/en/research/dime/data-and-analytics DIME Analytics] has also drafted the [https://github.com/worldbank/dime-standards/tree/master/dime-research-standards/pillar-3-research-reproducibility DIME Research Reproducibility Standards], which lists guidelines for ensuring '''reproducibility''' of research.<br />
<br />
=== Data security ===<br />
[[Data Security|Data security]] is another important aspect of responsible research. The [[Impact Evaluation Team|research team]] must ensure that members of the '''research team''' who are not listed by the [[IRB Approval|IRB]] can not access any confidential data. Data can be confidential for multiple reasons, but the most common reason is that it contains [[Personally Identifiable Information (PII)|personally identifiable information (PII)]]. Other reasons include that the data was shared under a data usage license that requires the data to be kept confidential. <br />
<br />
The most important aspect of '''data security''' is [[Encryption|encryption]]. [https://www.worldbank.org/en/research/dime/data-and-analytics DIME Analytics] has drafted the [https://github.com/worldbank/dime-standards/blob/master/dime-research-standards/pillar-4-data-security/dime-data-security-guidelines.md DIME Data Security Policy], which lists guidelines for ensuring '''data security''' during research.<br />
<br />
=== Data publication ===<br />
[[Publishing Data|Data publication]] is the release of data and [[Data Documentation|data documentation]] following [[Primary Data Collection | data collection]] and [[Data Analysis | analysis]]. It also involves [[De-identification|de-identifying]] sensitive, or [[Personally Identifiable Information (PII)|personal information]] before publication, and ensures that research is both ethical, as well as [[Reproducible Research|reproducible]]. While '''de-identifying''', the [[Impact Evaluation Team|research team]] must store an [[Encryption|encrypted]] version of the original raw data which includes '''personal identifiers''' like names, addresses, location, etc.<br />
<br />
It is also important to keep in mind the risks associated with '''data publication''', such as concerns about '''right to privacy''' of '''human subjects'''. [https://www.worldbank.org/en/research/dime/data-and-analytics DIME Analytics] has drafted the [https://github.com/worldbank/dime-standards/tree/master/dime-research-standards/pillar-5-data-publication DIME Data Publication Standards], which lists guidelines for '''data publication'''. Finally, the research team must also think about [[Data Ownership|data ownership]] through a set of clear rules for use of research data.<br />
<br />
== Related Pages ==<br />
[[Special:WhatLinksHere/Research_Ethics|Click here for pages that link to this topic]].<br />
<br />
== Additional Resources ==<br />
* Berkeley Initiative for Transparency in the Social Sciences (BITSS), [https://www.bitss.org/research-transparency-mooc/pre-registration-and-pre-analysis-plans/ Pre-Registration and Pre-Analysis Plans]<br />
* Berk Özler (World Bank), [http://blogs.worldbank.org/impactevaluations/taking-ethics-seriously-response-1 Taking ethics seriously: Response #1]<br />
* Berk Özler (World Bank), [https://blogs.worldbank.org/impactevaluations/research-adolescents-issues-surrounding-consent Research with adolescents: issues surrounding consent]<br />
* David Evans (Center for Global Development), [https://www.cgdev.org/blog/practical-suggestions-more-ethical-social-science-rcts Practical Suggestions for More Ethical Social Science RCTs]<br />
* David McKenzie (World Bank), [https://blogs.worldbank.org/impactevaluations/a-pre-analysis-plan-checklist A pre-analysis plan checklist]<br />
* DIME Analytics (World Bank), [https://osf.io/3jzgb/ Research Ethics in Practice]<br />
* DIME Analytics (World Bank), [https://osf.io/ybj8s/ Survey Ethics]<br />
* Emma Cohn and Douglas MacKay, [https://dmackay.web.unc.edu/ethics-of-field-experiments-a-bibliography/ Ethics of Field Experiments: A Bibliography]<br />
* Martin Ravallion (World Bank), [https://blogs.worldbank.org/impactevaluations/taking-ethical-validity-seriously Taking ethical validity seriously] <br />
* J-PAL, [https://www.povertyactionlab.org/ethics Ethics]<br />
* J-PAL, [https://www.povertyactionlab.org/resource/ethical-conduct-randomized-evaluations Ethical conduct of randomized evaluations]<br />
* Laterite, [https://www.laterite.com/blog/ethics-approvals-and-research-permits-in-east-africa/ Ethics approvals and research permits in East Africa]<br />
<br />
[[Category: Reproducible Research]]</div>Maria ruth joneshttps://dimewiki.worldbank.org/index.php?title=Primary_Data_Collection&diff=8442Primary Data Collection2022-04-26T20:59:20Z<p>Maria ruth jones: </p>
<hr />
<div>'''Primary data collection''' is the process of gathering data through [[Field Surveys|surveys]], interviews, or experiments. A typical example of primary data is '''household surveys'''. In this form of data collection, researchers can personally ensure that primary data meets the standards of [[Monitoring Data Quality | quality]], availability, [[Power Calculations in Stata | statistical power]] and [[Sampling & Power Calculations | sampling]] required for a particular research question. With globally increasing access to specialized [[Software Tools |survey tools]], [[Survey Firm | survey firms]], and [[Training Guidelines: Content and Structure|field manuals]], primary data has become the dominant source for empirical inquiry in development economics<br />
== Read First ==<br />
* [[Field Surveys|Field surveys]] are one of the most effective medium for '''primary data collection'''. Depending on the research question, these interviews may take the form of household surveys, business (firm) surveys, or agricultural (farm) surveys.<br />
* The [[Impact Evaluation Team|research team]] must [[Field Management|plan]] and [[Preparing for Field Data Collection|prepare]] for '''primary data collection''' in advance.<br />
* <code>[[iefieldkit]]</code> is a Stata package that aids '''primary data collection'''. It currently supports three major components of this process: [[Iefieldkit#Before Data Collection|testing survey instruments]]; [[Iefieldkit#During Data Collection|survey completion]]; and [[Data Cleaning|data-cleaning]] and [[Iefieldkit#After Data Collection|survey harmonization]].<br />
<br />
== Overview ==<br />
While impact evaluations often benefit from [[Secondary Data Sources|secondary sources of data]] like [[Administrative and Monitoring Data|administrative data]], census data, or household data, these sources may not always be available. In such cases, the [[Impact Evaluation Team|research team]] will need to collect data directly using well-designed [[Computer-Assisted Personal Interviews (CAPI)|interviews]] and [[Field Surveys|surveys]], and the '''research team''' typically owns the data that it collects. However, even then, the research team must keep in mind certain [[Research Ethics|ethical concerns]] related to owning and handling sensitive, or [[Personally Identifiable Information (PII)|personally identifiable information (PII)]].<br />
<br />
Before moving on to the discussion of concerns about ownership and handling, however, it is important to understand the process of '''collecting primary data'''. The process of '''primary data collection''' consists of several steps, from [[Questionnaire Design|questionnaire development]], to [[Enumerator Training|enumerator training]]. Each of these steps are listed below, and require detailed [[Field Management|planning]], and coordination among the members of the '''research team'''. <br />
<br />
== Develop Questionnaire == <br />
The first step of '''primary data collection''' is to [[Questionnaire Design|design a survey instrument]] (or '''questionnaire'''). It is important to remember that drafting a questionnaire from scratch can be a time-consuming process, so the [[Impact Evaluation Team|research team]] should try to use existing resources as far as possible. While '''developing''' the questionnaire, keep the following things in mind:<br />
* '''Plan.''' The '''research team''' should start with a clear understanding of the [[Theory of Change|theory of change]] for the project. List key outcomes of interest, and the main variables that can be used to measure these outcomes. A good starting point for this is the [[Pre-Analysis Plan|pre-analysis plan]].<br />
<br />
* '''Modules.''' Divide the questionnaire into individual modules, each with a group of questions that are related to one aspect of the [[Field Surveys|survey]]. Unless the context of the study is entirely new, perform a [[Literature Review for Questionnaire|literature review]] of existing well-tested and reliable surveys to prepare the general structure of the questionnaire. One example of a resource for past studies and questionnaires is the [https://microdata.worldbank.org/index.php/home World Bank Microdata Library].<br />
<br />
* '''Measurement challenges.''' Often, '''research teams''' face challenges in measuring certain outcomes, for instance, abstract concepts (like empowerment), or socially sensitive topics that people do not wish to talk about (like drug abuse). In such cases, try to use indicators that are easy to identify, or build a level of comfort with respondents before moving to the sensitive topics.<br />
<br />
* '''Translation.''' [[Questionnaire Translation|Translating]] the questionnaire is a very important step. The '''research team''' must hire only professional translators to translate the questionnaire into all local languages that are spoken in the study location.<br />
<br />
== Pilot Questionnaire ==<br />
[[Survey Pilot|Survey pilot]] is the process of carrying out interviews and tests on different components of a survey, including [[Checklist: Content-focused Pilot|content]] and [[Checklist:_Data-focused_Pilot|protocols]]. A good '''pilot''' provides the research team with important feedback before they start the process of [[Primary Data Collection|data collection]]. This feedback can help the [[Impact Evaluation Team|research team]] review and improve [[Questionnaire Design|instrument design]], [[Questionnaire Translation|translations]], as well as [[Survey Protocols|survey protocols]] related to [[Checklist:_Piloting_Survey_Protocols|interview scheduling]], [[Sampling|sampling]], and [[Geo Spatial Data|geo data]].<br />
<br />
A '''pilot''' has [[Survey Pilot#Stages of a Survey Pilot|three stages]] - '''pre-pilot''', [[Checklist: Content-focused Pilot|content-focused pilot]], and [[Checklist:_Data-focused_Pilot|data-focused pilot]]. Typically, the '''pilot''' is carried out before [[Procuring a Survey Firm|hiring a survey firm]]. The '''research team''' must draft a clear [[Survey Pilot#Timeline|timeline]] for the '''pilot''', and allocate enough time for each component of the pilot. [https://www.worldbank.org/en/research/dime/data-and-analytics DIME Analytics] has also created the following checklists to assist researchers and [[Survey Pilot Participants|enumerators]] in preparing for, and implementing a pilot:<br />
*[[Preparing_for_the_survey_checklist|Checklist: Preparing for a survey pilot]]<br />
*[[Checklist:_Content-focused_Pilot|Checklist: Refining questionnaire content]]<br />
*[[Checklist:_Data-focused_Pilot|Checklist: Refining questionnaire data]]<br />
<br />
== Pilot Recruitment Strategy ==<br />
Besides testing [[Questionnaire Design|content]] and [[Survey Protocols|protocols]], it is also important for the [[Impact Evaluation Team|research team]] to '''pilot recruitment strategy''' before starting data collection. This is especially important in the following cases: <br />
* '''Smaller sample.''' Before finalizing the target population, the '''research team''' often surveys a smaller sample of the population that is selected for an intervention. The research team can use this to finalize the '''sampling frame''' (which is a list of individuals or units in a population from which a sample can be drawn), [[Sampling_&_Power_Calculations | sample size]], and [[Power_Calculations | statistical power]] based on which they can then start data collection.<br />
<br />
* '''Low or unknown take-up.''' '''Take-up rate''' is the percentage of eligible people who accept a benefit, or participate in data collection. Sometimes the '''take-up rates''' can be low, or unknown. In such cases, the research team should reconsider the [[Sampling|sampling]] strategy, and test it before starting data collection. <br />
<br />
* '''Different participation rates.''' Sometimes participation can differ based on factors like gender, age, social status, etc. This requires the research to consider different strategies like [[Stratified Random Sample|stratified sampling]].<br />
<br />
One of the ways to test the '''recruitment strategy''' is to test 3 different recruitment strategies, say, '''A''', '''B''', and '''C'''. The research team can then finalize the strategy that has the highest '''take-up rates'''. Another method is identifying the ideal incentives which can ensure higher participation by the eligible population. After finalizing a recruitment strategy, the research team can move on to drafting the [[Survey Firm TOR|terms of reference (TOR)]] for the data collection.<br />
<br />
== TOR: Create Budget and Plan Fieldwork ==<br />
After finalizing the '''survey instrument''' and '''recruitment strategy''', the [[Impact Evaluation Team|research team]] must prepare a detailed [[Survey Firm TOR|terms of reference (TOR)]] for hiring a [[Survey Firm|survey firm]]. The '''terms of reference (TOR)''' define the structure of the project, as well as the responsibilities of the '''survey firm'''. While preparing the '''TOR''', the '''research team''' must create a [[Survey Budget|survey budget]], and [[Field Management|plan the fieldwork]]. <br />
=== Create budget ===<br />
The '''research team''' should calculate standard, as well as project-specific costs, and prepare a '''survey budget'''. In this stage, the research team should also consider what [[Sampling|sample size]] it can afford for the data collection. This allows the research team to calculate expected costs of conducting a study, and compare these with the proposals of [[Survey Firm|survey firms]] that respond to the '''terms of reference (TOR)'''.<br />
<br />
=== Plan fieldwork ===<br />
It is also important to plan '''fieldwork''' in advance to give potential '''survey firms''' an idea of the responsibilities and tasks involved in the data collection. For the [[Impact Evaluation Team#Field Coordinators|field coordinators (FCs)]], this includes deciding number of interviews each enumerator will conduct in a day, number of field teams, modes of transport, and keeping extra buffer time for possible delays. Similarly, for the survey firm, this involves defining basic parameters like [[Sampling & Power Calculations|sample size]], [[Sampling|sampling strategy]], [[Timeline of Survey Pilot|timeline]], etc.<br />
<br />
== Contract Survey Firm == <br />
After the [[Impact Evaluation Team|research team]] finalizes and issues the '''terms of reference (TOR)''', multiple [[Survey Firm|survey firms]] can express interest in signing a contract with the '''research team.''' The research team will then select one of these '''survey firms''', and sign a contract with the selected survey firm. This completes the process of [[Procuring a Survey Firm|procuring a survey firm]].<br />
<br />
After signing the contract, the research team and the survey firm should agree on the parameters defined in the '''terms of reference (TOR)''', the '''survey timeline''', and discuss possible scenarios and common issues that might arise during data collection. One such issue that the research team and the survey firm must discuss in detail is the [[Data Quality Assurance Plan|data quality assurance plan]].<br />
<br />
== Data Quality Assurance Plan ==<br />
The [[Impact Evaluation Team|research team]] must draft a [[Data Quality Assurance Plan|data quality assurance plan]], and share it with everyone in the '''research team''', as well as the [[Survey Firm|survey firm]] before starting with data collection. A '''data quality assurance plan''' considers everything that could go wrong ahead of time, and makes a plan to resolve these issues. Some of the issues that can affect [[Monitoring Data Quality|data quality]] include errors in [[Questionnaire Programming|programming]] or [[Questionnaire Translation|translation]], '''attrition''' (or dropping out of respondents during a [[Field Surveys|survey]], and faulty '''tablets''' used during [[Computer-Assisted Personal Interviews (CAPI)|computer-assisted personal interviews (CAPI)]], among others. A comprehensive '''data quality assurance plan''' has 3 major components for each of the following stages - '''before''', '''during''', and '''after''' data collection.<br />
=== Before data collection ===<br />
Before data collection, the '''research team''' can include the following in the '''data quality assurance plan''':<br />
* '''Survey design and programming.''' Make sure the [[Questionnaire Design|instrument design and structure]] are in line with the context of the study. Use the [[Survey Pilot|pilot]] to review and revise the '''instrument'''. Hire a professional translator to perform the [[Questionnaire Translation|translation]]. Check the [[Questionnaire Programming|programmed instrument]] for bugs, and make sure all '''skip patterns''' and '''repeat groups''' work properly.<br />
<br />
* '''Enumerator training.''' [[Enumerator Training|Train enumerators]] and conduct regular feedback sessions with them to refine the [[Checklist: Content-focused Pilot|survey content]] and [[Checklist:_Piloting_Survey_Protocols|protocols]]. Wherever possible, conduct [[Survey Pilot#Pen-and-Paper Pilots|pen-and-paper pilots]], since in that case it is easier for '''enumerators''' to write down the issues they are facing. Make sure enumerators conduct several practice interviews before the actual [[Field Surveys|fieldwork]] starts.<br />
<br />
=== During data collection ===<br />
During data collection, the '''research team''' can include the following in the '''data quality assurance plan''':<br />
* '''Communication and reporting.''' Clear communication is important to ensure that both '''enumerators''' and '''respondents''' are able to understand the questions in the survey instrument. It also allows [[Impact Evaluation Team#Field Coordinators (FCs)|field coordinators]] to regularly discuss issues faces by '''enumerators'''. For instance, enumerators may face issues like faulty equipment or connectivity issues, which can affect the quality of data.<br />
<br />
* '''Field monitoring.''' '''Field coordinators (FCs)''' and '''supervisors''' should monitor the performance of enumerators. There should be a clear list of parameters that supervisors will use to judge performance. They should also share useful feedback after (not during) the interviews, and ensure that respondents are able to understand the questions correctly. Ask supervisors to fill in a '''tracking sheet''' or a form that records observations about each enumerator who works under a supervisor. <br />
<br />
* '''Minimize attrition.''' There are several reasons for '''attrition''' of respondents. For instance, it is possible that the respondent moved away from the location of the study, or refuses to participate. It is important to first identify the reason for '''attrition'''. Generally, attrition rates of more than 5% are considered poor, and the research team must try to resolve these issues. High attrition rates can affect the quality of data and introduce bias in the results of a study.<br />
<br />
* '''Back checks and real-time checks.''' At the same time, it is also important to constantly [[Monitoring Data Quality|monitor quality]] of every new round of data shared by the '''field teams'''. In a [[Back Checks|back check]], an experienced enumerator asks some selected questions to the respondent again to compare the answers. Similarly, supervisors can conduct real-time data quality checks (or '''spot checks'''), and [[Monitoring_Data_Quality#High_Frequency_Checks|high frequency checks]] to check the quality of responses.<br />
<br />
=== After data collection ===<br />
After data collection ends, the [[Survey Firm|survey firm]] usually provides a final '''field report'''. This report can be used to improve data quality in the last stage of the data collection process. It can provide qualitative information to the '''research team''' about everything that could not been captured by the survey instrument, such as :<br />
* '''Issues in understanding.''' Sometimes respondents do not understand a question and answer randomly. This information is especially important for the research team if a study or experiment only shows marginal impact.<br />
<br />
* '''Limited option choices.''' Sometimes respondents may convey that the option choices for particular questions were not comprehensive, which can also affect the quality of data. <br />
<br />
* '''Other feedback.''' It also allows the research team to understand issues like size and structure of the communities that were part of the [[Sampling|sample]]. Such information is often useful to weight each group within a sample differently, which can improve accuracy of results.<br />
<br />
== Obtain Ethical Approval == <br />
Members of the [[Impact Evaluation Team|research team]] must ensure that they [[Protecting Human Research Subjects|protect the rights]] of all '''human subjects''' in a study, including the '''right to privacy'''. In this context, all living individuals whose sensitive or [[Personally Identifiable Information (PII)|personally identifiable information (PII)]] is contained in the data collected by the '''research team''' are considered '''human subjects'''. In this step, the '''research team''' must consider issues like [[IRB Approval|IRB approvals]], [[Informed Consent|informed consent]], and [[Data Security|data security]].<br />
<br />
=== IRB approvals ===<br />
The [[Impact Evaluation Team|research team]] must obtain [[IRB Approval|IRB approvals]] for studies that use [[Personally Identifiable Information (PII)|personally identifiable information]]. '''Institutional review boards (IRBs)''' are organizations that review and monitor research studies ensure the welfare of '''human subjects'''. <br />
<br />
In addition to '''IRB approvals''', the research team should also obtain approvals from local institutions in the location of the study. This will ensure that the study complies with local regulations, and does not violate any laws in that area, particular with respect to the '''right to privacy'''.<br />
<br />
=== Informed consent ===<br />
Before involving any individual in a research study, the [[Impact Evaluation Team|research team]] must obtain [[Informed Consent|informed consent]] from each individual. This means that the '''research team''' must clearly mention all possible risks and benefits from participating in a study, either for the [[Survey Pilot|survey pilot]] or as a [[Survey Pilot Participants|respondent]] in the actual data collection.<br />
<br />
=== Data security ===<br />
The [[Impact Evaluation Team|research team]] must understand the [[Research Ethics|ethics]] and rules for [[Data Security|data security]], and should use proper tools for [[Encryption | encryption]] and [[De-identification | de-identification]] of [[Personally Identifiable Information_(PII)|personally identifiable information (PII)]]. '''Data security''' also means ensuring that members of the '''research team''' who are not listed by the [[IRB Approval|IRB]] can not access any '''confidential data'''. Data can be '''confidential''' for multiple reasons, but the most common reason is that it contains '''personally identifiable information (PII)'''. Other reasons include that the data was shared under a data usage license that requires the data to be kept confidential.<br />
<br />
== Train Enumerators == <br />
Finally, the [[Impact Evaluation Team|research team]] must plan and conduct comprehensive [[Enumerator Training|enumerator training]]. '''Enumerator training''' is usually a joint effort between the '''research team''' and the [[Survey Firm|survey firm]]. The [[Training Guidelines: Content and Structure|content and structure]] of the training can be divided into the following sections: <br />
=== Objectives ===<br />
The purpose of the training should be to ensure that all '''field staff''' know all the [[Survey Protocols|survey protocols]]. Also ensure that '''enumerators''' understand all questions in the [[Questionnaire Design|survey instrument]]. They should also be comfortable with using '''tablets''' used in [[Computer-Assisted Personal Interviews (CAPI)|CAPI]], or paper forms used for [[Pen-and-Paper Personal Interviews (PAPI)|PAPI]]. Finally, ensure that all '''field staff''' know and understand their duties.<br />
<br />
=== Planning ===<br />
In terms of planning, the '''survey firm''' should coordinate with the research team on logistics, such as deciding a venue for the training, printing '''field manuals''', questionnaires, and the agenda for each session. It is a good idea to recruit '''enumerators''' and experienced trainers in advance. The [[Impact Evaluation Team#Field Coordinators (FCs)|field coordinator (FC)]] should finalize the '''field manual''', update the [[Enumerator Training#Enumerator Manual|training manual]], and also make sure the trainers are aware of the objectives and the context of the study. Finally, prepare quizzes for the assessments, and plan practice interviews for enumerators.<br />
=== Components ===<br />
Generally, the '''survey firm''' leads the training, while the '''field coordinator (FC)''' monitors the sessions. The training should explain the context and [[Piloting Survey Content|content]] of the questionnaire, [[Experimental Methods|methods]] used for data collection, [[Sampling|sample selection]], and [[Survey Protocols|protocols]]. It is also important to anticipate potential issues that enumerators may face, and train them on how to handle these issues.<br />
=== Assessment ===<br />
Conduct assessments and quizzes to select enumerators for the actual data collection. Select enumerators based on scores on these quizzes, observations of the '''field coordinator (FC)''' and '''supervisors''', communication skills, and familiarity with the '''survey instrument'''. Always train more enumerators than you need for the data collection. Provide regular feedback during the training to ensure transparency.<br />
=== Tips and ideas ===<br />
Finally, follow '''best practices''' in training. Examples of these practices include taking notes during the sessions, recording training sessions so that enumerators can watch them again later, and regular practice interviews.<br />
== Related Pages ==<br />
[[Special:WhatLinksHere/Primary_Data_Collection|Click here for pages that link to this topic]].<br />
<br />
== Additional Resources ==<br />
* DIME Analytics (World Bank), [https://osf.io/te3c7/ Preparing for Data Collection]<br />
* DIME Analytics (World Bank), [https://osf.io/wb86g/ Training Data Collectors]<br />
* DIME Analytics (World Bank), [https://osf.io/n7ctd/ SurveyCTO Guide For Data Collectors]<br />
* IPA-JPAL-SurveyCTO, [https://d37djvu3ytnwxt.cloudfront.net/assets/courseware/v1/30701099ebb94072fdfcf1ec96d8227a/asset-v1:MITx+JPAL102x+1T2017+type@asset+block/4.6_High_quality_data_accurate_data.pdf Collecting High Quality Data]<br />
* Oxfam, [http://policy-practice.oxfam.org.uk/publications/planning-survey-research-578973 Planning Survey Research]<br />
* Oxfam, [https://oxfamilibrary.openrepository.com/bitstream/handle/10546/620522/cs-going-digital-data-quality-data-collection-240718-en.pdf?sequence=1&isAllowed=y Case study: Improving data quality with digital data collection]<br />
* SurveyCTO, [https://www.youtube.com/watch?v=tHb-3bnfRLo Data quality with SurveyCTO]<br />
[[Category: Primary Data Collection ]]<br />
[[Category: Research Design]]</div>Maria ruth joneshttps://dimewiki.worldbank.org/index.php?title=Data_License_Agreement&diff=8113Data License Agreement2021-04-11T19:07:37Z<p>Maria ruth jones: </p>
<hr />
<div>'''Data license agreements (DLA)''', or data use agreements (DUA) are documents that describe what kind of data is being shared with recipients. A '''DLA''' clearly states the purpose and duration of access being provided to the recipient, along with restrictions and security protocols that the recipient of the data must follow. Other contracts, such as a '''non-disclosure agreement (NDA)''', may also be used to guarantee [[De-identification|confidentiality]] of sensitive data. <br />
<br />
== Read First ==<br />
* A '''non-disclosure agreement (NDA)''' is an agreement signed by a member of the [[Impact Evaluation Team|research team]] stating clearly that they will not share any sensitive information with anyone outside the '''research team'''.<br />
* A '''data license agreement (DLA)''' usually has two parties - one who provides the data ('''data provider'''), and the other who requests access to that data ('''data requestor''').<br />
* In a '''DLA''', the '''data provider''' retains full ownership of the dataset that is requested.<br />
* Projects that have multiple sources of data will require multiple '''DLAs'''.<br />
* Members of the [[Impact Evaluation Team|research team]] must follow certain best practices when drafting a '''DLA'''.<br />
* The [https://www.worldbank.org/en/home World Bank] uses the following '''DLA''' [https://worldbankgroup.sharepoint.com/teams/ddh/SiteAssets/SitePages/ddh/DataLicenseAgreementTemplate_v4.pdf?cid=68a54269-bbff-4b47-846d-cab248ad7de1 template].<br />
<br />
== Overview ==<br />
Every '''data license agreement (DLA)''' involves at least two sides: one, the researcher or organization providing the data - the '''data provider''', and two, the researcher or organization requesting access to the data - the '''data requestor'''. The '''data provider''' is responsible for permitting access to the data as a representative of the [[Protecting Human Research Subjects|research subjects]] or the agency responsible for [[Primary Data Collection|data collection]]. The data provider is also bound by laws and policies that deal with providing access to [[Personally Identifiable Information (PII)|PII data]] and other sensitive data such as test scores, health conditions, etc. In some cases, the data provider can take the help of a '''data intermediary''' or a '''data custodian''' to share the data on their behalf. '''Custodians''' and '''intermediaries''' support access to data for the requestors. They also reduce the burden on data providers by ensuring compliance with laws, and coordinating between multiple requestors and providers.<br />
<br />
== Scope == <br />
'''Data license agreements (DLAs)''' can differ in terms of structure and the level of detail depending on the context. However, they must clearly cover the following aspects:<br />
* '''The legal framework''' within which access to the data will be provided<br />
* '''Scope of the study''' for which requestor needs access to the data<br />
* '''Restrictions''' on what the requestor can do with the data<br />
* '''Constraints on publishing''' sensitive information and '''PII data'''<br />
<br />
'''NOTE:''' In addition to this, the '''data requestors''' must also submit their '''DLA''' to their organization's information security specialists for an internal review. In general, it is a good practice for the data requestor to prepare some form of '''documentation''' for data access, even if the '''data provider''' does not require it.<br />
<br />
== Drafting ==<br />
The process of '''drafting''' a '''data license agreement (DLA)''' can take up a lot of time. In some cases, negotiations between '''data providers''' and '''data requestors''' can also fall apart after months (or even years) of discussion. Therefore, advance planning can help both '''requestors''' and '''providers''' achieve secure agreements that ensure safety of [[Protecting Human Research Subjects|subjects]], as well as of the data and the study for which the data is required. Keep the following points in mind about preparing a secure '''DLA''': <br />
# '''Initiation:''' DLAs can be initiated by the provider, as well as the requester. <br />
# '''Established procedures:''' Data providers may already have established procedures for sharing data with a requestor, or with an organization working on behalf of the requestor.<br />
# '''Review:''' If a data provider has an established process for sharing data with requestors, the requesting organization must review the terms of these procedures. The requestor should offer additions or edits whereever they consider appropriate. However, requestors should note that it might not always be possible for providers to modify data-sharing procedures based on this feedback because of certain policies.<br />
# '''Legal framework:''' Data providers should be aware of the laws, regulations, and policies permitting use of their data. As soon as the providers receive a request, they should determine whether they already have established procedures for sharing data, and whether these procedures are legally valid. <br />
# '''Public procedures:''' While procedures for providing access to data may not always be publicly available, some agencies and organizations post their established procedures on their websites. This can significantly speed up and simplify the process of fulfilling the request for data access.<br />
<br />
== Logistics ==<br />
While discussing the '''data license agreement (DLA)''', the '''data provider''' and the '''data requestor''' should carefully discuss the '''logistics'''. This includes the following: <br />
* '''Time of data delivery:''' That is, when the requestor will get access to the data.<br />
* '''Method of extracting data:''' That is, how the data will be extracted from [[Encryption|storage]] before being shared with the requestor.<br />
* '''Expenses involved:''' That is, the expenses that might arise for both the requestor and the provider while the data-sharing agreement is in effect. This includes fees charged by administrative agencies and intermediaries for ensuring the agreement is successfully completed.<br />
In addition to this, both parties should keep in mind the following with regards to '''timeline''' and '''cost''' of drafting and carrying out a data-sharing arrangement:<br />
* '''Negotiations:''' Negotiating agreements like the '''DLA''' can take up to a year from the drafting stage to the execution stage, especially if there is no history of the two parties exchanging data in the past.<br />
* '''Pending requests:''' Even if the requestor and the provider have exchanged data in the past, there could be pending requests with the provider that can cause delays. <br />
* '''Interim steps:''' After signing a '''DLA''', requestors should allocate additional time after approval, but before data delivery, since the processes for fulfilling the request can take a significant amount of time to be completed. For example, data providers will need time to [[Data Documentation|document]] and [[Data Cleaning|format]] the requested data. The providers may also need time to extract the data from multiple databases, or from carefully [[Encryption|encrypted storage]] <br />
* '''Publication:''' The data requestor may also need to notify the data providers before any releasing any outputs or [[Publishing Data|publications]], and therefore additional time should be allocated to allow for these approvals.<br />
* '''Transparency:''' Finally, if both parties are transparent about the timeline and costs, and focus on making the '''DLA''' as clear as possible, then it makes it significantly easier to carry out the data-sharing arrangement. <br />
In conclusion, no matter how big the project is, or how large the requested dataset is, both parties should invest time and effort into preparing a sound '''data license agreement'''.<br />
<br />
== Related Pages ==<br />
[[Special:WhatLinksHere/Data_License_Agreement|Click here for pages that link to this topic.]]<br />
<br />
== Additional Resources ==<br />
* JPAL, [https://admindatahandbook.mit.edu/book/testing/index.html Handbook on Using Administrative Data for Research and Evidence-based Policy]<br />
* World Bank, [https://worldbankgroup.sharepoint.com/teams/ddh/SiteAssets/SitePages/ddh/DataLicenseAgreementTemplate_v4.pdf?cid=68a54269-bbff-4b47-846d-cab248ad7de1 World Bank's Data License Agreement]</div>Maria ruth joneshttps://dimewiki.worldbank.org/index.php?title=Survey_Pilot_Participants&diff=7897Survey Pilot Participants2021-01-16T20:38:59Z<p>Maria ruth jones: /* Read First */</p>
<hr />
<div>Typically, conducting a [[Survey Pilot|survey pilot]] requires several stages of planning and discussions. Apart from the [[Impact Evaluation Team|impact evaluation team]] (or research team), '''survey pilot participants''' include '''interviewers''', '''respondents''', and even '''local government agencies'''. For instance, during the actual [[Field Surveys|field data collection]], such as in a [[Computer-Assisted Personal Interviews (CAPI)|computer-assisted personal interview (CAPI)]], it is the interviewer who reads out the questions to the respondent. But in the '''piloting''' stages (before the actual survey), the [[Impact Evaluation Team#Field Coordinators (FCs)|field coordinators (FCs)]] and the [[Impact Evaluation Team#Principal Investigators (PIs)|principal investigators (PIs)]] oversee the overall [[Preparing for Field Data Collection|preparation]] and finalize the [[Survey Protocols|survey protocols]].<br />
<br />
== Read First ==<br />
* Each of the '''pilot participants''' has a specific role during each of the three stages.<br />
* '''Respondents''' should be as close to the '''target population''' as possible, but none of the sample for the actual survey should be included in the pilot<br />
* '''Interviewers''' should be fluent in all local languages of the study area and the language of the research team.<br />
* [[Impact_Evaluation_Team#Field_Coordinator|field coordinators (FCs)]] play a central role in organizing, managing, and supervising the pilot<br />
* '''Principal investigator (PI)''' should be briefed regularly during the pilot if not participating in person.<br />
* '''Other research team members''' such as the [[Impact Evaluation Team#Impact Evaluation Coordinator|impact evaluation coordinator]], research manager, programmers, and translators should also participate in pilot discussions.<br />
<br />
== Participant Roles ==<br />
{| class="wikitable" style="margin-left: auto; margin-right: auto; border: none;"<br />
|-<br />
|-<br />
! style="text-align:center;"|<br />
!<br />
! style="height:50px; width:250px; text-align:center;"| Stage 1 - Pre-Pilot<br />
! style="height:50px; width:250px; text-align:center;"| Stage 2 - Content-focused Pilot<br />
! style="height:50px; width:250px; text-align:center;"| Stage 3 - Data-focused Pilot<br />
|-<br />
| '''Respondents'''<br />
| <br />
| Respondents for the pilot should be as similar as possible to respondents for the actual survey in terms of age, gender, education, and socioeconomic (income) status. <br />
<br />
Neighboring villages, schools, or firms are good options for selecting respondents for the pilot.<br />
<br />
Never use any of these respondents in the final sample. <br />
| style="vertical-align:top;"| Test the protocols for [[Sampling|sampling]] in this stage. Use these protocols to select respondents for the pilot. <br />
<br />
<br />
<br />
This helps to find out if the '''sampling frame''' (list from which respondents are selected) needs to be revised. It also helps to reduce [[Selection Bias|selection bias]]. <br />
| style="vertical-align:top;"| Re-test your '''sampling protocols'''. <br />
<br />
<br />
<br />
<br />
<br />
Include revisions based on the previous round of '''piloting'''.<br />
|-<br />
| rowspan=4 | '''Enumerators (or interviewers)'''<br />
|style="height:100px;"| '''Language'''<br />
| Fully '''fluent''' (can read, write and/or speak) in the local language(s) and the language of research team.<br />
| Fully fluent in the local language(s) and the language of research team.<br />
| Fully fluent in the local language(s) and the language of research team.<br />
|-<br />
| '''Required experience'''<br />
|style="vertical-align:top;"| '''Qualitative''' experience as an enumerator. For instance, experience in identifying issues in wording and structure of questions.<br />
<br />
'''Sectoral''' knowledge. For instance, knowledge about agricultural patterns in a survey that aims to collect data on farm output.<br />
| style="vertical-align:top;"| '''Quantitative''' experience as an enumerator. For instance, experience in entering data electronically.<br />
<br />
<br />
Sectoral knowledge.<br />
| style="vertical-align:top;"|Quantitative experience as an enumerator.<br />
<br />
<br />
<br />
Experience working with phones or tablets.<br />
|-<br />
| '''Background'''<br />
| style="vertical-align:top;"| The pool of interviewers should be diverse in terms of age, race, religion, gender etc.<br />
| style="vertical-align:top;"| Interviewers should be mindful of (and learn about) cultural considerations. For instance, if women can interview men, or if men can interview women in the study area.<br />
| style="vertical-align:top;"| The background of the team of interviewers should be similar to that of the final survey team, which is decided based on previous stages of '''piloting'''.<br />
|-<br />
| '''Size of the team'''<br />
|style="vertical-align:top;"| Very small - not more than 2. <br />
<br />
A member of the research team should always accompany the enumerators.<br />
|style="vertical-align:top;"| Small - between 2 and 4. <br />
<br />
Ideally, a member of the research team should monitor each enumerator.<br />
|style="vertical-align:top;"| Large - between 4 and 8. <br />
<br />
More enumerators help simplify the process of [[Monitoring Data Quality|monitoring data quality]] and '''debugging''' (identifying errors) in the [[Questionnaire Programming|coded instrument]].<br />
|-<br />
| '''Survey firm'''<br />
| <br />
| Not part of this stage. <br />
<br />
This stage is typically conducted before survey firm is on board. <br />
| Not part of this stage. <br />
<br />
This stage is typically conducted before survey firm is on board.<br />
| Part of this stage. <br />
<br />
In fact, the survey firm should lead the pilot in this round. <br />
|-<br />
| '''Members of the research team'''<br />
|<br />
|style="vertical-align:top;"| The [[Impact Evaluation Team#Field Coordinators (FCs)|field coordinators (FCs)]]. Ideally they should participate in each stage.<br />
<br />
The [[Impact Evaluation Team#Principal Investigators (PIs)|principal investigators (PIs)]] should participate directly. They should also conduct daily discussions to '''debrief''' (exchange notes with) the enumerators. <br />
|style="vertical-align:top;"| The field coordinators (FCs).<br />
<br />
<br />
The [[Impact Evaluation Team#Impact Evaluation Coordinator|impact evaluation coordinator]] (or research manager) should participate directly. <br />
|style="vertical-align:top;"| The field coordinators (FCs).<br />
<br />
<br />
The '''programmer''' should participate directly (to debug the instrument in real time)<br />
<br />
Members of the research team who will be '''field supervisors''' in the actual survey should act as interviewers in this stage. This gives them experience with the instrument, which helps them conduct [[Enumerator Training|enumerator training]].<br />
|-<br />
| '''Others'''<br />
| <br />
| Local [[Impact Evaluation Team#Research Assistants (RAs|research assistants (RAs)]]. Includes short-term consultants (STCs) hired through the World Bank or through a local principal investigator (PI).<br />
<br />
Local government staff.<br />
<br />
University students.<br />
| Local research assistants (RAs). <br />
<br />
<br />
<br />
<br />
<br />
Local government staff.<br />
<br />
University students.<br />
| Local research assistants (RAs).<br />
<br />
<br />
<br />
<br />
<br />
Local government staff.<br />
<br />
University students. <br />
|}<br />
<br />
== Guidelines ==<br />
Finally, the '''pilot participants''' should keep certain '''best practices''' in mind while planning and conducting a [[Survey Pilot|survey pilot]]. These can improve the outcomes of the pilot, for instance, by reducing the down-time (gap) between surveys.<br />
* Throughout the process of planning the pilot, discuss with other members of the research team, and take notes of what needs to be part of the pilot.<br />
* Take careful notes of the clarifications that come up during the [[Structuring a Survey Pilot#Review Sessions|review sessions]]. These will be an important part of the [[Enumerator Training#Enumerator Manual|enumerator manual]] which '''supervisors''' use to [[Enumerator Training|train enumerator]].<br />
* Hire a local '''mobilizer''' to coordinate with respondents. '''Mobilizers''' explain the purpose behind conducting the survey to the respondents, and facilitate the process of obtaining consent. This is particularly helpful in urban areas, or in cases where respondents are busy.<br />
<br />
== Related Pages ==<br />
[[Special:WhatLinksHere/Survey_Pilot_Participants|Click here for pages that link to this topic.]]<br />
<br />
== Additional Resources ==<br />
* DIME Analytics (World Bank), [https://osf.io/wmpvh/ Team roles]<br />
* DIME Analytics (World Bank), [https://osf.io/63uv9/ Guidelines on piloting surveys]<br />
* DIME Analytics (World Bank), [https://osf.io/rqb5m/ Survey instrument design and pilot]<br />
* DIME Analytics (World Bank), [https://osf.io/6rvkj/ Survey preparing]<br />
* DIME Analytics (World Bank), [https://osf.io/ek9ab/ Monitoring survey content]<br />
* DIME Analytics (World Bank), [https://osf.io/4q6kb/ Monitoring data quality]<br />
[[Category: Survey Pilot]]</div>Maria ruth joneshttps://dimewiki.worldbank.org/index.php?title=Survey_Pilot&diff=7896Survey Pilot2021-01-16T20:35:17Z<p>Maria ruth jones: /* Participants */</p>
<hr />
<div>'''Survey pilot''' is the process of carrying out interviews and tests on different components of a survey, including [[Piloting Survey Content|content]] and [[Piloting Survey Protocols|protocols]]. A good '''pilot''' provides the research team with important feedback before they start the process of [[Primary Data Collection|primary data collection]]. This feedback can help the [[Impact Evaluation Team|research team]] review and improve the [[Questionnaire Design| design of the questionnaire]], [[Questionnaire Translation|translated questionnaires]], as well as [[Survey Protocols|survey protocols]] related to [[Piloting Survey Protocols#Interview scheduling|interview scheduling]], [[Sampling|sampling]], and [[Geo Spatial Data|geo data]].<br />
== Read First ==<br />
* A comprehensive '''survey pilot''' includes 3 stages: a pre-pilot, a content-focused pilot, and a data-focused pilot. <br />
* The '''pre-pilot''' is usually done during the process of [[Questionnaire Design|designing a questionnaire]], to refine the design, and wording of questions.<br />
* The '''content-focused pilot''' is conducted after [[Questionnaire Design|designing a questionnaire]], and sharing it with field and sector experts for their comments and inputs.<br />
* Finally, the '''data-focused pilot''' allows the [[Impact Evaluation Team|research team]] to review the data collected during the pilot, and identify possible errors in the [[Questionnaire Programming|programmed questionnaire]]. <br />
* A '''survey pilot''' is different from the '''field testing''' (practice interviews) that enumerators conduct at the end of [[Enumerator Training|enumerator training]]. The pilot should be complete before enumerator training begins. <br />
* [[Survey Pilot Participants|Participants of a survey pilot]] include the [[Impact Evaluation Team|research team]], '''interviewers''', and '''respondents'''.<br />
* Conduct the first two stages of the pilot before [[Questionnaire Programming|programming the questionnaire]]. Repeated programming is time-consuming and can create '''bugs''' (errors), for instance, by disturbing the order of questions.<br />
<br />
== Stages of a Survey Pilot ==<br />
A complete '''survey pilot''' is conducted over three stages - '''pre-pilot''', [[Piloting Survey Content|content-focused pilot]], and '''data-focused pilot'''. The table below discusses these stages in more detail.<br />
<br />
{| class="wikitable" style="margin-left: auto; margin-right: auto; border: none;"<br />
|-<br />
! style="height:50px; width:150px; text-align:center;" |<br />
! style="height:50px; width:250px; text-align:center;" | Stage 1: Pre-pilot<br />
! style="height:50px; width:250px; text-align:center;" | Stage 2: Content-focused pilot<br />
! style="height:50px; width:250px; text-align:center;" | Stage 3: Data-focused pilot<br />
|-<br />
| style="text-align:center;" | '''Objective'''<br />
| style="vertical-align:top;"| Answer broad questions about '''qualitative factors''' like [[Questionnaire Design|questionnaire design]] and the context for conducting the study. Discuss these with the concerned teams.<br />
<br />
Learn more about how people think to see what structure and flow makes sense.<br />
For example, do people think about their input use at plot-level? by crop? overall?<br />
<br />
It helps to think of the '''pre-pilot''' as a part of the process of [[Questionnaire Design|questionnaire design]]. For example, the '''pre-pilot''' can answer specific questions such as - important sources of income, relevant types of shocks, local food groups, and so on.<br />
<br />
In cases where the [[Impact Evaluation Team|research team]] is using a pre-existing questionnaire, they can skip this step. However, a '''pre-pilot''' is absolutely essential if the '''research team''' is designing a questionnaire from scratch, or if the questionnaire asks questions about issues that are difficult to measure.<br />
| style="vertical-align:top;"| Refine the order and wording of specific questions, the overall structure of the questionnaire, and [[Questionnaire Translation|translations]]. <br />
<br />
<br />
<br />
Check that the answer [[SurveyCTO Choice Lists|choices]] are '''comprehensive''', that is, they cover all possibilities. <br />
<br />
<br />
Flag any sensitive questions.<br />
<br />
<br />
Check how long it takes to answer the questions. Also check how answers differ between respondents ('''response variance''')<br />
| style="vertical-align:top;"| Check if the [[Questionnaire Programming|programmed instrument]] displays questions in the correct order, and follows the correct patterns (such as a group of questions which will always appear together, or will repeat).<br />
<br />
<br />
Load a sample data set (based on '''pilot interviews'''). Check if the data set has any missing fields. Perform all data quality checks, like [[Back Checks|back checks]] and [[Monitoring Data Quality#High Frequency Checks|high frequency checks]].<br />
|-<br />
| style="height:80px;text-align:center;" | '''Status of survey instrument'''<br />
| style="vertical-align:center;"| Early, printable version of the draft, and notes for further discussion.<br />
| style="vertical-align:center;"| A translated, printable draft. Ready to be '''programmed'''.<br />
| style="vertical-align:center;"| A translated, programmed, final draft. Ready for starting data collection.<br />
|-<br />
| style="height:80px;text-align:left;" | '''Mode'''<br />
| style="text-align:center;"| Pen-and-paper<br />
| style="text-align:center;"| Pen-and-paper<br />
| style="text-align:center;"| Electronic (phone/tablet)<br />
|}<br />
<br />
However, note that all three stages may not be necessary for every survey. The research team has to determine which stage to begin from on a case-to-case basis.<br />
* If the research team is using a brand new survey instrument, then they must start with Stage 1, the '''pre-pilot.'''<br />
* If the survey instrument is an adaptation of (based on) a different instrument which was used for a previous data collection for another project in the same region (province or country), the research team can start with Stage 2, the '''content-focused pilot'''. However, they must make sure that the survey instrument does not have any design issues and was shared by a reliable source.<br />
* If the survey instrument is an adaptation of an instrument used for a previous data collection for the same project, but the research team had to make significant revisions or additions, even then the research team should start with Stage 2, the '''content-focused pilot'''.<br />
* Only if the research team is '''piloting''' a '''follow-up''' survey, and there are no major changes from the '''baseline''' (or first round) survey, then in this case the research team may skip directly to Stage 3, the '''data-focused pilot'''.<br />
<br />
== Pilot on paper first == <br />
The first two stages of the survey pilot (pre-pilot and content-focused pilot) are best done on pen-and-paper is done on paper, regardless of the planned survey mode ([[Pen-and-Paper_Personal_Interviews_(PAPI)|pen-and-paper personal interviews (PAPI)]],<br />
[[Computer-Assisted_Personal_Interviews_(CAPI)|computer-assisted personal interviews (CAPI)]], [[Remote_Surveys|phone surveys (CATI)]]). A pen-and-paper pilot allows members of the research team and interviewers flexibility when it comes to recording answers and '''qualitative''' observations. <br />
<br />
Piloting on paper is particularly helpful for the following:<br />
* '''Open-ended responses.''' pre-pilot responses help the research team to make the list of choices for a question more detailed and comprehensive. Pen-and-paper pilots allow the interviewers to make note of these.<br />
* '''Restructuring.''' Pen-and-paper pilots allow the interviewers to draw lines and arrows between questions, which makes it easier to restructure the '''instrument'''.<br />
* '''Observations.''' Pen-and-paper pilots allow interviewers to record observations and feedback of interviewers in the margins. This allows the research team to take note of issues, such as the wording or flow of questions.<br />
<br />
== Timeline ==<br />
In order to conduct a successful [[Survey Pilot|survey pilot]], the [[Impact Evaluation Team|impact evaluation team]] must draft a detailed '''timeline for a survey pilot'''. The '''timeline''' should allow enough time for discussions and revisions about aspects like [[Questionnaire Design|instrument design]], [[Questionnaire Translation|translation]], and [[Survey Protocols|protocols]]. '''Piloting''' should ideally start 4-6 months before [[Field Surveys#Survey Launch|survey launch]] since this allows the research team to use the feedback from each [[Survey Pilot#Stages of a Survey Pilot|stage of a survey pilot]] to improve the survey content and [[Piloting Survey Protocols|protocols]]. <br />
<br />
The '''timeline''' is an important part of the process of [[Structuring a Survey Pilot|structuring a survey pilot]]. The actual [[Structuring a Survey Pilot#Duration of the Pilot|duration of the pilot]] will depend on factors like number of changes made, location of the study area, and availability of equipment for '''field teams'''. Based on '''best practices''', the following are the steps involved in conducting a [[Survey Pilot|survey pilot]] (from start to finish), and the recommended time for each step.<br />
{| class="wikitable" style="margin-left: auto; margin-right: auto; border: none;"<br />
!style="width:350px; text-align:center;"| Task<br />
!style="width:350px; text-align:center;"| Estimated Time<br />
|-<br />
|style="text-align:center;"| [[Survey Pilot#Stages of a Survey Pilot|Pre-pilot]]<br />
|style="text-align:center;"| 1-2 weeks<br />
|-<br />
|style="text-align:center;"| [[Questionnaire Design|Questionnaire design]]<br />
|style="text-align:center;"| 4 weeks<br />
|-<br />
|style="text-align:center;"| [[Piloting Survey Content|Content-focused pilot]] <br />
|style="text-align:center;"| 2-3 weeks<br />
|-<br />
|style="text-align:center;"| [[Structuring a Survey Pilot#Review Sessions|Review sessions]]<br />
|style="text-align:center;"| 1-2 weeks<br />
|-<br />
|style="text-align:center;"| [[Questionnaire Translation|Questionnaire translation]]<br />
|style="text-align:center;"| 2-3 weeks<br />
|-<br />
|style="text-align:center;"| [[IRB Approval|IRB approval]]<br />
|style="text-align:center;"| Depends on the complexity of the survey. Applying for [[IRB Approval|approvals]], [[Questionnaire Programming|programming]], and the [[Survey Pilot#Stages of a Survey Pilot|data-focused pilot]] - should all happen at the same time.<br />
|-<br />
|style="text-align:center;"| [[Questionnaire Programming|Questionnaire programming]]<br />
|style="text-align:center;"| 4-6 weeks<br />
|-<br />
|style="text-align:center;"| [[Survey Pilot#Stages of a Survey Pilot|Data-focused pilot]]<br />
|style="text-align:center;"| 2-3 weeks<br />
|-<br />
|style="text-align:center;"| [[Enumerator Training|Enumerator training]]<br />
|style="text-align:center;"| 2 weeks<br />
|}<br />
<br />
Do not confuse the pilot with '''field testing''' (practice interviews) which is conducted at the end of [[Enumerator_Training |enumerator training]]!<br />
<br />
== Structure ==<br />
The process of '''structuring a survey pilot''' involves agreeing upon the '''logistics''' (or practical aspects) of a [[Survey Pilot|survey pilot]], including '''duration''', '''approvals''', '''review sessions''', and '''training.''' In order to conduct a successful pilot, the [[Impact Evaluation Team|impact evaluation team]] (or research team) must discuss each of these aspects in detail to ensure that the '''survey pilot''' is conducted smoothly. A good pilot helps to improve the quality of the [[Primary Data Collection|data collection process]] and the [[Survey Protocols|survey protocols]]. Go to [[Structuring a Survey Pilot|Structuring a Survey Pilot]] for a detailed discussion of recommended protocols.<br />
<br />
== Participants ==<br />
Typically, a '''survey pilot''' is conducted before a [[Survey Firm|survey firm]] is brought on-board. Each of the pilot participants plays an important role in the implementation of a successful pilot. Typical participants include respondents, interviewiers, field coordinator, principal investigator, and other research team members. For a detailed discussion of the role of each participant, and how to select appropriate respondents and interviewers for a pilot, go to the article on [[Survey Pilot Participants| Survey Pilot Participants]].<br />
<br />
== Guidelines ==<br />
Finally, it is important to keep the following '''best practices''' in mind while planning and conducting a pilot:<br />
* Throughout the process of [[Questionnaire Design|designing the questionnaire]], take notes of what needs to be explored or tested during the pilot.<br />
* Encourage interviewers to probe and follow-up much more than they would in a typical interview<br />
* Encourage respondents to think out-loud, to understand how they are coming up with their answer, and ask enumerators to record these notes.<br />
* In some cultures, it will be appropriate and useful to ask respondents their feedback at the end of the interview<br />
* Take careful notes of the discussions and clarifications that come up during the pilot. These will be an important part of the [[Enumerator Training#Enumerator Manual|enumerator manual]] which is used during [[Enumerator Training|enumerator training]].<br />
* It is useful for research team members to sit in on pilot interviews even if they do not speak the local language. Take note of: questions that take longer than expected; any moments at which the respondent seems frustrated, confused, or uncomfortable; any questions that generate significant discussion; and the overall flow of modules and survey instrument.<br />
* Use the '''data-focused pilot''' to test [[Back Checks|back check]] templates. <br />
* Hire a local '''mobilizer''' to coordinate with respondents. '''Mobilizers''' explain the purpose behind conducting the survey, and facilitate the process of obtaining consent. This is particularly helpful in urban areas, or in cases where respondents are busy. This can improve the outcomes of piloting, for instance, by reducing gap between surveys.<br />
<br />
[https://www.worldbank.org/en/research/dime/data-and-analytics DIME Analytics] has also created the following checklists to assist researchers and enumerators in preparing for, and implementing a pilot:<br />
*[[Preparing_for_the_survey_checklist|Checklist: Preparing for a survey pilot]]<br />
*[[Checklist: Refine the Questionnaire (Content)|Checklist: Refining questionnaire content]]<br />
*[[Checklist: Refine the Questionnaire (Data)|Checklist: Refining questionnaire data]]<br />
<br />
== Related Pages ==<br />
[[Special:WhatLinksHere/Survey_Pilot|Click here for pages that link to this topic.]]<br />
<br />
== Additional Resources == <br />
* DIME Analytics (World Bank), [https://osf.io/63uv9/ Survey pilot guidelines]<br />
* DIME Analytics (World Bank), [https://osf.io/rqb5m/ Survey instrument design and pilot]<br />
<br />
[[Category: Primary Data Collection]] [[Category: Questionnaire Design]]</div>Maria ruth joneshttps://dimewiki.worldbank.org/index.php?title=Survey_Pilot&diff=7895Survey Pilot2021-01-16T20:34:59Z<p>Maria ruth jones: /* Participants */</p>
<hr />
<div>'''Survey pilot''' is the process of carrying out interviews and tests on different components of a survey, including [[Piloting Survey Content|content]] and [[Piloting Survey Protocols|protocols]]. A good '''pilot''' provides the research team with important feedback before they start the process of [[Primary Data Collection|primary data collection]]. This feedback can help the [[Impact Evaluation Team|research team]] review and improve the [[Questionnaire Design| design of the questionnaire]], [[Questionnaire Translation|translated questionnaires]], as well as [[Survey Protocols|survey protocols]] related to [[Piloting Survey Protocols#Interview scheduling|interview scheduling]], [[Sampling|sampling]], and [[Geo Spatial Data|geo data]].<br />
== Read First ==<br />
* A comprehensive '''survey pilot''' includes 3 stages: a pre-pilot, a content-focused pilot, and a data-focused pilot. <br />
* The '''pre-pilot''' is usually done during the process of [[Questionnaire Design|designing a questionnaire]], to refine the design, and wording of questions.<br />
* The '''content-focused pilot''' is conducted after [[Questionnaire Design|designing a questionnaire]], and sharing it with field and sector experts for their comments and inputs.<br />
* Finally, the '''data-focused pilot''' allows the [[Impact Evaluation Team|research team]] to review the data collected during the pilot, and identify possible errors in the [[Questionnaire Programming|programmed questionnaire]]. <br />
* A '''survey pilot''' is different from the '''field testing''' (practice interviews) that enumerators conduct at the end of [[Enumerator Training|enumerator training]]. The pilot should be complete before enumerator training begins. <br />
* [[Survey Pilot Participants|Participants of a survey pilot]] include the [[Impact Evaluation Team|research team]], '''interviewers''', and '''respondents'''.<br />
* Conduct the first two stages of the pilot before [[Questionnaire Programming|programming the questionnaire]]. Repeated programming is time-consuming and can create '''bugs''' (errors), for instance, by disturbing the order of questions.<br />
<br />
== Stages of a Survey Pilot ==<br />
A complete '''survey pilot''' is conducted over three stages - '''pre-pilot''', [[Piloting Survey Content|content-focused pilot]], and '''data-focused pilot'''. The table below discusses these stages in more detail.<br />
<br />
{| class="wikitable" style="margin-left: auto; margin-right: auto; border: none;"<br />
|-<br />
! style="height:50px; width:150px; text-align:center;" |<br />
! style="height:50px; width:250px; text-align:center;" | Stage 1: Pre-pilot<br />
! style="height:50px; width:250px; text-align:center;" | Stage 2: Content-focused pilot<br />
! style="height:50px; width:250px; text-align:center;" | Stage 3: Data-focused pilot<br />
|-<br />
| style="text-align:center;" | '''Objective'''<br />
| style="vertical-align:top;"| Answer broad questions about '''qualitative factors''' like [[Questionnaire Design|questionnaire design]] and the context for conducting the study. Discuss these with the concerned teams.<br />
<br />
Learn more about how people think to see what structure and flow makes sense.<br />
For example, do people think about their input use at plot-level? by crop? overall?<br />
<br />
It helps to think of the '''pre-pilot''' as a part of the process of [[Questionnaire Design|questionnaire design]]. For example, the '''pre-pilot''' can answer specific questions such as - important sources of income, relevant types of shocks, local food groups, and so on.<br />
<br />
In cases where the [[Impact Evaluation Team|research team]] is using a pre-existing questionnaire, they can skip this step. However, a '''pre-pilot''' is absolutely essential if the '''research team''' is designing a questionnaire from scratch, or if the questionnaire asks questions about issues that are difficult to measure.<br />
| style="vertical-align:top;"| Refine the order and wording of specific questions, the overall structure of the questionnaire, and [[Questionnaire Translation|translations]]. <br />
<br />
<br />
<br />
Check that the answer [[SurveyCTO Choice Lists|choices]] are '''comprehensive''', that is, they cover all possibilities. <br />
<br />
<br />
Flag any sensitive questions.<br />
<br />
<br />
Check how long it takes to answer the questions. Also check how answers differ between respondents ('''response variance''')<br />
| style="vertical-align:top;"| Check if the [[Questionnaire Programming|programmed instrument]] displays questions in the correct order, and follows the correct patterns (such as a group of questions which will always appear together, or will repeat).<br />
<br />
<br />
Load a sample data set (based on '''pilot interviews'''). Check if the data set has any missing fields. Perform all data quality checks, like [[Back Checks|back checks]] and [[Monitoring Data Quality#High Frequency Checks|high frequency checks]].<br />
|-<br />
| style="height:80px;text-align:center;" | '''Status of survey instrument'''<br />
| style="vertical-align:center;"| Early, printable version of the draft, and notes for further discussion.<br />
| style="vertical-align:center;"| A translated, printable draft. Ready to be '''programmed'''.<br />
| style="vertical-align:center;"| A translated, programmed, final draft. Ready for starting data collection.<br />
|-<br />
| style="height:80px;text-align:left;" | '''Mode'''<br />
| style="text-align:center;"| Pen-and-paper<br />
| style="text-align:center;"| Pen-and-paper<br />
| style="text-align:center;"| Electronic (phone/tablet)<br />
|}<br />
<br />
However, note that all three stages may not be necessary for every survey. The research team has to determine which stage to begin from on a case-to-case basis.<br />
* If the research team is using a brand new survey instrument, then they must start with Stage 1, the '''pre-pilot.'''<br />
* If the survey instrument is an adaptation of (based on) a different instrument which was used for a previous data collection for another project in the same region (province or country), the research team can start with Stage 2, the '''content-focused pilot'''. However, they must make sure that the survey instrument does not have any design issues and was shared by a reliable source.<br />
* If the survey instrument is an adaptation of an instrument used for a previous data collection for the same project, but the research team had to make significant revisions or additions, even then the research team should start with Stage 2, the '''content-focused pilot'''.<br />
* Only if the research team is '''piloting''' a '''follow-up''' survey, and there are no major changes from the '''baseline''' (or first round) survey, then in this case the research team may skip directly to Stage 3, the '''data-focused pilot'''.<br />
<br />
== Pilot on paper first == <br />
The first two stages of the survey pilot (pre-pilot and content-focused pilot) are best done on pen-and-paper is done on paper, regardless of the planned survey mode ([[Pen-and-Paper_Personal_Interviews_(PAPI)|pen-and-paper personal interviews (PAPI)]],<br />
[[Computer-Assisted_Personal_Interviews_(CAPI)|computer-assisted personal interviews (CAPI)]], [[Remote_Surveys|phone surveys (CATI)]]). A pen-and-paper pilot allows members of the research team and interviewers flexibility when it comes to recording answers and '''qualitative''' observations. <br />
<br />
Piloting on paper is particularly helpful for the following:<br />
* '''Open-ended responses.''' pre-pilot responses help the research team to make the list of choices for a question more detailed and comprehensive. Pen-and-paper pilots allow the interviewers to make note of these.<br />
* '''Restructuring.''' Pen-and-paper pilots allow the interviewers to draw lines and arrows between questions, which makes it easier to restructure the '''instrument'''.<br />
* '''Observations.''' Pen-and-paper pilots allow interviewers to record observations and feedback of interviewers in the margins. This allows the research team to take note of issues, such as the wording or flow of questions.<br />
<br />
== Timeline ==<br />
In order to conduct a successful [[Survey Pilot|survey pilot]], the [[Impact Evaluation Team|impact evaluation team]] must draft a detailed '''timeline for a survey pilot'''. The '''timeline''' should allow enough time for discussions and revisions about aspects like [[Questionnaire Design|instrument design]], [[Questionnaire Translation|translation]], and [[Survey Protocols|protocols]]. '''Piloting''' should ideally start 4-6 months before [[Field Surveys#Survey Launch|survey launch]] since this allows the research team to use the feedback from each [[Survey Pilot#Stages of a Survey Pilot|stage of a survey pilot]] to improve the survey content and [[Piloting Survey Protocols|protocols]]. <br />
<br />
The '''timeline''' is an important part of the process of [[Structuring a Survey Pilot|structuring a survey pilot]]. The actual [[Structuring a Survey Pilot#Duration of the Pilot|duration of the pilot]] will depend on factors like number of changes made, location of the study area, and availability of equipment for '''field teams'''. Based on '''best practices''', the following are the steps involved in conducting a [[Survey Pilot|survey pilot]] (from start to finish), and the recommended time for each step.<br />
{| class="wikitable" style="margin-left: auto; margin-right: auto; border: none;"<br />
!style="width:350px; text-align:center;"| Task<br />
!style="width:350px; text-align:center;"| Estimated Time<br />
|-<br />
|style="text-align:center;"| [[Survey Pilot#Stages of a Survey Pilot|Pre-pilot]]<br />
|style="text-align:center;"| 1-2 weeks<br />
|-<br />
|style="text-align:center;"| [[Questionnaire Design|Questionnaire design]]<br />
|style="text-align:center;"| 4 weeks<br />
|-<br />
|style="text-align:center;"| [[Piloting Survey Content|Content-focused pilot]] <br />
|style="text-align:center;"| 2-3 weeks<br />
|-<br />
|style="text-align:center;"| [[Structuring a Survey Pilot#Review Sessions|Review sessions]]<br />
|style="text-align:center;"| 1-2 weeks<br />
|-<br />
|style="text-align:center;"| [[Questionnaire Translation|Questionnaire translation]]<br />
|style="text-align:center;"| 2-3 weeks<br />
|-<br />
|style="text-align:center;"| [[IRB Approval|IRB approval]]<br />
|style="text-align:center;"| Depends on the complexity of the survey. Applying for [[IRB Approval|approvals]], [[Questionnaire Programming|programming]], and the [[Survey Pilot#Stages of a Survey Pilot|data-focused pilot]] - should all happen at the same time.<br />
|-<br />
|style="text-align:center;"| [[Questionnaire Programming|Questionnaire programming]]<br />
|style="text-align:center;"| 4-6 weeks<br />
|-<br />
|style="text-align:center;"| [[Survey Pilot#Stages of a Survey Pilot|Data-focused pilot]]<br />
|style="text-align:center;"| 2-3 weeks<br />
|-<br />
|style="text-align:center;"| [[Enumerator Training|Enumerator training]]<br />
|style="text-align:center;"| 2 weeks<br />
|}<br />
<br />
Do not confuse the pilot with '''field testing''' (practice interviews) which is conducted at the end of [[Enumerator_Training |enumerator training]]!<br />
<br />
== Structure ==<br />
The process of '''structuring a survey pilot''' involves agreeing upon the '''logistics''' (or practical aspects) of a [[Survey Pilot|survey pilot]], including '''duration''', '''approvals''', '''review sessions''', and '''training.''' In order to conduct a successful pilot, the [[Impact Evaluation Team|impact evaluation team]] (or research team) must discuss each of these aspects in detail to ensure that the '''survey pilot''' is conducted smoothly. A good pilot helps to improve the quality of the [[Primary Data Collection|data collection process]] and the [[Survey Protocols|survey protocols]]. Go to [[Structuring a Survey Pilot|Structuring a Survey Pilot]] for a detailed discussion of recommended protocols.<br />
<br />
== Participants ==<br />
Typically, a '''survey pilot''' is conducted before a [[Survey Firm|survey firm]] is brought on-board. Each of the pilot participants plays an important role in the implementation of a successful pilot. Typical participants include respondents, interviewiers, field coordinator, principal investigator, and other research team members. For a detailed discussion of the role of each participant, and how to select appropriate respondents and interviewers for a pilot, go to the article on [Survey Pilot Participants| Survey Pilot Participants].<br />
<br />
== Guidelines ==<br />
Finally, it is important to keep the following '''best practices''' in mind while planning and conducting a pilot:<br />
* Throughout the process of [[Questionnaire Design|designing the questionnaire]], take notes of what needs to be explored or tested during the pilot.<br />
* Encourage interviewers to probe and follow-up much more than they would in a typical interview<br />
* Encourage respondents to think out-loud, to understand how they are coming up with their answer, and ask enumerators to record these notes.<br />
* In some cultures, it will be appropriate and useful to ask respondents their feedback at the end of the interview<br />
* Take careful notes of the discussions and clarifications that come up during the pilot. These will be an important part of the [[Enumerator Training#Enumerator Manual|enumerator manual]] which is used during [[Enumerator Training|enumerator training]].<br />
* It is useful for research team members to sit in on pilot interviews even if they do not speak the local language. Take note of: questions that take longer than expected; any moments at which the respondent seems frustrated, confused, or uncomfortable; any questions that generate significant discussion; and the overall flow of modules and survey instrument.<br />
* Use the '''data-focused pilot''' to test [[Back Checks|back check]] templates. <br />
* Hire a local '''mobilizer''' to coordinate with respondents. '''Mobilizers''' explain the purpose behind conducting the survey, and facilitate the process of obtaining consent. This is particularly helpful in urban areas, or in cases where respondents are busy. This can improve the outcomes of piloting, for instance, by reducing gap between surveys.<br />
<br />
[https://www.worldbank.org/en/research/dime/data-and-analytics DIME Analytics] has also created the following checklists to assist researchers and enumerators in preparing for, and implementing a pilot:<br />
*[[Preparing_for_the_survey_checklist|Checklist: Preparing for a survey pilot]]<br />
*[[Checklist: Refine the Questionnaire (Content)|Checklist: Refining questionnaire content]]<br />
*[[Checklist: Refine the Questionnaire (Data)|Checklist: Refining questionnaire data]]<br />
<br />
== Related Pages ==<br />
[[Special:WhatLinksHere/Survey_Pilot|Click here for pages that link to this topic.]]<br />
<br />
== Additional Resources == <br />
* DIME Analytics (World Bank), [https://osf.io/63uv9/ Survey pilot guidelines]<br />
* DIME Analytics (World Bank), [https://osf.io/rqb5m/ Survey instrument design and pilot]<br />
<br />
[[Category: Primary Data Collection]] [[Category: Questionnaire Design]]</div>Maria ruth joneshttps://dimewiki.worldbank.org/index.php?title=Survey_Pilot&diff=7894Survey Pilot2021-01-16T20:34:19Z<p>Maria ruth jones: /* Participants */</p>
<hr />
<div>'''Survey pilot''' is the process of carrying out interviews and tests on different components of a survey, including [[Piloting Survey Content|content]] and [[Piloting Survey Protocols|protocols]]. A good '''pilot''' provides the research team with important feedback before they start the process of [[Primary Data Collection|primary data collection]]. This feedback can help the [[Impact Evaluation Team|research team]] review and improve the [[Questionnaire Design| design of the questionnaire]], [[Questionnaire Translation|translated questionnaires]], as well as [[Survey Protocols|survey protocols]] related to [[Piloting Survey Protocols#Interview scheduling|interview scheduling]], [[Sampling|sampling]], and [[Geo Spatial Data|geo data]].<br />
== Read First ==<br />
* A comprehensive '''survey pilot''' includes 3 stages: a pre-pilot, a content-focused pilot, and a data-focused pilot. <br />
* The '''pre-pilot''' is usually done during the process of [[Questionnaire Design|designing a questionnaire]], to refine the design, and wording of questions.<br />
* The '''content-focused pilot''' is conducted after [[Questionnaire Design|designing a questionnaire]], and sharing it with field and sector experts for their comments and inputs.<br />
* Finally, the '''data-focused pilot''' allows the [[Impact Evaluation Team|research team]] to review the data collected during the pilot, and identify possible errors in the [[Questionnaire Programming|programmed questionnaire]]. <br />
* A '''survey pilot''' is different from the '''field testing''' (practice interviews) that enumerators conduct at the end of [[Enumerator Training|enumerator training]]. The pilot should be complete before enumerator training begins. <br />
* [[Survey Pilot Participants|Participants of a survey pilot]] include the [[Impact Evaluation Team|research team]], '''interviewers''', and '''respondents'''.<br />
* Conduct the first two stages of the pilot before [[Questionnaire Programming|programming the questionnaire]]. Repeated programming is time-consuming and can create '''bugs''' (errors), for instance, by disturbing the order of questions.<br />
<br />
== Stages of a Survey Pilot ==<br />
A complete '''survey pilot''' is conducted over three stages - '''pre-pilot''', [[Piloting Survey Content|content-focused pilot]], and '''data-focused pilot'''. The table below discusses these stages in more detail.<br />
<br />
{| class="wikitable" style="margin-left: auto; margin-right: auto; border: none;"<br />
|-<br />
! style="height:50px; width:150px; text-align:center;" |<br />
! style="height:50px; width:250px; text-align:center;" | Stage 1: Pre-pilot<br />
! style="height:50px; width:250px; text-align:center;" | Stage 2: Content-focused pilot<br />
! style="height:50px; width:250px; text-align:center;" | Stage 3: Data-focused pilot<br />
|-<br />
| style="text-align:center;" | '''Objective'''<br />
| style="vertical-align:top;"| Answer broad questions about '''qualitative factors''' like [[Questionnaire Design|questionnaire design]] and the context for conducting the study. Discuss these with the concerned teams.<br />
<br />
Learn more about how people think to see what structure and flow makes sense.<br />
For example, do people think about their input use at plot-level? by crop? overall?<br />
<br />
It helps to think of the '''pre-pilot''' as a part of the process of [[Questionnaire Design|questionnaire design]]. For example, the '''pre-pilot''' can answer specific questions such as - important sources of income, relevant types of shocks, local food groups, and so on.<br />
<br />
In cases where the [[Impact Evaluation Team|research team]] is using a pre-existing questionnaire, they can skip this step. However, a '''pre-pilot''' is absolutely essential if the '''research team''' is designing a questionnaire from scratch, or if the questionnaire asks questions about issues that are difficult to measure.<br />
| style="vertical-align:top;"| Refine the order and wording of specific questions, the overall structure of the questionnaire, and [[Questionnaire Translation|translations]]. <br />
<br />
<br />
<br />
Check that the answer [[SurveyCTO Choice Lists|choices]] are '''comprehensive''', that is, they cover all possibilities. <br />
<br />
<br />
Flag any sensitive questions.<br />
<br />
<br />
Check how long it takes to answer the questions. Also check how answers differ between respondents ('''response variance''')<br />
| style="vertical-align:top;"| Check if the [[Questionnaire Programming|programmed instrument]] displays questions in the correct order, and follows the correct patterns (such as a group of questions which will always appear together, or will repeat).<br />
<br />
<br />
Load a sample data set (based on '''pilot interviews'''). Check if the data set has any missing fields. Perform all data quality checks, like [[Back Checks|back checks]] and [[Monitoring Data Quality#High Frequency Checks|high frequency checks]].<br />
|-<br />
| style="height:80px;text-align:center;" | '''Status of survey instrument'''<br />
| style="vertical-align:center;"| Early, printable version of the draft, and notes for further discussion.<br />
| style="vertical-align:center;"| A translated, printable draft. Ready to be '''programmed'''.<br />
| style="vertical-align:center;"| A translated, programmed, final draft. Ready for starting data collection.<br />
|-<br />
| style="height:80px;text-align:left;" | '''Mode'''<br />
| style="text-align:center;"| Pen-and-paper<br />
| style="text-align:center;"| Pen-and-paper<br />
| style="text-align:center;"| Electronic (phone/tablet)<br />
|}<br />
<br />
However, note that all three stages may not be necessary for every survey. The research team has to determine which stage to begin from on a case-to-case basis.<br />
* If the research team is using a brand new survey instrument, then they must start with Stage 1, the '''pre-pilot.'''<br />
* If the survey instrument is an adaptation of (based on) a different instrument which was used for a previous data collection for another project in the same region (province or country), the research team can start with Stage 2, the '''content-focused pilot'''. However, they must make sure that the survey instrument does not have any design issues and was shared by a reliable source.<br />
* If the survey instrument is an adaptation of an instrument used for a previous data collection for the same project, but the research team had to make significant revisions or additions, even then the research team should start with Stage 2, the '''content-focused pilot'''.<br />
* Only if the research team is '''piloting''' a '''follow-up''' survey, and there are no major changes from the '''baseline''' (or first round) survey, then in this case the research team may skip directly to Stage 3, the '''data-focused pilot'''.<br />
<br />
== Pilot on paper first == <br />
The first two stages of the survey pilot (pre-pilot and content-focused pilot) are best done on pen-and-paper is done on paper, regardless of the planned survey mode ([[Pen-and-Paper_Personal_Interviews_(PAPI)|pen-and-paper personal interviews (PAPI)]],<br />
[[Computer-Assisted_Personal_Interviews_(CAPI)|computer-assisted personal interviews (CAPI)]], [[Remote_Surveys|phone surveys (CATI)]]). A pen-and-paper pilot allows members of the research team and interviewers flexibility when it comes to recording answers and '''qualitative''' observations. <br />
<br />
Piloting on paper is particularly helpful for the following:<br />
* '''Open-ended responses.''' pre-pilot responses help the research team to make the list of choices for a question more detailed and comprehensive. Pen-and-paper pilots allow the interviewers to make note of these.<br />
* '''Restructuring.''' Pen-and-paper pilots allow the interviewers to draw lines and arrows between questions, which makes it easier to restructure the '''instrument'''.<br />
* '''Observations.''' Pen-and-paper pilots allow interviewers to record observations and feedback of interviewers in the margins. This allows the research team to take note of issues, such as the wording or flow of questions.<br />
<br />
== Timeline ==<br />
In order to conduct a successful [[Survey Pilot|survey pilot]], the [[Impact Evaluation Team|impact evaluation team]] must draft a detailed '''timeline for a survey pilot'''. The '''timeline''' should allow enough time for discussions and revisions about aspects like [[Questionnaire Design|instrument design]], [[Questionnaire Translation|translation]], and [[Survey Protocols|protocols]]. '''Piloting''' should ideally start 4-6 months before [[Field Surveys#Survey Launch|survey launch]] since this allows the research team to use the feedback from each [[Survey Pilot#Stages of a Survey Pilot|stage of a survey pilot]] to improve the survey content and [[Piloting Survey Protocols|protocols]]. <br />
<br />
The '''timeline''' is an important part of the process of [[Structuring a Survey Pilot|structuring a survey pilot]]. The actual [[Structuring a Survey Pilot#Duration of the Pilot|duration of the pilot]] will depend on factors like number of changes made, location of the study area, and availability of equipment for '''field teams'''. Based on '''best practices''', the following are the steps involved in conducting a [[Survey Pilot|survey pilot]] (from start to finish), and the recommended time for each step.<br />
{| class="wikitable" style="margin-left: auto; margin-right: auto; border: none;"<br />
!style="width:350px; text-align:center;"| Task<br />
!style="width:350px; text-align:center;"| Estimated Time<br />
|-<br />
|style="text-align:center;"| [[Survey Pilot#Stages of a Survey Pilot|Pre-pilot]]<br />
|style="text-align:center;"| 1-2 weeks<br />
|-<br />
|style="text-align:center;"| [[Questionnaire Design|Questionnaire design]]<br />
|style="text-align:center;"| 4 weeks<br />
|-<br />
|style="text-align:center;"| [[Piloting Survey Content|Content-focused pilot]] <br />
|style="text-align:center;"| 2-3 weeks<br />
|-<br />
|style="text-align:center;"| [[Structuring a Survey Pilot#Review Sessions|Review sessions]]<br />
|style="text-align:center;"| 1-2 weeks<br />
|-<br />
|style="text-align:center;"| [[Questionnaire Translation|Questionnaire translation]]<br />
|style="text-align:center;"| 2-3 weeks<br />
|-<br />
|style="text-align:center;"| [[IRB Approval|IRB approval]]<br />
|style="text-align:center;"| Depends on the complexity of the survey. Applying for [[IRB Approval|approvals]], [[Questionnaire Programming|programming]], and the [[Survey Pilot#Stages of a Survey Pilot|data-focused pilot]] - should all happen at the same time.<br />
|-<br />
|style="text-align:center;"| [[Questionnaire Programming|Questionnaire programming]]<br />
|style="text-align:center;"| 4-6 weeks<br />
|-<br />
|style="text-align:center;"| [[Survey Pilot#Stages of a Survey Pilot|Data-focused pilot]]<br />
|style="text-align:center;"| 2-3 weeks<br />
|-<br />
|style="text-align:center;"| [[Enumerator Training|Enumerator training]]<br />
|style="text-align:center;"| 2 weeks<br />
|}<br />
<br />
Do not confuse the pilot with '''field testing''' (practice interviews) which is conducted at the end of [[Enumerator_Training |enumerator training]]!<br />
<br />
== Structure ==<br />
The process of '''structuring a survey pilot''' involves agreeing upon the '''logistics''' (or practical aspects) of a [[Survey Pilot|survey pilot]], including '''duration''', '''approvals''', '''review sessions''', and '''training.''' In order to conduct a successful pilot, the [[Impact Evaluation Team|impact evaluation team]] (or research team) must discuss each of these aspects in detail to ensure that the '''survey pilot''' is conducted smoothly. A good pilot helps to improve the quality of the [[Primary Data Collection|data collection process]] and the [[Survey Protocols|survey protocols]]. Go to [[Structuring a Survey Pilot|Structuring a Survey Pilot]] for a detailed discussion of recommended protocols.<br />
<br />
== Participants ==<br />
Typically, a '''survey pilot''' is conducted before a [[Survey Firm|survey firm]] is brought on-board. Each of the [[Survey Pilot Participants|survey pilot participants]] plays an important role in the implementation of a successful pilot. Typical participants include respondents, interviewiers, field coordinator, principal investigator, and other research team members. For a detailed discussion of the role of each participant, and how to select appropriate respondents and interviewers for a pilot, go to the article on [https://dimewiki.worldbank.org/wiki/Survey_Pilot_Participants|Survey Pilot Participants].<br />
<br />
== Guidelines ==<br />
Finally, it is important to keep the following '''best practices''' in mind while planning and conducting a pilot:<br />
* Throughout the process of [[Questionnaire Design|designing the questionnaire]], take notes of what needs to be explored or tested during the pilot.<br />
* Encourage interviewers to probe and follow-up much more than they would in a typical interview<br />
* Encourage respondents to think out-loud, to understand how they are coming up with their answer, and ask enumerators to record these notes.<br />
* In some cultures, it will be appropriate and useful to ask respondents their feedback at the end of the interview<br />
* Take careful notes of the discussions and clarifications that come up during the pilot. These will be an important part of the [[Enumerator Training#Enumerator Manual|enumerator manual]] which is used during [[Enumerator Training|enumerator training]].<br />
* It is useful for research team members to sit in on pilot interviews even if they do not speak the local language. Take note of: questions that take longer than expected; any moments at which the respondent seems frustrated, confused, or uncomfortable; any questions that generate significant discussion; and the overall flow of modules and survey instrument.<br />
* Use the '''data-focused pilot''' to test [[Back Checks|back check]] templates. <br />
* Hire a local '''mobilizer''' to coordinate with respondents. '''Mobilizers''' explain the purpose behind conducting the survey, and facilitate the process of obtaining consent. This is particularly helpful in urban areas, or in cases where respondents are busy. This can improve the outcomes of piloting, for instance, by reducing gap between surveys.<br />
<br />
[https://www.worldbank.org/en/research/dime/data-and-analytics DIME Analytics] has also created the following checklists to assist researchers and enumerators in preparing for, and implementing a pilot:<br />
*[[Preparing_for_the_survey_checklist|Checklist: Preparing for a survey pilot]]<br />
*[[Checklist: Refine the Questionnaire (Content)|Checklist: Refining questionnaire content]]<br />
*[[Checklist: Refine the Questionnaire (Data)|Checklist: Refining questionnaire data]]<br />
<br />
== Related Pages ==<br />
[[Special:WhatLinksHere/Survey_Pilot|Click here for pages that link to this topic.]]<br />
<br />
== Additional Resources == <br />
* DIME Analytics (World Bank), [https://osf.io/63uv9/ Survey pilot guidelines]<br />
* DIME Analytics (World Bank), [https://osf.io/rqb5m/ Survey instrument design and pilot]<br />
<br />
[[Category: Primary Data Collection]] [[Category: Questionnaire Design]]</div>Maria ruth joneshttps://dimewiki.worldbank.org/index.php?title=Survey_Pilot&diff=7893Survey Pilot2021-01-16T20:34:07Z<p>Maria ruth jones: /* Participants */</p>
<hr />
<div>'''Survey pilot''' is the process of carrying out interviews and tests on different components of a survey, including [[Piloting Survey Content|content]] and [[Piloting Survey Protocols|protocols]]. A good '''pilot''' provides the research team with important feedback before they start the process of [[Primary Data Collection|primary data collection]]. This feedback can help the [[Impact Evaluation Team|research team]] review and improve the [[Questionnaire Design| design of the questionnaire]], [[Questionnaire Translation|translated questionnaires]], as well as [[Survey Protocols|survey protocols]] related to [[Piloting Survey Protocols#Interview scheduling|interview scheduling]], [[Sampling|sampling]], and [[Geo Spatial Data|geo data]].<br />
== Read First ==<br />
* A comprehensive '''survey pilot''' includes 3 stages: a pre-pilot, a content-focused pilot, and a data-focused pilot. <br />
* The '''pre-pilot''' is usually done during the process of [[Questionnaire Design|designing a questionnaire]], to refine the design, and wording of questions.<br />
* The '''content-focused pilot''' is conducted after [[Questionnaire Design|designing a questionnaire]], and sharing it with field and sector experts for their comments and inputs.<br />
* Finally, the '''data-focused pilot''' allows the [[Impact Evaluation Team|research team]] to review the data collected during the pilot, and identify possible errors in the [[Questionnaire Programming|programmed questionnaire]]. <br />
* A '''survey pilot''' is different from the '''field testing''' (practice interviews) that enumerators conduct at the end of [[Enumerator Training|enumerator training]]. The pilot should be complete before enumerator training begins. <br />
* [[Survey Pilot Participants|Participants of a survey pilot]] include the [[Impact Evaluation Team|research team]], '''interviewers''', and '''respondents'''.<br />
* Conduct the first two stages of the pilot before [[Questionnaire Programming|programming the questionnaire]]. Repeated programming is time-consuming and can create '''bugs''' (errors), for instance, by disturbing the order of questions.<br />
<br />
== Stages of a Survey Pilot ==<br />
A complete '''survey pilot''' is conducted over three stages - '''pre-pilot''', [[Piloting Survey Content|content-focused pilot]], and '''data-focused pilot'''. The table below discusses these stages in more detail.<br />
<br />
{| class="wikitable" style="margin-left: auto; margin-right: auto; border: none;"<br />
|-<br />
! style="height:50px; width:150px; text-align:center;" |<br />
! style="height:50px; width:250px; text-align:center;" | Stage 1: Pre-pilot<br />
! style="height:50px; width:250px; text-align:center;" | Stage 2: Content-focused pilot<br />
! style="height:50px; width:250px; text-align:center;" | Stage 3: Data-focused pilot<br />
|-<br />
| style="text-align:center;" | '''Objective'''<br />
| style="vertical-align:top;"| Answer broad questions about '''qualitative factors''' like [[Questionnaire Design|questionnaire design]] and the context for conducting the study. Discuss these with the concerned teams.<br />
<br />
Learn more about how people think to see what structure and flow makes sense.<br />
For example, do people think about their input use at plot-level? by crop? overall?<br />
<br />
It helps to think of the '''pre-pilot''' as a part of the process of [[Questionnaire Design|questionnaire design]]. For example, the '''pre-pilot''' can answer specific questions such as - important sources of income, relevant types of shocks, local food groups, and so on.<br />
<br />
In cases where the [[Impact Evaluation Team|research team]] is using a pre-existing questionnaire, they can skip this step. However, a '''pre-pilot''' is absolutely essential if the '''research team''' is designing a questionnaire from scratch, or if the questionnaire asks questions about issues that are difficult to measure.<br />
| style="vertical-align:top;"| Refine the order and wording of specific questions, the overall structure of the questionnaire, and [[Questionnaire Translation|translations]]. <br />
<br />
<br />
<br />
Check that the answer [[SurveyCTO Choice Lists|choices]] are '''comprehensive''', that is, they cover all possibilities. <br />
<br />
<br />
Flag any sensitive questions.<br />
<br />
<br />
Check how long it takes to answer the questions. Also check how answers differ between respondents ('''response variance''')<br />
| style="vertical-align:top;"| Check if the [[Questionnaire Programming|programmed instrument]] displays questions in the correct order, and follows the correct patterns (such as a group of questions which will always appear together, or will repeat).<br />
<br />
<br />
Load a sample data set (based on '''pilot interviews'''). Check if the data set has any missing fields. Perform all data quality checks, like [[Back Checks|back checks]] and [[Monitoring Data Quality#High Frequency Checks|high frequency checks]].<br />
|-<br />
| style="height:80px;text-align:center;" | '''Status of survey instrument'''<br />
| style="vertical-align:center;"| Early, printable version of the draft, and notes for further discussion.<br />
| style="vertical-align:center;"| A translated, printable draft. Ready to be '''programmed'''.<br />
| style="vertical-align:center;"| A translated, programmed, final draft. Ready for starting data collection.<br />
|-<br />
| style="height:80px;text-align:left;" | '''Mode'''<br />
| style="text-align:center;"| Pen-and-paper<br />
| style="text-align:center;"| Pen-and-paper<br />
| style="text-align:center;"| Electronic (phone/tablet)<br />
|}<br />
<br />
However, note that all three stages may not be necessary for every survey. The research team has to determine which stage to begin from on a case-to-case basis.<br />
* If the research team is using a brand new survey instrument, then they must start with Stage 1, the '''pre-pilot.'''<br />
* If the survey instrument is an adaptation of (based on) a different instrument which was used for a previous data collection for another project in the same region (province or country), the research team can start with Stage 2, the '''content-focused pilot'''. However, they must make sure that the survey instrument does not have any design issues and was shared by a reliable source.<br />
* If the survey instrument is an adaptation of an instrument used for a previous data collection for the same project, but the research team had to make significant revisions or additions, even then the research team should start with Stage 2, the '''content-focused pilot'''.<br />
* Only if the research team is '''piloting''' a '''follow-up''' survey, and there are no major changes from the '''baseline''' (or first round) survey, then in this case the research team may skip directly to Stage 3, the '''data-focused pilot'''.<br />
<br />
== Pilot on paper first == <br />
The first two stages of the survey pilot (pre-pilot and content-focused pilot) are best done on pen-and-paper is done on paper, regardless of the planned survey mode ([[Pen-and-Paper_Personal_Interviews_(PAPI)|pen-and-paper personal interviews (PAPI)]],<br />
[[Computer-Assisted_Personal_Interviews_(CAPI)|computer-assisted personal interviews (CAPI)]], [[Remote_Surveys|phone surveys (CATI)]]). A pen-and-paper pilot allows members of the research team and interviewers flexibility when it comes to recording answers and '''qualitative''' observations. <br />
<br />
Piloting on paper is particularly helpful for the following:<br />
* '''Open-ended responses.''' pre-pilot responses help the research team to make the list of choices for a question more detailed and comprehensive. Pen-and-paper pilots allow the interviewers to make note of these.<br />
* '''Restructuring.''' Pen-and-paper pilots allow the interviewers to draw lines and arrows between questions, which makes it easier to restructure the '''instrument'''.<br />
* '''Observations.''' Pen-and-paper pilots allow interviewers to record observations and feedback of interviewers in the margins. This allows the research team to take note of issues, such as the wording or flow of questions.<br />
<br />
== Timeline ==<br />
In order to conduct a successful [[Survey Pilot|survey pilot]], the [[Impact Evaluation Team|impact evaluation team]] must draft a detailed '''timeline for a survey pilot'''. The '''timeline''' should allow enough time for discussions and revisions about aspects like [[Questionnaire Design|instrument design]], [[Questionnaire Translation|translation]], and [[Survey Protocols|protocols]]. '''Piloting''' should ideally start 4-6 months before [[Field Surveys#Survey Launch|survey launch]] since this allows the research team to use the feedback from each [[Survey Pilot#Stages of a Survey Pilot|stage of a survey pilot]] to improve the survey content and [[Piloting Survey Protocols|protocols]]. <br />
<br />
The '''timeline''' is an important part of the process of [[Structuring a Survey Pilot|structuring a survey pilot]]. The actual [[Structuring a Survey Pilot#Duration of the Pilot|duration of the pilot]] will depend on factors like number of changes made, location of the study area, and availability of equipment for '''field teams'''. Based on '''best practices''', the following are the steps involved in conducting a [[Survey Pilot|survey pilot]] (from start to finish), and the recommended time for each step.<br />
{| class="wikitable" style="margin-left: auto; margin-right: auto; border: none;"<br />
!style="width:350px; text-align:center;"| Task<br />
!style="width:350px; text-align:center;"| Estimated Time<br />
|-<br />
|style="text-align:center;"| [[Survey Pilot#Stages of a Survey Pilot|Pre-pilot]]<br />
|style="text-align:center;"| 1-2 weeks<br />
|-<br />
|style="text-align:center;"| [[Questionnaire Design|Questionnaire design]]<br />
|style="text-align:center;"| 4 weeks<br />
|-<br />
|style="text-align:center;"| [[Piloting Survey Content|Content-focused pilot]] <br />
|style="text-align:center;"| 2-3 weeks<br />
|-<br />
|style="text-align:center;"| [[Structuring a Survey Pilot#Review Sessions|Review sessions]]<br />
|style="text-align:center;"| 1-2 weeks<br />
|-<br />
|style="text-align:center;"| [[Questionnaire Translation|Questionnaire translation]]<br />
|style="text-align:center;"| 2-3 weeks<br />
|-<br />
|style="text-align:center;"| [[IRB Approval|IRB approval]]<br />
|style="text-align:center;"| Depends on the complexity of the survey. Applying for [[IRB Approval|approvals]], [[Questionnaire Programming|programming]], and the [[Survey Pilot#Stages of a Survey Pilot|data-focused pilot]] - should all happen at the same time.<br />
|-<br />
|style="text-align:center;"| [[Questionnaire Programming|Questionnaire programming]]<br />
|style="text-align:center;"| 4-6 weeks<br />
|-<br />
|style="text-align:center;"| [[Survey Pilot#Stages of a Survey Pilot|Data-focused pilot]]<br />
|style="text-align:center;"| 2-3 weeks<br />
|-<br />
|style="text-align:center;"| [[Enumerator Training|Enumerator training]]<br />
|style="text-align:center;"| 2 weeks<br />
|}<br />
<br />
Do not confuse the pilot with '''field testing''' (practice interviews) which is conducted at the end of [[Enumerator_Training |enumerator training]]!<br />
<br />
== Structure ==<br />
The process of '''structuring a survey pilot''' involves agreeing upon the '''logistics''' (or practical aspects) of a [[Survey Pilot|survey pilot]], including '''duration''', '''approvals''', '''review sessions''', and '''training.''' In order to conduct a successful pilot, the [[Impact Evaluation Team|impact evaluation team]] (or research team) must discuss each of these aspects in detail to ensure that the '''survey pilot''' is conducted smoothly. A good pilot helps to improve the quality of the [[Primary Data Collection|data collection process]] and the [[Survey Protocols|survey protocols]]. Go to [[Structuring a Survey Pilot|Structuring a Survey Pilot]] for a detailed discussion of recommended protocols.<br />
<br />
== Participants ==<br />
Typically, a '''survey pilot''' is conducted before a [[Survey Firm|survey firm]] is brought on-board. Each of the [[Survey Pilot Participants|survey pilot participants]] plays an important role in the implementation of a successful pilot. Typical participants include respondents, interviewiers, field coordinator, principal investigator, and other research team members. For a detailed discussion of the role of each participant, and how to select appropriate respondents and interviewers for a pilot, go to the article on [[https://dimewiki.worldbank.org/wiki/Survey_Pilot_Participants|Survey Pilot Participants]].<br />
<br />
== Guidelines ==<br />
Finally, it is important to keep the following '''best practices''' in mind while planning and conducting a pilot:<br />
* Throughout the process of [[Questionnaire Design|designing the questionnaire]], take notes of what needs to be explored or tested during the pilot.<br />
* Encourage interviewers to probe and follow-up much more than they would in a typical interview<br />
* Encourage respondents to think out-loud, to understand how they are coming up with their answer, and ask enumerators to record these notes.<br />
* In some cultures, it will be appropriate and useful to ask respondents their feedback at the end of the interview<br />
* Take careful notes of the discussions and clarifications that come up during the pilot. These will be an important part of the [[Enumerator Training#Enumerator Manual|enumerator manual]] which is used during [[Enumerator Training|enumerator training]].<br />
* It is useful for research team members to sit in on pilot interviews even if they do not speak the local language. Take note of: questions that take longer than expected; any moments at which the respondent seems frustrated, confused, or uncomfortable; any questions that generate significant discussion; and the overall flow of modules and survey instrument.<br />
* Use the '''data-focused pilot''' to test [[Back Checks|back check]] templates. <br />
* Hire a local '''mobilizer''' to coordinate with respondents. '''Mobilizers''' explain the purpose behind conducting the survey, and facilitate the process of obtaining consent. This is particularly helpful in urban areas, or in cases where respondents are busy. This can improve the outcomes of piloting, for instance, by reducing gap between surveys.<br />
<br />
[https://www.worldbank.org/en/research/dime/data-and-analytics DIME Analytics] has also created the following checklists to assist researchers and enumerators in preparing for, and implementing a pilot:<br />
*[[Preparing_for_the_survey_checklist|Checklist: Preparing for a survey pilot]]<br />
*[[Checklist: Refine the Questionnaire (Content)|Checklist: Refining questionnaire content]]<br />
*[[Checklist: Refine the Questionnaire (Data)|Checklist: Refining questionnaire data]]<br />
<br />
== Related Pages ==<br />
[[Special:WhatLinksHere/Survey_Pilot|Click here for pages that link to this topic.]]<br />
<br />
== Additional Resources == <br />
* DIME Analytics (World Bank), [https://osf.io/63uv9/ Survey pilot guidelines]<br />
* DIME Analytics (World Bank), [https://osf.io/rqb5m/ Survey instrument design and pilot]<br />
<br />
[[Category: Primary Data Collection]] [[Category: Questionnaire Design]]</div>Maria ruth joneshttps://dimewiki.worldbank.org/index.php?title=Survey_Pilot&diff=7892Survey Pilot2021-01-16T20:31:25Z<p>Maria ruth jones: /* Read First */</p>
<hr />
<div>'''Survey pilot''' is the process of carrying out interviews and tests on different components of a survey, including [[Piloting Survey Content|content]] and [[Piloting Survey Protocols|protocols]]. A good '''pilot''' provides the research team with important feedback before they start the process of [[Primary Data Collection|primary data collection]]. This feedback can help the [[Impact Evaluation Team|research team]] review and improve the [[Questionnaire Design| design of the questionnaire]], [[Questionnaire Translation|translated questionnaires]], as well as [[Survey Protocols|survey protocols]] related to [[Piloting Survey Protocols#Interview scheduling|interview scheduling]], [[Sampling|sampling]], and [[Geo Spatial Data|geo data]].<br />
== Read First ==<br />
* A comprehensive '''survey pilot''' includes 3 stages: a pre-pilot, a content-focused pilot, and a data-focused pilot. <br />
* The '''pre-pilot''' is usually done during the process of [[Questionnaire Design|designing a questionnaire]], to refine the design, and wording of questions.<br />
* The '''content-focused pilot''' is conducted after [[Questionnaire Design|designing a questionnaire]], and sharing it with field and sector experts for their comments and inputs.<br />
* Finally, the '''data-focused pilot''' allows the [[Impact Evaluation Team|research team]] to review the data collected during the pilot, and identify possible errors in the [[Questionnaire Programming|programmed questionnaire]]. <br />
* A '''survey pilot''' is different from the '''field testing''' (practice interviews) that enumerators conduct at the end of [[Enumerator Training|enumerator training]]. The pilot should be complete before enumerator training begins. <br />
* [[Survey Pilot Participants|Participants of a survey pilot]] include the [[Impact Evaluation Team|research team]], '''interviewers''', and '''respondents'''.<br />
* Conduct the first two stages of the pilot before [[Questionnaire Programming|programming the questionnaire]]. Repeated programming is time-consuming and can create '''bugs''' (errors), for instance, by disturbing the order of questions.<br />
<br />
== Stages of a Survey Pilot ==<br />
A complete '''survey pilot''' is conducted over three stages - '''pre-pilot''', [[Piloting Survey Content|content-focused pilot]], and '''data-focused pilot'''. The table below discusses these stages in more detail.<br />
<br />
{| class="wikitable" style="margin-left: auto; margin-right: auto; border: none;"<br />
|-<br />
! style="height:50px; width:150px; text-align:center;" |<br />
! style="height:50px; width:250px; text-align:center;" | Stage 1: Pre-pilot<br />
! style="height:50px; width:250px; text-align:center;" | Stage 2: Content-focused pilot<br />
! style="height:50px; width:250px; text-align:center;" | Stage 3: Data-focused pilot<br />
|-<br />
| style="text-align:center;" | '''Objective'''<br />
| style="vertical-align:top;"| Answer broad questions about '''qualitative factors''' like [[Questionnaire Design|questionnaire design]] and the context for conducting the study. Discuss these with the concerned teams.<br />
<br />
Learn more about how people think to see what structure and flow makes sense.<br />
For example, do people think about their input use at plot-level? by crop? overall?<br />
<br />
It helps to think of the '''pre-pilot''' as a part of the process of [[Questionnaire Design|questionnaire design]]. For example, the '''pre-pilot''' can answer specific questions such as - important sources of income, relevant types of shocks, local food groups, and so on.<br />
<br />
In cases where the [[Impact Evaluation Team|research team]] is using a pre-existing questionnaire, they can skip this step. However, a '''pre-pilot''' is absolutely essential if the '''research team''' is designing a questionnaire from scratch, or if the questionnaire asks questions about issues that are difficult to measure.<br />
| style="vertical-align:top;"| Refine the order and wording of specific questions, the overall structure of the questionnaire, and [[Questionnaire Translation|translations]]. <br />
<br />
<br />
<br />
Check that the answer [[SurveyCTO Choice Lists|choices]] are '''comprehensive''', that is, they cover all possibilities. <br />
<br />
<br />
Flag any sensitive questions.<br />
<br />
<br />
Check how long it takes to answer the questions. Also check how answers differ between respondents ('''response variance''')<br />
| style="vertical-align:top;"| Check if the [[Questionnaire Programming|programmed instrument]] displays questions in the correct order, and follows the correct patterns (such as a group of questions which will always appear together, or will repeat).<br />
<br />
<br />
Load a sample data set (based on '''pilot interviews'''). Check if the data set has any missing fields. Perform all data quality checks, like [[Back Checks|back checks]] and [[Monitoring Data Quality#High Frequency Checks|high frequency checks]].<br />
|-<br />
| style="height:80px;text-align:center;" | '''Status of survey instrument'''<br />
| style="vertical-align:center;"| Early, printable version of the draft, and notes for further discussion.<br />
| style="vertical-align:center;"| A translated, printable draft. Ready to be '''programmed'''.<br />
| style="vertical-align:center;"| A translated, programmed, final draft. Ready for starting data collection.<br />
|-<br />
| style="height:80px;text-align:left;" | '''Mode'''<br />
| style="text-align:center;"| Pen-and-paper<br />
| style="text-align:center;"| Pen-and-paper<br />
| style="text-align:center;"| Electronic (phone/tablet)<br />
|}<br />
<br />
However, note that all three stages may not be necessary for every survey. The research team has to determine which stage to begin from on a case-to-case basis.<br />
* If the research team is using a brand new survey instrument, then they must start with Stage 1, the '''pre-pilot.'''<br />
* If the survey instrument is an adaptation of (based on) a different instrument which was used for a previous data collection for another project in the same region (province or country), the research team can start with Stage 2, the '''content-focused pilot'''. However, they must make sure that the survey instrument does not have any design issues and was shared by a reliable source.<br />
* If the survey instrument is an adaptation of an instrument used for a previous data collection for the same project, but the research team had to make significant revisions or additions, even then the research team should start with Stage 2, the '''content-focused pilot'''.<br />
* Only if the research team is '''piloting''' a '''follow-up''' survey, and there are no major changes from the '''baseline''' (or first round) survey, then in this case the research team may skip directly to Stage 3, the '''data-focused pilot'''.<br />
<br />
== Pilot on paper first == <br />
The first two stages of the survey pilot (pre-pilot and content-focused pilot) are best done on pen-and-paper is done on paper, regardless of the planned survey mode ([[Pen-and-Paper_Personal_Interviews_(PAPI)|pen-and-paper personal interviews (PAPI)]],<br />
[[Computer-Assisted_Personal_Interviews_(CAPI)|computer-assisted personal interviews (CAPI)]], [[Remote_Surveys|phone surveys (CATI)]]). A pen-and-paper pilot allows members of the research team and interviewers flexibility when it comes to recording answers and '''qualitative''' observations. <br />
<br />
Piloting on paper is particularly helpful for the following:<br />
* '''Open-ended responses.''' pre-pilot responses help the research team to make the list of choices for a question more detailed and comprehensive. Pen-and-paper pilots allow the interviewers to make note of these.<br />
* '''Restructuring.''' Pen-and-paper pilots allow the interviewers to draw lines and arrows between questions, which makes it easier to restructure the '''instrument'''.<br />
* '''Observations.''' Pen-and-paper pilots allow interviewers to record observations and feedback of interviewers in the margins. This allows the research team to take note of issues, such as the wording or flow of questions.<br />
<br />
== Timeline ==<br />
In order to conduct a successful [[Survey Pilot|survey pilot]], the [[Impact Evaluation Team|impact evaluation team]] must draft a detailed '''timeline for a survey pilot'''. The '''timeline''' should allow enough time for discussions and revisions about aspects like [[Questionnaire Design|instrument design]], [[Questionnaire Translation|translation]], and [[Survey Protocols|protocols]]. '''Piloting''' should ideally start 4-6 months before [[Field Surveys#Survey Launch|survey launch]] since this allows the research team to use the feedback from each [[Survey Pilot#Stages of a Survey Pilot|stage of a survey pilot]] to improve the survey content and [[Piloting Survey Protocols|protocols]]. <br />
<br />
The '''timeline''' is an important part of the process of [[Structuring a Survey Pilot|structuring a survey pilot]]. The actual [[Structuring a Survey Pilot#Duration of the Pilot|duration of the pilot]] will depend on factors like number of changes made, location of the study area, and availability of equipment for '''field teams'''. Based on '''best practices''', the following are the steps involved in conducting a [[Survey Pilot|survey pilot]] (from start to finish), and the recommended time for each step.<br />
{| class="wikitable" style="margin-left: auto; margin-right: auto; border: none;"<br />
!style="width:350px; text-align:center;"| Task<br />
!style="width:350px; text-align:center;"| Estimated Time<br />
|-<br />
|style="text-align:center;"| [[Survey Pilot#Stages of a Survey Pilot|Pre-pilot]]<br />
|style="text-align:center;"| 1-2 weeks<br />
|-<br />
|style="text-align:center;"| [[Questionnaire Design|Questionnaire design]]<br />
|style="text-align:center;"| 4 weeks<br />
|-<br />
|style="text-align:center;"| [[Piloting Survey Content|Content-focused pilot]] <br />
|style="text-align:center;"| 2-3 weeks<br />
|-<br />
|style="text-align:center;"| [[Structuring a Survey Pilot#Review Sessions|Review sessions]]<br />
|style="text-align:center;"| 1-2 weeks<br />
|-<br />
|style="text-align:center;"| [[Questionnaire Translation|Questionnaire translation]]<br />
|style="text-align:center;"| 2-3 weeks<br />
|-<br />
|style="text-align:center;"| [[IRB Approval|IRB approval]]<br />
|style="text-align:center;"| Depends on the complexity of the survey. Applying for [[IRB Approval|approvals]], [[Questionnaire Programming|programming]], and the [[Survey Pilot#Stages of a Survey Pilot|data-focused pilot]] - should all happen at the same time.<br />
|-<br />
|style="text-align:center;"| [[Questionnaire Programming|Questionnaire programming]]<br />
|style="text-align:center;"| 4-6 weeks<br />
|-<br />
|style="text-align:center;"| [[Survey Pilot#Stages of a Survey Pilot|Data-focused pilot]]<br />
|style="text-align:center;"| 2-3 weeks<br />
|-<br />
|style="text-align:center;"| [[Enumerator Training|Enumerator training]]<br />
|style="text-align:center;"| 2 weeks<br />
|}<br />
<br />
Do not confuse the pilot with '''field testing''' (practice interviews) which is conducted at the end of [[Enumerator_Training |enumerator training]]!<br />
<br />
== Structure ==<br />
The process of '''structuring a survey pilot''' involves agreeing upon the '''logistics''' (or practical aspects) of a [[Survey Pilot|survey pilot]], including '''duration''', '''approvals''', '''review sessions''', and '''training.''' In order to conduct a successful pilot, the [[Impact Evaluation Team|impact evaluation team]] (or research team) must discuss each of these aspects in detail to ensure that the '''survey pilot''' is conducted smoothly. A good pilot helps to improve the quality of the [[Primary Data Collection|data collection process]] and the [[Survey Protocols|survey protocols]]. Go to [[Structuring a Survey Pilot|Structuring a Survey Pilot]] for a detailed discussion of recommended protocols.<br />
<br />
== Participants ==<br />
Typically, a '''survey pilot''' is conducted before a [[Survey Firm|survey firm]] is brought on-board. Each of the [[Survey Pilot Participants|survey pilot participants]] plays an important role in the implementation of a successful pilot. <br />
* '''Respondents.''' They should be as close to the '''target population''' as possible. None of the respondents from a pilot should be in the sample for the actual survey.<br />
* '''Interviewers.''' They should be fluent in all local languages of the study area and the language of the research team. The size of the team of interviewers depends on the stage of pilot.<br />
* '''Field coordinator (FC).''' The [[Impact_Evaluation_Team#Field_Coordinator|field coordinators (FCs)]] play a central role as part of the research team. They are responsible for updating the research team on all monitoring and evaluation '''(M&E)''' activities in the field.<br />
* '''Principal investigator (PI).''' The [[Impact_Evaluation_Team#Principal_Investigator|principal investigators (PIs)]] oversee the overall functioning of the '''impact evaluation''' (study). They should brief the FCs regularly during the pilot.<br />
* '''Other research team members.'''Ideally, other members of the [[Impact Evaluation Team|impact evaluation team]] such as the [[Impact Evaluation Team#Impact Evaluation Coordinator|impact evaluation coordinator]], research manager, programmers, and translators should also participate.<br />
<br />
== Guidelines ==<br />
Finally, it is important to keep the following '''best practices''' in mind while planning and conducting a pilot:<br />
* Throughout the process of [[Questionnaire Design|designing the questionnaire]], take notes of what needs to be explored or tested during the pilot.<br />
* Encourage interviewers to probe and follow-up much more than they would in a typical interview<br />
* Encourage respondents to think out-loud, to understand how they are coming up with their answer, and ask enumerators to record these notes.<br />
* In some cultures, it will be appropriate and useful to ask respondents their feedback at the end of the interview<br />
* Take careful notes of the discussions and clarifications that come up during the pilot. These will be an important part of the [[Enumerator Training#Enumerator Manual|enumerator manual]] which is used during [[Enumerator Training|enumerator training]].<br />
* It is useful for research team members to sit in on pilot interviews even if they do not speak the local language. Take note of: questions that take longer than expected; any moments at which the respondent seems frustrated, confused, or uncomfortable; any questions that generate significant discussion; and the overall flow of modules and survey instrument.<br />
* Use the '''data-focused pilot''' to test [[Back Checks|back check]] templates. <br />
* Hire a local '''mobilizer''' to coordinate with respondents. '''Mobilizers''' explain the purpose behind conducting the survey, and facilitate the process of obtaining consent. This is particularly helpful in urban areas, or in cases where respondents are busy. This can improve the outcomes of piloting, for instance, by reducing gap between surveys.<br />
<br />
[https://www.worldbank.org/en/research/dime/data-and-analytics DIME Analytics] has also created the following checklists to assist researchers and enumerators in preparing for, and implementing a pilot:<br />
*[[Preparing_for_the_survey_checklist|Checklist: Preparing for a survey pilot]]<br />
*[[Checklist: Refine the Questionnaire (Content)|Checklist: Refining questionnaire content]]<br />
*[[Checklist: Refine the Questionnaire (Data)|Checklist: Refining questionnaire data]]<br />
<br />
== Related Pages ==<br />
[[Special:WhatLinksHere/Survey_Pilot|Click here for pages that link to this topic.]]<br />
<br />
== Additional Resources == <br />
* DIME Analytics (World Bank), [https://osf.io/63uv9/ Survey pilot guidelines]<br />
* DIME Analytics (World Bank), [https://osf.io/rqb5m/ Survey instrument design and pilot]<br />
<br />
[[Category: Primary Data Collection]] [[Category: Questionnaire Design]]</div>Maria ruth joneshttps://dimewiki.worldbank.org/index.php?title=Survey_Pilot&diff=7891Survey Pilot2021-01-16T20:27:37Z<p>Maria ruth jones: /* Timeline */</p>
<hr />
<div>'''Survey pilot''' is the process of carrying out interviews and tests on different components of a survey, including [[Piloting Survey Content|content]] and [[Piloting Survey Protocols|protocols]]. A good '''pilot''' provides the research team with important feedback before they start the process of [[Primary Data Collection|primary data collection]]. This feedback can help the [[Impact Evaluation Team|research team]] review and improve the [[Questionnaire Design| design of the questionnaire]], [[Questionnaire Translation|translated questionnaires]], as well as [[Survey Protocols|survey protocols]] related to [[Piloting Survey Protocols#Interview scheduling|interview scheduling]], [[Sampling|sampling]], and [[Geo Spatial Data|geo data]].<br />
== Read First ==<br />
* A comprehensive '''survey pilot''' includes 3 stages: a pre-pilot, a content-focused pilot, and a data-focused pilot. <br />
* The '''pre-pilot''' is usually done during the process of [[Questionnaire Design|designing a questionnaire]], to refine the design, and wording of questions.<br />
* The [[Piloting Survey Content|content-focused pilot]] is conducted after [[Questionnaire Design|designing a questionnaire]], and sharing it with field and sector experts for their comments and inputs.<br />
* Finally, the '''data-focused pilot''' allows the [[Impact Evaluation Team|research team]] to review the data collected during the pilot, and identify possible errors in the [[Questionnaire Programming|programmed questionnaire]]. <br />
* A '''survey pilot''' is different from the '''field testing''' (practice interviews) that enumerators conduct at the end of [[Enumerator Training|enumerator training]]. The pilot should be complete before enumerator training begins. <br />
* [[Survey Pilot Participants|Participants of a survey pilot]] include the [[Impact Evaluation Team|research team]], '''interviewers''', and '''respondents'''.<br />
* Conduct the first two stages of the pilot before [[Questionnaire Programming|programming the questionnaire]]. Repeated programming is time-consuming and can create '''bugs''' (errors), for instance, by disturbing the order of questions.<br />
<br />
== Stages of a Survey Pilot ==<br />
A complete '''survey pilot''' is conducted over three stages - '''pre-pilot''', [[Piloting Survey Content|content-focused pilot]], and '''data-focused pilot'''. The table below discusses these stages in more detail.<br />
<br />
{| class="wikitable" style="margin-left: auto; margin-right: auto; border: none;"<br />
|-<br />
! style="height:50px; width:150px; text-align:center;" |<br />
! style="height:50px; width:250px; text-align:center;" | Stage 1: Pre-pilot<br />
! style="height:50px; width:250px; text-align:center;" | Stage 2: Content-focused pilot<br />
! style="height:50px; width:250px; text-align:center;" | Stage 3: Data-focused pilot<br />
|-<br />
| style="text-align:center;" | '''Objective'''<br />
| style="vertical-align:top;"| Answer broad questions about '''qualitative factors''' like [[Questionnaire Design|questionnaire design]] and the context for conducting the study. Discuss these with the concerned teams.<br />
<br />
Learn more about how people think to see what structure and flow makes sense.<br />
For example, do people think about their input use at plot-level? by crop? overall?<br />
<br />
It helps to think of the '''pre-pilot''' as a part of the process of [[Questionnaire Design|questionnaire design]]. For example, the '''pre-pilot''' can answer specific questions such as - important sources of income, relevant types of shocks, local food groups, and so on.<br />
<br />
In cases where the [[Impact Evaluation Team|research team]] is using a pre-existing questionnaire, they can skip this step. However, a '''pre-pilot''' is absolutely essential if the '''research team''' is designing a questionnaire from scratch, or if the questionnaire asks questions about issues that are difficult to measure.<br />
| style="vertical-align:top;"| Refine the order and wording of specific questions, the overall structure of the questionnaire, and [[Questionnaire Translation|translations]]. <br />
<br />
<br />
<br />
Check that the answer [[SurveyCTO Choice Lists|choices]] are '''comprehensive''', that is, they cover all possibilities. <br />
<br />
<br />
Flag any sensitive questions.<br />
<br />
<br />
Check how long it takes to answer the questions. Also check how answers differ between respondents ('''response variance''')<br />
| style="vertical-align:top;"| Check if the [[Questionnaire Programming|programmed instrument]] displays questions in the correct order, and follows the correct patterns (such as a group of questions which will always appear together, or will repeat).<br />
<br />
<br />
Load a sample data set (based on '''pilot interviews'''). Check if the data set has any missing fields. Perform all data quality checks, like [[Back Checks|back checks]] and [[Monitoring Data Quality#High Frequency Checks|high frequency checks]].<br />
|-<br />
| style="height:80px;text-align:center;" | '''Status of survey instrument'''<br />
| style="vertical-align:center;"| Early, printable version of the draft, and notes for further discussion.<br />
| style="vertical-align:center;"| A translated, printable draft. Ready to be '''programmed'''.<br />
| style="vertical-align:center;"| A translated, programmed, final draft. Ready for starting data collection.<br />
|-<br />
| style="height:80px;text-align:left;" | '''Mode'''<br />
| style="text-align:center;"| Pen-and-paper<br />
| style="text-align:center;"| Pen-and-paper<br />
| style="text-align:center;"| Electronic (phone/tablet)<br />
|}<br />
<br />
However, note that all three stages may not be necessary for every survey. The research team has to determine which stage to begin from on a case-to-case basis.<br />
* If the research team is using a brand new survey instrument, then they must start with Stage 1, the '''pre-pilot.'''<br />
* If the survey instrument is an adaptation of (based on) a different instrument which was used for a previous data collection for another project in the same region (province or country), the research team can start with Stage 2, the '''content-focused pilot'''. However, they must make sure that the survey instrument does not have any design issues and was shared by a reliable source.<br />
* If the survey instrument is an adaptation of an instrument used for a previous data collection for the same project, but the research team had to make significant revisions or additions, even then the research team should start with Stage 2, the '''content-focused pilot'''.<br />
* Only if the research team is '''piloting''' a '''follow-up''' survey, and there are no major changes from the '''baseline''' (or first round) survey, then in this case the research team may skip directly to Stage 3, the '''data-focused pilot'''.<br />
<br />
== Pilot on paper first == <br />
The first two stages of the survey pilot (pre-pilot and content-focused pilot) are best done on pen-and-paper is done on paper, regardless of the planned survey mode ([[Pen-and-Paper_Personal_Interviews_(PAPI)|pen-and-paper personal interviews (PAPI)]],<br />
[[Computer-Assisted_Personal_Interviews_(CAPI)|computer-assisted personal interviews (CAPI)]], [[Remote_Surveys|phone surveys (CATI)]]). A pen-and-paper pilot allows members of the research team and interviewers flexibility when it comes to recording answers and '''qualitative''' observations. <br />
<br />
Piloting on paper is particularly helpful for the following:<br />
* '''Open-ended responses.''' pre-pilot responses help the research team to make the list of choices for a question more detailed and comprehensive. Pen-and-paper pilots allow the interviewers to make note of these.<br />
* '''Restructuring.''' Pen-and-paper pilots allow the interviewers to draw lines and arrows between questions, which makes it easier to restructure the '''instrument'''.<br />
* '''Observations.''' Pen-and-paper pilots allow interviewers to record observations and feedback of interviewers in the margins. This allows the research team to take note of issues, such as the wording or flow of questions.<br />
<br />
== Timeline ==<br />
In order to conduct a successful [[Survey Pilot|survey pilot]], the [[Impact Evaluation Team|impact evaluation team]] must draft a detailed '''timeline for a survey pilot'''. The '''timeline''' should allow enough time for discussions and revisions about aspects like [[Questionnaire Design|instrument design]], [[Questionnaire Translation|translation]], and [[Survey Protocols|protocols]]. '''Piloting''' should ideally start 4-6 months before [[Field Surveys#Survey Launch|survey launch]] since this allows the research team to use the feedback from each [[Survey Pilot#Stages of a Survey Pilot|stage of a survey pilot]] to improve the survey content and [[Piloting Survey Protocols|protocols]]. <br />
<br />
The '''timeline''' is an important part of the process of [[Structuring a Survey Pilot|structuring a survey pilot]]. The actual [[Structuring a Survey Pilot#Duration of the Pilot|duration of the pilot]] will depend on factors like number of changes made, location of the study area, and availability of equipment for '''field teams'''. Based on '''best practices''', the following are the steps involved in conducting a [[Survey Pilot|survey pilot]] (from start to finish), and the recommended time for each step.<br />
{| class="wikitable" style="margin-left: auto; margin-right: auto; border: none;"<br />
!style="width:350px; text-align:center;"| Task<br />
!style="width:350px; text-align:center;"| Estimated Time<br />
|-<br />
|style="text-align:center;"| [[Survey Pilot#Stages of a Survey Pilot|Pre-pilot]]<br />
|style="text-align:center;"| 1-2 weeks<br />
|-<br />
|style="text-align:center;"| [[Questionnaire Design|Questionnaire design]]<br />
|style="text-align:center;"| 4 weeks<br />
|-<br />
|style="text-align:center;"| [[Piloting Survey Content|Content-focused pilot]] <br />
|style="text-align:center;"| 2-3 weeks<br />
|-<br />
|style="text-align:center;"| [[Structuring a Survey Pilot#Review Sessions|Review sessions]]<br />
|style="text-align:center;"| 1-2 weeks<br />
|-<br />
|style="text-align:center;"| [[Questionnaire Translation|Questionnaire translation]]<br />
|style="text-align:center;"| 2-3 weeks<br />
|-<br />
|style="text-align:center;"| [[IRB Approval|IRB approval]]<br />
|style="text-align:center;"| Depends on the complexity of the survey. Applying for [[IRB Approval|approvals]], [[Questionnaire Programming|programming]], and the [[Survey Pilot#Stages of a Survey Pilot|data-focused pilot]] - should all happen at the same time.<br />
|-<br />
|style="text-align:center;"| [[Questionnaire Programming|Questionnaire programming]]<br />
|style="text-align:center;"| 4-6 weeks<br />
|-<br />
|style="text-align:center;"| [[Survey Pilot#Stages of a Survey Pilot|Data-focused pilot]]<br />
|style="text-align:center;"| 2-3 weeks<br />
|-<br />
|style="text-align:center;"| [[Enumerator Training|Enumerator training]]<br />
|style="text-align:center;"| 2 weeks<br />
|}<br />
<br />
Do not confuse the pilot with '''field testing''' (practice interviews) which is conducted at the end of [[Enumerator_Training |enumerator training]]!<br />
<br />
== Structure ==<br />
The process of '''structuring a survey pilot''' involves agreeing upon the '''logistics''' (or practical aspects) of a [[Survey Pilot|survey pilot]], including '''duration''', '''approvals''', '''review sessions''', and '''training.''' In order to conduct a successful pilot, the [[Impact Evaluation Team|impact evaluation team]] (or research team) must discuss each of these aspects in detail to ensure that the '''survey pilot''' is conducted smoothly. A good pilot helps to improve the quality of the [[Primary Data Collection|data collection process]] and the [[Survey Protocols|survey protocols]]. Go to [[Structuring a Survey Pilot|Structuring a Survey Pilot]] for a detailed discussion of recommended protocols.<br />
<br />
== Participants ==<br />
Typically, a '''survey pilot''' is conducted before a [[Survey Firm|survey firm]] is brought on-board. Each of the [[Survey Pilot Participants|survey pilot participants]] plays an important role in the implementation of a successful pilot. <br />
* '''Respondents.''' They should be as close to the '''target population''' as possible. None of the respondents from a pilot should be in the sample for the actual survey.<br />
* '''Interviewers.''' They should be fluent in all local languages of the study area and the language of the research team. The size of the team of interviewers depends on the stage of pilot.<br />
* '''Field coordinator (FC).''' The [[Impact_Evaluation_Team#Field_Coordinator|field coordinators (FCs)]] play a central role as part of the research team. They are responsible for updating the research team on all monitoring and evaluation '''(M&E)''' activities in the field.<br />
* '''Principal investigator (PI).''' The [[Impact_Evaluation_Team#Principal_Investigator|principal investigators (PIs)]] oversee the overall functioning of the '''impact evaluation''' (study). They should brief the FCs regularly during the pilot.<br />
* '''Other research team members.'''Ideally, other members of the [[Impact Evaluation Team|impact evaluation team]] such as the [[Impact Evaluation Team#Impact Evaluation Coordinator|impact evaluation coordinator]], research manager, programmers, and translators should also participate.<br />
<br />
== Guidelines ==<br />
Finally, it is important to keep the following '''best practices''' in mind while planning and conducting a pilot:<br />
* Throughout the process of [[Questionnaire Design|designing the questionnaire]], take notes of what needs to be explored or tested during the pilot.<br />
* Encourage interviewers to probe and follow-up much more than they would in a typical interview<br />
* Encourage respondents to think out-loud, to understand how they are coming up with their answer, and ask enumerators to record these notes.<br />
* In some cultures, it will be appropriate and useful to ask respondents their feedback at the end of the interview<br />
* Take careful notes of the discussions and clarifications that come up during the pilot. These will be an important part of the [[Enumerator Training#Enumerator Manual|enumerator manual]] which is used during [[Enumerator Training|enumerator training]].<br />
* It is useful for research team members to sit in on pilot interviews even if they do not speak the local language. Take note of: questions that take longer than expected; any moments at which the respondent seems frustrated, confused, or uncomfortable; any questions that generate significant discussion; and the overall flow of modules and survey instrument.<br />
* Use the '''data-focused pilot''' to test [[Back Checks|back check]] templates. <br />
* Hire a local '''mobilizer''' to coordinate with respondents. '''Mobilizers''' explain the purpose behind conducting the survey, and facilitate the process of obtaining consent. This is particularly helpful in urban areas, or in cases where respondents are busy. This can improve the outcomes of piloting, for instance, by reducing gap between surveys.<br />
<br />
[https://www.worldbank.org/en/research/dime/data-and-analytics DIME Analytics] has also created the following checklists to assist researchers and enumerators in preparing for, and implementing a pilot:<br />
*[[Preparing_for_the_survey_checklist|Checklist: Preparing for a survey pilot]]<br />
*[[Checklist: Refine the Questionnaire (Content)|Checklist: Refining questionnaire content]]<br />
*[[Checklist: Refine the Questionnaire (Data)|Checklist: Refining questionnaire data]]<br />
<br />
== Related Pages ==<br />
[[Special:WhatLinksHere/Survey_Pilot|Click here for pages that link to this topic.]]<br />
<br />
== Additional Resources == <br />
* DIME Analytics (World Bank), [https://osf.io/63uv9/ Survey pilot guidelines]<br />
* DIME Analytics (World Bank), [https://osf.io/rqb5m/ Survey instrument design and pilot]<br />
<br />
[[Category: Primary Data Collection]] [[Category: Questionnaire Design]]</div>Maria ruth joneshttps://dimewiki.worldbank.org/index.php?title=Survey_Pilot&diff=7890Survey Pilot2021-01-16T20:22:14Z<p>Maria ruth jones: /* Structure */</p>
<hr />
<div>'''Survey pilot''' is the process of carrying out interviews and tests on different components of a survey, including [[Piloting Survey Content|content]] and [[Piloting Survey Protocols|protocols]]. A good '''pilot''' provides the research team with important feedback before they start the process of [[Primary Data Collection|primary data collection]]. This feedback can help the [[Impact Evaluation Team|research team]] review and improve the [[Questionnaire Design| design of the questionnaire]], [[Questionnaire Translation|translated questionnaires]], as well as [[Survey Protocols|survey protocols]] related to [[Piloting Survey Protocols#Interview scheduling|interview scheduling]], [[Sampling|sampling]], and [[Geo Spatial Data|geo data]].<br />
== Read First ==<br />
* A comprehensive '''survey pilot''' includes 3 stages: a pre-pilot, a content-focused pilot, and a data-focused pilot. <br />
* The '''pre-pilot''' is usually done during the process of [[Questionnaire Design|designing a questionnaire]], to refine the design, and wording of questions.<br />
* The [[Piloting Survey Content|content-focused pilot]] is conducted after [[Questionnaire Design|designing a questionnaire]], and sharing it with field and sector experts for their comments and inputs.<br />
* Finally, the '''data-focused pilot''' allows the [[Impact Evaluation Team|research team]] to review the data collected during the pilot, and identify possible errors in the [[Questionnaire Programming|programmed questionnaire]]. <br />
* A '''survey pilot''' is different from the '''field testing''' (practice interviews) that enumerators conduct at the end of [[Enumerator Training|enumerator training]]. The pilot should be complete before enumerator training begins. <br />
* [[Survey Pilot Participants|Participants of a survey pilot]] include the [[Impact Evaluation Team|research team]], '''interviewers''', and '''respondents'''.<br />
* Conduct the first two stages of the pilot before [[Questionnaire Programming|programming the questionnaire]]. Repeated programming is time-consuming and can create '''bugs''' (errors), for instance, by disturbing the order of questions.<br />
<br />
== Stages of a Survey Pilot ==<br />
A complete '''survey pilot''' is conducted over three stages - '''pre-pilot''', [[Piloting Survey Content|content-focused pilot]], and '''data-focused pilot'''. The table below discusses these stages in more detail.<br />
<br />
{| class="wikitable" style="margin-left: auto; margin-right: auto; border: none;"<br />
|-<br />
! style="height:50px; width:150px; text-align:center;" |<br />
! style="height:50px; width:250px; text-align:center;" | Stage 1: Pre-pilot<br />
! style="height:50px; width:250px; text-align:center;" | Stage 2: Content-focused pilot<br />
! style="height:50px; width:250px; text-align:center;" | Stage 3: Data-focused pilot<br />
|-<br />
| style="text-align:center;" | '''Objective'''<br />
| style="vertical-align:top;"| Answer broad questions about '''qualitative factors''' like [[Questionnaire Design|questionnaire design]] and the context for conducting the study. Discuss these with the concerned teams.<br />
<br />
Learn more about how people think to see what structure and flow makes sense.<br />
For example, do people think about their input use at plot-level? by crop? overall?<br />
<br />
It helps to think of the '''pre-pilot''' as a part of the process of [[Questionnaire Design|questionnaire design]]. For example, the '''pre-pilot''' can answer specific questions such as - important sources of income, relevant types of shocks, local food groups, and so on.<br />
<br />
In cases where the [[Impact Evaluation Team|research team]] is using a pre-existing questionnaire, they can skip this step. However, a '''pre-pilot''' is absolutely essential if the '''research team''' is designing a questionnaire from scratch, or if the questionnaire asks questions about issues that are difficult to measure.<br />
| style="vertical-align:top;"| Refine the order and wording of specific questions, the overall structure of the questionnaire, and [[Questionnaire Translation|translations]]. <br />
<br />
<br />
<br />
Check that the answer [[SurveyCTO Choice Lists|choices]] are '''comprehensive''', that is, they cover all possibilities. <br />
<br />
<br />
Flag any sensitive questions.<br />
<br />
<br />
Check how long it takes to answer the questions. Also check how answers differ between respondents ('''response variance''')<br />
| style="vertical-align:top;"| Check if the [[Questionnaire Programming|programmed instrument]] displays questions in the correct order, and follows the correct patterns (such as a group of questions which will always appear together, or will repeat).<br />
<br />
<br />
Load a sample data set (based on '''pilot interviews'''). Check if the data set has any missing fields. Perform all data quality checks, like [[Back Checks|back checks]] and [[Monitoring Data Quality#High Frequency Checks|high frequency checks]].<br />
|-<br />
| style="height:80px;text-align:center;" | '''Status of survey instrument'''<br />
| style="vertical-align:center;"| Early, printable version of the draft, and notes for further discussion.<br />
| style="vertical-align:center;"| A translated, printable draft. Ready to be '''programmed'''.<br />
| style="vertical-align:center;"| A translated, programmed, final draft. Ready for starting data collection.<br />
|-<br />
| style="height:80px;text-align:left;" | '''Mode'''<br />
| style="text-align:center;"| Pen-and-paper<br />
| style="text-align:center;"| Pen-and-paper<br />
| style="text-align:center;"| Electronic (phone/tablet)<br />
|}<br />
<br />
However, note that all three stages may not be necessary for every survey. The research team has to determine which stage to begin from on a case-to-case basis.<br />
* If the research team is using a brand new survey instrument, then they must start with Stage 1, the '''pre-pilot.'''<br />
* If the survey instrument is an adaptation of (based on) a different instrument which was used for a previous data collection for another project in the same region (province or country), the research team can start with Stage 2, the '''content-focused pilot'''. However, they must make sure that the survey instrument does not have any design issues and was shared by a reliable source.<br />
* If the survey instrument is an adaptation of an instrument used for a previous data collection for the same project, but the research team had to make significant revisions or additions, even then the research team should start with Stage 2, the '''content-focused pilot'''.<br />
* Only if the research team is '''piloting''' a '''follow-up''' survey, and there are no major changes from the '''baseline''' (or first round) survey, then in this case the research team may skip directly to Stage 3, the '''data-focused pilot'''.<br />
<br />
== Pilot on paper first == <br />
The first two stages of the survey pilot (pre-pilot and content-focused pilot) are best done on pen-and-paper is done on paper, regardless of the planned survey mode ([[Pen-and-Paper_Personal_Interviews_(PAPI)|pen-and-paper personal interviews (PAPI)]],<br />
[[Computer-Assisted_Personal_Interviews_(CAPI)|computer-assisted personal interviews (CAPI)]], [[Remote_Surveys|phone surveys (CATI)]]). A pen-and-paper pilot allows members of the research team and interviewers flexibility when it comes to recording answers and '''qualitative''' observations. <br />
<br />
Piloting on paper is particularly helpful for the following:<br />
* '''Open-ended responses.''' pre-pilot responses help the research team to make the list of choices for a question more detailed and comprehensive. Pen-and-paper pilots allow the interviewers to make note of these.<br />
* '''Restructuring.''' Pen-and-paper pilots allow the interviewers to draw lines and arrows between questions, which makes it easier to restructure the '''instrument'''.<br />
* '''Observations.''' Pen-and-paper pilots allow interviewers to record observations and feedback of interviewers in the margins. This allows the research team to take note of issues, such as the wording or flow of questions.<br />
<br />
== Timeline ==<br />
'''Piloting''' should start 4-6 weeks before [[Field Surveys#Survey Launch|survey launch]]. In fact, starting a pilot early allows the research team to use the feedback from each stage to improve the [[Piloting Survey Content|survey instrument]] and [[Piloting Survey Protocols| survey protocols]]. This makes the process of implementing a survey more efficient. Therefore, the research team must draft a detailed [[Timeline of Survey Pilot|timeline for the pilot]] to allow enough time for discussions and revisions. Do not confuse the pilot with '''field testing''' (practice interviews) which is conducted at the end of [[Enumerator_Training |enumerator training]].<br />
<br />
== Structure ==<br />
The process of '''structuring a survey pilot''' involves agreeing upon the '''logistics''' (or practical aspects) of a [[Survey Pilot|survey pilot]], including '''duration''', '''approvals''', '''review sessions''', and '''training.''' In order to conduct a successful pilot, the [[Impact Evaluation Team|impact evaluation team]] (or research team) must discuss each of these aspects in detail to ensure that the '''survey pilot''' is conducted smoothly. A good pilot helps to improve the quality of the [[Primary Data Collection|data collection process]] and the [[Survey Protocols|survey protocols]]. Go to [[Structuring a Survey Pilot|Structuring a Survey Pilot]] for a detailed discussion of recommended protocols.<br />
<br />
== Participants ==<br />
Typically, a '''survey pilot''' is conducted before a [[Survey Firm|survey firm]] is brought on-board. Each of the [[Survey Pilot Participants|survey pilot participants]] plays an important role in the implementation of a successful pilot. <br />
* '''Respondents.''' They should be as close to the '''target population''' as possible. None of the respondents from a pilot should be in the sample for the actual survey.<br />
* '''Interviewers.''' They should be fluent in all local languages of the study area and the language of the research team. The size of the team of interviewers depends on the stage of pilot.<br />
* '''Field coordinator (FC).''' The [[Impact_Evaluation_Team#Field_Coordinator|field coordinators (FCs)]] play a central role as part of the research team. They are responsible for updating the research team on all monitoring and evaluation '''(M&E)''' activities in the field.<br />
* '''Principal investigator (PI).''' The [[Impact_Evaluation_Team#Principal_Investigator|principal investigators (PIs)]] oversee the overall functioning of the '''impact evaluation''' (study). They should brief the FCs regularly during the pilot.<br />
* '''Other research team members.'''Ideally, other members of the [[Impact Evaluation Team|impact evaluation team]] such as the [[Impact Evaluation Team#Impact Evaluation Coordinator|impact evaluation coordinator]], research manager, programmers, and translators should also participate.<br />
<br />
== Guidelines ==<br />
Finally, it is important to keep the following '''best practices''' in mind while planning and conducting a pilot:<br />
* Throughout the process of [[Questionnaire Design|designing the questionnaire]], take notes of what needs to be explored or tested during the pilot.<br />
* Encourage interviewers to probe and follow-up much more than they would in a typical interview<br />
* Encourage respondents to think out-loud, to understand how they are coming up with their answer, and ask enumerators to record these notes.<br />
* In some cultures, it will be appropriate and useful to ask respondents their feedback at the end of the interview<br />
* Take careful notes of the discussions and clarifications that come up during the pilot. These will be an important part of the [[Enumerator Training#Enumerator Manual|enumerator manual]] which is used during [[Enumerator Training|enumerator training]].<br />
* It is useful for research team members to sit in on pilot interviews even if they do not speak the local language. Take note of: questions that take longer than expected; any moments at which the respondent seems frustrated, confused, or uncomfortable; any questions that generate significant discussion; and the overall flow of modules and survey instrument.<br />
* Use the '''data-focused pilot''' to test [[Back Checks|back check]] templates. <br />
* Hire a local '''mobilizer''' to coordinate with respondents. '''Mobilizers''' explain the purpose behind conducting the survey, and facilitate the process of obtaining consent. This is particularly helpful in urban areas, or in cases where respondents are busy. This can improve the outcomes of piloting, for instance, by reducing gap between surveys.<br />
<br />
[https://www.worldbank.org/en/research/dime/data-and-analytics DIME Analytics] has also created the following checklists to assist researchers and enumerators in preparing for, and implementing a pilot:<br />
*[[Preparing_for_the_survey_checklist|Checklist: Preparing for a survey pilot]]<br />
*[[Checklist: Refine the Questionnaire (Content)|Checklist: Refining questionnaire content]]<br />
*[[Checklist: Refine the Questionnaire (Data)|Checklist: Refining questionnaire data]]<br />
<br />
== Related Pages ==<br />
[[Special:WhatLinksHere/Survey_Pilot|Click here for pages that link to this topic.]]<br />
<br />
== Additional Resources == <br />
* DIME Analytics (World Bank), [https://osf.io/63uv9/ Survey pilot guidelines]<br />
* DIME Analytics (World Bank), [https://osf.io/rqb5m/ Survey instrument design and pilot]<br />
<br />
[[Category: Primary Data Collection]] [[Category: Questionnaire Design]]</div>Maria ruth joneshttps://dimewiki.worldbank.org/index.php?title=Survey_Pilot&diff=7889Survey Pilot2021-01-16T20:18:26Z<p>Maria ruth jones: /* Guidelines */</p>
<hr />
<div>'''Survey pilot''' is the process of carrying out interviews and tests on different components of a survey, including [[Piloting Survey Content|content]] and [[Piloting Survey Protocols|protocols]]. A good '''pilot''' provides the research team with important feedback before they start the process of [[Primary Data Collection|primary data collection]]. This feedback can help the [[Impact Evaluation Team|research team]] review and improve the [[Questionnaire Design| design of the questionnaire]], [[Questionnaire Translation|translated questionnaires]], as well as [[Survey Protocols|survey protocols]] related to [[Piloting Survey Protocols#Interview scheduling|interview scheduling]], [[Sampling|sampling]], and [[Geo Spatial Data|geo data]].<br />
== Read First ==<br />
* A comprehensive '''survey pilot''' includes 3 stages: a pre-pilot, a content-focused pilot, and a data-focused pilot. <br />
* The '''pre-pilot''' is usually done during the process of [[Questionnaire Design|designing a questionnaire]], to refine the design, and wording of questions.<br />
* The [[Piloting Survey Content|content-focused pilot]] is conducted after [[Questionnaire Design|designing a questionnaire]], and sharing it with field and sector experts for their comments and inputs.<br />
* Finally, the '''data-focused pilot''' allows the [[Impact Evaluation Team|research team]] to review the data collected during the pilot, and identify possible errors in the [[Questionnaire Programming|programmed questionnaire]]. <br />
* A '''survey pilot''' is different from the '''field testing''' (practice interviews) that enumerators conduct at the end of [[Enumerator Training|enumerator training]]. The pilot should be complete before enumerator training begins. <br />
* [[Survey Pilot Participants|Participants of a survey pilot]] include the [[Impact Evaluation Team|research team]], '''interviewers''', and '''respondents'''.<br />
* Conduct the first two stages of the pilot before [[Questionnaire Programming|programming the questionnaire]]. Repeated programming is time-consuming and can create '''bugs''' (errors), for instance, by disturbing the order of questions.<br />
<br />
== Stages of a Survey Pilot ==<br />
A complete '''survey pilot''' is conducted over three stages - '''pre-pilot''', [[Piloting Survey Content|content-focused pilot]], and '''data-focused pilot'''. The table below discusses these stages in more detail.<br />
<br />
{| class="wikitable" style="margin-left: auto; margin-right: auto; border: none;"<br />
|-<br />
! style="height:50px; width:150px; text-align:center;" |<br />
! style="height:50px; width:250px; text-align:center;" | Stage 1: Pre-pilot<br />
! style="height:50px; width:250px; text-align:center;" | Stage 2: Content-focused pilot<br />
! style="height:50px; width:250px; text-align:center;" | Stage 3: Data-focused pilot<br />
|-<br />
| style="text-align:center;" | '''Objective'''<br />
| style="vertical-align:top;"| Answer broad questions about '''qualitative factors''' like [[Questionnaire Design|questionnaire design]] and the context for conducting the study. Discuss these with the concerned teams.<br />
<br />
Learn more about how people think to see what structure and flow makes sense.<br />
For example, do people think about their input use at plot-level? by crop? overall?<br />
<br />
It helps to think of the '''pre-pilot''' as a part of the process of [[Questionnaire Design|questionnaire design]]. For example, the '''pre-pilot''' can answer specific questions such as - important sources of income, relevant types of shocks, local food groups, and so on.<br />
<br />
In cases where the [[Impact Evaluation Team|research team]] is using a pre-existing questionnaire, they can skip this step. However, a '''pre-pilot''' is absolutely essential if the '''research team''' is designing a questionnaire from scratch, or if the questionnaire asks questions about issues that are difficult to measure.<br />
| style="vertical-align:top;"| Refine the order and wording of specific questions, the overall structure of the questionnaire, and [[Questionnaire Translation|translations]]. <br />
<br />
<br />
<br />
Check that the answer [[SurveyCTO Choice Lists|choices]] are '''comprehensive''', that is, they cover all possibilities. <br />
<br />
<br />
Flag any sensitive questions.<br />
<br />
<br />
Check how long it takes to answer the questions. Also check how answers differ between respondents ('''response variance''')<br />
| style="vertical-align:top;"| Check if the [[Questionnaire Programming|programmed instrument]] displays questions in the correct order, and follows the correct patterns (such as a group of questions which will always appear together, or will repeat).<br />
<br />
<br />
Load a sample data set (based on '''pilot interviews'''). Check if the data set has any missing fields. Perform all data quality checks, like [[Back Checks|back checks]] and [[Monitoring Data Quality#High Frequency Checks|high frequency checks]].<br />
|-<br />
| style="height:80px;text-align:center;" | '''Status of survey instrument'''<br />
| style="vertical-align:center;"| Early, printable version of the draft, and notes for further discussion.<br />
| style="vertical-align:center;"| A translated, printable draft. Ready to be '''programmed'''.<br />
| style="vertical-align:center;"| A translated, programmed, final draft. Ready for starting data collection.<br />
|-<br />
| style="height:80px;text-align:left;" | '''Mode'''<br />
| style="text-align:center;"| Pen-and-paper<br />
| style="text-align:center;"| Pen-and-paper<br />
| style="text-align:center;"| Electronic (phone/tablet)<br />
|}<br />
<br />
However, note that all three stages may not be necessary for every survey. The research team has to determine which stage to begin from on a case-to-case basis.<br />
* If the research team is using a brand new survey instrument, then they must start with Stage 1, the '''pre-pilot.'''<br />
* If the survey instrument is an adaptation of (based on) a different instrument which was used for a previous data collection for another project in the same region (province or country), the research team can start with Stage 2, the '''content-focused pilot'''. However, they must make sure that the survey instrument does not have any design issues and was shared by a reliable source.<br />
* If the survey instrument is an adaptation of an instrument used for a previous data collection for the same project, but the research team had to make significant revisions or additions, even then the research team should start with Stage 2, the '''content-focused pilot'''.<br />
* Only if the research team is '''piloting''' a '''follow-up''' survey, and there are no major changes from the '''baseline''' (or first round) survey, then in this case the research team may skip directly to Stage 3, the '''data-focused pilot'''.<br />
<br />
== Pilot on paper first == <br />
The first two stages of the survey pilot (pre-pilot and content-focused pilot) are best done on pen-and-paper is done on paper, regardless of the planned survey mode ([[Pen-and-Paper_Personal_Interviews_(PAPI)|pen-and-paper personal interviews (PAPI)]],<br />
[[Computer-Assisted_Personal_Interviews_(CAPI)|computer-assisted personal interviews (CAPI)]], [[Remote_Surveys|phone surveys (CATI)]]). A pen-and-paper pilot allows members of the research team and interviewers flexibility when it comes to recording answers and '''qualitative''' observations. <br />
<br />
Piloting on paper is particularly helpful for the following:<br />
* '''Open-ended responses.''' pre-pilot responses help the research team to make the list of choices for a question more detailed and comprehensive. Pen-and-paper pilots allow the interviewers to make note of these.<br />
* '''Restructuring.''' Pen-and-paper pilots allow the interviewers to draw lines and arrows between questions, which makes it easier to restructure the '''instrument'''.<br />
* '''Observations.''' Pen-and-paper pilots allow interviewers to record observations and feedback of interviewers in the margins. This allows the research team to take note of issues, such as the wording or flow of questions.<br />
<br />
== Timeline ==<br />
'''Piloting''' should start 4-6 weeks before [[Field Surveys#Survey Launch|survey launch]]. In fact, starting a pilot early allows the research team to use the feedback from each stage to improve the [[Piloting Survey Content|survey instrument]] and [[Piloting Survey Protocols| survey protocols]]. This makes the process of implementing a survey more efficient. Therefore, the research team must draft a detailed [[Timeline of Survey Pilot|timeline for the pilot]] to allow enough time for discussions and revisions. Do not confuse the pilot with '''field testing''' (practice interviews) which is conducted at the end of [[Enumerator_Training |enumerator training]].<br />
<br />
== Structure ==<br />
In order to conduct a successful '''survey pilot''', the research team must decide on the [[Structuring a Survey Pilot|structure of the survey pilot]]. This involves agreeing upon the '''logistics''' for the following aspects of the pilot:<br />
* '''Duration of the pilot.''' The pilot should ideally go on until the research team agrees that there are no changes left to be made.<br />
* '''Approvals.''' Obtain all necessary approvals, for example, [[Survey Budget|budget]] approval from the organization funding the study. Sometimes the research team must obtain an [[IRB Approval|IRB approval]] even for a pilot. <br />
* '''Review sessions.''' Set aside time for feedback sessions and revisions. The research team should make sure everyone involved in the '''survey pilot''' gets a chance to express their concerns. Therefore, it is a good practice to draft guidelines on the schedule and frequency of review sessions.<br />
* '''Training.''' As part of [[Enumerator Training|enumerator training]], set aside time specifically for discussing best practices for conducting a '''survey pilot'''.<br />
<br />
== Participants ==<br />
Typically, a '''survey pilot''' is conducted before a [[Survey Firm|survey firm]] is brought on-board. Each of the [[Survey Pilot Participants|survey pilot participants]] plays an important role in the implementation of a successful pilot. <br />
* '''Respondents.''' They should be as close to the '''target population''' as possible. None of the respondents from a pilot should be in the sample for the actual survey.<br />
* '''Interviewers.''' They should be fluent in all local languages of the study area and the language of the research team. The size of the team of interviewers depends on the stage of pilot.<br />
* '''Field coordinator (FC).''' The [[Impact_Evaluation_Team#Field_Coordinator|field coordinators (FCs)]] play a central role as part of the research team. They are responsible for updating the research team on all monitoring and evaluation '''(M&E)''' activities in the field.<br />
* '''Principal investigator (PI).''' The [[Impact_Evaluation_Team#Principal_Investigator|principal investigators (PIs)]] oversee the overall functioning of the '''impact evaluation''' (study). They should brief the FCs regularly during the pilot.<br />
* '''Other research team members.'''Ideally, other members of the [[Impact Evaluation Team|impact evaluation team]] such as the [[Impact Evaluation Team#Impact Evaluation Coordinator|impact evaluation coordinator]], research manager, programmers, and translators should also participate.<br />
<br />
== Guidelines ==<br />
Finally, it is important to keep the following '''best practices''' in mind while planning and conducting a pilot:<br />
* Throughout the process of [[Questionnaire Design|designing the questionnaire]], take notes of what needs to be explored or tested during the pilot.<br />
* Encourage interviewers to probe and follow-up much more than they would in a typical interview<br />
* Encourage respondents to think out-loud, to understand how they are coming up with their answer, and ask enumerators to record these notes.<br />
* In some cultures, it will be appropriate and useful to ask respondents their feedback at the end of the interview<br />
* Take careful notes of the discussions and clarifications that come up during the pilot. These will be an important part of the [[Enumerator Training#Enumerator Manual|enumerator manual]] which is used during [[Enumerator Training|enumerator training]].<br />
* It is useful for research team members to sit in on pilot interviews even if they do not speak the local language. Take note of: questions that take longer than expected; any moments at which the respondent seems frustrated, confused, or uncomfortable; any questions that generate significant discussion; and the overall flow of modules and survey instrument.<br />
* Use the '''data-focused pilot''' to test [[Back Checks|back check]] templates. <br />
* Hire a local '''mobilizer''' to coordinate with respondents. '''Mobilizers''' explain the purpose behind conducting the survey, and facilitate the process of obtaining consent. This is particularly helpful in urban areas, or in cases where respondents are busy. This can improve the outcomes of piloting, for instance, by reducing gap between surveys.<br />
<br />
[https://www.worldbank.org/en/research/dime/data-and-analytics DIME Analytics] has also created the following checklists to assist researchers and enumerators in preparing for, and implementing a pilot:<br />
*[[Preparing_for_the_survey_checklist|Checklist: Preparing for a survey pilot]]<br />
*[[Checklist: Refine the Questionnaire (Content)|Checklist: Refining questionnaire content]]<br />
*[[Checklist: Refine the Questionnaire (Data)|Checklist: Refining questionnaire data]]<br />
<br />
== Related Pages ==<br />
[[Special:WhatLinksHere/Survey_Pilot|Click here for pages that link to this topic.]]<br />
<br />
== Additional Resources == <br />
* DIME Analytics (World Bank), [https://osf.io/63uv9/ Survey pilot guidelines]<br />
* DIME Analytics (World Bank), [https://osf.io/rqb5m/ Survey instrument design and pilot]<br />
<br />
[[Category: Primary Data Collection]] [[Category: Questionnaire Design]]</div>Maria ruth joneshttps://dimewiki.worldbank.org/index.php?title=Survey_Pilot&diff=7888Survey Pilot2021-01-16T20:09:57Z<p>Maria ruth jones: /* Guidelines */</p>
<hr />
<div>'''Survey pilot''' is the process of carrying out interviews and tests on different components of a survey, including [[Piloting Survey Content|content]] and [[Piloting Survey Protocols|protocols]]. A good '''pilot''' provides the research team with important feedback before they start the process of [[Primary Data Collection|primary data collection]]. This feedback can help the [[Impact Evaluation Team|research team]] review and improve the [[Questionnaire Design| design of the questionnaire]], [[Questionnaire Translation|translated questionnaires]], as well as [[Survey Protocols|survey protocols]] related to [[Piloting Survey Protocols#Interview scheduling|interview scheduling]], [[Sampling|sampling]], and [[Geo Spatial Data|geo data]].<br />
== Read First ==<br />
* A comprehensive '''survey pilot''' includes 3 stages: a pre-pilot, a content-focused pilot, and a data-focused pilot. <br />
* The '''pre-pilot''' is usually done during the process of [[Questionnaire Design|designing a questionnaire]], to refine the design, and wording of questions.<br />
* The [[Piloting Survey Content|content-focused pilot]] is conducted after [[Questionnaire Design|designing a questionnaire]], and sharing it with field and sector experts for their comments and inputs.<br />
* Finally, the '''data-focused pilot''' allows the [[Impact Evaluation Team|research team]] to review the data collected during the pilot, and identify possible errors in the [[Questionnaire Programming|programmed questionnaire]]. <br />
* A '''survey pilot''' is different from the '''field testing''' (practice interviews) that enumerators conduct at the end of [[Enumerator Training|enumerator training]]. The pilot should be complete before enumerator training begins. <br />
* [[Survey Pilot Participants|Participants of a survey pilot]] include the [[Impact Evaluation Team|research team]], '''interviewers''', and '''respondents'''.<br />
* Conduct the first two stages of the pilot before [[Questionnaire Programming|programming the questionnaire]]. Repeated programming is time-consuming and can create '''bugs''' (errors), for instance, by disturbing the order of questions.<br />
<br />
== Stages of a Survey Pilot ==<br />
A complete '''survey pilot''' is conducted over three stages - '''pre-pilot''', [[Piloting Survey Content|content-focused pilot]], and '''data-focused pilot'''. The table below discusses these stages in more detail.<br />
<br />
{| class="wikitable" style="margin-left: auto; margin-right: auto; border: none;"<br />
|-<br />
! style="height:50px; width:150px; text-align:center;" |<br />
! style="height:50px; width:250px; text-align:center;" | Stage 1: Pre-pilot<br />
! style="height:50px; width:250px; text-align:center;" | Stage 2: Content-focused pilot<br />
! style="height:50px; width:250px; text-align:center;" | Stage 3: Data-focused pilot<br />
|-<br />
| style="text-align:center;" | '''Objective'''<br />
| style="vertical-align:top;"| Answer broad questions about '''qualitative factors''' like [[Questionnaire Design|questionnaire design]] and the context for conducting the study. Discuss these with the concerned teams.<br />
<br />
Learn more about how people think to see what structure and flow makes sense.<br />
For example, do people think about their input use at plot-level? by crop? overall?<br />
<br />
It helps to think of the '''pre-pilot''' as a part of the process of [[Questionnaire Design|questionnaire design]]. For example, the '''pre-pilot''' can answer specific questions such as - important sources of income, relevant types of shocks, local food groups, and so on.<br />
<br />
In cases where the [[Impact Evaluation Team|research team]] is using a pre-existing questionnaire, they can skip this step. However, a '''pre-pilot''' is absolutely essential if the '''research team''' is designing a questionnaire from scratch, or if the questionnaire asks questions about issues that are difficult to measure.<br />
| style="vertical-align:top;"| Refine the order and wording of specific questions, the overall structure of the questionnaire, and [[Questionnaire Translation|translations]]. <br />
<br />
<br />
<br />
Check that the answer [[SurveyCTO Choice Lists|choices]] are '''comprehensive''', that is, they cover all possibilities. <br />
<br />
<br />
Flag any sensitive questions.<br />
<br />
<br />
Check how long it takes to answer the questions. Also check how answers differ between respondents ('''response variance''')<br />
| style="vertical-align:top;"| Check if the [[Questionnaire Programming|programmed instrument]] displays questions in the correct order, and follows the correct patterns (such as a group of questions which will always appear together, or will repeat).<br />
<br />
<br />
Load a sample data set (based on '''pilot interviews'''). Check if the data set has any missing fields. Perform all data quality checks, like [[Back Checks|back checks]] and [[Monitoring Data Quality#High Frequency Checks|high frequency checks]].<br />
|-<br />
| style="height:80px;text-align:center;" | '''Status of survey instrument'''<br />
| style="vertical-align:center;"| Early, printable version of the draft, and notes for further discussion.<br />
| style="vertical-align:center;"| A translated, printable draft. Ready to be '''programmed'''.<br />
| style="vertical-align:center;"| A translated, programmed, final draft. Ready for starting data collection.<br />
|-<br />
| style="height:80px;text-align:left;" | '''Mode'''<br />
| style="text-align:center;"| Pen-and-paper<br />
| style="text-align:center;"| Pen-and-paper<br />
| style="text-align:center;"| Electronic (phone/tablet)<br />
|}<br />
<br />
However, note that all three stages may not be necessary for every survey. The research team has to determine which stage to begin from on a case-to-case basis.<br />
* If the research team is using a brand new survey instrument, then they must start with Stage 1, the '''pre-pilot.'''<br />
* If the survey instrument is an adaptation of (based on) a different instrument which was used for a previous data collection for another project in the same region (province or country), the research team can start with Stage 2, the '''content-focused pilot'''. However, they must make sure that the survey instrument does not have any design issues and was shared by a reliable source.<br />
* If the survey instrument is an adaptation of an instrument used for a previous data collection for the same project, but the research team had to make significant revisions or additions, even then the research team should start with Stage 2, the '''content-focused pilot'''.<br />
* Only if the research team is '''piloting''' a '''follow-up''' survey, and there are no major changes from the '''baseline''' (or first round) survey, then in this case the research team may skip directly to Stage 3, the '''data-focused pilot'''.<br />
<br />
== Pilot on paper first == <br />
The first two stages of the survey pilot (pre-pilot and content-focused pilot) are best done on pen-and-paper is done on paper, regardless of the planned survey mode ([[Pen-and-Paper_Personal_Interviews_(PAPI)|pen-and-paper personal interviews (PAPI)]],<br />
[[Computer-Assisted_Personal_Interviews_(CAPI)|computer-assisted personal interviews (CAPI)]], [[Remote_Surveys|phone surveys (CATI)]]). A pen-and-paper pilot allows members of the research team and interviewers flexibility when it comes to recording answers and '''qualitative''' observations. <br />
<br />
Piloting on paper is particularly helpful for the following:<br />
* '''Open-ended responses.''' pre-pilot responses help the research team to make the list of choices for a question more detailed and comprehensive. Pen-and-paper pilots allow the interviewers to make note of these.<br />
* '''Restructuring.''' Pen-and-paper pilots allow the interviewers to draw lines and arrows between questions, which makes it easier to restructure the '''instrument'''.<br />
* '''Observations.''' Pen-and-paper pilots allow interviewers to record observations and feedback of interviewers in the margins. This allows the research team to take note of issues, such as the wording or flow of questions.<br />
<br />
== Timeline ==<br />
'''Piloting''' should start 4-6 weeks before [[Field Surveys#Survey Launch|survey launch]]. In fact, starting a pilot early allows the research team to use the feedback from each stage to improve the [[Piloting Survey Content|survey instrument]] and [[Piloting Survey Protocols| survey protocols]]. This makes the process of implementing a survey more efficient. Therefore, the research team must draft a detailed [[Timeline of Survey Pilot|timeline for the pilot]] to allow enough time for discussions and revisions. Do not confuse the pilot with '''field testing''' (practice interviews) which is conducted at the end of [[Enumerator_Training |enumerator training]].<br />
<br />
== Structure ==<br />
In order to conduct a successful '''survey pilot''', the research team must decide on the [[Structuring a Survey Pilot|structure of the survey pilot]]. This involves agreeing upon the '''logistics''' for the following aspects of the pilot:<br />
* '''Duration of the pilot.''' The pilot should ideally go on until the research team agrees that there are no changes left to be made.<br />
* '''Approvals.''' Obtain all necessary approvals, for example, [[Survey Budget|budget]] approval from the organization funding the study. Sometimes the research team must obtain an [[IRB Approval|IRB approval]] even for a pilot. <br />
* '''Review sessions.''' Set aside time for feedback sessions and revisions. The research team should make sure everyone involved in the '''survey pilot''' gets a chance to express their concerns. Therefore, it is a good practice to draft guidelines on the schedule and frequency of review sessions.<br />
* '''Training.''' As part of [[Enumerator Training|enumerator training]], set aside time specifically for discussing best practices for conducting a '''survey pilot'''.<br />
<br />
== Participants ==<br />
Typically, a '''survey pilot''' is conducted before a [[Survey Firm|survey firm]] is brought on-board. Each of the [[Survey Pilot Participants|survey pilot participants]] plays an important role in the implementation of a successful pilot. <br />
* '''Respondents.''' They should be as close to the '''target population''' as possible. None of the respondents from a pilot should be in the sample for the actual survey.<br />
* '''Interviewers.''' They should be fluent in all local languages of the study area and the language of the research team. The size of the team of interviewers depends on the stage of pilot.<br />
* '''Field coordinator (FC).''' The [[Impact_Evaluation_Team#Field_Coordinator|field coordinators (FCs)]] play a central role as part of the research team. They are responsible for updating the research team on all monitoring and evaluation '''(M&E)''' activities in the field.<br />
* '''Principal investigator (PI).''' The [[Impact_Evaluation_Team#Principal_Investigator|principal investigators (PIs)]] oversee the overall functioning of the '''impact evaluation''' (study). They should brief the FCs regularly during the pilot.<br />
* '''Other research team members.'''Ideally, other members of the [[Impact Evaluation Team|impact evaluation team]] such as the [[Impact Evaluation Team#Impact Evaluation Coordinator|impact evaluation coordinator]], research manager, programmers, and translators should also participate.<br />
<br />
== Guidelines ==<br />
Finally, it is important to keep the following '''best practices''' in mind while planning and conducting a pilot:<br />
* Throughout the process of [[Questionnaire Design|designing the questionnaire]], discuss with other members of the research team, and take notes of what needs to be part of the pilot.<br />
* Encourage interviewers to probe and follow-up much more than they would in a typical interview<br />
* Encourage respondents to think out-loud, to understand how they are coming up with their answer, and ask enumerators to record these notes.<br />
* In some cultures, it will be appropriate and useful to ask respondents their feedback at the end of the interview<br />
* Take careful notes of the discussions and clarifications that come up during the pilot. These will be an important part of the [[Enumerator Training#Enumerator Manual|enumerator manual]] which is used during [[Enumerator Training|enumerator training]].<br />
* Use the '''data-focused pilot''' to test [[Back Checks|back check]] templates. <br />
* Hire a local '''mobilizer''' to coordinate with respondents. '''Mobilizers''' explain the purpose behind conducting the survey, and facilitate the process of obtaining consent. This is particularly helpful in urban areas, or in cases where respondents are busy. This can improve the outcomes of piloting, for instance, by reducing gap between surveys.<br />
<br />
[https://www.worldbank.org/en/research/dime/data-and-analytics DIME Analytics] has also created the following checklists to assist researchers and enumerators in preparing for, and implementing a pilot:<br />
*[[Preparing_for_the_survey_checklist|Checklist: Preparing for a survey pilot]]<br />
*[[Checklist: Refine the Questionnaire (Content)|Checklist: Refining questionnaire content]]<br />
*[[Checklist: Refine the Questionnaire (Data)|Checklist: Refining questionnaire data]]<br />
<br />
== Related Pages ==<br />
[[Special:WhatLinksHere/Survey_Pilot|Click here for pages that link to this topic.]]<br />
<br />
== Additional Resources == <br />
* DIME Analytics (World Bank), [https://osf.io/63uv9/ Survey pilot guidelines]<br />
* DIME Analytics (World Bank), [https://osf.io/rqb5m/ Survey instrument design and pilot]<br />
<br />
[[Category: Primary Data Collection]] [[Category: Questionnaire Design]]</div>Maria ruth joneshttps://dimewiki.worldbank.org/index.php?title=Survey_Pilot&diff=7887Survey Pilot2021-01-16T20:07:53Z<p>Maria ruth jones: /* Stages of a Survey Pilot */</p>
<hr />
<div>'''Survey pilot''' is the process of carrying out interviews and tests on different components of a survey, including [[Piloting Survey Content|content]] and [[Piloting Survey Protocols|protocols]]. A good '''pilot''' provides the research team with important feedback before they start the process of [[Primary Data Collection|primary data collection]]. This feedback can help the [[Impact Evaluation Team|research team]] review and improve the [[Questionnaire Design| design of the questionnaire]], [[Questionnaire Translation|translated questionnaires]], as well as [[Survey Protocols|survey protocols]] related to [[Piloting Survey Protocols#Interview scheduling|interview scheduling]], [[Sampling|sampling]], and [[Geo Spatial Data|geo data]].<br />
== Read First ==<br />
* A comprehensive '''survey pilot''' includes 3 stages: a pre-pilot, a content-focused pilot, and a data-focused pilot. <br />
* The '''pre-pilot''' is usually done during the process of [[Questionnaire Design|designing a questionnaire]], to refine the design, and wording of questions.<br />
* The [[Piloting Survey Content|content-focused pilot]] is conducted after [[Questionnaire Design|designing a questionnaire]], and sharing it with field and sector experts for their comments and inputs.<br />
* Finally, the '''data-focused pilot''' allows the [[Impact Evaluation Team|research team]] to review the data collected during the pilot, and identify possible errors in the [[Questionnaire Programming|programmed questionnaire]]. <br />
* A '''survey pilot''' is different from the '''field testing''' (practice interviews) that enumerators conduct at the end of [[Enumerator Training|enumerator training]]. The pilot should be complete before enumerator training begins. <br />
* [[Survey Pilot Participants|Participants of a survey pilot]] include the [[Impact Evaluation Team|research team]], '''interviewers''', and '''respondents'''.<br />
* Conduct the first two stages of the pilot before [[Questionnaire Programming|programming the questionnaire]]. Repeated programming is time-consuming and can create '''bugs''' (errors), for instance, by disturbing the order of questions.<br />
<br />
== Stages of a Survey Pilot ==<br />
A complete '''survey pilot''' is conducted over three stages - '''pre-pilot''', [[Piloting Survey Content|content-focused pilot]], and '''data-focused pilot'''. The table below discusses these stages in more detail.<br />
<br />
{| class="wikitable" style="margin-left: auto; margin-right: auto; border: none;"<br />
|-<br />
! style="height:50px; width:150px; text-align:center;" |<br />
! style="height:50px; width:250px; text-align:center;" | Stage 1: Pre-pilot<br />
! style="height:50px; width:250px; text-align:center;" | Stage 2: Content-focused pilot<br />
! style="height:50px; width:250px; text-align:center;" | Stage 3: Data-focused pilot<br />
|-<br />
| style="text-align:center;" | '''Objective'''<br />
| style="vertical-align:top;"| Answer broad questions about '''qualitative factors''' like [[Questionnaire Design|questionnaire design]] and the context for conducting the study. Discuss these with the concerned teams.<br />
<br />
Learn more about how people think to see what structure and flow makes sense.<br />
For example, do people think about their input use at plot-level? by crop? overall?<br />
<br />
It helps to think of the '''pre-pilot''' as a part of the process of [[Questionnaire Design|questionnaire design]]. For example, the '''pre-pilot''' can answer specific questions such as - important sources of income, relevant types of shocks, local food groups, and so on.<br />
<br />
In cases where the [[Impact Evaluation Team|research team]] is using a pre-existing questionnaire, they can skip this step. However, a '''pre-pilot''' is absolutely essential if the '''research team''' is designing a questionnaire from scratch, or if the questionnaire asks questions about issues that are difficult to measure.<br />
| style="vertical-align:top;"| Refine the order and wording of specific questions, the overall structure of the questionnaire, and [[Questionnaire Translation|translations]]. <br />
<br />
<br />
<br />
Check that the answer [[SurveyCTO Choice Lists|choices]] are '''comprehensive''', that is, they cover all possibilities. <br />
<br />
<br />
Flag any sensitive questions.<br />
<br />
<br />
Check how long it takes to answer the questions. Also check how answers differ between respondents ('''response variance''')<br />
| style="vertical-align:top;"| Check if the [[Questionnaire Programming|programmed instrument]] displays questions in the correct order, and follows the correct patterns (such as a group of questions which will always appear together, or will repeat).<br />
<br />
<br />
Load a sample data set (based on '''pilot interviews'''). Check if the data set has any missing fields. Perform all data quality checks, like [[Back Checks|back checks]] and [[Monitoring Data Quality#High Frequency Checks|high frequency checks]].<br />
|-<br />
| style="height:80px;text-align:center;" | '''Status of survey instrument'''<br />
| style="vertical-align:center;"| Early, printable version of the draft, and notes for further discussion.<br />
| style="vertical-align:center;"| A translated, printable draft. Ready to be '''programmed'''.<br />
| style="vertical-align:center;"| A translated, programmed, final draft. Ready for starting data collection.<br />
|-<br />
| style="height:80px;text-align:left;" | '''Mode'''<br />
| style="text-align:center;"| Pen-and-paper<br />
| style="text-align:center;"| Pen-and-paper<br />
| style="text-align:center;"| Electronic (phone/tablet)<br />
|}<br />
<br />
However, note that all three stages may not be necessary for every survey. The research team has to determine which stage to begin from on a case-to-case basis.<br />
* If the research team is using a brand new survey instrument, then they must start with Stage 1, the '''pre-pilot.'''<br />
* If the survey instrument is an adaptation of (based on) a different instrument which was used for a previous data collection for another project in the same region (province or country), the research team can start with Stage 2, the '''content-focused pilot'''. However, they must make sure that the survey instrument does not have any design issues and was shared by a reliable source.<br />
* If the survey instrument is an adaptation of an instrument used for a previous data collection for the same project, but the research team had to make significant revisions or additions, even then the research team should start with Stage 2, the '''content-focused pilot'''.<br />
* Only if the research team is '''piloting''' a '''follow-up''' survey, and there are no major changes from the '''baseline''' (or first round) survey, then in this case the research team may skip directly to Stage 3, the '''data-focused pilot'''.<br />
<br />
== Pilot on paper first == <br />
The first two stages of the survey pilot (pre-pilot and content-focused pilot) are best done on pen-and-paper is done on paper, regardless of the planned survey mode ([[Pen-and-Paper_Personal_Interviews_(PAPI)|pen-and-paper personal interviews (PAPI)]],<br />
[[Computer-Assisted_Personal_Interviews_(CAPI)|computer-assisted personal interviews (CAPI)]], [[Remote_Surveys|phone surveys (CATI)]]). A pen-and-paper pilot allows members of the research team and interviewers flexibility when it comes to recording answers and '''qualitative''' observations. <br />
<br />
Piloting on paper is particularly helpful for the following:<br />
* '''Open-ended responses.''' pre-pilot responses help the research team to make the list of choices for a question more detailed and comprehensive. Pen-and-paper pilots allow the interviewers to make note of these.<br />
* '''Restructuring.''' Pen-and-paper pilots allow the interviewers to draw lines and arrows between questions, which makes it easier to restructure the '''instrument'''.<br />
* '''Observations.''' Pen-and-paper pilots allow interviewers to record observations and feedback of interviewers in the margins. This allows the research team to take note of issues, such as the wording or flow of questions.<br />
<br />
== Timeline ==<br />
'''Piloting''' should start 4-6 weeks before [[Field Surveys#Survey Launch|survey launch]]. In fact, starting a pilot early allows the research team to use the feedback from each stage to improve the [[Piloting Survey Content|survey instrument]] and [[Piloting Survey Protocols| survey protocols]]. This makes the process of implementing a survey more efficient. Therefore, the research team must draft a detailed [[Timeline of Survey Pilot|timeline for the pilot]] to allow enough time for discussions and revisions. Do not confuse the pilot with '''field testing''' (practice interviews) which is conducted at the end of [[Enumerator_Training |enumerator training]].<br />
<br />
== Structure ==<br />
In order to conduct a successful '''survey pilot''', the research team must decide on the [[Structuring a Survey Pilot|structure of the survey pilot]]. This involves agreeing upon the '''logistics''' for the following aspects of the pilot:<br />
* '''Duration of the pilot.''' The pilot should ideally go on until the research team agrees that there are no changes left to be made.<br />
* '''Approvals.''' Obtain all necessary approvals, for example, [[Survey Budget|budget]] approval from the organization funding the study. Sometimes the research team must obtain an [[IRB Approval|IRB approval]] even for a pilot. <br />
* '''Review sessions.''' Set aside time for feedback sessions and revisions. The research team should make sure everyone involved in the '''survey pilot''' gets a chance to express their concerns. Therefore, it is a good practice to draft guidelines on the schedule and frequency of review sessions.<br />
* '''Training.''' As part of [[Enumerator Training|enumerator training]], set aside time specifically for discussing best practices for conducting a '''survey pilot'''.<br />
<br />
== Participants ==<br />
Typically, a '''survey pilot''' is conducted before a [[Survey Firm|survey firm]] is brought on-board. Each of the [[Survey Pilot Participants|survey pilot participants]] plays an important role in the implementation of a successful pilot. <br />
* '''Respondents.''' They should be as close to the '''target population''' as possible. None of the respondents from a pilot should be in the sample for the actual survey.<br />
* '''Interviewers.''' They should be fluent in all local languages of the study area and the language of the research team. The size of the team of interviewers depends on the stage of pilot.<br />
* '''Field coordinator (FC).''' The [[Impact_Evaluation_Team#Field_Coordinator|field coordinators (FCs)]] play a central role as part of the research team. They are responsible for updating the research team on all monitoring and evaluation '''(M&E)''' activities in the field.<br />
* '''Principal investigator (PI).''' The [[Impact_Evaluation_Team#Principal_Investigator|principal investigators (PIs)]] oversee the overall functioning of the '''impact evaluation''' (study). They should brief the FCs regularly during the pilot.<br />
* '''Other research team members.'''Ideally, other members of the [[Impact Evaluation Team|impact evaluation team]] such as the [[Impact Evaluation Team#Impact Evaluation Coordinator|impact evaluation coordinator]], research manager, programmers, and translators should also participate.<br />
<br />
== Guidelines ==<br />
Finally, it is important to keep the following '''best practices''' in mind while planning and conducting a pilot:<br />
* Throughout the process of [[Questionnaire Design|designing the questionnaire]], discuss with other members of the research team, and take notes of what needs to be part of the pilot.<br />
* Take careful notes of the discussions and clarifications that come up during the pilot. These will be an important part of the [[Enumerator Training#Enumerator Manual|enumerator manual]] which is used during [[Enumerator Training|enumerator training]].<br />
* Use the '''data-focused pilot''' to test [[Back Checks|back check]] templates. <br />
* Hire a local '''mobilizer''' to coordinate with respondents. '''Mobilizers''' explain the purpose behind conducting the survey, and facilitate the process of obtaining consent. This is particularly helpful in urban areas, or in cases where respondents are busy. This can improve the outcomes of piloting, for instance, by reducing gap between surveys.<br />
<br />
[https://www.worldbank.org/en/research/dime/data-and-analytics DIME Analytics] has also created the following checklists to assist researchers and enumerators in preparing for, and implementing a pilot:<br />
*[[Preparing_for_the_survey_checklist|Checklist: Preparing for a survey pilot]]<br />
*[[Checklist: Refine the Questionnaire (Content)|Checklist: Refining questionnaire content]]<br />
*[[Checklist: Refine the Questionnaire (Data)|Checklist: Refining questionnaire data]]<br />
<br />
== Related Pages ==<br />
[[Special:WhatLinksHere/Survey_Pilot|Click here for pages that link to this topic.]]<br />
<br />
== Additional Resources == <br />
* DIME Analytics (World Bank), [https://osf.io/63uv9/ Survey pilot guidelines]<br />
* DIME Analytics (World Bank), [https://osf.io/rqb5m/ Survey instrument design and pilot]<br />
<br />
[[Category: Primary Data Collection]] [[Category: Questionnaire Design]]</div>Maria ruth joneshttps://dimewiki.worldbank.org/index.php?title=Back_Checks&diff=7484Back Checks2020-09-01T18:14:10Z<p>Maria ruth jones: /* Designing the Back-Check Survey */ update to definition of types of questions to backcheck</p>
<hr />
<div>Back-checks are a [[Monitoring Data Quality | quality]] control method implemented to verify the quality and legitimacy of [[Primary Data Collection | data collected]] during a survey. Throughout the course of fieldwork, a back-check team returns to a randomly-selected subset of households for which data has been collected. The back-check team re-interviews these respondents with a short subset of survey questions, otherwise known as a back-check survey. Back-checks are used to verify the quality and legitimacy of key data collected in the actual survey. This page will provide points on how to coordinate, sample for, and design questionnaires for back-checks.<br />
<br />
==Read First ==<br />
*Back-checks are an important tool to detect fraud (i.e. enumerators sitting under a tree and filling out questionnaires themselves).<br />
*Back-checks help researchers to assess the accuracy and quality of the data collected. <br />
*Back-checks can be conducted by in-person visits or phone calls. A complementary approach to in-person back checks is conducting [[Random Audio Audits]].<br />
*Problems identified through back checks can be remedied by further [[Enumerator Training | training enumerators]] or replacing low-performing or problematic enumerators.<br />
<br />
==Coordinating Back-Checks==<br />
*The total duration of each back-check survey should be around 10-15 minutes.<br />
*The back-checks should be conducted by a team of specialized back-check enumerators. The back-check enumerators should be experienced, skilled enumerators.<br />
*The back-check team should be independent from the rest of the survey staff. They should be trained separately and have minimal contact with the survey team. <br />
*Administer 20% of back-checks within the first two weeks of fieldwork. This helps the research team to identify early whether the [[Questionnaire Design | questionnaire]] is effective, whether enumerators are doing their jobs well, and which changes to make to ensure high quality data collection.<br />
<br />
==Sampling for Back-Checks==<br />
*Aim to back-check 10-20% of the total observations.<br />
*The back-check sample should be [[Stratified Random Sample | stratified]] across survey teams/enumerators. Every team and every enumerator must be back-checked as soon as possible and regularly.<br />
*Include missing respondents in the back-check sample to verify that enumerators are not biasing your sample by not tracking hard-to-find respondents. Also include observations flagged in other quality tests like [[Monitoring Data Quality#Guidelines#High Frequency Checks | high frequency checks]] and observations collected by enumerators suspected of cheating.<br />
<br />
==Designing the Back-Check Survey==<br />
Back-check questions are drawn from the original [[Questionnaire Design | questionnaire]]. There are four types of questions that should be included in a back-check to gauge data and enumerator quality:<br />
<br />
*Questions to identify respondent and interview information: <br />
: These questions verify the identity of the respondent and check if, when, and where the original survey took place. This is useful to check for fraud and verify reported completion rates. <br />
<br />
* Questions to detect fraud<br />
:Include questions that ask for straightforward information with no expected variation or room for error. These should be questions that do not require particularly skilled enumeration, and do not vary over time (specifically the time period between the main interview and the backcheck). Examples include type of dwelling, education level, marital status, occupation, whether the respondent has children or not, etc. The specific variables to include will depend on the survey instrument and context. If values differ between the questionnaire and the backcheck survey, they indicate poor quality data, a serious enumerator problem, and/or potentially falsified work. <br />
<br />
* Questions to detect errors in survey execution<br />
:These are questions for which capable enumerators should get the true answer. These should be questions which involve relatively complex logic or consistency checks. If values for these questions differ between the questionnaire and the backcheck survey, they indicate that the enumerator may need more training.<br />
<br />
* Questions to detect problems with the questionnaire or key outcomes<br />
:These should be a selection of questions that are key outcome variables for the survey. The backcheck provides an additional accuracy checks, and are useful to flag difficulties and/or inconsistencies in enumerator interpretation of the questions. If these values differ between the questionnaire and the backcheck, it indicates the need for further enumerator training or, in particular cases, questionnaire modification. <br />
<br />
* Questions that determine repeated sections of the questionnaire<br />
: These should be included to check whether enumerators are falsifying data to reduce the length of interviews. For example, if there is a long series of questions about each household member, verify that the number of household members is correct. If an agricultural survey asks for production information by plot, verify the number of plots is correct. <br />
<br />
Note that it is important that enumerators do not know what questions will be audited. To that end, you may consider randomizing questions or changing the back-check survey regularly during data collection.<br />
<br />
== Analyzing Back-Check Data ==<br />
<br />
After completing a back-check, you can compare the back-check data to the original survey data. This can be done by using the Stata command <code>bcstats</code>, developed by [http://www.poverty-action.org/ Innovations for Poverty Action]. This command produces a dataset of the comparisons between the back-check and original survey data. The command also completes enumerator checks and stability checks for variables.<br />
<br />
The steps are as follows:<br />
<nowiki><br />
ssc install bcstats <br />
bcstats, surveydata(filename) bcdata(filename) id(varlist) [options]<br />
</nowiki><br />
To learn about the options for <code>bcstats</code> and back-checks, please type <code> help bcstats </code> on Stata after installing the command.<br />
<br />
==Back to Parent==<br />
This article is part of the topic [[Field Management]].<br />
<br />
== Additional Resources ==<br />
*DIME Analytics’ [https://github.com/worldbank/DIME-Resources/blob/master/stata1-4-quality.pdf Real Time Data Quality Checks]<br />
*DIME Analytics’ [https://github.com/worldbank/DIME-Resources/blob/master/stata2-4-quality.pdf Data Quality Assurance]<br />
* World Health Organization's [http://unstats.un.org/unsd/hhsurveys/pdf/Chapter_10.pdf '''Quality Assurance in Surveys: standards, guidelines, and procedures''']. This chapter provides, in detail, the approach and methodology on quality control during surveys.<br />
*[https://ideas.repec.org/c/boc/bocode/s458173.html bcstats], a Stata program written by an IPA staff member for conducting back checks on survey data.<br />
[[Category: Field Management ]]</div>Maria ruth joneshttps://dimewiki.worldbank.org/index.php?title=Back_Checks&diff=7482Back Checks2020-09-01T15:52:07Z<p>Maria ruth jones: /* Coordinating Back-Checks */</p>
<hr />
<div>Back-checks are a [[Monitoring Data Quality | quality]] control method implemented to verify the quality and legitimacy of [[Primary Data Collection | data collected]] during a survey. Throughout the course of fieldwork, a back-check team returns to a randomly-selected subset of households for which data has been collected. The back-check team re-interviews these respondents with a short subset of survey questions, otherwise known as a back-check survey. Back-checks are used to verify the quality and legitimacy of key data collected in the actual survey. This page will provide points on how to coordinate, sample for, and design questionnaires for back-checks.<br />
<br />
==Read First ==<br />
*Back-checks are an important tool to detect fraud (i.e. enumerators sitting under a tree and filling out questionnaires themselves).<br />
*Back-checks help researchers to assess the accuracy and quality of the data collected. <br />
*Back-checks can be conducted by in-person visits or phone calls. A complementary approach to in-person back checks is conducting [[Random Audio Audits]].<br />
*Problems identified through back checks can be remedied by further [[Enumerator Training | training enumerators]] or replacing low-performing or problematic enumerators.<br />
<br />
==Coordinating Back-Checks==<br />
*The total duration of each back-check survey should be around 10-15 minutes.<br />
*The back-checks should be conducted by a team of specialized back-check enumerators. The back-check enumerators should be experienced, skilled enumerators.<br />
*The back-check team should be independent from the rest of the survey staff. They should be trained separately and have minimal contact with the survey team. <br />
*Administer 20% of back-checks within the first two weeks of fieldwork. This helps the research team to identify early whether the [[Questionnaire Design | questionnaire]] is effective, whether enumerators are doing their jobs well, and which changes to make to ensure high quality data collection.<br />
<br />
==Sampling for Back-Checks==<br />
*Aim to back-check 10-20% of the total observations.<br />
*The back-check sample should be [[Stratified Random Sample | stratified]] across survey teams/enumerators. Every team and every enumerator must be back-checked as soon as possible and regularly.<br />
*Include missing respondents in the back-check sample to verify that enumerators are not biasing your sample by not tracking hard-to-find respondents. Also include observations flagged in other quality tests like [[Monitoring Data Quality#Guidelines#High Frequency Checks | high frequency checks]] and observations collected by enumerators suspected of cheating.<br />
<br />
==Designing the Back-Check Survey==<br />
Back-check questions are drawn from the original [[Questionnaire Design | questionnaire]]. [http://www.poverty-action.org/ Innovation for Poverty Action] identifies four types of questions that should be included in a back-check to best gauge data and enumerator quality:<br />
<br />
*Questions to identify respondent and interview information: <br />
: These questions verify the identity of the respondent and check if, when, and where the original survey took place. <br />
<br />
*Type 1 Variable Questions: <br />
:These questions ask straightforward information with no expected variation or room for error. They may include questions about education level, marital status, occupation, whether the respondent has children or not, etc. If Type 1 variable values differ between the questionnaire and the backcheck survey, they indicate poor quality data, a serious enumerator problem, and potentially falsified work. <br />
<br />
*Type 2 Variable Questions: <br />
:These are questions for which capable enumerators should get the true answer. If the Type 2 response value differ between the questionnaire and the backcheck survey, they indicate that the enumerator may need more training.<br />
<br />
*Type 3 Variable Questions: <br />
:These questions are expected to be difficult. They help research teams to understand if the questionnaire is effectively designed and if enumerators are interpreting difficult and/or nuanced questions correctly and uniformly. If Type 3 variable values differ between the questionnaire and the backcheck, they indicate the need for further enumerator training or, in particular cases, questionnaire modification. <br />
<br />
Back-check surveys may also test for translation issues by including questions that could be interpreted differently by different surveyors. Finally, to test whether enumerators are falsifying data to shorten interviews, back-check questions that determine repeated sections of the questionnaire. For example, if there is a long series of questions about household members, verify the correct number of household members. If an agricultural survey asks for production information by plot, verify the number of plots. <br />
<br />
Note that it is important that enumerators do not know what questions will be audited. To that end, you may consider randomly changing the back-check survey regularly during data collection.<br />
<br />
== Analyzing Back-Check Data ==<br />
<br />
After completing a back-check, you can compare the back-check data to the original survey data. This can be done by using the Stata command <code>bcstats</code>, developed by [http://www.poverty-action.org/ Innovations for Poverty Action]. This command produces a dataset of the comparisons between the back-check and original survey data. The command also completes enumerator checks and stability checks for variables.<br />
<br />
The steps are as follows:<br />
<nowiki><br />
ssc install bcstats <br />
bcstats, surveydata(filename) bcdata(filename) id(varlist) [options]<br />
</nowiki><br />
To learn about the options for <code>bcstats</code> and back-checks, please type <code> help bcstats </code> on Stata after installing the command.<br />
<br />
==Back to Parent==<br />
This article is part of the topic [[Field Management]].<br />
<br />
== Additional Resources ==<br />
*DIME Analytics’ [https://github.com/worldbank/DIME-Resources/blob/master/stata1-4-quality.pdf Real Time Data Quality Checks]<br />
*DIME Analytics’ [https://github.com/worldbank/DIME-Resources/blob/master/stata2-4-quality.pdf Data Quality Assurance]<br />
* World Health Organization's [http://unstats.un.org/unsd/hhsurveys/pdf/Chapter_10.pdf '''Quality Assurance in Surveys: standards, guidelines, and procedures''']. This chapter provides, in detail, the approach and methodology on quality control during surveys.<br />
*[https://ideas.repec.org/c/boc/bocode/s458173.html bcstats], a Stata program written by an IPA staff member for conducting back checks on survey data.<br />
[[Category: Field Management ]]</div>Maria ruth joneshttps://dimewiki.worldbank.org/index.php?title=Back_Checks&diff=7481Back Checks2020-09-01T15:48:49Z<p>Maria ruth jones: /* Read First */</p>
<hr />
<div>Back-checks are a [[Monitoring Data Quality | quality]] control method implemented to verify the quality and legitimacy of [[Primary Data Collection | data collected]] during a survey. Throughout the course of fieldwork, a back-check team returns to a randomly-selected subset of households for which data has been collected. The back-check team re-interviews these respondents with a short subset of survey questions, otherwise known as a back-check survey. Back-checks are used to verify the quality and legitimacy of key data collected in the actual survey. This page will provide points on how to coordinate, sample for, and design questionnaires for back-checks.<br />
<br />
==Read First ==<br />
*Back-checks are an important tool to detect fraud (i.e. enumerators sitting under a tree and filling out questionnaires themselves).<br />
*Back-checks help researchers to assess the accuracy and quality of the data collected. <br />
*Back-checks can be conducted by in-person visits or phone calls. A complementary approach to in-person back checks is conducting [[Random Audio Audits]].<br />
*Problems identified through back checks can be remedied by further [[Enumerator Training | training enumerators]] or replacing low-performing or problematic enumerators.<br />
<br />
==Coordinating Back-Checks==<br />
*The total duration of each back-check survey should be around 10-15 minutes.<br />
*The back-checks should be conducted by a specialized team of a few exclusively back-checking enumerators. The back-check enumerators should be of the highest trust and quality.<br />
*Administer 20% of back-checks within the first two weeks of fieldwork. This helps the research team to identify early whether the [[Questionnaire Design | questionnaire]] is effective, whether enumerators are doing their jobs well, and which changes to make to ensure high quality data collection.<br />
<br />
==Sampling for Back-Checks==<br />
*Aim to back-check 10-20% of the total observations.<br />
*The back-check sample should be [[Stratified Random Sample | stratified]] across survey teams/enumerators. Every team and every enumerator must be back-checked as soon as possible and regularly.<br />
*Include missing respondents in the back-check sample to verify that enumerators are not biasing your sample by not tracking hard-to-find respondents. Also include observations flagged in other quality tests like [[Monitoring Data Quality#Guidelines#High Frequency Checks | high frequency checks]] and observations collected by enumerators suspected of cheating.<br />
<br />
==Designing the Back-Check Survey==<br />
Back-check questions are drawn from the original [[Questionnaire Design | questionnaire]]. [http://www.poverty-action.org/ Innovation for Poverty Action] identifies four types of questions that should be included in a back-check to best gauge data and enumerator quality:<br />
<br />
*Questions to identify respondent and interview information: <br />
: These questions verify the identity of the respondent and check if, when, and where the original survey took place. <br />
<br />
*Type 1 Variable Questions: <br />
:These questions ask straightforward information with no expected variation or room for error. They may include questions about education level, marital status, occupation, whether the respondent has children or not, etc. If Type 1 variable values differ between the questionnaire and the backcheck survey, they indicate poor quality data, a serious enumerator problem, and potentially falsified work. <br />
<br />
*Type 2 Variable Questions: <br />
:These are questions for which capable enumerators should get the true answer. If the Type 2 response value differ between the questionnaire and the backcheck survey, they indicate that the enumerator may need more training.<br />
<br />
*Type 3 Variable Questions: <br />
:These questions are expected to be difficult. They help research teams to understand if the questionnaire is effectively designed and if enumerators are interpreting difficult and/or nuanced questions correctly and uniformly. If Type 3 variable values differ between the questionnaire and the backcheck, they indicate the need for further enumerator training or, in particular cases, questionnaire modification. <br />
<br />
Back-check surveys may also test for translation issues by including questions that could be interpreted differently by different surveyors. Finally, to test whether enumerators are falsifying data to shorten interviews, back-check questions that determine repeated sections of the questionnaire. For example, if there is a long series of questions about household members, verify the correct number of household members. If an agricultural survey asks for production information by plot, verify the number of plots. <br />
<br />
Note that it is important that enumerators do not know what questions will be audited. To that end, you may consider randomly changing the back-check survey regularly during data collection.<br />
<br />
== Analyzing Back-Check Data ==<br />
<br />
After completing a back-check, you can compare the back-check data to the original survey data. This can be done by using the Stata command <code>bcstats</code>, developed by [http://www.poverty-action.org/ Innovations for Poverty Action]. This command produces a dataset of the comparisons between the back-check and original survey data. The command also completes enumerator checks and stability checks for variables.<br />
<br />
The steps are as follows:<br />
<nowiki><br />
ssc install bcstats <br />
bcstats, surveydata(filename) bcdata(filename) id(varlist) [options]<br />
</nowiki><br />
To learn about the options for <code>bcstats</code> and back-checks, please type <code> help bcstats </code> on Stata after installing the command.<br />
<br />
==Back to Parent==<br />
This article is part of the topic [[Field Management]].<br />
<br />
== Additional Resources ==<br />
*DIME Analytics’ [https://github.com/worldbank/DIME-Resources/blob/master/stata1-4-quality.pdf Real Time Data Quality Checks]<br />
*DIME Analytics’ [https://github.com/worldbank/DIME-Resources/blob/master/stata2-4-quality.pdf Data Quality Assurance]<br />
* World Health Organization's [http://unstats.un.org/unsd/hhsurveys/pdf/Chapter_10.pdf '''Quality Assurance in Surveys: standards, guidelines, and procedures''']. This chapter provides, in detail, the approach and methodology on quality control during surveys.<br />
*[https://ideas.repec.org/c/boc/bocode/s458173.html bcstats], a Stata program written by an IPA staff member for conducting back checks on survey data.<br />
[[Category: Field Management ]]</div>Maria ruth joneshttps://dimewiki.worldbank.org/index.php?title=Survey_Pilot&diff=7455Survey Pilot2020-08-21T02:05:57Z<p>Maria ruth jones: /* Stages of a Survey Pilot */</p>
<hr />
<div>'''Survey pilot''' is the process of carrying out interviews and tests on different components of a survey, including [[Piloting Survey Content|content]] and [[Piloting Survey Protocols|protocols]]. A good '''pilot''' provides the research team with important feedback before they start the process of [[Primary Data Collection|data collection]]. This feedback can help the [[Impact Evaluation Team|research team]] review and improve [[Questionnaire Design|instrument design]], [[Questionnaire Translation|translations]], as well as [[Survey Protocols|survey protocols]] related to [[Piloting Survey Protocols#Interview scheduling|interview scheduling]], [[Sampling|sampling]], and [[Geo Spatial Data|geo data]].<br />
== Read First ==<br />
* A '''survey pilot''' is necessary for both [[Field Surveys|field surveys]] and [[Remote Surveys|remote surveys]].<br />
* A comprehensive '''survey pilot''' includes 3 stages: a pre-pilot, a content-focused pilot, and a data-focused pilot. <br />
* A survey pilot is different from the '''field testing''' (practice interviews) that enumerators conduct at the end of [[Enumerator Training|enumerator training]]. The pilot should be complete before enumerator training begins. <br />
* [[Survey Pilot Participants|Participants of a survey pilot]] include the [[Impact Evaluation Team|research team]], '''interviewers''', and '''respondents'''.<br />
* Conduct the first two stages of the pilot before [[Questionnaire Programming|programming the questionnaire]]. Repeated programming is time-consuming and can create '''bugs''' (errors), for instance, by disturbing the order of questions.<br />
<br />
<br />
<br />
<br />
== Stages of a Survey Pilot ==<br />
A complete '''survey pilot''' is conducted over three stages - '''pre-pilot''', [[Piloting Survey Content|content-focused pilot]], and '''data-focused pilot'''. The table below discusses these stages in more detail.<br />
<br />
{| class="wikitable" style="margin-left: auto; margin-right: auto; border: none;"<br />
|-<br />
! style="height:50px; width:150px; text-align:center;" |<br />
! style="height:50px; width:250px; text-align:center;" | Stage 1: Pre-pilot<br />
! style="height:50px; width:250px; text-align:center;" | Stage 2: Content-focused pilot<br />
! style="height:50px; width:250px; text-align:center;" | Stage 3: Data-focused pilot<br />
|-<br />
| style="text-align:center;" | '''Objective'''<br />
| style="vertical-align:top;"| Answer broad questions about '''qualitative factors''' like [[Questionnaire Design|survey design]] and the context for conducting the study. Discuss these with the concerned teams.<br />
| style="vertical-align:top;"| Refine the order and wording of specific questions, the overall structure of the questionnaire, and [[Questionnaire Translation|translations]]. <br />
<br />
<br />
<br />
Check that the answer choices are '''complete''', that is, they cover all possibilities. <br />
<br />
<br />
Check how long it takes to answer the questions. Also check how answers differ between respondents ('''response variance''')<br />
| style="vertical-align:top;"| Check if the [[Questionnaire Programming|programmed instrument]] displays questions in the correct order, and follows the correct patterns (such as a group of questions which will always appear together, or will repeat).<br />
<br />
Load a sample data set (based on '''pilot interviews'''). Check if the data set has any missing fields. Perform all data quality checks, like [[Back Checks|back checks]] and [[Monitoring Data Quality#High Frequency Checks|high frequency checks]].<br />
|-<br />
| style="height:80px;text-align:center;" | '''Status of survey instrument'''<br />
| style="vertical-align:center;"| Early, printable version of the draft, and notes for further discussion.<br />
| style="vertical-align:center;"| A translated, printable draft. Ready to be '''programmed'''.<br />
| style="vertical-align:center;"| A translated, programmed, final draft. Ready for starting data collection.<br />
|-<br />
| style="height:80px;text-align:center;" | '''Mode'''<br />
| style="text-align:center;"| Pen-and-paper<br />
| style="text-align:center;"| Pen-and-paper<br />
| style="text-align:center;"| Tablet/phone<br />
|}<br />
<br />
However, note that all three stages may not be necessary for every survey. The research team has to determine which stage to begin from on a case-to-case basis.<br />
* If the research team is using a brand new survey instrument, then they must start with Stage 1, the '''pre-pilot.'''<br />
* If the survey instrument is an adaptation of (based on) a different instrument which was used for a previous data collection for another project in the same region (province or country), the research team can start with Stage 2, the '''content-focused pilot'''. However, they must make sure that the survey instrument does not have any design issues and was shared by a reliable source.<br />
* If the survey instrument is an adaptation of an instrument used for a previous data collection for the same project, but the research team had to make significant revisions or additions, even then the research team should start with Stage 2, the '''content-focused pilot'''.<br />
* Only if the research team is '''piloting''' a '''follow-up''' survey, and there are no major changes from the '''baseline''' (or first round) survey, then in this case the research team may skip directly to Stage 3, the '''data-focused pilot'''.<br />
<br />
== Pilot on paper first == <br />
The first two stages of the survey pilot (pre-pilot and content-focused pilot) are best done on pen-and-paper is done on paper, regardless of the planned survey mode ([[Pen-and-Paper_Personal_Interviews_(PAPI)|pen-and-paper personal interviews (PAPI)]],<br />
[[Computer-Assisted_Personal_Interviews_(CAPI)|computer-assisted personal interviews (CAPI)]], [[Remote_Surveys|phone surveys (CATI)]]). A pen-and-paper pilot allows members of the research team and interviewers flexibility when it comes to recording answers and '''qualitative''' observations. <br />
<br />
Piloting on paper is particularly helpful for the following:<br />
* '''Open-ended responses.''' pre-pilot responses help the research team to make the list of choices for a question more detailed and comprehensive. Pen-and-paper pilots allow the interviewers to make note of these.<br />
* '''Restructuring.''' Pen-and-paper pilots allow the interviewers to draw lines and arrows between questions, which makes it easier to restructure the '''instrument'''.<br />
* '''Observations.''' Pen-and-paper pilots allow interviewers to record observations and feedback of interviewers in the margins. This allows the research team to take note of issues, such as the wording or flow of questions.<br />
<br />
== Timeline ==<br />
'''Piloting''' should start 4-6 weeks before [[Field Surveys#Survey Launch|survey launch]]. In fact, starting a pilot early allows the research team to use the feedback from each stage to improve the [[Piloting Survey Content|survey instrument]] and [[Piloting Survey Protocols| survey protocols]]. This makes the process of implementing a survey more efficient. Therefore, the research team must draft a detailed [[Timeline of Survey Pilot|timeline for the pilot]] to allow enough time for discussions and revisions. Do not confuse the pilot with '''field testing''' (practice interviews) which is conducted at the end of [[Enumerator_Training |enumerator training]].<br />
<br />
== Structure ==<br />
In order to conduct a successful '''survey pilot''', the research team must decide on the [[Structuring a Survey Pilot|structure of the survey pilot]]. This involves agreeing upon the '''logistics''' for the following aspects of the pilot:<br />
* '''Duration of the pilot.''' The pilot should ideally go on until the research team agrees that there are no changes left to be made.<br />
* '''Approvals.''' Obtain all necessary approvals, for example, [[Survey Budget|budget]] approval from the organization funding the study. Sometimes the research team must obtain an [[IRB Approval|IRB approval]] even for a pilot. <br />
* '''Review sessions.''' Set aside time for feedback sessions and revisions. The research team should make sure everyone involved in the '''survey pilot''' gets a chance to express their concerns. Therefore, it is a good practice to draft guidelines on the schedule and frequency of review sessions.<br />
* '''Training.''' As part of [[Enumerator Training|enumerator training]], set aside time specifically for discussing best practices for conducting a '''survey pilot'''.<br />
<br />
== Participants ==<br />
Typically, a '''survey pilot''' is conducted before a [[Survey Firm|survey firm]] is brought on-board. Each of the [[Survey Pilot Participants|survey pilot participants]] plays an important role in the implementation of a successful pilot. <br />
* '''Respondents.''' They should be as close to the '''target population''' as possible. None of the respondents from a pilot should be in the sample for the actual survey.<br />
* '''Interviewers.''' They should be fluent in all local languages of the study area and the language of the research team. The size of the team of interviewers depends on the stage of pilot.<br />
* '''Field coordinator (FC).''' The [[Impact_Evaluation_Team#Field_Coordinator|field coordinators (FCs)]] play a central role as part of the research team. They are responsible for updating the research team on all monitoring and evaluation '''(M&E)''' activities in the field.<br />
* '''Principal investigator (PI).''' The [[Impact_Evaluation_Team#Principal_Investigator|principal investigators (PIs)]] oversee the overall functioning of the '''impact evaluation''' (study). They should brief the FCs regularly during the pilot.<br />
* '''Other research team members.'''Ideally, other members of the [[Impact Evaluation Team|impact evaluation team]] such as the [[Impact Evaluation Team#Impact Evaluation Coordinator|impact evaluation coordinator]], research manager, programmers, and translators should also participate.<br />
<br />
== Guidelines ==<br />
Finally, it is important to keep the following '''best practices''' in mind while planning and conducting a pilot:<br />
* Throughout the process of [[Questionnaire Design|designing the questionnaire]], discuss with other members of the research team, and take notes of what needs to be part of the pilot.<br />
* Take careful notes of the discussions and clarifications that come up during the pilot. These will be an important part of the [[Enumerator Training#Enumerator Manual|enumerator manual]] which is used during [[Enumerator Training|enumerator training]].<br />
* Use the '''data-focused pilot''' to test [[Back Checks|back check]] templates. <br />
* Hire a local '''mobilizer''' to coordinate with respondents. '''Mobilizers''' explain the purpose behind conducting the survey, and facilitate the process of obtaining consent. This is particularly helpful in urban areas, or in cases where respondents are busy. This can improve the outcomes of piloting, for instance, by reducing gap between surveys.<br />
<br />
[https://www.worldbank.org/en/research/dime/data-and-analytics DIME Analytics] has also created the following checklists to assist researchers and enumerators in preparing for, and implementing a pilot:<br />
*[[Preparing_for_the_survey_checklist|Checklist: Preparing for a survey pilot]]<br />
*[[Checklist: Refine the Questionnaire (Content)|Checklist: Refining questionnaire content]]<br />
*[[Checklist: Refine the Questionnaire (Data)|Checklist: Refining questionnaire data]]<br />
<br />
== Related Pages ==<br />
[[Special:WhatLinksHere/Survey_Pilot|Click here for pages that link to this topic.]]<br />
<br />
== Additional Resources == <br />
* DIME Analytics (World Bank), [https://osf.io/63uv9/ Survey pilot guidelines]<br />
* DIME Analytics (World Bank), [https://osf.io/rqb5m/ Survey instrument design and pilot]<br />
<br />
[[Category: Primary Data Collection]] [[Category: Questionnaire Design]]</div>Maria ruth joneshttps://dimewiki.worldbank.org/index.php?title=Survey_Pilot&diff=7454Survey Pilot2020-08-21T02:05:40Z<p>Maria ruth jones: /* Read First */</p>
<hr />
<div>'''Survey pilot''' is the process of carrying out interviews and tests on different components of a survey, including [[Piloting Survey Content|content]] and [[Piloting Survey Protocols|protocols]]. A good '''pilot''' provides the research team with important feedback before they start the process of [[Primary Data Collection|data collection]]. This feedback can help the [[Impact Evaluation Team|research team]] review and improve [[Questionnaire Design|instrument design]], [[Questionnaire Translation|translations]], as well as [[Survey Protocols|survey protocols]] related to [[Piloting Survey Protocols#Interview scheduling|interview scheduling]], [[Sampling|sampling]], and [[Geo Spatial Data|geo data]].<br />
== Read First ==<br />
* A '''survey pilot''' is necessary for both [[Field Surveys|field surveys]] and [[Remote Surveys|remote surveys]].<br />
* A comprehensive '''survey pilot''' includes 3 stages: a pre-pilot, a content-focused pilot, and a data-focused pilot. <br />
* A survey pilot is different from the '''field testing''' (practice interviews) that enumerators conduct at the end of [[Enumerator Training|enumerator training]]. The pilot should be complete before enumerator training begins. <br />
* [[Survey Pilot Participants|Participants of a survey pilot]] include the [[Impact Evaluation Team|research team]], '''interviewers''', and '''respondents'''.<br />
* Conduct the first two stages of the pilot before [[Questionnaire Programming|programming the questionnaire]]. Repeated programming is time-consuming and can create '''bugs''' (errors), for instance, by disturbing the order of questions.<br />
<br />
== Stages of a Survey Pilot ==<br />
A complete '''survey pilot''' is conducted over three stages - '''pre-pilot''', [[Piloting Survey Content|content-focused pilot]], and '''data-focused pilot'''. The table below discusses these stages in more detail.<br />
<br />
{| class="wikitable" style="margin-left: auto; margin-right: auto; border: none;"<br />
|-<br />
! style="height:50px; width:150px; text-align:center;" |<br />
! style="height:50px; width:250px; text-align:center;" | Stage 1: Pre-pilot<br />
! style="height:50px; width:250px; text-align:center;" | Stage 2: Content-focused pilot<br />
! style="height:50px; width:250px; text-align:center;" | Stage 3: Data-focused pilot<br />
|-<br />
| style="text-align:center;" | '''Objective'''<br />
| style="vertical-align:top;"| Answer broad questions about '''qualitative factors''' like [[Questionnaire Design|survey design]] and the context for conducting the study. Discuss these with the concerned teams.<br />
| style="vertical-align:top;"| Refine the order and wording of specific questions, the overall structure of the questionnaire, and [[Questionnaire Translation|translations]]. <br />
<br />
<br />
<br />
Check that the answer choices are '''complete''', that is, they cover all possibilities. <br />
<br />
<br />
Check how long it takes to answer the questions. Also check how answers differ between respondents ('''response variance''')<br />
| style="vertical-align:top;"| Check if the [[Questionnaire Programming|programmed instrument]] displays questions in the correct order, and follows the correct patterns (such as a group of questions which will always appear together, or will repeat).<br />
<br />
Load a sample data set (based on '''pilot interviews'''). Check if the data set has any missing fields. Perform all data quality checks, like [[Back Checks|back checks]] and [[Monitoring Data Quality#High Frequency Checks|high frequency checks]].<br />
|-<br />
| style="height:80px;text-align:center;" | '''Status of survey instrument'''<br />
| style="vertical-align:center;"| Early, printable version of the draft, and notes for further discussion.<br />
| style="vertical-align:center;"| A translated, printable draft. Ready to be '''programmed'''.<br />
| style="vertical-align:center;"| A translated, programmed, final draft. Ready for starting data collection.<br />
|-<br />
| style="height:80px;text-align:center;" | '''Mode'''<br />
| style="text-align:center;"| Pen-and-paper<br />
| style="text-align:center;"| Pen-and-paper<br />
| style="text-align:center;"| Tablet/phone<br />
|}<br />
<br />
However, note that all three stages may not be necessary for every survey. The research team has to determine which stage to begin from on a case-to-case basis.<br />
* If the research team is using a brand new survey instrument, then they must start with Stage 1, the '''pre-pilot.'''<br />
* If the survey instrument is an adaptation of (based on) a different instrument which was used for a previous data collection for another project in the same region (province or country), the research team can start with Stage 2, the '''content-focused pilot'''. However, they must make sure that the survey instrument does not have any design issues and was shared by a reliable source.<br />
* If the survey instrument is an adaptation of an instrument used for a previous data collection for the same project, but the research team had to make significant revisions or additions, even then the research team should start with Stage 2, the '''content-focused pilot'''.<br />
* Only if the research team is '''piloting''' a '''follow-up''' survey, and there are no major changes from the '''baseline''' (or first round) survey, then in this case the research team may skip directly to Stage 3, the '''data-focused pilot'''.<br />
<br />
== Pilot on paper first == <br />
The first two stages of the survey pilot (pre-pilot and content-focused pilot) are best done on pen-and-paper is done on paper, regardless of the planned survey mode ([[Pen-and-Paper_Personal_Interviews_(PAPI)|pen-and-paper personal interviews (PAPI)]],<br />
[[Computer-Assisted_Personal_Interviews_(CAPI)|computer-assisted personal interviews (CAPI)]], [[Remote_Surveys|phone surveys (CATI)]]). A pen-and-paper pilot allows members of the research team and interviewers flexibility when it comes to recording answers and '''qualitative''' observations. <br />
<br />
Piloting on paper is particularly helpful for the following:<br />
* '''Open-ended responses.''' pre-pilot responses help the research team to make the list of choices for a question more detailed and comprehensive. Pen-and-paper pilots allow the interviewers to make note of these.<br />
* '''Restructuring.''' Pen-and-paper pilots allow the interviewers to draw lines and arrows between questions, which makes it easier to restructure the '''instrument'''.<br />
* '''Observations.''' Pen-and-paper pilots allow interviewers to record observations and feedback of interviewers in the margins. This allows the research team to take note of issues, such as the wording or flow of questions.<br />
<br />
== Timeline ==<br />
'''Piloting''' should start 4-6 weeks before [[Field Surveys#Survey Launch|survey launch]]. In fact, starting a pilot early allows the research team to use the feedback from each stage to improve the [[Piloting Survey Content|survey instrument]] and [[Piloting Survey Protocols| survey protocols]]. This makes the process of implementing a survey more efficient. Therefore, the research team must draft a detailed [[Timeline of Survey Pilot|timeline for the pilot]] to allow enough time for discussions and revisions. Do not confuse the pilot with '''field testing''' (practice interviews) which is conducted at the end of [[Enumerator_Training |enumerator training]].<br />
<br />
== Structure ==<br />
In order to conduct a successful '''survey pilot''', the research team must decide on the [[Structuring a Survey Pilot|structure of the survey pilot]]. This involves agreeing upon the '''logistics''' for the following aspects of the pilot:<br />
* '''Duration of the pilot.''' The pilot should ideally go on until the research team agrees that there are no changes left to be made.<br />
* '''Approvals.''' Obtain all necessary approvals, for example, [[Survey Budget|budget]] approval from the organization funding the study. Sometimes the research team must obtain an [[IRB Approval|IRB approval]] even for a pilot. <br />
* '''Review sessions.''' Set aside time for feedback sessions and revisions. The research team should make sure everyone involved in the '''survey pilot''' gets a chance to express their concerns. Therefore, it is a good practice to draft guidelines on the schedule and frequency of review sessions.<br />
* '''Training.''' As part of [[Enumerator Training|enumerator training]], set aside time specifically for discussing best practices for conducting a '''survey pilot'''.<br />
<br />
== Participants ==<br />
Typically, a '''survey pilot''' is conducted before a [[Survey Firm|survey firm]] is brought on-board. Each of the [[Survey Pilot Participants|survey pilot participants]] plays an important role in the implementation of a successful pilot. <br />
* '''Respondents.''' They should be as close to the '''target population''' as possible. None of the respondents from a pilot should be in the sample for the actual survey.<br />
* '''Interviewers.''' They should be fluent in all local languages of the study area and the language of the research team. The size of the team of interviewers depends on the stage of pilot.<br />
* '''Field coordinator (FC).''' The [[Impact_Evaluation_Team#Field_Coordinator|field coordinators (FCs)]] play a central role as part of the research team. They are responsible for updating the research team on all monitoring and evaluation '''(M&E)''' activities in the field.<br />
* '''Principal investigator (PI).''' The [[Impact_Evaluation_Team#Principal_Investigator|principal investigators (PIs)]] oversee the overall functioning of the '''impact evaluation''' (study). They should brief the FCs regularly during the pilot.<br />
* '''Other research team members.'''Ideally, other members of the [[Impact Evaluation Team|impact evaluation team]] such as the [[Impact Evaluation Team#Impact Evaluation Coordinator|impact evaluation coordinator]], research manager, programmers, and translators should also participate.<br />
<br />
== Guidelines ==<br />
Finally, it is important to keep the following '''best practices''' in mind while planning and conducting a pilot:<br />
* Throughout the process of [[Questionnaire Design|designing the questionnaire]], discuss with other members of the research team, and take notes of what needs to be part of the pilot.<br />
* Take careful notes of the discussions and clarifications that come up during the pilot. These will be an important part of the [[Enumerator Training#Enumerator Manual|enumerator manual]] which is used during [[Enumerator Training|enumerator training]].<br />
* Use the '''data-focused pilot''' to test [[Back Checks|back check]] templates. <br />
* Hire a local '''mobilizer''' to coordinate with respondents. '''Mobilizers''' explain the purpose behind conducting the survey, and facilitate the process of obtaining consent. This is particularly helpful in urban areas, or in cases where respondents are busy. This can improve the outcomes of piloting, for instance, by reducing gap between surveys.<br />
<br />
[https://www.worldbank.org/en/research/dime/data-and-analytics DIME Analytics] has also created the following checklists to assist researchers and enumerators in preparing for, and implementing a pilot:<br />
*[[Preparing_for_the_survey_checklist|Checklist: Preparing for a survey pilot]]<br />
*[[Checklist: Refine the Questionnaire (Content)|Checklist: Refining questionnaire content]]<br />
*[[Checklist: Refine the Questionnaire (Data)|Checklist: Refining questionnaire data]]<br />
<br />
== Related Pages ==<br />
[[Special:WhatLinksHere/Survey_Pilot|Click here for pages that link to this topic.]]<br />
<br />
== Additional Resources == <br />
* DIME Analytics (World Bank), [https://osf.io/63uv9/ Survey pilot guidelines]<br />
* DIME Analytics (World Bank), [https://osf.io/rqb5m/ Survey instrument design and pilot]<br />
<br />
[[Category: Primary Data Collection]] [[Category: Questionnaire Design]]</div>Maria ruth joneshttps://dimewiki.worldbank.org/index.php?title=Survey_Pilot&diff=7453Survey Pilot2020-08-21T02:04:43Z<p>Maria ruth jones: /* Read First */</p>
<hr />
<div>'''Survey pilot''' is the process of carrying out interviews and tests on different components of a survey, including [[Piloting Survey Content|content]] and [[Piloting Survey Protocols|protocols]]. A good '''pilot''' provides the research team with important feedback before they start the process of [[Primary Data Collection|data collection]]. This feedback can help the [[Impact Evaluation Team|research team]] review and improve [[Questionnaire Design|instrument design]], [[Questionnaire Translation|translations]], as well as [[Survey Protocols|survey protocols]] related to [[Piloting Survey Protocols#Interview scheduling|interview scheduling]], [[Sampling|sampling]], and [[Geo Spatial Data|geo data]].<br />
== Read First ==<br />
* A '''survey pilot''' is necessary for both [[Field Surveys|field surveys]] and [[Remote Surveys|remote surveys]].<br />
* A comprehensive '''survey pilot''' includes 3 stages: a pre-pilot, a content-focused pilot, and a data-focused pilot. <br />
* The survey pilot is different from the '''field testing''' (practice interviews) that enumerators conduct at the end of [[Enumerator Training|enumerator training]]. The pilot should be complete before enumerator training begins. <br />
* [[Survey Pilot Participants|Participants of a survey pilot]] include the [[Impact Evaluation Team|research team]], '''interviewers''', and '''respondents'''.<br />
* Conduct the first two stages of the pilot before [[Questionnaire Programming|programming the questionnaire]]. Repeated programming is time-consuming and can create '''bugs''' (errors), for instance, by disturbing the order of questions.<br />
<br />
== Stages of a Survey Pilot ==<br />
A complete '''survey pilot''' is conducted over three stages - '''pre-pilot''', [[Piloting Survey Content|content-focused pilot]], and '''data-focused pilot'''. The table below discusses these stages in more detail.<br />
<br />
{| class="wikitable" style="margin-left: auto; margin-right: auto; border: none;"<br />
|-<br />
! style="height:50px; width:150px; text-align:center;" |<br />
! style="height:50px; width:250px; text-align:center;" | Stage 1: Pre-pilot<br />
! style="height:50px; width:250px; text-align:center;" | Stage 2: Content-focused pilot<br />
! style="height:50px; width:250px; text-align:center;" | Stage 3: Data-focused pilot<br />
|-<br />
| style="text-align:center;" | '''Objective'''<br />
| style="vertical-align:top;"| Answer broad questions about '''qualitative factors''' like [[Questionnaire Design|survey design]] and the context for conducting the study. Discuss these with the concerned teams.<br />
| style="vertical-align:top;"| Refine the order and wording of specific questions, the overall structure of the questionnaire, and [[Questionnaire Translation|translations]]. <br />
<br />
<br />
<br />
Check that the answer choices are '''complete''', that is, they cover all possibilities. <br />
<br />
<br />
Check how long it takes to answer the questions. Also check how answers differ between respondents ('''response variance''')<br />
| style="vertical-align:top;"| Check if the [[Questionnaire Programming|programmed instrument]] displays questions in the correct order, and follows the correct patterns (such as a group of questions which will always appear together, or will repeat).<br />
<br />
Load a sample data set (based on '''pilot interviews'''). Check if the data set has any missing fields. Perform all data quality checks, like [[Back Checks|back checks]] and [[Monitoring Data Quality#High Frequency Checks|high frequency checks]].<br />
|-<br />
| style="height:80px;text-align:center;" | '''Status of survey instrument'''<br />
| style="vertical-align:center;"| Early, printable version of the draft, and notes for further discussion.<br />
| style="vertical-align:center;"| A translated, printable draft. Ready to be '''programmed'''.<br />
| style="vertical-align:center;"| A translated, programmed, final draft. Ready for starting data collection.<br />
|-<br />
| style="height:80px;text-align:center;" | '''Mode'''<br />
| style="text-align:center;"| Pen-and-paper<br />
| style="text-align:center;"| Pen-and-paper<br />
| style="text-align:center;"| Tablet/phone<br />
|}<br />
<br />
However, note that all three stages may not be necessary for every survey. The research team has to determine which stage to begin from on a case-to-case basis.<br />
* If the research team is using a brand new survey instrument, then they must start with Stage 1, the '''pre-pilot.'''<br />
* If the survey instrument is an adaptation of (based on) a different instrument which was used for a previous data collection for another project in the same region (province or country), the research team can start with Stage 2, the '''content-focused pilot'''. However, they must make sure that the survey instrument does not have any design issues and was shared by a reliable source.<br />
* If the survey instrument is an adaptation of an instrument used for a previous data collection for the same project, but the research team had to make significant revisions or additions, even then the research team should start with Stage 2, the '''content-focused pilot'''.<br />
* Only if the research team is '''piloting''' a '''follow-up''' survey, and there are no major changes from the '''baseline''' (or first round) survey, then in this case the research team may skip directly to Stage 3, the '''data-focused pilot'''.<br />
<br />
== Pilot on paper first == <br />
The first two stages of the survey pilot (pre-pilot and content-focused pilot) are best done on pen-and-paper is done on paper, regardless of the planned survey mode ([[Pen-and-Paper_Personal_Interviews_(PAPI)|pen-and-paper personal interviews (PAPI)]],<br />
[[Computer-Assisted_Personal_Interviews_(CAPI)|computer-assisted personal interviews (CAPI)]], [[Remote_Surveys|phone surveys (CATI)]]). A pen-and-paper pilot allows members of the research team and interviewers flexibility when it comes to recording answers and '''qualitative''' observations. <br />
<br />
Piloting on paper is particularly helpful for the following:<br />
* '''Open-ended responses.''' pre-pilot responses help the research team to make the list of choices for a question more detailed and comprehensive. Pen-and-paper pilots allow the interviewers to make note of these.<br />
* '''Restructuring.''' Pen-and-paper pilots allow the interviewers to draw lines and arrows between questions, which makes it easier to restructure the '''instrument'''.<br />
* '''Observations.''' Pen-and-paper pilots allow interviewers to record observations and feedback of interviewers in the margins. This allows the research team to take note of issues, such as the wording or flow of questions.<br />
<br />
== Timeline ==<br />
'''Piloting''' should start 4-6 weeks before [[Field Surveys#Survey Launch|survey launch]]. In fact, starting a pilot early allows the research team to use the feedback from each stage to improve the [[Piloting Survey Content|survey instrument]] and [[Piloting Survey Protocols| survey protocols]]. This makes the process of implementing a survey more efficient. Therefore, the research team must draft a detailed [[Timeline of Survey Pilot|timeline for the pilot]] to allow enough time for discussions and revisions. Do not confuse the pilot with '''field testing''' (practice interviews) which is conducted at the end of [[Enumerator_Training |enumerator training]].<br />
<br />
== Structure ==<br />
In order to conduct a successful '''survey pilot''', the research team must decide on the [[Structuring a Survey Pilot|structure of the survey pilot]]. This involves agreeing upon the '''logistics''' for the following aspects of the pilot:<br />
* '''Duration of the pilot.''' The pilot should ideally go on until the research team agrees that there are no changes left to be made.<br />
* '''Approvals.''' Obtain all necessary approvals, for example, [[Survey Budget|budget]] approval from the organization funding the study. Sometimes the research team must obtain an [[IRB Approval|IRB approval]] even for a pilot. <br />
* '''Review sessions.''' Set aside time for feedback sessions and revisions. The research team should make sure everyone involved in the '''survey pilot''' gets a chance to express their concerns. Therefore, it is a good practice to draft guidelines on the schedule and frequency of review sessions.<br />
* '''Training.''' As part of [[Enumerator Training|enumerator training]], set aside time specifically for discussing best practices for conducting a '''survey pilot'''.<br />
<br />
== Participants ==<br />
Typically, a '''survey pilot''' is conducted before a [[Survey Firm|survey firm]] is brought on-board. Each of the [[Survey Pilot Participants|survey pilot participants]] plays an important role in the implementation of a successful pilot. <br />
* '''Respondents.''' They should be as close to the '''target population''' as possible. None of the respondents from a pilot should be in the sample for the actual survey.<br />
* '''Interviewers.''' They should be fluent in all local languages of the study area and the language of the research team. The size of the team of interviewers depends on the stage of pilot.<br />
* '''Field coordinator (FC).''' The [[Impact_Evaluation_Team#Field_Coordinator|field coordinators (FCs)]] play a central role as part of the research team. They are responsible for updating the research team on all monitoring and evaluation '''(M&E)''' activities in the field.<br />
* '''Principal investigator (PI).''' The [[Impact_Evaluation_Team#Principal_Investigator|principal investigators (PIs)]] oversee the overall functioning of the '''impact evaluation''' (study). They should brief the FCs regularly during the pilot.<br />
* '''Other research team members.'''Ideally, other members of the [[Impact Evaluation Team|impact evaluation team]] such as the [[Impact Evaluation Team#Impact Evaluation Coordinator|impact evaluation coordinator]], research manager, programmers, and translators should also participate.<br />
<br />
== Guidelines ==<br />
Finally, it is important to keep the following '''best practices''' in mind while planning and conducting a pilot:<br />
* Throughout the process of [[Questionnaire Design|designing the questionnaire]], discuss with other members of the research team, and take notes of what needs to be part of the pilot.<br />
* Take careful notes of the discussions and clarifications that come up during the pilot. These will be an important part of the [[Enumerator Training#Enumerator Manual|enumerator manual]] which is used during [[Enumerator Training|enumerator training]].<br />
* Use the '''data-focused pilot''' to test [[Back Checks|back check]] templates. <br />
* Hire a local '''mobilizer''' to coordinate with respondents. '''Mobilizers''' explain the purpose behind conducting the survey, and facilitate the process of obtaining consent. This is particularly helpful in urban areas, or in cases where respondents are busy. This can improve the outcomes of piloting, for instance, by reducing gap between surveys.<br />
<br />
[https://www.worldbank.org/en/research/dime/data-and-analytics DIME Analytics] has also created the following checklists to assist researchers and enumerators in preparing for, and implementing a pilot:<br />
*[[Preparing_for_the_survey_checklist|Checklist: Preparing for a survey pilot]]<br />
*[[Checklist: Refine the Questionnaire (Content)|Checklist: Refining questionnaire content]]<br />
*[[Checklist: Refine the Questionnaire (Data)|Checklist: Refining questionnaire data]]<br />
<br />
== Related Pages ==<br />
[[Special:WhatLinksHere/Survey_Pilot|Click here for pages that link to this topic.]]<br />
<br />
== Additional Resources == <br />
* DIME Analytics (World Bank), [https://osf.io/63uv9/ Survey pilot guidelines]<br />
* DIME Analytics (World Bank), [https://osf.io/rqb5m/ Survey instrument design and pilot]<br />
<br />
[[Category: Primary Data Collection]] [[Category: Questionnaire Design]]</div>Maria ruth joneshttps://dimewiki.worldbank.org/index.php?title=Survey_Pilot&diff=7452Survey Pilot2020-08-21T02:02:01Z<p>Maria ruth jones: /*Pilot on paper first */</p>
<hr />
<div>'''Survey pilot''' is the process of carrying out interviews and tests on different components of a survey, including [[Piloting Survey Content|content]] and [[Piloting Survey Protocols|protocols]]. A good '''pilot''' provides the research team with important feedback before they start the process of [[Primary Data Collection|data collection]]. This feedback can help the [[Impact Evaluation Team|research team]] review and improve [[Questionnaire Design|instrument design]], [[Questionnaire Translation|translations]], as well as [[Survey Protocols|survey protocols]] related to [[Piloting Survey Protocols#Interview scheduling|interview scheduling]], [[Sampling|sampling]], and [[Geo Spatial Data|geo data]].<br />
== Read First ==<br />
* A '''pilot''' can be conducted for both [[Field Surveys|field surveys]] and [[Remote Surveys|remote surveys]].<br />
* [[Survey Pilot Participants|Participants of a survey pilot]] include the [[Impact Evaluation Team|research team]], '''interviewers''', and '''respondents'''.<br />
* Note that a '''survey pilot''' is different from the '''field testing''' (practice interviews) that enumerators conduct at the end of [[Enumerator Training|enumerator training]]. <br />
* Conduct the pilot before moving on to '''enumerator training'''. <br />
* Conduct the pilot before [[Questionnaire Programming|programming the questionnaire]]. Repeated programming is time-consuming and can create '''bugs''' (errors), for instance, by disturbing the order of questions.<br />
<br />
== Stages of a Survey Pilot ==<br />
A complete '''survey pilot''' is conducted over three stages - '''pre-pilot''', [[Piloting Survey Content|content-focused pilot]], and '''data-focused pilot'''. The table below discusses these stages in more detail.<br />
<br />
{| class="wikitable" style="margin-left: auto; margin-right: auto; border: none;"<br />
|-<br />
! style="height:50px; width:150px; text-align:center;" |<br />
! style="height:50px; width:250px; text-align:center;" | Stage 1: Pre-pilot<br />
! style="height:50px; width:250px; text-align:center;" | Stage 2: Content-focused pilot<br />
! style="height:50px; width:250px; text-align:center;" | Stage 3: Data-focused pilot<br />
|-<br />
| style="text-align:center;" | '''Objective'''<br />
| style="vertical-align:top;"| Answer broad questions about '''qualitative factors''' like [[Questionnaire Design|survey design]] and the context for conducting the study. Discuss these with the concerned teams.<br />
| style="vertical-align:top;"| Refine the order and wording of specific questions, the overall structure of the questionnaire, and [[Questionnaire Translation|translations]]. <br />
<br />
<br />
<br />
Check that the answer choices are '''complete''', that is, they cover all possibilities. <br />
<br />
<br />
Check how long it takes to answer the questions. Also check how answers differ between respondents ('''response variance''')<br />
| style="vertical-align:top;"| Check if the [[Questionnaire Programming|programmed instrument]] displays questions in the correct order, and follows the correct patterns (such as a group of questions which will always appear together, or will repeat).<br />
<br />
Load a sample data set (based on '''pilot interviews'''). Check if the data set has any missing fields. Perform all data quality checks, like [[Back Checks|back checks]] and [[Monitoring Data Quality#High Frequency Checks|high frequency checks]].<br />
|-<br />
| style="height:80px;text-align:center;" | '''Status of survey instrument'''<br />
| style="vertical-align:center;"| Early, printable version of the draft, and notes for further discussion.<br />
| style="vertical-align:center;"| A translated, printable draft. Ready to be '''programmed'''.<br />
| style="vertical-align:center;"| A translated, programmed, final draft. Ready for starting data collection.<br />
|-<br />
| style="height:80px;text-align:center;" | '''Mode'''<br />
| style="text-align:center;"| Pen-and-paper<br />
| style="text-align:center;"| Pen-and-paper<br />
| style="text-align:center;"| Tablet/phone<br />
|}<br />
<br />
However, note that all three stages may not be necessary for every survey. The research team has to determine which stage to begin from on a case-to-case basis.<br />
* If the research team is using a brand new survey instrument, then they must start with Stage 1, the '''pre-pilot.'''<br />
* If the survey instrument is an adaptation of (based on) a different instrument which was used for a previous data collection for another project in the same region (province or country), the research team can start with Stage 2, the '''content-focused pilot'''. However, they must make sure that the survey instrument does not have any design issues and was shared by a reliable source.<br />
* If the survey instrument is an adaptation of an instrument used for a previous data collection for the same project, but the research team had to make significant revisions or additions, even then the research team should start with Stage 2, the '''content-focused pilot'''.<br />
* Only if the research team is '''piloting''' a '''follow-up''' survey, and there are no major changes from the '''baseline''' (or first round) survey, then in this case the research team may skip directly to Stage 3, the '''data-focused pilot'''.<br />
<br />
== Pilot on paper first == <br />
The first two stages of the survey pilot (pre-pilot and content-focused pilot) are best done on pen-and-paper is done on paper, regardless of the planned survey mode ([[Pen-and-Paper_Personal_Interviews_(PAPI)|pen-and-paper personal interviews (PAPI)]],<br />
[[Computer-Assisted_Personal_Interviews_(CAPI)|computer-assisted personal interviews (CAPI)]], [[Remote_Surveys|phone surveys (CATI)]]). A pen-and-paper pilot allows members of the research team and interviewers flexibility when it comes to recording answers and '''qualitative''' observations. <br />
<br />
Piloting on paper is particularly helpful for the following:<br />
* '''Open-ended responses.''' pre-pilot responses help the research team to make the list of choices for a question more detailed and comprehensive. Pen-and-paper pilots allow the interviewers to make note of these.<br />
* '''Restructuring.''' Pen-and-paper pilots allow the interviewers to draw lines and arrows between questions, which makes it easier to restructure the '''instrument'''.<br />
* '''Observations.''' Pen-and-paper pilots allow interviewers to record observations and feedback of interviewers in the margins. This allows the research team to take note of issues, such as the wording or flow of questions.<br />
<br />
== Timeline ==<br />
'''Piloting''' should start 4-6 weeks before [[Field Surveys#Survey Launch|survey launch]]. In fact, starting a pilot early allows the research team to use the feedback from each stage to improve the [[Piloting Survey Content|survey instrument]] and [[Piloting Survey Protocols| survey protocols]]. This makes the process of implementing a survey more efficient. Therefore, the research team must draft a detailed [[Timeline of Survey Pilot|timeline for the pilot]] to allow enough time for discussions and revisions. Do not confuse the pilot with '''field testing''' (practice interviews) which is conducted at the end of [[Enumerator_Training |enumerator training]].<br />
<br />
== Structure ==<br />
In order to conduct a successful '''survey pilot''', the research team must decide on the [[Structuring a Survey Pilot|structure of the survey pilot]]. This involves agreeing upon the '''logistics''' for the following aspects of the pilot:<br />
* '''Duration of the pilot.''' The pilot should ideally go on until the research team agrees that there are no changes left to be made.<br />
* '''Approvals.''' Obtain all necessary approvals, for example, [[Survey Budget|budget]] approval from the organization funding the study. Sometimes the research team must obtain an [[IRB Approval|IRB approval]] even for a pilot. <br />
* '''Review sessions.''' Set aside time for feedback sessions and revisions. The research team should make sure everyone involved in the '''survey pilot''' gets a chance to express their concerns. Therefore, it is a good practice to draft guidelines on the schedule and frequency of review sessions.<br />
* '''Training.''' As part of [[Enumerator Training|enumerator training]], set aside time specifically for discussing best practices for conducting a '''survey pilot'''.<br />
<br />
== Participants ==<br />
Typically, a '''survey pilot''' is conducted before a [[Survey Firm|survey firm]] is brought on-board. Each of the [[Survey Pilot Participants|survey pilot participants]] plays an important role in the implementation of a successful pilot. <br />
* '''Respondents.''' They should be as close to the '''target population''' as possible. None of the respondents from a pilot should be in the sample for the actual survey.<br />
* '''Interviewers.''' They should be fluent in all local languages of the study area and the language of the research team. The size of the team of interviewers depends on the stage of pilot.<br />
* '''Field coordinator (FC).''' The [[Impact_Evaluation_Team#Field_Coordinator|field coordinators (FCs)]] play a central role as part of the research team. They are responsible for updating the research team on all monitoring and evaluation '''(M&E)''' activities in the field.<br />
* '''Principal investigator (PI).''' The [[Impact_Evaluation_Team#Principal_Investigator|principal investigators (PIs)]] oversee the overall functioning of the '''impact evaluation''' (study). They should brief the FCs regularly during the pilot.<br />
* '''Other research team members.'''Ideally, other members of the [[Impact Evaluation Team|impact evaluation team]] such as the [[Impact Evaluation Team#Impact Evaluation Coordinator|impact evaluation coordinator]], research manager, programmers, and translators should also participate.<br />
<br />
== Guidelines ==<br />
Finally, it is important to keep the following '''best practices''' in mind while planning and conducting a pilot:<br />
* Throughout the process of [[Questionnaire Design|designing the questionnaire]], discuss with other members of the research team, and take notes of what needs to be part of the pilot.<br />
* Take careful notes of the discussions and clarifications that come up during the pilot. These will be an important part of the [[Enumerator Training#Enumerator Manual|enumerator manual]] which is used during [[Enumerator Training|enumerator training]].<br />
* Use the '''data-focused pilot''' to test [[Back Checks|back check]] templates. <br />
* Hire a local '''mobilizer''' to coordinate with respondents. '''Mobilizers''' explain the purpose behind conducting the survey, and facilitate the process of obtaining consent. This is particularly helpful in urban areas, or in cases where respondents are busy. This can improve the outcomes of piloting, for instance, by reducing gap between surveys.<br />
<br />
[https://www.worldbank.org/en/research/dime/data-and-analytics DIME Analytics] has also created the following checklists to assist researchers and enumerators in preparing for, and implementing a pilot:<br />
*[[Preparing_for_the_survey_checklist|Checklist: Preparing for a survey pilot]]<br />
*[[Checklist: Refine the Questionnaire (Content)|Checklist: Refining questionnaire content]]<br />
*[[Checklist: Refine the Questionnaire (Data)|Checklist: Refining questionnaire data]]<br />
<br />
== Related Pages ==<br />
[[Special:WhatLinksHere/Survey_Pilot|Click here for pages that link to this topic.]]<br />
<br />
== Additional Resources == <br />
* DIME Analytics (World Bank), [https://osf.io/63uv9/ Survey pilot guidelines]<br />
* DIME Analytics (World Bank), [https://osf.io/rqb5m/ Survey instrument design and pilot]<br />
<br />
[[Category: Primary Data Collection]] [[Category: Questionnaire Design]]</div>Maria ruth joneshttps://dimewiki.worldbank.org/index.php?title=Survey_Pilot&diff=7451Survey Pilot2020-08-20T19:40:55Z<p>Maria ruth jones: /* Pre-Pilots */</p>
<hr />
<div>'''Survey pilot''' is the process of carrying out interviews and tests on different components of a survey, including [[Piloting Survey Content|content]] and [[Piloting Survey Protocols|protocols]]. A good '''pilot''' provides the research team with important feedback before they start the process of [[Primary Data Collection|data collection]]. This feedback can help the [[Impact Evaluation Team|research team]] review and improve [[Questionnaire Design|instrument design]], [[Questionnaire Translation|translations]], as well as [[Survey Protocols|survey protocols]] related to [[Piloting Survey Protocols#Interview scheduling|interview scheduling]], [[Sampling|sampling]], and [[Geo Spatial Data|geo data]].<br />
== Read First ==<br />
* A '''pilot''' can be conducted for both [[Field Surveys|field surveys]] and [[Remote Surveys|remote surveys]].<br />
* [[Survey Pilot Participants|Participants of a survey pilot]] include the [[Impact Evaluation Team|research team]], '''interviewers''', and '''respondents'''.<br />
* Note that a '''survey pilot''' is different from the '''field testing''' (practice interviews) that enumerators conduct at the end of [[Enumerator Training|enumerator training]]. <br />
* Conduct the pilot before moving on to '''enumerator training'''. <br />
* Conduct the pilot before [[Questionnaire Programming|programming the questionnaire]]. Repeated programming is time-consuming and can create '''bugs''' (errors), for instance, by disturbing the order of questions.<br />
<br />
== Stages of a Survey Pilot ==<br />
A complete '''survey pilot''' is conducted over three stages - '''pre-pilot''', [[Piloting Survey Content|content-focused pilot]], and '''data-focused pilot'''. The table below discusses these stages in more detail.<br />
<br />
{| class="wikitable" style="margin-left: auto; margin-right: auto; border: none;"<br />
|-<br />
! style="height:50px; width:150px; text-align:center;" |<br />
! style="height:50px; width:250px; text-align:center;" | Stage 1: Pre-pilot<br />
! style="height:50px; width:250px; text-align:center;" | Stage 2: Content-focused pilot<br />
! style="height:50px; width:250px; text-align:center;" | Stage 3: Data-focused pilot<br />
|-<br />
| style="text-align:center;" | '''Objective'''<br />
| style="vertical-align:top;"| Answer broad questions about '''qualitative factors''' like [[Questionnaire Design|survey design]] and the context for conducting the study. Discuss these with the concerned teams.<br />
| style="vertical-align:top;"| Refine the order and wording of specific questions, the overall structure of the questionnaire, and [[Questionnaire Translation|translations]]. <br />
<br />
<br />
<br />
Check that the answer choices are '''complete''', that is, they cover all possibilities. <br />
<br />
<br />
Check how long it takes to answer the questions. Also check how answers differ between respondents ('''response variance''')<br />
| style="vertical-align:top;"| Check if the [[Questionnaire Programming|programmed instrument]] displays questions in the correct order, and follows the correct patterns (such as a group of questions which will always appear together, or will repeat).<br />
<br />
Load a sample data set (based on '''pilot interviews'''). Check if the data set has any missing fields. Perform all data quality checks, like [[Back Checks|back checks]] and [[Monitoring Data Quality#High Frequency Checks|high frequency checks]].<br />
|-<br />
| style="height:80px;text-align:center;" | '''Status of survey instrument'''<br />
| style="vertical-align:center;"| Early, printable version of the draft, and notes for further discussion.<br />
| style="vertical-align:center;"| A translated, printable draft. Ready to be '''programmed'''.<br />
| style="vertical-align:center;"| A translated, programmed, final draft. Ready for starting data collection.<br />
|-<br />
| style="height:80px;text-align:center;" | '''Mode'''<br />
| style="text-align:center;"| Pen-and-paper<br />
| style="text-align:center;"| Pen-and-paper<br />
| style="text-align:center;"| Tablet/phone<br />
|}<br />
<br />
However, note that all three stages may not be necessary for every survey. The research team has to determine which stage to begin from on a case-to-case basis.<br />
* If the research team is using a brand new survey instrument, then they must start with Stage 1, the '''pre-pilot.'''<br />
* If the survey instrument is an adaptation of (based on) a different instrument which was used for a previous data collection for another project in the same region (province or country), the research team can start with Stage 2, the '''content-focused pilot'''. However, they must make sure that the survey instrument does not have any design issues and was shared by a reliable source.<br />
* If the survey instrument is an adaptation of an instrument used for a previous data collection for the same project, but the research team had to make significant revisions or additions, even then the research team should start with Stage 2, the '''content-focused pilot'''.<br />
* Only if the research team is '''piloting''' a '''follow-up''' survey, and there are no major changes from the '''baseline''' (or first round) survey, then in this case the research team may skip directly to Stage 3, the '''data-focused pilot'''.<br />
<br />
== Pre-Pilots == <br />
The pre-pilot is the first stage of piloting and is particularly important when developing new survey instruments. The pre-pilot is done on paper (rather than electronically), to allow members of the research team and interviewers flexibility when it comes to recording answers and '''qualitative''' observations. Pen-and-paper pre-pilots are useful regardless of the ultimate survey mode ([[Pen-and-Paper_Personal_Interviews_(PAPI)|pen-and-paper personal interviews (PAPI)]],<br />
[[Computer-Assisted_Personal_Interviews_(CAPI)|computer-assisted personal interviews (CAPI)]], [[Remote_Surveys|phone surveys (CATI)]]). The pre-pilot is particularly helpful for the following:<br />
* '''Open-ended responses.''' pre-pilot responses help the research team to make the list of choices for a question more detailed and comprehensive. Pen-and-paper pilots allow the interviewers to make note of these.<br />
* '''Restructuring.''' Pen-and-paper pilots allow the interviewers to draw lines and arrows between questions, which makes it easier to restructure the '''instrument'''.<br />
* '''Observations.''' Pen-and-paper pilots allow interviewers to record observations and feedback of interviewers in the margins. This allows the research team to take note of issues, such as the wording or flow of questions.<br />
<br />
== Timeline ==<br />
'''Piloting''' should start 4-6 weeks before [[Field Surveys#Survey Launch|survey launch]]. In fact, starting a pilot early allows the research team to use the feedback from each stage to improve the [[Piloting Survey Content|survey instrument]] and [[Piloting Survey Protocols| survey protocols]]. This makes the process of implementing a survey more efficient. Therefore, the research team must draft a detailed [[Timeline of Survey Pilot|timeline for the pilot]] to allow enough time for discussions and revisions. Do not confuse the pilot with '''field testing''' (practice interviews) which is conducted at the end of [[Enumerator_Training |enumerator training]].<br />
<br />
== Structure ==<br />
In order to conduct a successful '''survey pilot''', the research team must decide on the [[Structuring a Survey Pilot|structure of the survey pilot]]. This involves agreeing upon the '''logistics''' for the following aspects of the pilot:<br />
* '''Duration of the pilot.''' The pilot should ideally go on until the research team agrees that there are no changes left to be made.<br />
* '''Approvals.''' Obtain all necessary approvals, for example, [[Survey Budget|budget]] approval from the organization funding the study. Sometimes the research team must obtain an [[IRB Approval|IRB approval]] even for a pilot. <br />
* '''Review sessions.''' Set aside time for feedback sessions and revisions. The research team should make sure everyone involved in the '''survey pilot''' gets a chance to express their concerns. Therefore, it is a good practice to draft guidelines on the schedule and frequency of review sessions.<br />
* '''Training.''' As part of [[Enumerator Training|enumerator training]], set aside time specifically for discussing best practices for conducting a '''survey pilot'''.<br />
<br />
== Participants ==<br />
Typically, a '''survey pilot''' is conducted before a [[Survey Firm|survey firm]] is brought on-board. Each of the [[Survey Pilot Participants|survey pilot participants]] plays an important role in the implementation of a successful pilot. <br />
* '''Respondents.''' They should be as close to the '''target population''' as possible. None of the respondents from a pilot should be in the sample for the actual survey.<br />
* '''Interviewers.''' They should be fluent in all local languages of the study area and the language of the research team. The size of the team of interviewers depends on the stage of pilot.<br />
* '''Field coordinator (FC).''' The [[Impact_Evaluation_Team#Field_Coordinator|field coordinators (FCs)]] play a central role as part of the research team. They are responsible for updating the research team on all monitoring and evaluation '''(M&E)''' activities in the field.<br />
* '''Principal investigator (PI).''' The [[Impact_Evaluation_Team#Principal_Investigator|principal investigators (PIs)]] oversee the overall functioning of the '''impact evaluation''' (study). They should brief the FCs regularly during the pilot.<br />
* '''Other research team members.'''Ideally, other members of the [[Impact Evaluation Team|impact evaluation team]] such as the [[Impact Evaluation Team#Impact Evaluation Coordinator|impact evaluation coordinator]], research manager, programmers, and translators should also participate.<br />
<br />
== Guidelines ==<br />
Finally, it is important to keep the following '''best practices''' in mind while planning and conducting a pilot:<br />
* Throughout the process of [[Questionnaire Design|designing the questionnaire]], discuss with other members of the research team, and take notes of what needs to be part of the pilot.<br />
* Take careful notes of the discussions and clarifications that come up during the pilot. These will be an important part of the [[Enumerator Training#Enumerator Manual|enumerator manual]] which is used during [[Enumerator Training|enumerator training]].<br />
* Use the '''data-focused pilot''' to test [[Back Checks|back check]] templates. <br />
* Hire a local '''mobilizer''' to coordinate with respondents. '''Mobilizers''' explain the purpose behind conducting the survey, and facilitate the process of obtaining consent. This is particularly helpful in urban areas, or in cases where respondents are busy. This can improve the outcomes of piloting, for instance, by reducing gap between surveys.<br />
<br />
[https://www.worldbank.org/en/research/dime/data-and-analytics DIME Analytics] has also created the following checklists to assist researchers and enumerators in preparing for, and implementing a pilot:<br />
*[[Preparing_for_the_survey_checklist|Checklist: Preparing for a survey pilot]]<br />
*[[Checklist: Refine the Questionnaire (Content)|Checklist: Refining questionnaire content]]<br />
*[[Checklist: Refine the Questionnaire (Data)|Checklist: Refining questionnaire data]]<br />
<br />
== Related Pages ==<br />
[[Special:WhatLinksHere/Survey_Pilot|Click here for pages that link to this topic.]]<br />
<br />
== Additional Resources == <br />
* DIME Analytics (World Bank), [https://osf.io/63uv9/ Survey pilot guidelines]<br />
* DIME Analytics (World Bank), [https://osf.io/rqb5m/ Survey instrument design and pilot]<br />
<br />
[[Category: Primary Data Collection]] [[Category: Questionnaire Design]]</div>Maria ruth joneshttps://dimewiki.worldbank.org/index.php?title=Survey_Pilot&diff=7449Survey Pilot2020-08-20T19:24:26Z<p>Maria ruth jones: /* Pre-pilots */</p>
<hr />
<div>'''Survey pilot''' is the process of carrying out interviews and tests on different components of a survey, including [[Piloting Survey Content|content]] and [[Piloting Survey Protocols|protocols]]. A good '''pilot''' provides the research team with important feedback before they start the process of [[Primary Data Collection|data collection]]. This feedback can help the [[Impact Evaluation Team|research team]] review and improve [[Questionnaire Design|instrument design]], [[Questionnaire Translation|translations]], as well as [[Survey Protocols|survey protocols]] related to [[Piloting Survey Protocols#Interview scheduling|interview scheduling]], [[Sampling|sampling]], and [[Geo Spatial Data|geo data]].<br />
== Read First ==<br />
* A '''pilot''' can be conducted for both [[Field Surveys|field surveys]] and [[Remote Surveys|remote surveys]].<br />
* [[Survey Pilot Participants|Participants of a survey pilot]] include the [[Impact Evaluation Team|research team]], '''interviewers''', and '''respondents'''.<br />
* Note that a '''survey pilot''' is different from the '''field testing''' (practice interviews) that enumerators conduct at the end of [[Enumerator Training|enumerator training]]. <br />
* Conduct the pilot before moving on to '''enumerator training'''. <br />
* Conduct the pilot before [[Questionnaire Programming|programming the questionnaire]]. Repeated programming is time-consuming and can create '''bugs''' (errors), for instance, by disturbing the order of questions.<br />
<br />
== Stages of a Survey Pilot ==<br />
A complete '''survey pilot''' is conducted over three stages - '''pre-pilot''', [[Piloting Survey Content|content-focused pilot]], and '''data-focused pilot'''. The table below discusses these stages in more detail.<br />
<br />
{| class="wikitable" style="margin-left: auto; margin-right: auto; border: none;"<br />
|-<br />
! style="height:50px; width:150px; text-align:center;" |<br />
! style="height:50px; width:250px; text-align:center;" | Stage 1: Pre-pilot<br />
! style="height:50px; width:250px; text-align:center;" | Stage 2: Content-focused pilot<br />
! style="height:50px; width:250px; text-align:center;" | Stage 3: Data-focused pilot<br />
|-<br />
| style="text-align:center;" | '''Objective'''<br />
| style="vertical-align:top;"| Answer broad questions about '''qualitative factors''' like [[Questionnaire Design|survey design]] and the context for conducting the study. Discuss these with the concerned teams.<br />
| style="vertical-align:top;"| Refine the order and wording of specific questions, the overall structure of the questionnaire, and [[Questionnaire Translation|translations]]. <br />
<br />
<br />
<br />
Check that the answer choices are '''complete''', that is, they cover all possibilities. <br />
<br />
<br />
Check how long it takes to answer the questions. Also check how answers differ between respondents ('''response variance''')<br />
| style="vertical-align:top;"| Check if the [[Questionnaire Programming|programmed instrument]] displays questions in the correct order, and follows the correct patterns (such as a group of questions which will always appear together, or will repeat).<br />
<br />
Load a sample data set (based on '''pilot interviews'''). Check if the data set has any missing fields. Perform all data quality checks, like [[Back Checks|back checks]] and [[Monitoring Data Quality#High Frequency Checks|high frequency checks]].<br />
|-<br />
| style="height:80px;text-align:center;" | '''Status of survey instrument'''<br />
| style="vertical-align:center;"| Early, printable version of the draft, and notes for further discussion.<br />
| style="vertical-align:center;"| A translated, printable draft. Ready to be '''programmed'''.<br />
| style="vertical-align:center;"| A translated, programmed, final draft. Ready for starting data collection.<br />
|-<br />
| style="height:80px;text-align:center;" | '''Mode'''<br />
| style="text-align:center;"| Pen-and-paper<br />
| style="text-align:center;"| Pen-and-paper<br />
| style="text-align:center;"| Tablet/phone<br />
|}<br />
<br />
However, note that all three stages may not be necessary for every survey. The research team has to determine which stage to begin from on a case-to-case basis.<br />
* If the research team is using a brand new survey instrument, then they must start with Stage 1, the '''pre-pilot.'''<br />
* If the survey instrument is an adaptation of (based on) a different instrument which was used for a previous data collection for another project in the same region (province or country), the research team can start with Stage 2, the '''content-focused pilot'''. However, they must make sure that the survey instrument does not have any design issues and was shared by a reliable source.<br />
* If the survey instrument is an adaptation of an instrument used for a previous data collection for the same project, but the research team had to make significant revisions or additions, even then the research team should start with Stage 2, the '''content-focused pilot'''.<br />
* Only if the research team is '''piloting''' a '''follow-up''' survey, and there are no major changes from the '''baseline''' (or first round) survey, then in this case the research team may skip directly to Stage 3, the '''data-focused pilot'''.<br />
<br />
== Pre-Pilots == <br />
The pre-pilot is the first stage of piloting and is particularly important when developing new survey instruments. The pre-pilot is done on paper (rather than electronically), to allow members of the research team and interviewers flexibility when it comes to recording answers and '''qualitative''' observations. Pen-and-paper pre-pilots are useful regardless of the ultimate survey mode([[Pen-and-Paper_Personal_Interviews_(PAPI)|pen-and-paper personal interviews (PAPI)]], <br />
[[Computer-Assisted Personal Interviews (CAPI)|computer-assisted personal interviews (CAPI)]], [[Remote Surveys#Phone Surveys (CATI)|phone surveys (CATI)]]. The pre-pilot is particularly helpful for the following:<br />
* '''Open-ended responses.''' pre-pilot responses help the research team to make the list of choices for a question more detailed and comprehensive. Pen-and-paper pilots allow the interviewers to make note of these.<br />
* '''Restructuring.''' Pen-and-paper pilots allow the interviewers to draw lines and arrows between questions, which makes it easier to restructure the '''instrument'''.<br />
* '''Observations.''' Pen-and-paper pilots allow interviewers to record observations and feedback of interviewers in the margins. This allows the research team to take note of issues, such as the wording or flow of questions.<br />
<br />
== Timeline ==<br />
'''Piloting''' should start 4-6 weeks before [[Field Surveys#Survey Launch|survey launch]]. In fact, starting a pilot early allows the research team to use the feedback from each stage to improve the [[Piloting Survey Content|survey instrument]] and [[Piloting Survey Protocols| survey protocols]]. This makes the process of implementing a survey more efficient. Therefore, the research team must draft a detailed [[Timeline of Survey Pilot|timeline for the pilot]] to allow enough time for discussions and revisions. Do not confuse the pilot with '''field testing''' (practice interviews) which is conducted at the end of [[Enumerator_Training |enumerator training]].<br />
<br />
== Structure ==<br />
In order to conduct a successful '''survey pilot''', the research team must decide on the [[Structuring a Survey Pilot|structure of the survey pilot]]. This involves agreeing upon the '''logistics''' for the following aspects of the pilot:<br />
* '''Duration of the pilot.''' The pilot should ideally go on until the research team agrees that there are no changes left to be made.<br />
* '''Approvals.''' Obtain all necessary approvals, for example, [[Survey Budget|budget]] approval from the organization funding the study. Sometimes the research team must obtain an [[IRB Approval|IRB approval]] even for a pilot. <br />
* '''Review sessions.''' Set aside time for feedback sessions and revisions. The research team should make sure everyone involved in the '''survey pilot''' gets a chance to express their concerns. Therefore, it is a good practice to draft guidelines on the schedule and frequency of review sessions.<br />
* '''Training.''' As part of [[Enumerator Training|enumerator training]], set aside time specifically for discussing best practices for conducting a '''survey pilot'''.<br />
<br />
== Participants ==<br />
Typically, a '''survey pilot''' is conducted before a [[Survey Firm|survey firm]] is brought on-board. Each of the [[Survey Pilot Participants|survey pilot participants]] plays an important role in the implementation of a successful pilot. <br />
* '''Respondents.''' They should be as close to the '''target population''' as possible. None of the respondents from a pilot should be in the sample for the actual survey.<br />
* '''Interviewers.''' They should be fluent in all local languages of the study area and the language of the research team. The size of the team of interviewers depends on the stage of pilot.<br />
* '''Field coordinator (FC).''' The [[Impact_Evaluation_Team#Field_Coordinator|field coordinators (FCs)]] play a central role as part of the research team. They are responsible for updating the research team on all monitoring and evaluation '''(M&E)''' activities in the field.<br />
* '''Principal investigator (PI).''' The [[Impact_Evaluation_Team#Principal_Investigator|principal investigators (PIs)]] oversee the overall functioning of the '''impact evaluation''' (study). They should brief the FCs regularly during the pilot.<br />
* '''Other research team members.'''Ideally, other members of the [[Impact Evaluation Team|impact evaluation team]] such as the [[Impact Evaluation Team#Impact Evaluation Coordinator|impact evaluation coordinator]], research manager, programmers, and translators should also participate.<br />
<br />
== Guidelines ==<br />
Finally, it is important to keep the following '''best practices''' in mind while planning and conducting a pilot:<br />
* Throughout the process of [[Questionnaire Design|designing the questionnaire]], discuss with other members of the research team, and take notes of what needs to be part of the pilot.<br />
* Take careful notes of the discussions and clarifications that come up during the pilot. These will be an important part of the [[Enumerator Training#Enumerator Manual|enumerator manual]] which is used during [[Enumerator Training|enumerator training]].<br />
* Use the '''data-focused pilot''' to test [[Back Checks|back check]] templates. <br />
* Hire a local '''mobilizer''' to coordinate with respondents. '''Mobilizers''' explain the purpose behind conducting the survey, and facilitate the process of obtaining consent. This is particularly helpful in urban areas, or in cases where respondents are busy. This can improve the outcomes of piloting, for instance, by reducing gap between surveys.<br />
<br />
[https://www.worldbank.org/en/research/dime/data-and-analytics DIME Analytics] has also created the following checklists to assist researchers and enumerators in preparing for, and implementing a pilot:<br />
*[[Preparing_for_the_survey_checklist|Checklist: Preparing for a survey pilot]]<br />
*[[Checklist: Refine the Questionnaire (Content)|Checklist: Refining questionnaire content]]<br />
*[[Checklist: Refine the Questionnaire (Data)|Checklist: Refining questionnaire data]]<br />
<br />
== Related Pages ==<br />
[[Special:WhatLinksHere/Survey_Pilot|Click here for pages that link to this topic.]]<br />
<br />
== Additional Resources == <br />
* DIME Analytics (World Bank), [https://osf.io/63uv9/ Survey pilot guidelines]<br />
* DIME Analytics (World Bank), [https://osf.io/rqb5m/ Survey instrument design and pilot]<br />
<br />
[[Category: Primary Data Collection]] [[Category: Questionnaire Design]]</div>Maria ruth joneshttps://dimewiki.worldbank.org/index.php?title=Naming_Conventions&diff=5606Naming Conventions2020-02-20T19:33:58Z<p>Maria ruth jones: Added content on file naming conventions for code-compatible files and a new reference.</p>
<hr />
<div><onlyinclude>Impact Evaluation projects should follow a clear file naming convention as many team members will need to understand and interact with files over the project lifetime. It is very important to use a naming convention that not only you understand but someone looking at the files after years also understands. </onlyinclude><br />
<br />
== Read First ==<br />
*Follow clear and consistent naming conventions for all files<br />
*Pay special attention to naming conventions for code-compatible files<br />
*Use version control softwares(such as Git/Github) instead of naming the folders _v01, _v02, old, new, etc. <br />
<br />
== Naming requirements for code files ==<br />
Files accessed by code have special naming requirements, since different software and operating systems read file names in different ways.<br />
<br />
=== Spaces ===<br />
Introducing spaces between words in a file name (including the folder path) can break a file's path when it's read by code, so while a Word document may be called ``2019-10-30 Sampling Procedure Description.docx'',<br />
a related code file would have a name like ``sampling-endline.do''.<br />
<br />
=== Timestamps ===<br />
Adding timestamps to binary files as in the example above can be useful, as it is not straightforward to track changes using version control software. <br />
For plaintext files version-controlled using Git, timestamps are an unnecessary distraction.<br />
Output tables, graphs, and documentations are an exception to this and it is good practice to date them. Instead of using versions, they should be dated for clarity. For example = "_2017June8" rather than "_v02".<br />
<br />
=== Capitalization ===<br />
Code-compatible files should never include capital letters, as strings and file paths are case-sensitive in some software. <br />
<br />
=== Sortability ===<br />
Use names that can be sorted easily, for example through alphabetization. The best names from a coding perspective are usually the opposite of those from an English perspective. For example, for a deidentified household dataset from the baseline round, you should prefer a name like ``baseline-household-deidentified.dta'', rather than the opposite way around as occurs in natural language. This ensures that all baseline data stays together, then all baseline-household data, and finally provides unique information about this specific file.<br />
<br />
==Version Control == <br />
<br />
It is generally recommended to use version control software such as Git/GitHub instead of naming the folders _v01, _v02, _old, _new, _final, etc. The only exception to this would be tables, graphs, and documentations and it is good practice to date those. For these kinds of files, it should be dated i.e. ''filename''_2017June08 rather than ''filename''_v02. <br />
<br />
=== Using Github ===<br />
<br />
Github is an excellent tool used for version control for documentation, and do files. It also allows users to collaborate on codes together making it easier for multiple people to work on the same code. More information on what Github does and how to get started with Github can be found [https://guides.github.com/activities/hello-world/ here].<br />
<br />
=== Using Box / Dropbox Version Control features ===<br />
<br />
Box and Dropbox are both file hosting/cloud storage services which provide version control features. <br />
<br />
'''''Important : ''''' It is important that you pay attention to your subscription of DropBox/Box when using them for version control as they only store previous versions for certain time. For example - Free Dropbox stores different versions of a files from the last 30 days. Similarly, Box does not allow users with free accounts to access versions of their files. <br />
<br />
More information on Dropbox's version control, here are some useful links on [https://www.dropbox.com/en/help/11 recovering older versions of a file], [https://www.dropbox.com/en/help/9114 version history information], and [https://www.dropbox.com/help/113 extended version history for Dropbox Pro users].<br />
<br />
More information on Box's version control, the number of histories saved, and tracking older versions can be found [https://community.box.com/t5/Managing-Your-Content/How-To-Track-Your-Files-and-File-Versions-Version-History/ta-p/329 here].<br />
<br />
<br />
== Back to Parent ==<br />
This article is part of the topic [[Data Management]]<br />
<br />
== Additional Resources ==<br />
* [https://www2.stat.duke.edu/~rcs46/lectures\_2015/01-markdown-git/slides/naming-slides/naming-slides.pdf| Guidelines on naming conventions] from Duke University <br />
<br />
[[Category: Data Management ]]</div>Maria ruth joneshttps://dimewiki.worldbank.org/index.php?title=Randomized_Control_Trials&diff=52Randomized Control Trials2016-12-13T23:26:46Z<p>Maria ruth jones: </p>
<hr />
<div>Randomized Control Trials (RCTs) are considered the 'gold standard' for impact evaluation.<br />
<br />
= RCT-related Topics =<br />
== Individual-level RCTs ==<br />
=== Examples of individual RCTs in the DIME Portfolio ===<br />
<br />
== Clustered RCTs ==<br />
=== Examples of clustered RCTs in the DIME Portfolio ===<br />
<br />
= External Resources =<br />
* Running Randomized Evaluations - Rachel Glennester [http://runningres.com/]<br />
<br />
= See Also = <br />
*[[Randomization in Stata]]<br />
*[[Randomization in Excel]]<br />
*[[Randomization in SurveyCTO]]<br />
<br />
= References =</div>Maria ruth joneshttps://dimewiki.worldbank.org/index.php?title=Randomized_Control_Trials&diff=51Randomized Control Trials2016-12-13T23:24:04Z<p>Maria ruth jones: </p>
<hr />
<div>Randomized Control Trials (RCTs) are considered the 'gold standard' for impact evaluation.<br />
<br />
= RCT-related Topics =<br />
== Individual-level RCTs ==<br />
<br />
== Clustered RCTs ==<br />
<br />
= External Resources =<br />
* Running Randomized Evaluations - Rachel Glennester [http://runningres.com/]<br />
<br />
= See Also = <br />
*[[Randomization in Stata]]<br />
*[[Randomization in Excel]]<br />
*[[Randomization in SurveyCTO]]<br />
<br />
= References =</div>Maria ruth joneshttps://dimewiki.worldbank.org/index.php?title=Randomized_Control_Trials&diff=50Randomized Control Trials2016-12-13T23:08:45Z<p>Maria ruth jones: </p>
<hr />
<div>Randomized Control Trials (RCTs) are considered the 'gold standard' for impact evaluation.<br />
<br />
= RCT-related Topics =<br />
== Individual-level RCTs ==<br />
<br />
== Clustered RCTs ==<br />
<br />
= External Resources =<br />
* [http://runningres.com/]<br />
<br />
= See Also = <br />
*[[Randomization in Stata]]<br />
*[[Randomization in Excel]]<br />
*[[Randomization in SurveyCTO]]<br />
<br />
= References =</div>Maria ruth joneshttps://dimewiki.worldbank.org/index.php?title=Randomized_Control_Trials&diff=49Randomized Control Trials2016-12-13T23:07:26Z<p>Maria ruth jones: </p>
<hr />
<div>Randomized Control Trials (RCTs) are considered the 'gold standard' for impact evaluation.<br />
<br />
= RCT-related Topics =<br />
== Individual-level RCTs ==<br />
<br />
== Clustered RCTs ==<br />
<br />
= External Resources =<br />
<br />
= See Also = <br />
*[[Randomization in Stata]]<br />
*[[Randomization in Excel]]<br />
*[[Randomization in SurveyCTO]]<br />
<br />
= References =</div>Maria ruth joneshttps://dimewiki.worldbank.org/index.php?title=Randomized_Control_Trials&diff=48Randomized Control Trials2016-12-13T23:07:04Z<p>Maria ruth jones: </p>
<hr />
<div>Randomized Control Trials (RCTs) are considered the 'gold standard' for impact evaluation.<br />
<br />
= RCT-related Topics =<br />
== Individual-level RCTs ==<br />
<br />
== Clustered RCTs ==<br />
<br />
= External Resources =<br />
<br />
= See Also = <br />
[[Randomization in Stata]]<br />
[[Randomization in Excel]]<br />
[[Randomization in SurveyCTO]]<br />
<br />
= References =</div>Maria ruth joneshttps://dimewiki.worldbank.org/index.php?title=Randomized_Control_Trials&diff=47Randomized Control Trials2016-12-13T23:06:07Z<p>Maria ruth jones: </p>
<hr />
<div>Randomized Control Trials (RCTs) are considered the 'gold standard' for impact evaluation.<br />
<br />
= RCT-related Topics =<br />
== Individual-level RCTs ==<br />
<br />
== Clustered RCTs ==<br />
<br />
= External Resources =<br />
<br />
= See Also = <br />
<br />
= References =</div>Maria ruth joneshttps://dimewiki.worldbank.org/index.php?title=Randomized_Control_Trials&diff=45Randomized Control Trials2016-12-13T23:03:13Z<p>Maria ruth jones: </p>
<hr />
<div>Randomized Control Trials (RCTs) are considered the 'gold standard' for impact evaluation.<br />
<br />
= Individual-level RCTs =<br />
<br />
= Clustered RCTs =<br />
<br />
= External Resources =<br />
<br />
= See Also = <br />
<br />
= References =</div>Maria ruth joneshttps://dimewiki.worldbank.org/index.php?title=Randomized_Control_Trials&diff=43Randomized Control Trials2016-12-13T23:02:02Z<p>Maria ruth jones: Created page with "= Overview = Randomized Control Trials (RCTs) are considered the 'gold standard' for impact evaluation. = Individual-level RCTs = = Clustered RCTs = = External Resources = ..."</p>
<hr />
<div>= Overview =<br />
Randomized Control Trials (RCTs) are considered the 'gold standard' for impact evaluation.<br />
<br />
= Individual-level RCTs =<br />
<br />
= Clustered RCTs =<br />
<br />
= External Resources =<br />
<br />
= See Also = <br />
<br />
= References =</div>Maria ruth jones