Difference between revisions of "Survey Firm TOR"
| (14 intermediate revisions by 4 users not shown) | |||
| Line 1: | Line 1: | ||
A [[Survey Firm|survey firm]] term of reference (TOR) defines the structure of the project and breaks down the responsibilities of all parties involved, including that of the [[Impact Evaluation Team|impact evaluation team]]. The TOR is an important document for [[Procuring a Survey Firm | survey firm procurement]] and ongoing monitoring of survey firm performance. This page suggests the scope of work for the survey firm and impact evaluation team, discusses deliverables, and provides an example excerpt from a survey firm TOR. | A [[Survey Firm|survey firm]] term of reference (TOR) defines the structure of the project and breaks down the responsibilities of all parties involved, including that of the [[Impact Evaluation Team|impact evaluation team]]. The TOR is an important document for [[Procuring a Survey Firm | survey firm procurement]] and ongoing monitoring of '''survey firm''' performance. This page suggests the scope of work for the '''survey firm''' and '''impact evaluation team''', discusses deliverables, and provides an example excerpt from a survey firm TOR. | ||
== Read First == | == Read First == | ||
*Prepare the TOR early in the firm procurement process. | *Prepare the TOR early in the [[Survey Firm Procurement|firm procurement]] process. | ||
*Within the TOR, clearly delineate tasks between the survey firm and the impact evaluation team; define deliverables carefully; and specify all expectations and consequences if the expectations are not met. | *Within the TOR, clearly delineate tasks between the [[Survey Firm|survey firm]] and the [[Impact Evaluation Team|impact evaluation team]]; define deliverables carefully; and specify all expectations and consequences if the expectations are not met. | ||
*Refer to the TOR throughout field work. Any changes to survey protocols or sampling that deviate from the TOR must be confirmed in writing. | *Refer to the TOR throughout field work. Any changes to [[Survey Protocols|survey protocols]] or [[Sampling|sampling]] that deviate from the TOR must be confirmed in writing. | ||
==Overview== | ==Overview== | ||
A detailed TOR is essential for smoothly running fieldwork. Prepare the TOR before publishing the | A detailed TOR is essential for smoothly running fieldwork. Prepare the TOR before publishing the Request for Expression of Interest (rEOI) for [[Survey Firm|survey firms]]; publish it after shortlisting [[Survey Fir|firms]] based on the expression of interest (EOI). After hiring the '''survey firm''', monitor and provide feedback early and often, using the TOR as an anchor for expectations. Any changes to [[Survey Protocols|survey protocols]] or [[Sampling|sampling]] that deviate from the TOR must be confirmed in writing. If there are cost implications, a contract modification may be required. | ||
== Defining the Scope of Work == | == Defining the Scope of Work == | ||
| Line 15: | Line 15: | ||
=== Survey Firm Tasks === | === Survey Firm Tasks === | ||
# Obtain necessary permits or clearance for the survey | # Obtain necessary permits or clearance for the [[Survey Pilot|survey]] | ||
# [[Questionnaire Translation|Translate]] and [[Survey Pilot|field test]] all questionnaires | # [[Questionnaire Translation|Translate]] and [[Survey Pilot|field test]] all [[Questionnaire Design|questionnaires]] | ||
# Create detailed field procedure plan | # Create detailed field procedure plan along with a timeline. | ||
# Recruit, [[Enumerator Training|train]], and contract experienced field staff | # Recruit, [[Enumerator Training|train]], and contract experienced field staff | ||
# Manage and oversee household survey data collection | # Manage and oversee household '''survey''' [[Primary Data Collection|data collection]] | ||
=== Impact Evaluation Team Tasks === | === Impact Evaluation Team Tasks === | ||
# [[Questionnaire Design|Design questionnaire]] | # [[Questionnaire Design|Design questionnaire]] | ||
# | # [[Survey Pilot | Manage survey pilot]] | ||
# [[Questionnaire Programming| Program]] | # [[Questionnaire Programming| Program questionnaire]] | ||
# [[Sampling | # [[Sampling | Sample]] | ||
# [[Monitoring Data Quality|Monitor]] | # [[Monitoring Data Quality|Monitor data quality]] | ||
# [[Data Cleaning| Clean]] | # [[Data Cleaning| Clean data]] | ||
# | # [[Data Analysis| Conduct data analysis]] | ||
The field coordinator should work closely with survey firm every step of the process. | The [[Impact Evaluation Team#Field Coordinators (FCs)|field coordinator]] should work closely with [[Survey Firm|survey firm]] every step of the process. | ||
== Defining the Deliverables == | == Defining the Deliverables == | ||
Review of deliverables is the formal and enforceable way to provide feedback to the survey firm. It is important to assess quality and provide feedback early and often. | Review of deliverables is the formal and enforceable way to provide feedback to the [[Survey Firm|survey firm]]. It is important to [[Monitoring Data Quality|assess quality]] and provide feedback early and often. | ||
Deliverables should be spaced throughout the contract rather than only at the end of the survey. Each major stage of the process should include a deliverable. Deliverables should be specified as a detailed list in the TOR, with intermediate outputs. Finally, deliverables should have clearly indicated quality standards. Consequences should be clearly outlined in the case that the firm does not meet standards. | Deliverables should be spaced throughout the contract rather than only at the end of the [[Survey Pilot|survey]]. Each major stage of the process should include a deliverable. Deliverables should be specified as a detailed list in the TOR, with intermediate outputs. Finally, deliverables should have clearly indicated quality standards. Consequences should be clearly outlined in the case that the firm does not meet standards. | ||
[[File:Example_Survey_Firm_Deliverables.png|Example List of Deliverables | 800px]] | [[File:Example_Survey_Firm_Deliverables.png|Example List of Deliverables | 800px]] | ||
== TOR Example Excerpt == | == TOR Example Excerpt == | ||
===Activity 5 : Household survey data collection=== | ===Activity 5 : Household survey data collection=== | ||
Train all enumerators, supervisors, and data manager on the administration of questionnaires provided by the research team. | [[Enumerator Training|Train all enumerators]], supervisors, and data manager on the administration of [[Questionnaire Design|questionnaires]] provided by the [[Impact Evaluation Team|research team]]. | ||
The training should also serve as a screening process for skilled interviewers. Consequently, the survey | The training should also serve as a screening process for skilled interviewers. Consequently, the [[Survey Firm|survey firm]] must recruit more interviewers for the training than will be ultimately hired for the project. Five '''enumerators''' should be included in the training as a reserve. | ||
The following components must be included in the training: | The following components must be included in the training: | ||
# Theoretical: training should include a review the theory of the questionnaire and each question in order for trainees to fully understand the objective of each question. Standard quantitative interviewing techniques and field protocols should also be covered. | # Theoretical: training should include a review the theory of the '''questionnaire''' and each question in order for trainees to fully understand the objective of each question. Standard quantitative interviewing techniques and field protocols should also be covered. | ||
#Classroom practice: training should include individual and group exercises for trainees to become familiar with the practice of asking questions and filling questionnaires. This part of the training may include in-class demonstrations, where the questionnaire is projected and one interviewer completes the | #Classroom practice: training should include individual and group exercises for trainees to become familiar with the practice of asking questions and filling '''questionnaires'''. This part of the training may include in-class demonstrations, where the '''questionnaire''' is projected and one interviewer completes the it in front of the classroom. The training may also use vignettes, where the company designs case scenarios based on typical households (perhaps those found during the supervisor training or [[Survey Pilot|piloting]]) and have interviewers complete the '''questionnaire''' based on the vignette. Finally, the trainees should conduct pilot interviews on the same subject, and have the interviewers complete a '''questionnaire''' for the interview to test consistency across interviewers. | ||
# Field practice: after the theoretical and classroom practice, the interviewers should go to the field to administer the full questionnaire to a small number of households (outside the study sample). The pre-test should not focus on major adjustments to the questionnaire, but rather simulate the administration of the questionnaire under normal circumstances. All field team members must demonstrate they clearly understand their roles and are correctly following the survey protocols. | # Field practice: after the theoretical and classroom practice, the interviewers should go to the field to administer the full '''questionnaire''' to a small number of households (outside the study [[Sampling|sample]]). The pre-test should not focus on major adjustments to the '''questionnaire''', but rather simulate the administration of the '''questionnaire''' under normal circumstances. All field team members must demonstrate they clearly understand their roles and are correctly following the [[Survey Protocols|survey protocols]]. | ||
# Evaluation: following the training, interviewers, supervisors and data managers should be evaluated based on their understanding of the questionnaire and their ability to correctly record data. | # Evaluation: following the training, interviewers, supervisors and data managers should be evaluated based on their understanding of the '''questionnaire''' and their ability to correctly record data. | ||
The training period will conclude only when the field teams have demonstrated mastery of the designated tasks. Decisions as to which field staff will take part in the data collection must be made on the basis of this evaluation. | The training period will conclude only when the field teams have demonstrated mastery of the designated tasks. Decisions as to which field staff will take part in the [[Primary Data Collection|data collection]] must be made on the basis of this evaluation. | ||
== Back to Parent == | == Back to Parent == | ||
| Line 58: | Line 59: | ||
== Additional Resources == | == Additional Resources == | ||
*DIME Analytics' [https://github.com/worldbank/DIME-Resources/blob/master/Survey%20firm%20Technical%20Review%20_%20Scoring%20Matrix.xlsx Survey Firm Technical Review Scoring Matrix] | |||
*DIME Analytics' [https://github.com/worldbank/DIME-Resources/blob/master/survey-firms.pdf Working with Survey Firms] | |||
[[Category: Primary Data Collection]] | [[Category: Primary Data Collection]] | ||
Latest revision as of 17:22, 27 June 2024
A survey firm term of reference (TOR) defines the structure of the project and breaks down the responsibilities of all parties involved, including that of the impact evaluation team. The TOR is an important document for survey firm procurement and ongoing monitoring of survey firm performance. This page suggests the scope of work for the survey firm and impact evaluation team, discusses deliverables, and provides an example excerpt from a survey firm TOR.
Read First
- Prepare the TOR early in the firm procurement process.
- Within the TOR, clearly delineate tasks between the survey firm and the impact evaluation team; define deliverables carefully; and specify all expectations and consequences if the expectations are not met.
- Refer to the TOR throughout field work. Any changes to survey protocols or sampling that deviate from the TOR must be confirmed in writing.
Overview
A detailed TOR is essential for smoothly running fieldwork. Prepare the TOR before publishing the Request for Expression of Interest (rEOI) for survey firms; publish it after shortlisting firms based on the expression of interest (EOI). After hiring the survey firm, monitor and provide feedback early and often, using the TOR as an anchor for expectations. Any changes to survey protocols or sampling that deviate from the TOR must be confirmed in writing. If there are cost implications, a contract modification may be required.
Defining the Scope of Work
Survey Firm Tasks
- Obtain necessary permits or clearance for the survey
- Translate and field test all questionnaires
- Create detailed field procedure plan along with a timeline.
- Recruit, train, and contract experienced field staff
- Manage and oversee household survey data collection
Impact Evaluation Team Tasks
- Design questionnaire
- Manage survey pilot
- Program questionnaire
- Sample
- Monitor data quality
- Clean data
- Conduct data analysis
The field coordinator should work closely with survey firm every step of the process.
Defining the Deliverables
Review of deliverables is the formal and enforceable way to provide feedback to the survey firm. It is important to assess quality and provide feedback early and often.
Deliverables should be spaced throughout the contract rather than only at the end of the survey. Each major stage of the process should include a deliverable. Deliverables should be specified as a detailed list in the TOR, with intermediate outputs. Finally, deliverables should have clearly indicated quality standards. Consequences should be clearly outlined in the case that the firm does not meet standards.
TOR Example Excerpt
Activity 5 : Household survey data collection
Train all enumerators, supervisors, and data manager on the administration of questionnaires provided by the research team. The training should also serve as a screening process for skilled interviewers. Consequently, the survey firm must recruit more interviewers for the training than will be ultimately hired for the project. Five enumerators should be included in the training as a reserve. The following components must be included in the training:
- Theoretical: training should include a review the theory of the questionnaire and each question in order for trainees to fully understand the objective of each question. Standard quantitative interviewing techniques and field protocols should also be covered.
- Classroom practice: training should include individual and group exercises for trainees to become familiar with the practice of asking questions and filling questionnaires. This part of the training may include in-class demonstrations, where the questionnaire is projected and one interviewer completes the it in front of the classroom. The training may also use vignettes, where the company designs case scenarios based on typical households (perhaps those found during the supervisor training or piloting) and have interviewers complete the questionnaire based on the vignette. Finally, the trainees should conduct pilot interviews on the same subject, and have the interviewers complete a questionnaire for the interview to test consistency across interviewers.
- Field practice: after the theoretical and classroom practice, the interviewers should go to the field to administer the full questionnaire to a small number of households (outside the study sample). The pre-test should not focus on major adjustments to the questionnaire, but rather simulate the administration of the questionnaire under normal circumstances. All field team members must demonstrate they clearly understand their roles and are correctly following the survey protocols.
- Evaluation: following the training, interviewers, supervisors and data managers should be evaluated based on their understanding of the questionnaire and their ability to correctly record data.
The training period will conclude only when the field teams have demonstrated mastery of the designated tasks. Decisions as to which field staff will take part in the data collection must be made on the basis of this evaluation.
Back to Parent
This article is part of the topic Primary Data Collection
Additional Resources
- DIME Analytics' Survey Firm Technical Review Scoring Matrix
- DIME Analytics' Working with Survey Firms