Dimewiki

Difference between revisions of "Survey Firm TOR"

Jump to: navigation, search
 
(24 intermediate revisions by 5 users not shown)
Line 1: Line 1:


A [[Survey Firm|survey firm]] term of reference (TOR) defines the structure of the project and breaks down the responsibilities of all parties involved, including that of the [[Impact Evaluation Team|impact evaluation team]]. The TOR is an important document for [[Procuring a Survey Firm | survey firm procurement]] and ongoing monitoring of '''survey firm''' performance. This page suggests the scope of work for the '''survey firm''' and '''impact evaluation team''', discusses deliverables, and provides an example excerpt from a survey firm TOR.


== Read First ==
== Read First ==
Detailed TORs are essential! Specify all expectations and consequences if expectations are not met. Then monitor, monitor, monitor! Provide feedback early and often. Note that any changes to survey protocols or sampling that deviate from the TOR must be confirmed in writing. If there are cost implications, a contract modification may be required.
*Prepare the TOR early in the [[Survey Firm Procurement|firm procurement]] process.
*Within the TOR, clearly delineate tasks between the [[Survey Firm|survey firm]] and the [[Impact Evaluation Team|impact evaluation team]]; define deliverables carefully; and specify all expectations and consequences if the expectations are not met.
*Refer to the TOR throughout field work. Any changes to [[Survey Protocols|survey protocols]] or [[Sampling|sampling]] that deviate from the TOR must be confirmed in writing.  


== Guidelines ==
==Overview==
=== Scope of Work ===


Tasks best done by [[Survey Firm]]
A detailed TOR is essential for smoothly running fieldwork. Prepare the TOR before publishing the Request for Expression of Interest (rEOI) for [[Survey Firm|survey firms]]; publish it after shortlisting [[Survey Fir|firms]] based on the expression of interest (EOI). After hiring the '''survey firm''', monitor and provide feedback early and often, using the TOR as an anchor for expectations. Any changes to [[Survey Protocols|survey protocols]] or [[Sampling|sampling]] that deviate from the TOR must be confirmed in writing. If there are cost implications, a contract modification may be required.
# Obtaining necessary permits or clearance for the survey
# Translation & Field Testing of all questionnaires
# Detailed Field Procedure Plan
# Recruitment, training, and contracting of experienced field staff
# Household survey data collection


== Defining the Scope of Work ==


Tasks best done by the [[Impact Evaluation Team]]
=== Survey Firm Tasks ===
# [[Questionnaire Design]]
# [[Survey Pilot]]
# [[Questionnaire Programming]]
# [[Sampling & Power Calculations|Sampling]]
# [[Monitoring Data Quality]]
# [[Data Cleaning]]
# [[Data Analysis]]


The Field Coordinator should work closely with survey firm every step of the process.
# Obtain necessary permits or clearance for the [[Survey Pilot|survey]]
# [[Questionnaire Translation|Translate]] and [[Survey Pilot|field test]] all [[Questionnaire Design|questionnaires]]
# Create detailed field procedure plan along with a timeline.
# Recruit, [[Enumerator Training|train]], and contract experienced field staff
# Manage and oversee household '''survey''' [[Primary Data Collection|data collection]]


=== Impact Evaluation Team Tasks ===


==== Example of recommended level of detail ====
# [[Questionnaire Design|Design questionnaire]]
'''Activity 5 : Household survey  data collection'''
# [[Survey Pilot | Manage survey pilot]]
Train all enumerators, supervisors, and data manager on the administration of questionnaires provided by the research team.  
# [[Questionnaire Programming| Program questionnaire]]
The training should also serve as a screening process for skilled interviewers. Consequently, the survey company must recruit more interviewers for the training than will be ultimately hired for the project. Five enumerators should be included in the training as a reserve.
# [[Sampling | Sample]]
# [[Monitoring Data Quality|Monitor data quality]]
# [[Data Cleaning| Clean data]]
# [[Data Analysis| Conduct data analysis]]
 
The [[Impact Evaluation Team#Field Coordinators (FCs)|field coordinator]] should work closely with [[Survey Firm|survey firm]] every step of the process.
 
== Defining the Deliverables ==
 
Review of deliverables is the formal and enforceable way to provide feedback to the [[Survey Firm|survey firm]]. It is important to [[Monitoring Data Quality|assess quality]] and provide feedback early and often.
 
Deliverables should be spaced throughout the contract rather than only at the end of the [[Survey Pilot|survey]]. Each major stage of the process should include a deliverable. Deliverables should be specified as a detailed list in the TOR, with intermediate outputs. Finally, deliverables should have clearly indicated quality standards. Consequences should be clearly outlined in the case that the firm does not meet standards.
 
[[File:Example_Survey_Firm_Deliverables.png|Example List of Deliverables | 800px]]
 
== TOR Example Excerpt ==
 
===Activity 5 : Household survey  data collection===
[[Enumerator Training|Train all enumerators]], supervisors, and data manager on the administration of [[Questionnaire Design|questionnaires]] provided by the [[Impact Evaluation Team|research team]].  
The training should also serve as a screening process for skilled interviewers. Consequently, the [[Survey Firm|survey firm]] must recruit more interviewers for the training than will be ultimately hired for the project. Five '''enumerators''' should be included in the training as a reserve.
The following components must be included in the training:
The following components must be included in the training:
* Theoretical: Training should include a review the theory of the questionnaire and each question in order for trainees to fully understand the objective of each question. Standard quantitative interviewing techniques and field protocols should also be covered.
* Classroom practice: Training should include individual and group exercises for trainees to become familiar with the practice of asking questions and filling questionnaires. This part of the training may include in-class demonstrations, where the questionnaire is projected and one interviewer completes the questionnaire in front of the classroom.  The training may also use vignettes, where the company designs case scenarios based on typical households (perhaps those found during the supervisor training or piloting) and have interviewers complete the questionnaire based on the vignette.  Finally, the trainees should conduct pilot interviews on the same subject, and have the interviewers complete a questionnaire for the interview to test consistency across interviewers. 
* Field practice: After the theoretical and classroom practice, the interviewers should go to the field to administer the full questionnaire to a small number of households (outside the study sample).  The pre-test should not focus on major adjustments to the questionnaire, but rather simulate the administration of the questionnaire under normal circumstances.  All field team members must demonstrate they clearly understand their roles and are correctly following the survey protocols.
* Evaluation: Following the training, interviewers, supervisors and data managers should be evaluated based on their understanding of the questionnaire and their ability to correctly record data.
The training period will conclude only when the field teams have demonstrated mastery of the designated tasks. Decisions as to which field staff will take part in the data collection must be made on the basis of this evaluation.
=== Deliverables ===
Review of deliverables is the formal (and enforceable) way to provide feedback to the firm. It is important to assess quality and provide feedback early and often.


Deliverables should be ...
# Theoretical: training should include a review the theory of the '''questionnaire''' and each question in order for trainees to fully understand the objective of each question. Standard quantitative interviewing techniques and field protocols should also be covered.
* Spaced throughout the contract (not only the final report at the end of the survey). Include a deliverable for each major stage of the process.   
#Classroom practice: training should include individual and group exercises for trainees to become familiar with the practice of asking questions and filling '''questionnaires'''. This part of the training may include in-class demonstrations, where the '''questionnaire''' is projected and one interviewer completes the it in front of the classroom.  The training may also use vignettes, where the company designs case scenarios based on typical households (perhaps those found during the supervisor training or [[Survey Pilot|piloting]]) and have interviewers complete the '''questionnaire''' based on the vignette.  Finally, the trainees should conduct pilot interviews on the same subject, and have the interviewers complete a '''questionnaire''' for the interview to test consistency across interviewers.
* Specified as a detailed list, with intermediate outputs
# Field practice: after the theoretical and classroom practice, the interviewers should go to the field to administer the full '''questionnaire''' to a small number of households (outside the study [[Sampling|sample]]). The pre-test should not focus on major adjustments to the '''questionnaire''', but rather simulate the administration of the '''questionnaire''' under normal circumstances.  All field team members must demonstrate they clearly understand their roles and are correctly following the [[Survey Protocols|survey protocols]].
* Clearly indicated quality standards and consequences for not meeting the standards
# Evaluation: following the training, interviewers, supervisors and data managers should be evaluated based on their understanding of the '''questionnaire''' and their ability to correctly record data.


[[Example_Survey_Firm_Deliverables.png|Example List of Deliverables]]
The training period will conclude only when the field teams have demonstrated mastery of the designated tasks. Decisions as to which field staff will take part in the [[Primary Data Collection|data collection]] must be made on the basis of this evaluation.


== Back to Parent ==
== Back to Parent ==
This article is part of the topic [[Survey Firm Procurement]]
This article is part of the topic [[Primary Data Collection]]


== Additional Resources ==
== Additional Resources ==
* list here other articles related to this topic, with a brief description and link
*DIME Analytics' [https://github.com/worldbank/DIME-Resources/blob/master/Survey%20firm%20Technical%20Review%20_%20Scoring%20Matrix.xlsx Survey Firm Technical Review Scoring Matrix]
 
*DIME Analytics' [https://github.com/worldbank/DIME-Resources/blob/master/survey-firms.pdf Working with Survey Firms]
[[Category: Survey Firm Procurement]]
[[Category: Primary Data Collection]]

Latest revision as of 17:22, 27 June 2024

A survey firm term of reference (TOR) defines the structure of the project and breaks down the responsibilities of all parties involved, including that of the impact evaluation team. The TOR is an important document for survey firm procurement and ongoing monitoring of survey firm performance. This page suggests the scope of work for the survey firm and impact evaluation team, discusses deliverables, and provides an example excerpt from a survey firm TOR.

Read First

  • Prepare the TOR early in the firm procurement process.
  • Within the TOR, clearly delineate tasks between the survey firm and the impact evaluation team; define deliverables carefully; and specify all expectations and consequences if the expectations are not met.
  • Refer to the TOR throughout field work. Any changes to survey protocols or sampling that deviate from the TOR must be confirmed in writing.

Overview

A detailed TOR is essential for smoothly running fieldwork. Prepare the TOR before publishing the Request for Expression of Interest (rEOI) for survey firms; publish it after shortlisting firms based on the expression of interest (EOI). After hiring the survey firm, monitor and provide feedback early and often, using the TOR as an anchor for expectations. Any changes to survey protocols or sampling that deviate from the TOR must be confirmed in writing. If there are cost implications, a contract modification may be required.

Defining the Scope of Work

Survey Firm Tasks

  1. Obtain necessary permits or clearance for the survey
  2. Translate and field test all questionnaires
  3. Create detailed field procedure plan along with a timeline.
  4. Recruit, train, and contract experienced field staff
  5. Manage and oversee household survey data collection

Impact Evaluation Team Tasks

  1. Design questionnaire
  2. Manage survey pilot
  3. Program questionnaire
  4. Sample
  5. Monitor data quality
  6. Clean data
  7. Conduct data analysis

The field coordinator should work closely with survey firm every step of the process.

Defining the Deliverables

Review of deliverables is the formal and enforceable way to provide feedback to the survey firm. It is important to assess quality and provide feedback early and often.

Deliverables should be spaced throughout the contract rather than only at the end of the survey. Each major stage of the process should include a deliverable. Deliverables should be specified as a detailed list in the TOR, with intermediate outputs. Finally, deliverables should have clearly indicated quality standards. Consequences should be clearly outlined in the case that the firm does not meet standards.

Example List of Deliverables

TOR Example Excerpt

Activity 5 : Household survey data collection

Train all enumerators, supervisors, and data manager on the administration of questionnaires provided by the research team. The training should also serve as a screening process for skilled interviewers. Consequently, the survey firm must recruit more interviewers for the training than will be ultimately hired for the project. Five enumerators should be included in the training as a reserve. The following components must be included in the training:

  1. Theoretical: training should include a review the theory of the questionnaire and each question in order for trainees to fully understand the objective of each question. Standard quantitative interviewing techniques and field protocols should also be covered.
  2. Classroom practice: training should include individual and group exercises for trainees to become familiar with the practice of asking questions and filling questionnaires. This part of the training may include in-class demonstrations, where the questionnaire is projected and one interviewer completes the it in front of the classroom. The training may also use vignettes, where the company designs case scenarios based on typical households (perhaps those found during the supervisor training or piloting) and have interviewers complete the questionnaire based on the vignette. Finally, the trainees should conduct pilot interviews on the same subject, and have the interviewers complete a questionnaire for the interview to test consistency across interviewers.
  3. Field practice: after the theoretical and classroom practice, the interviewers should go to the field to administer the full questionnaire to a small number of households (outside the study sample). The pre-test should not focus on major adjustments to the questionnaire, but rather simulate the administration of the questionnaire under normal circumstances. All field team members must demonstrate they clearly understand their roles and are correctly following the survey protocols.
  4. Evaluation: following the training, interviewers, supervisors and data managers should be evaluated based on their understanding of the questionnaire and their ability to correctly record data.

The training period will conclude only when the field teams have demonstrated mastery of the designated tasks. Decisions as to which field staff will take part in the data collection must be made on the basis of this evaluation.

Back to Parent

This article is part of the topic Primary Data Collection

Additional Resources

Retrieved from "https://dimewiki.worldbank.org/index.php?title=Survey_Firm_TOR&oldid=9424"