Difference between revisions of "Data Quality Assurance Plan"

Jump to: navigation, search
 
(One intermediate revision by one other user not shown)
Line 1: Line 1:
 +
Quality data is data that is not systematically biased and does not misstate representativeness or coverage. A data quality assurance plan considers everything in [[Primary Data Collection | data collection]] that could go wrong ahead of time and makes a plan to preempt these issues. The plan should be shared with all impact evaluation stakeholders – including the [[Impact Evaluation Team | impact evaluation team]] and the [[Survey Firm | survey firm]] – before data collection starts. This page covers the reasons behind low quality data and outlines key processes to include in the data quality assurance plan.
  
 
== Read First ==
 
== Read First ==
<onlyinclude>
+
*Data can be low quality because of respondent fatigue, enumerator error, or poor quality control.
Many things can go wrong during [[Primary Data Collection]]. The purpose of a data quality assurance plan is to think about everything that could go wrong ahead of time, and make a plan to preempt it. The plan should be shared with all impact evaluation stakeholders, including the [[Impact Evaluation Team]] and the [[Survey Firm]] before data collection starts. It is essential to delineate how data quality will be assessed and what actions will be taken when problems arise.  
+
*Good [[Questionnaire Design | questionnaire design]] and [[Questionnaire Programming | programming]], [[Back Checks | back-checks]] and high frequency checks help to ensure high quality data.
</onlyinclude>
 
  
== Guidelines ==
+
==Why Can Data be Low Quality?==
Your data quality assurance plan should include
+
The quality of [[Primary Data Collection | primary data]] is at risk of low quality because of three main reasons:
* [[Back Checks]]
+
*Respondents are human, so they have imperfect recall and motivation and/or can become tired or annoyed.
* [[Monitoring Data Quality]]
+
*Enumerators are human, so they can make mistakes, quickly fill answers to unasked questions when they’re sure they know the answer, or even just conduct fake interviews. Granted, they have really hard jobs! They are often travelling to new places and meeting people who are more or less friendly; dealing with weather, pollution, congestion and other conditions; tracking down hard-to-find respondents; and sometimes trying to decipher unclear instructions.
* For follow-up surveys, special consideration should be paid to [[Tracking]] and [[Attrition]]
+
*Research assistants are human, so they often fail to implement quality-control efforts in a timely manner, they stay away from conflict and don’t confront underperforming staff and they operate with chronic shortages of time and prior experience.
 +
There are three key lines of defense help to overcome these challenges to data quality: survey design, field management, and high frequency checks.
 +
==3 Elements of Data Quality Assurance Plan==
 +
===Survey Design===
 +
Within the [[Questionnaire Design | survey design]]:
 +
*use constraints and relevance.
 +
*preload data when possible to reduce error.
 +
*when relevant, use SurveyCTO’s case management to pre-assign surveys and/or treatments to individuals.
 +
Make sure the questions are clearly worded in [[Questionnaire Translation | all languages]] used and do not leave room for nuanced interpretations. [[Survey Pilot | Pilot]] the survey intensely to fine tune these questions and logic. Then, train [[Enumerator Training | enumerators]] well in order to ensure that the questionnaire is clear, intuitive, and easy to conduct as intended.
 +
===Field Management===
 +
During [[Primary Data Collection | data collection]], maintain close contact with enumerators and managers and give them a “buy-in” to quality; check in constantly and make them feel valued. Consider that unless egregious or fraudulent, the errors are not the field team’s but rather yours. As part of the data quality assurance plan, take the following steps during field management:
 +
*Spot checks/accompaniments:
 +
**By you: spend the first two weeks shadowing your enumerators and observing how each of them administers the survey. Give them useful feedback after – not during – the interviews
 +
**By team leaders: team leaders are experienced enumerators who have diverse tasks, the most important of which is observing interviews and supporting their team members in perfecting their interviewing abilities. Give team leaders a form to write down their observations for each enumerator and ask them to submit these forms.
 +
*Tracking sheets: give tracking sheets to enumerators on which they should track every day the information of completed surveys, units not found, etc. Enumerators should enter this information every day as a core task and present it frequently to team leaders for accountability.
 +
*[[Back Checks | Back-checks]]:
 +
**Compare answers with original survey. Every team and surveyor must be back-checked as soon as possible and regularly
 +
*Cover 10% of the same, with 20% being administered in the first two weeks of field work. *Cover a random sub-sample, include missing respondents to verify that your team is not biasing the sample by not tracking hard to find respondents, check observations flagged in other quality tests, and run backchecks on enumerators suspected of cheating.
 +
*See more information on [[Back Checks]].
 +
===High Frequency Checks===
 +
High frequency checks allow the research team to check quality, catch key mistakes early, and provide visible performance metrics for enumerators. For more information on how to implement high frequency checks, see [[Monitoring Data Quality#High Frequency Checks | High Frequency Checks]].
  
 
== Back to Parent ==
 
== Back to Parent ==
Line 19: Line 39:
 
*DIME Analytics’ [https://github.com/worldbank/DIME-Resources/blob/master/stata1-4-quality.pdf Real Time Data Quality Checks]
 
*DIME Analytics’ [https://github.com/worldbank/DIME-Resources/blob/master/stata1-4-quality.pdf Real Time Data Quality Checks]
 
*DIME Analytics’ [https://github.com/worldbank/DIME-Resources/blob/master/stata2-4-quality.pdf Data Quality Assurance]
 
*DIME Analytics’ [https://github.com/worldbank/DIME-Resources/blob/master/stata2-4-quality.pdf Data Quality Assurance]
 +
*SurveyCTO’s [https://www.surveycto.com/best-practices/survey-design-for-quality-data-part-2 Survey Design for Quality Data]
 +
[[Category: Research Design]]
 
[[Category: Field Management ]]
 
[[Category: Field Management ]]

Latest revision as of 14:50, 13 April 2021

Quality data is data that is not systematically biased and does not misstate representativeness or coverage. A data quality assurance plan considers everything in data collection that could go wrong ahead of time and makes a plan to preempt these issues. The plan should be shared with all impact evaluation stakeholders – including the impact evaluation team and the survey firm – before data collection starts. This page covers the reasons behind low quality data and outlines key processes to include in the data quality assurance plan.

Read First

Why Can Data be Low Quality?

The quality of primary data is at risk of low quality because of three main reasons:

  • Respondents are human, so they have imperfect recall and motivation and/or can become tired or annoyed.
  • Enumerators are human, so they can make mistakes, quickly fill answers to unasked questions when they’re sure they know the answer, or even just conduct fake interviews. Granted, they have really hard jobs! They are often travelling to new places and meeting people who are more or less friendly; dealing with weather, pollution, congestion and other conditions; tracking down hard-to-find respondents; and sometimes trying to decipher unclear instructions.
  • Research assistants are human, so they often fail to implement quality-control efforts in a timely manner, they stay away from conflict and don’t confront underperforming staff and they operate with chronic shortages of time and prior experience.

There are three key lines of defense help to overcome these challenges to data quality: survey design, field management, and high frequency checks.

3 Elements of Data Quality Assurance Plan

Survey Design

Within the survey design:

  • use constraints and relevance.
  • preload data when possible to reduce error.
  • when relevant, use SurveyCTO’s case management to pre-assign surveys and/or treatments to individuals.

Make sure the questions are clearly worded in all languages used and do not leave room for nuanced interpretations. Pilot the survey intensely to fine tune these questions and logic. Then, train enumerators well in order to ensure that the questionnaire is clear, intuitive, and easy to conduct as intended.

Field Management

During data collection, maintain close contact with enumerators and managers and give them a “buy-in” to quality; check in constantly and make them feel valued. Consider that unless egregious or fraudulent, the errors are not the field team’s but rather yours. As part of the data quality assurance plan, take the following steps during field management:

  • Spot checks/accompaniments:
    • By you: spend the first two weeks shadowing your enumerators and observing how each of them administers the survey. Give them useful feedback after – not during – the interviews
    • By team leaders: team leaders are experienced enumerators who have diverse tasks, the most important of which is observing interviews and supporting their team members in perfecting their interviewing abilities. Give team leaders a form to write down their observations for each enumerator and ask them to submit these forms.
  • Tracking sheets: give tracking sheets to enumerators on which they should track every day the information of completed surveys, units not found, etc. Enumerators should enter this information every day as a core task and present it frequently to team leaders for accountability.
  • Back-checks:
    • Compare answers with original survey. Every team and surveyor must be back-checked as soon as possible and regularly
  • Cover 10% of the same, with 20% being administered in the first two weeks of field work. *Cover a random sub-sample, include missing respondents to verify that your team is not biasing the sample by not tracking hard to find respondents, check observations flagged in other quality tests, and run backchecks on enumerators suspected of cheating.
  • See more information on Back Checks.

High Frequency Checks

High frequency checks allow the research team to check quality, catch key mistakes early, and provide visible performance metrics for enumerators. For more information on how to implement high frequency checks, see High Frequency Checks.

Back to Parent

This article is part of the topic Field Management

Additional Resources