Dimewiki

Difference between revisions of "Monitoring Data Quality"

Jump to: navigation, search
Line 1: Line 1:
<onlyinclude>
Data quality monitoring is an ongoing process during [[Primary Data Collection | primary data collection ]] that entails conducting [[Back Checks | back-checks]] in the field and regularly running high-frequency checks on the data (i.e. response quality checks, programming checks and enumerator checks). This process, part of a broader [[Data Quality Assurance Plan | data quality assurance plan]], helps the research team to identify and correct any [[Enumerator Training | enumerator issues]] or remaining quirks in questionnaire [[Questionnaire Design | design]] and [[Questionnaire Programming | programming]]. With the information collected during data quality monitoring, research teams can ensure that they are obtaining the highest possible quality of data. This page outlines key elements of data quality monitoring: back-checks, high frequency checks, project checks, and treatment monitoring.
Data quality monitoring involves both checks done in the field and high-frequency checks run on the data. The primary field activity for monitoring data quality is [[Back Checks]]. High-frequency data checks include: response quality checks, programming checks and enumerator checks.  
</onlyinclude>
== Read First ==
* All surveys should do [[Back Checks]] for at least 10% of the sample, aiming to stratify the backcheck sample by enumerator
* [[Computer-Assisted Personal Interviews (CAPI)|CAPI]] and [[Computer-Assisted Field Entry (CAFE)|CAFE]] Surveys should also do data checks while the survey is in the field. Data checks may be run for [[Pen-and-Paper Personal Interviews (PAPI)| PAPI]] Surveys once the data entry is complete to assess data quality, though it will be too late to make changes.
*The best time to design and code the data checks is in parallel to the [[Questionnaire Design]] and the [[Questionnaire Programming]]. Data quality checks are often completed too late to be relevant or omit important tests when not written in parallel to the questionnaire.
*Data quality checks should be run daily, as the enumerator will still remember the interview if we have questions, and she/he is likely to still be able to go back to the respondent in case we have questions. These checks serve as additional enumerator support mechanisms that allow team members and enumerators to notice data discrepancies as they arise and resolve the issue(s) immediately.


==Guidelines ==  
==Read First==
*Data quality checks should be run daily, as the enumerator will still remember the interview should any questions arise. Further, she/he can likely return to the respondent in if necessary. These checks serve as additional enumerator support mechanisms that allow team members and enumerators to notice data discrepancies as they arise and resolve the issue(s) immediately.
*The best time to design and code the back-checks and high frequency checks is in parallel to the [[Questionnaire Design | design]] and the [[Questionnaire Programming | programming]]. Data quality checks may omit important tests or be irrelevant if not written in parallel with the questionnaire.
*Create a dashboard with enumerator data check results to facilitate a continual feedback process for enumerators and survey teams: this maintains transparency, accountability, and can help boost motivation.


=== Back Checks ===
== Back-Checks ==
[[Back Checks]], also known as Survey Audits, are a second visit to the household to confirm the interview was conducted and verify key pieces of information. Best practice is for back checks to be completed by an independent third party.
Back-checks, also known as survey audits, are a quality control method implemented to verify the quality and legitimacy of data collected during a survey. Throughout the course of fieldwork, a back-check team returns to a randomly-selected subset of households for which data has been collected. The back-check team re-interviews these respondents with a short subset of survey questions, otherwise known as a back-check survey. Back-checks are used to verify the quality and legitimacy of key data collected in the actual survey. For more information on how to collect and analyze data via back-checks, see [[Back Checks]].


==High Frequency Checks==
Research teams should run high frequency checks (HFC) from the office daily. Prepare the HFC code via Stata or R once the questionnaire is finalized but before it goes to the field. You should also prepare instructions for the HFCs in case someone else needs to run it while you are in the field and/or without internet connection. During data collection, download data and daily run the HFC to report flags. This should be a one-click process. Within the HFC, include four main types of checks: response quality checks, programming checks, enumerator checks, and duplicate/survey log checks.


===High Frequency Checks===
=== Response Quality Checks ===
There are many different best practices on what checks to do during data quality checks. There is no best practice that is best in every single case, but this section presents a general framework useful to anyone that are new to data quality checks, for three main types of checks.  
Response quality checks monitor the consistency of responses across the survey instrument and the range within the responses fall.
*Consistency of responses across the survey instrument: most consistency tests can and should be built into the [[Questionnaire Programming | questionnaire programming]] via logic and constraints. However, some checks may be overly complex to program in the survey instrument, particularly when comparing responses across rosters or dealing with multiple units. For example, imagine we ask about plot and harvest size and allow the respondent to answer in the unit of his/her choice. In order to test if the harvest in terms of kilos per hectare is plausible, we need to convert harvest and plot size to kilos and hectares, which may be challenging to program within the questionnaire itself. As a rule of thumb, program as many checks as possible into the survey instrument. Then include the rest in the HFC do file or script.
*Reasonable ranges of responses: while range checks should always be programmed into the survey instrument, typically questionnaires employ 'soft' constraints (i.e. warning enumerators that the response is unusual but can continue). Thus, HFC data checks should include checks for extreme values and outliers and confirm whether they make sense in context. Data checks should also check the range for constructed indicators; multiplication or division can create or expose outliers even when the numerator and denominator are reasonable. For example, say a household reported a plot size of 0.05 hectares (the low end of an acceptable range) and produced 1000kg of maize (within an acceptable range): the yield for the plot would be 20,000kg/ha. This is an extreme outlier.


==== Response Quality Checks ====
=== Programming Checks ===
As we do not know the 'true' answer, it is challenging to check the quality of responses. However there are a few areas we should check:
Programming checks help the research team to understand if they have designed and programmed the questionnaire properly. Most programming errors should be caught when [[Survey Pilot | testing the questionnaire]], but it is impossible to test all possible outcomes before data collection. Including programming checks in the HFC is especially important when the team has made last-minute edits to the survey instrument.
# Consistency of responses across the survey instrument.
## For example, if a respondent answered that no one in the household worked outside of the household in a labor module, but then in the income reported earnings from wage labor, that would be inconsistent. Most of these tests can and should be built into the questionnaire programming. However, some checks may be overly complex to program in the survey instrument, particularly when comparing responses across rosters or dealing with multiple units. Let's say that we ask about plot size and harvest size and allow the respondent to answer in the unit of his/her choice. In order to test if the harvest in terms of kilos per hectare is plausible, we need to convert harvest and plot size to kilos and hectares, which may be challenging to program. The rule of thumb is to program as many checks as possible into the survey instrument, and to include all the rest in a Stata .do file or R script which will be run against the data on a daily basis.  


# Reasonable ranges of responses
=== Enumerators Checks ===
## While range checks should always be programmed into the survey instrument, the typical recommendation is to use 'soft' constraints in the survey instrument (i.e. enumerators are warned that the response is unusual but can continue). Data checks should include checks for extreme values and outliers, and confirm whether those make sense in context. In addition, it is important to check range for constructed indicators as multiplication or division can create outliers even when the numerator and denominator are reasonable. For example, say a household reported they had a plot that was 0.05 hectares (the low end of an acceptable range) and produced 1000kg of maize (within an acceptable range): the yield for the plot would be 20,000kg/ha (extreme outlier).  
Enumerator checks help the research team determine if any individual enumerator's data is significantly different from other enumerators' data in the datasets or different from the mean of a given question. These checks should:
*Check percentage of “don’t know” and refusal responses by enumerator.
*Check the distribution of responses for key questions by enumerator.
*Check the number of surveys per day by enumerator.
*Check the average interview duration by enumerator.
*Check the duration of consent by enumerator.
*Check the duration of other modules by enumerator (anthropometrics, games, etc.).
These statistics can be outputted into an enumerator dashboard. Keeping track of survey team metrics and frequently discussing them with enumerators and team leaders maintains accountability, transparency, and can boost motivation. See more on SurveyCTO’s tracking dashboard [https://www.surveycto.com/best-practices/hacking-google-sheets-for-real-time-dashboards/ here].
===Duplicates and Survey Log Checks ====
Duplicate and survey log checks confirm that all the data from the field is on the survey in a sound manner. They should:
*Test that all data from the field is on the server: match survey data logs from the field with survey data logs on the server to confirm that all the data from the field has been transferred to the server.
*Test for target number: since surveys are submitted in daily waves, keep track of the numbers of surveys submitted and the target number of surveys needed for an area to be completed. 
*Test for [[Duplicates and Survey Logs | duplicates]]: since SurveyCTO/ODK data provides a number of duplicates, check for duplicates using <code>[[ieduplicates]]</code>.
Verifying these details as soon as possible is critical: since the enumerator is most likely close by if you run daily checks, it is easy for him/her to re-interview and get missing data if the HFC renders this necessary.  


==Project Checks==
Make sure to look at the broader status and progress of the project itself. These statistics can help the research team see bigger picture trends, including:
*Overall survey progress relative to planned sample
*Summaries of key research variables
*Two-way summaries of survey variables by demographic/geographic characteristics
*Attrition rates by type and treatment status
*Comparisons of variables with known distributions
*Geographical distribution of observations via maps/GIS – are all observations where they’re meant to be?


==== Programming Checks ====
==Treatment Monitoring==
This type of tests help us understand if we have designed and programmed the questionnaire properly. Most programming errors should be caught when testing the questionnaire, but it is impossible to test all possible outcomes before data collection. This is especially important when there are last-minute edits to the survey instrument.
For the validity of the research, research teams should also monitor that the units assigned to treatment actually receive the treatment – or at least an offer – and that control households do not. Treatment contamination occurs when some of the control receives the treatment. This should be reduced to as great an extent as possible. Research teams may monitor treatment via physical checks or administrative checks:
 
*Physical checks can ensure that the treatment was applied as stated in impact evaluation protocol to the units selected for treatment, in the correct order, according to the script, and using marketing materials as planned.
==== Enumerators Checks ====
*[[Administrative and Monitoring Data | Administrative data]] checks can check attendance, account records and use pre-/post- tests to ensure that only units to which treatment was offered participated in the intervention. The exact details of this method depend on both the intervention and the available data. With interventions that involve savings accounts, for example, you may check the details of the accounts opened to ensure that the account opener is related to treatment household in some way.  
Enumerator checks help us determine if any individual enumerator record data that are different from other enumerators in the data sets or different from the mean of a given question.
 
The enumerators are the eyes and the ears of the project in the field, and it is absolutely necessary for high data quality that our enumerators are making their best effort. Enumerator checks are meant as a tool to support enumerators that need extra help.  
 
=== Practical Examples of Data Quality Checks ===
====Duplicates and Survey Logs ====
It is very important to do quality checks on data during the survey implementation as it is difficult to fix the problem/re-collect the data if the error is found after the completion of the survey.
*Testing for Duplicates - Since SurveyCTO/ODK data has a number of duplicates, the first thing you need to do is check for duplicates and remove the duplicates. Use the command [[ieduplicates]]
*Test that all data from the field is on the server - Survey data logs from the field can then be matched with the logs from the survey data logs on the server to see if the all the data from the field has been transferred to the server.
*Test for target number - Since surveys are submitted in daily waves, you should keep track of the numbers of surveys that are submitted and the target number of surveys needed for an area to be completed. 
 
'''''Tip:''''' Verifying that the data is complete should be done the day of or the day after the survey. Since, the interviewer is most likely close by, it would be easy to re-interview and get missing data if significant chunks of data were missing.
 
To see how to remove duplicates and check that all the field data is on servers, please see the main article at '''[[Duplicates and Survey Logs]].'''


== Back to Parent ==
== Back to Parent ==
Line 53: Line 59:
*DIME's [http://web.worldbank.org/archive/website01542/WEB/IMAGES/SURVEY.PDF Planning for, Preparing & Monitoring Household Surveys]
*DIME's [http://web.worldbank.org/archive/website01542/WEB/IMAGES/SURVEY.PDF Planning for, Preparing & Monitoring Household Surveys]
*DIME Analytics’ [https://github.com/worldbank/DIME-Resources/blob/master/stata1-4-quality.pdf Real Time Data Quality Checks]
*DIME Analytics’ [https://github.com/worldbank/DIME-Resources/blob/master/stata1-4-quality.pdf Real Time Data Quality Checks]
[[Category: Field Management ]]
*SurveyCTO’s [https://www.surveycto.com/case-studies/monitoring-and-visualization/ Monitoring and Visualization] case studies

Revision as of 14:44, 6 June 2019

Data quality monitoring is an ongoing process during primary data collection that entails conducting back-checks in the field and regularly running high-frequency checks on the data (i.e. response quality checks, programming checks and enumerator checks). This process, part of a broader data quality assurance plan, helps the research team to identify and correct any enumerator issues or remaining quirks in questionnaire design and programming. With the information collected during data quality monitoring, research teams can ensure that they are obtaining the highest possible quality of data. This page outlines key elements of data quality monitoring: back-checks, high frequency checks, project checks, and treatment monitoring.

Read First

  • Data quality checks should be run daily, as the enumerator will still remember the interview should any questions arise. Further, she/he can likely return to the respondent in if necessary. These checks serve as additional enumerator support mechanisms that allow team members and enumerators to notice data discrepancies as they arise and resolve the issue(s) immediately.
  • The best time to design and code the back-checks and high frequency checks is in parallel to the design and the programming. Data quality checks may omit important tests or be irrelevant if not written in parallel with the questionnaire.
  • Create a dashboard with enumerator data check results to facilitate a continual feedback process for enumerators and survey teams: this maintains transparency, accountability, and can help boost motivation.

Back-Checks

Back-checks, also known as survey audits, are a quality control method implemented to verify the quality and legitimacy of data collected during a survey. Throughout the course of fieldwork, a back-check team returns to a randomly-selected subset of households for which data has been collected. The back-check team re-interviews these respondents with a short subset of survey questions, otherwise known as a back-check survey. Back-checks are used to verify the quality and legitimacy of key data collected in the actual survey. For more information on how to collect and analyze data via back-checks, see Back Checks.

High Frequency Checks

Research teams should run high frequency checks (HFC) from the office daily. Prepare the HFC code via Stata or R once the questionnaire is finalized but before it goes to the field. You should also prepare instructions for the HFCs in case someone else needs to run it while you are in the field and/or without internet connection. During data collection, download data and daily run the HFC to report flags. This should be a one-click process. Within the HFC, include four main types of checks: response quality checks, programming checks, enumerator checks, and duplicate/survey log checks.

Response Quality Checks

Response quality checks monitor the consistency of responses across the survey instrument and the range within the responses fall.

  • Consistency of responses across the survey instrument: most consistency tests can and should be built into the questionnaire programming via logic and constraints. However, some checks may be overly complex to program in the survey instrument, particularly when comparing responses across rosters or dealing with multiple units. For example, imagine we ask about plot and harvest size and allow the respondent to answer in the unit of his/her choice. In order to test if the harvest in terms of kilos per hectare is plausible, we need to convert harvest and plot size to kilos and hectares, which may be challenging to program within the questionnaire itself. As a rule of thumb, program as many checks as possible into the survey instrument. Then include the rest in the HFC do file or script.
  • Reasonable ranges of responses: while range checks should always be programmed into the survey instrument, typically questionnaires employ 'soft' constraints (i.e. warning enumerators that the response is unusual but can continue). Thus, HFC data checks should include checks for extreme values and outliers and confirm whether they make sense in context. Data checks should also check the range for constructed indicators; multiplication or division can create or expose outliers even when the numerator and denominator are reasonable. For example, say a household reported a plot size of 0.05 hectares (the low end of an acceptable range) and produced 1000kg of maize (within an acceptable range): the yield for the plot would be 20,000kg/ha. This is an extreme outlier.

Programming Checks

Programming checks help the research team to understand if they have designed and programmed the questionnaire properly. Most programming errors should be caught when testing the questionnaire, but it is impossible to test all possible outcomes before data collection. Including programming checks in the HFC is especially important when the team has made last-minute edits to the survey instrument.

Enumerators Checks

Enumerator checks help the research team determine if any individual enumerator's data is significantly different from other enumerators' data in the datasets or different from the mean of a given question. These checks should:

  • Check percentage of “don’t know” and refusal responses by enumerator.
  • Check the distribution of responses for key questions by enumerator.
  • Check the number of surveys per day by enumerator.
  • Check the average interview duration by enumerator.
  • Check the duration of consent by enumerator.
  • Check the duration of other modules by enumerator (anthropometrics, games, etc.).

These statistics can be outputted into an enumerator dashboard. Keeping track of survey team metrics and frequently discussing them with enumerators and team leaders maintains accountability, transparency, and can boost motivation. See more on SurveyCTO’s tracking dashboard here.

Duplicates and Survey Log Checks =

Duplicate and survey log checks confirm that all the data from the field is on the survey in a sound manner. They should:

  • Test that all data from the field is on the server: match survey data logs from the field with survey data logs on the server to confirm that all the data from the field has been transferred to the server.
  • Test for target number: since surveys are submitted in daily waves, keep track of the numbers of surveys submitted and the target number of surveys needed for an area to be completed.
  • Test for duplicates: since SurveyCTO/ODK data provides a number of duplicates, check for duplicates using ieduplicates.

Verifying these details as soon as possible is critical: since the enumerator is most likely close by if you run daily checks, it is easy for him/her to re-interview and get missing data if the HFC renders this necessary.

Project Checks

Make sure to look at the broader status and progress of the project itself. These statistics can help the research team see bigger picture trends, including:

  • Overall survey progress relative to planned sample
  • Summaries of key research variables
  • Two-way summaries of survey variables by demographic/geographic characteristics
  • Attrition rates by type and treatment status
  • Comparisons of variables with known distributions
  • Geographical distribution of observations via maps/GIS – are all observations where they’re meant to be?

Treatment Monitoring

For the validity of the research, research teams should also monitor that the units assigned to treatment actually receive the treatment – or at least an offer – and that control households do not. Treatment contamination occurs when some of the control receives the treatment. This should be reduced to as great an extent as possible. Research teams may monitor treatment via physical checks or administrative checks:

  • Physical checks can ensure that the treatment was applied as stated in impact evaluation protocol to the units selected for treatment, in the correct order, according to the script, and using marketing materials as planned.
  • Administrative data checks can check attendance, account records and use pre-/post- tests to ensure that only units to which treatment was offered participated in the intervention. The exact details of this method depend on both the intervention and the available data. With interventions that involve savings accounts, for example, you may check the details of the accounts opened to ensure that the account opener is related to treatment household in some way.

Back to Parent

This article is part of the topic Field Management

Additional Resources

Retrieved from "https://dimewiki.worldbank.org/index.php?title=Monitoring_Data_Quality&oldid=5378"