Dimewiki

Difference between revisions of "High Frequency Checks"

Jump to: navigation, search
 
(17 intermediate revisions by 3 users not shown)
Line 1: Line 1:
Research teams should run high frequency checks (HFC) from the office daily. Prepare the HFC code via Stata or R once the questionnaire is finalized but before it goes to the field. You should also prepare instructions for the HFCs in case someone else needs to run it while you are in the field and/or without internet connectivity. During data collection, download data and daily run the HFC to report flags. This should be a one-click process. Within the HFC, include four main types of checks: response quality checks, programming checks, enumerator checks, and duplicate/survey log checks.
Before starting with [[Primary Data Collection|data collection]], the [[Impact Evaluation Team|research team]] should work with the '''field team''' to design and code [[High Frequency Checks|high frequency checks]], as part of the [[Data Quality Assurance Plan|data quality assurance plan]]. The best time to design and code these '''high frequency checks''' is in parallel to the process of [[Questionnaire Design | questionnaire design]] and [[Questionnaire Programming | programming]]. These checks should be run every time new data is received, to provide real-time information to the '''field team''' and '''research team''' for all [[Field Surveys|surveys]].  
== Read First ==
* Along with [[Back Checks|back checks]], '''High frequency checks (HFCs)''' are an important aspect of the [[Data Quality Assurance Plan|data quality assurance plan]].  
*  Before launching the [[Field Surveys|survey]], carry out a [[Checklist:_Data-focused_Pilot|data-focused pilot]] to plan for '''high frequency checks'''.
* [https://www.poverty-action.org/ IPA] has created a Stata package called <code>[https://github.com/PovertyAction/high-frequency-checks ipacheck]</code> to carry out '''high frequency checks''' on [[Primary Data Collection|survey data]].
 
== General Principles ==
While [[Survey Firm|survey firms]] often conduct internal quality checks, it is important for the [[Impact Evaluation Team|research team]] to conduct their own list of '''high frequency checks''' to improve the quality of [[Publishing Data|research outputs]]. For this purpose, [https://www.worldbank.org/en/research/dime DIME] has proposed the following four general principles for [[Monitoring Data Quality|conducting quality checks]] on various data sources:
=== Completeness ===
Compare collected responses with the sample to make sure all data points actually appear in the dataset. Verify that there are no duplicate observations. Check whether the data covers all timeframes (in case of '''time series data'''), all locations (such as all villages in a district), and all [[Unit of Observation|units of observation]]. One quick way to check for '''completeness''' is by visualizing the dataset to identify gaps.
 
'''For example''', the figures below, shows the timeframes and locations where data on cell phone use is missing.
[[File:Completeness.jpeg|500px|thumb|center|'''Figure 1: Check for completeness''']]


== Read First ==
=== Consistency ===  
* Data-focused pilot.
For each variable, make sure that the data is consistent for different rounds of [[Primary Data Collection|data collection]]. For example, if a farmer reported that they sowed 1 hectare of maize in a '''baseline survey''', but reported that they harvest 20 hectares of maize in an '''endline survey'''. Before receiving the data, use another source of data for these variables to note down expected ranges. Then once the data is received, check that for each variable, the data points fall within expected ranges.
* Monitoring Data Quality
 
* Data Quality Assurance Plan
'''For example''', the figure below checks for '''consistency''' between the number of active users of cell phones, and the number of daily calls and messages. In this case, the points in purple are '''not consistent''' because every call or message must have a user who generates it.
[[File:Consistency.jpg|500px|thumb|center|'''Figure 1: Check for consistency''']]
=== Anomalous data points ===
One of the most common ways to deal with '''anomalous data points''' (like '''outliers''') is through carefully [[Questionnaire Programming|programmed questionnaires]] that set the accepted range and/or type of values for each question. However, there can still be outliers because of random issues during the survey. Use '''standard deviation''' as a benchmark - for instance, by flagging observations that are more than 2 standard deviations higher or lower than the mean.  Regularly [[Back Checks|back check]] incoming data to also identify outliers. Visualizing the data points in this case can again help with identifying outliers.
 
'''For example''', the figure below shows the speeds of 4 vehicles over time. Any data point that lies outside the gray area in this figure can be easily identified as an '''anomalous data point''' since it is more than 2 standard deviations over the mean.
[[File:Anomalous.jpg|500px|thumb|center|'''Figure 1: Check for anomalous data points''']]
=== Real-time ===
'''Real-time''' checks on quality can be done by discussing any issues that come up with the [[Survey Firm|survey firm]] just a few hours after receiving the data from the [[Field Surveys|field]]. However, it might not always be possible to do this, especially when using [[Secondary Data Sources|secondary data]]. In this case, run checks for '''completeness''', '''consistency''', and '''anomalous data points''' as soon as the data is received. Discuss the data points that are flagged in these checks in '''real-time''' to improve [[Monitoring Data Quality|data quality]].


== Types ==  
== Types ==  
=== Response Quality Checks ===
The following are the different categories of '''high frequency checks''' that [[Impact Evaluation Teams|research teams]] can use to improve [[Monitoring Data Quality|data quality]]
Response quality checks monitor the consistency of responses across the survey instrument and the range within the responses fall.  
* '''Response quality checks:''' These checks are designed to ensure that responses are consistent for a particular '''survey instrument''', and that the responses fall within a particular range. For example, checks to ensure that all variables are [[Standardization|standardized]], and there are no outliers in the data. Share a daily log of such errors, and check if it is an issue with '''enumerators''', in which case there might be a need to [[Enumerator Training|re-train enumerators]].
*Consistency of responses across the survey instrument: most consistency tests can and should be built into the [[Questionnaire Programming | questionnaire programming]] via logic and constraints. However, some checks may be overly complex to program in the survey instrument, particularly when comparing responses across rosters or dealing with multiple units. For example, imagine we ask about plot and harvest size and allow the respondent to answer in the unit of his/her choice. In order to test if the harvest in terms of kilos per hectare is plausible, we need to convert harvest and plot size to kilos and hectares, which may be challenging to program within the questionnaire itself. As a rule of thumb, program as many checks as possible into the survey instrument. Then include the rest in the HFC do file or script.  
 
*Reasonable ranges of responses: while range checks should always be programmed into the survey instrument, typically questionnaires employ 'soft' constraints (i.e. warning enumerators that the response is unusual but can continue). Thus, HFC data checks should include checks for extreme values and outliers and confirm whether they make sense in context. Data checks should also check the range for constructed indicators; multiplication or division can create or expose outliers even when the numerator and denominator are reasonable. For example, say a household reported a plot size of 0.05 hectares (the low end of an acceptable range) and produced 1000kg of maize (within an acceptable range): the yield for the plot would be 20,000kg/ha. This is an extreme outlier.
* '''Enumerator checks:''' These are designed to check if data shared by any particular enumerator is significantly different from the data shared by other enumerators. Some parameters to check enumerator performance include percentage of '''"Don't know"''' responses, or average interview duration. In the first case, there might be a need to [[Questionnaire Design#Draft Instrument|re-draft the questions]], while in the second case, there might be a need to '''re-train enumerators'''.


=== Programming Checks ===
* '''Programming checks:''' These test for issues in logic or '''skip patterns''' that were not spotted during [[Questionnaire Programming|questionnaire programming]]. These also include checking for data that is missing because the '''programmed version''' of the instrument skips certain questions. In this case, program the instrument again, and double-check for errors.
Programming checks help the research team to understand if they have designed and programmed the questionnaire properly. Most programming errors should be caught when [[Survey Pilot | testing the questionnaire]], but it is impossible to test all possible outcomes before data collection. Including programming checks in the HFC is especially important when the team has made last-minute edits to the survey instrument.


=== Enumerators Checks ===
* '''Duplicates and survey log checks:''' These ensure that all the data collected from the field is on the [[SurveyCTO Server Management|server]], and that there are no duplicates. In case some data is missing, or the [[Field Surveys|survey forms]] are incomplete, share a report with the field team and identify the reasons for low completion rate. In case there are duplicate IDs, [[Ieduplicates|identify]] and [[Iecompdup|resolve]] them.
Enumerator checks help the research team determine if any individual enumerator's data is significantly different from other enumerators' data in the datasets or different from the mean of a given question. These checks should:
*Check percentage of “don’t know” and refusal responses by the enumerator.
*Check the distribution of responses for key questions by enumerator.
*Check the number of surveys per day by the enumerator.
*Check the average interview duration by the enumerator.
*Check the duration of consent by the enumerator.
*Check the duration of other modules by enumerator (anthropometrics, games, etc.).
These statistics can be output into an enumerator dashboard. Keeping track of survey team metrics and frequently discussing them with enumerators and team leaders maintain accountability, transparency, and can boost motivation. See more on SurveyCTO’s tracking dashboard [https://www.surveycto.com/best-practices/hacking-google-sheets-for-real-time-dashboards/ here].
===Duplicates and Survey Log Checks ===
Duplicate and survey log checks confirm that all the data from the field is on the survey in a sound manner. They should:
*Test that all data from the field is on the server: match survey data logs from the field with survey data logs on the server to confirm that all the data from the field has been transferred to the server.
*Test for target number: since surveys are submitted in daily waves, keep track of the numbers of surveys submitted and the target number of surveys needed for an area to be completed. 
*Test for [[Duplicates and Survey Logs | duplicates]]: since SurveyCTO/ODK data provides a number of duplicates, check for duplicates using <code>[[ieduplicates]]</code>.
Verifying these details as soon as possible is critical: since the enumerator is most likely close by if you run daily checks, it is easy for him/her to re-interview and get missing data if the HFC renders this necessary.


=== Other Checks ===
* '''Other checks related to the project:''' These include checking if start date and end date of an interview are the same, or ensuring that there is at least one variable that has a [[ID Variable Properties|unique ID]]. Further, in the case of [[Administrative and Monitoring Data | administrative data]], there can be daily checks to check and compare data with previous records, and ensure that participation in a study was only offered to those who were selected for '''treatment'''. For example, in an intervention involving cash transfers, check the details of the accounts to ensure that the account is in the name of a person who was selected for treatment.
For the validity of the research, research teams should also monitor that the units assigned to treatment actually receive the treatment – or at least an offer – and that control households do not. Treatment contamination occurs when some of the control receives the treatment. This should be reduced to as great an extent as possible. Research teams may monitor treatment via physical checks or administrative checks:
== Useful Tips ==
*Physical checks can ensure that the treatment was applied as stated in impact evaluation protocol to the units selected for treatment, in the correct order, according to the script, and using marketing materials as planned.
Keep the following in mind regarding '''high frequency checks (HFCs)''':
*[[Administrative and Monitoring Data | Administrative data]] checks can check attendance, account records and use pre-/post- tests to ensure that only units to which treatment was offered participated in the intervention. The exact details of this method depend on both the intervention and the available data. With interventions that involve savings accounts, for example, you may check the details of the accounts opened to ensure that the account opener is related to treatment household in some way.
* '''Prepare HFCs before start of the survey.''' Prepare the code and instructions for '''HFCs''' before starting with [[Preparing for Field Data Collection|field data collection]].
* '''Regularly monitor and resolve errors.''' Monitor the reporting of errors, and deal with them daily.
* '''HFCs should be a one-click process.''' The '''HFCs''' should be a one-click process. [https://www.poverty-action.org/ IPA] has created <code>[https://github.com/PovertyAction/high-frequency-checks/wiki ipacheck]</code>, a Stata package for running '''HFCs''' on research data.
* '''Prioritize.''' It might not always be possible to perform all checks. So prioritize the ones that are most likely to affect data quality.


== Related Pages ==  
== Related Pages ==
[[Special:WhatLinksHere/High_Frequency_Checks|Click here for pages that link to this topic.]]


== Additional Resources ==
== Additional Resources ==
* DIME Analytics (World Bank) [https://osf.io/t48ug Data Quality Checks]
* DIME Analytics (World Bank), [https://osf.io/j8t5f Assuring Data Quality]
* Innovation for Poverty Action, [https://github.com/PovertyAction/high-frequency-checks/wiki/Background Template for high frequency checks]
* Innovation for Poverty Action, [https://github.com/PovertyAction/high-frequency-checks/wiki High frequency checks and <code>ipacheck</code>]
* J-PAL, [https://www.povertyactionlab.org/resource/data-quality-checks Data Quality Checks]
* SurveyCTO, [https://www.youtube.com/watch?v=tHb-3bnfRLo Data quality with SurveyCTO]
* SurveyCTO, [https://www.surveycto.com/case-studies/monitoring-and-visualization/ Monitoring and Visualization]
* SurveyCTO, [https://www.surveycto.com/videos/world-bank-high-frequecy-checks-webinar/?utm_source=mailchimp&utm_medium=email&utm_campaign=february2023-mailchimp-mid-month-email&utm_content=world-bank-webinar-on-demand&sctosrc=mailchimp_email World Bank Researchers Teach How to Create High Frequency Checks Dashboards]
[[Category: Research Design]]

Latest revision as of 18:29, 28 June 2023

Before starting with data collection, the research team should work with the field team to design and code high frequency checks, as part of the data quality assurance plan. The best time to design and code these high frequency checks is in parallel to the process of questionnaire design and programming. These checks should be run every time new data is received, to provide real-time information to the field team and research team for all surveys.

Read First

General Principles

While survey firms often conduct internal quality checks, it is important for the research team to conduct their own list of high frequency checks to improve the quality of research outputs. For this purpose, DIME has proposed the following four general principles for conducting quality checks on various data sources:

Completeness

Compare collected responses with the sample to make sure all data points actually appear in the dataset. Verify that there are no duplicate observations. Check whether the data covers all timeframes (in case of time series data), all locations (such as all villages in a district), and all units of observation. One quick way to check for completeness is by visualizing the dataset to identify gaps.

For example, the figures below, shows the timeframes and locations where data on cell phone use is missing.

Figure 1: Check for completeness

Consistency

For each variable, make sure that the data is consistent for different rounds of data collection. For example, if a farmer reported that they sowed 1 hectare of maize in a baseline survey, but reported that they harvest 20 hectares of maize in an endline survey. Before receiving the data, use another source of data for these variables to note down expected ranges. Then once the data is received, check that for each variable, the data points fall within expected ranges.

For example, the figure below checks for consistency between the number of active users of cell phones, and the number of daily calls and messages. In this case, the points in purple are not consistent because every call or message must have a user who generates it.

Figure 1: Check for consistency

Anomalous data points

One of the most common ways to deal with anomalous data points (like outliers) is through carefully programmed questionnaires that set the accepted range and/or type of values for each question. However, there can still be outliers because of random issues during the survey. Use standard deviation as a benchmark - for instance, by flagging observations that are more than 2 standard deviations higher or lower than the mean. Regularly back check incoming data to also identify outliers. Visualizing the data points in this case can again help with identifying outliers.

For example, the figure below shows the speeds of 4 vehicles over time. Any data point that lies outside the gray area in this figure can be easily identified as an anomalous data point since it is more than 2 standard deviations over the mean.

Figure 1: Check for anomalous data points

Real-time

Real-time checks on quality can be done by discussing any issues that come up with the survey firm just a few hours after receiving the data from the field. However, it might not always be possible to do this, especially when using secondary data. In this case, run checks for completeness, consistency, and anomalous data points as soon as the data is received. Discuss the data points that are flagged in these checks in real-time to improve data quality.

Types

The following are the different categories of high frequency checks that research teams can use to improve data quality

  • Response quality checks: These checks are designed to ensure that responses are consistent for a particular survey instrument, and that the responses fall within a particular range. For example, checks to ensure that all variables are standardized, and there are no outliers in the data. Share a daily log of such errors, and check if it is an issue with enumerators, in which case there might be a need to re-train enumerators.
  • Enumerator checks: These are designed to check if data shared by any particular enumerator is significantly different from the data shared by other enumerators. Some parameters to check enumerator performance include percentage of "Don't know" responses, or average interview duration. In the first case, there might be a need to re-draft the questions, while in the second case, there might be a need to re-train enumerators.
  • Programming checks: These test for issues in logic or skip patterns that were not spotted during questionnaire programming. These also include checking for data that is missing because the programmed version of the instrument skips certain questions. In this case, program the instrument again, and double-check for errors.
  • Duplicates and survey log checks: These ensure that all the data collected from the field is on the server, and that there are no duplicates. In case some data is missing, or the survey forms are incomplete, share a report with the field team and identify the reasons for low completion rate. In case there are duplicate IDs, identify and resolve them.
  • Other checks related to the project: These include checking if start date and end date of an interview are the same, or ensuring that there is at least one variable that has a unique ID. Further, in the case of administrative data, there can be daily checks to check and compare data with previous records, and ensure that participation in a study was only offered to those who were selected for treatment. For example, in an intervention involving cash transfers, check the details of the accounts to ensure that the account is in the name of a person who was selected for treatment.

Useful Tips

Keep the following in mind regarding high frequency checks (HFCs):

  • Prepare HFCs before start of the survey. Prepare the code and instructions for HFCs before starting with field data collection.
  • Regularly monitor and resolve errors. Monitor the reporting of errors, and deal with them daily.
  • HFCs should be a one-click process. The HFCs should be a one-click process. IPA has created ipacheck, a Stata package for running HFCs on research data.
  • Prioritize. It might not always be possible to perform all checks. So prioritize the ones that are most likely to affect data quality.

Related Pages

Click here for pages that link to this topic.

Additional Resources

Retrieved from "https://dimewiki.worldbank.org/index.php?title=High_Frequency_Checks&oldid=8625"