Primary Data Collection
Primary data collection is the process of gathering data through surveys, interviews, or experiments. A typical example of primary data is household surveys. In this form of data collection, researchers can personally ensure that primary data meets the standards of quality, availability, statistical power and sampling required for a particular research question. With globally increasing access to specialized survey tools, survey firms, and field manuals, primary data has become the dominant source for empirical inquiry in development economics.
- The DIME Research Standards provide a comprehensive checklist to ensure that collection and handling of research data is in line with global best practices.
- Field surveys are one of the most effective medium for primary data collection. Depending on the research question, these interviews may take the form of household surveys, business (firm) surveys, or agricultural (farm) surveys.
- The research team must plan and prepare for primary data collection in advance.
iefieldkitis a Stata package that aids primary data collection. It currently supports three major components of this process: testing survey instruments; survey completion; and data-cleaning and survey harmonization.
While impact evaluations often benefit from secondary sources of data like administrative data, census data, or household data, these sources may not always be available. In such cases, the research team will need to collect data directly using well-designed interviews and surveys, and the research team typically owns the data that it collects. However, even then, the research team must keep in mind certain ethical concerns related to owning and handling sensitive, or personally identifiable information (PII).
Before moving on to the discussion of concerns about ownership and handling, however, it is important to understand the process of collecting primary data. The process of primary data collection consists of several steps, from questionnaire development, to enumerator training. Each of these steps are listed below, and require detailed planning, and coordination among the members of the research team.
The first step of primary data collection is to design a survey instrument (or questionnaire). It is important to remember that drafting a questionnaire from scratch can be a time-consuming process, so the research team should try to use existing resources as far as possible. While developing the questionnaire, keep the following things in mind:
- Modules. Divide the questionnaire into individual modules, each with a group of questions that are related to one aspect of the survey. Unless the context of the study is entirely new, perform a literature review of existing well-tested and reliable surveys to prepare the general structure of the questionnaire. One example of a resource for past studies and questionnaires is the World Bank Microdata Library.
- Measurement challenges. Often, research teams face challenges in measuring certain outcomes, for instance, abstract concepts (like empowerment), or socially sensitive topics that people do not wish to talk about (like drug abuse). In such cases, try to use indicators that are easy to identify, or build a level of comfort with respondents before moving to the sensitive topics.
- Translation. Translating the questionnaire is a very important step. The research team must hire only professional translators to translate the questionnaire into all local languages that are spoken in the study location.
Survey pilot is the process of carrying out interviews and tests on different components of a survey, including content and protocols. A good pilot provides the research team with important feedback before they start the process of data collection. This feedback can help the research team review and improve instrument design, translations, as well as survey protocols related to interview scheduling, sampling, and geo data.
A pilot has three stages - pre-pilot, content-focused pilot, and data-focused pilot. Typically, the pilot is carried out before hiring a survey firm. The research team must draft a clear timeline for the pilot, and allocate enough time for each component of the pilot. DIME Analytics] has also created the following checklists to assist researchers and enumerators in preparing for, and implementing a pilot:
- Checklist: Preparing for a survey pilot
- Checklist: Refining questionnaire content
- Checklist: Refining questionnaire data
Pilot Recruitment Strategy
- Smaller sample. Before finalizing the target population, the research team often surveys a smaller sample of the population that is selected for an intervention. This is often done to double-check sampling and power calculations, and verify whether the strategy for recruiting individuals for the data collection is effective or not.
- Low or unknown take-up. Take-up rate is the percentage of eligible people who accept a benefit, or participate in data collection. Sometimes the take-up rates can be low, or unknown. In such cases, the research team should reconsider the sampling strategy, and test it before starting data collection.
- Different participation rates. Sometimes participation can differ based on factors like gender, age, social status, etc. This requires the research to consider different strategies like stratified sampling.
One of the ways to do this is to test 3 different recruitment strategies, say, A, B, and C. The research team can then finalize the strategy that has the highest take-up rates. Another method is identifying the ideal incentives which can ensure higher participation by the eligible population.
TOR and Procurement
Researchers must prepare a survey budget before procuring a survey firm. This step allows researchers to calculate expected costs of conducting a study, and compare these with the proposals of firms that submit an expression of interest (EOI).
Determine relevant parameters of a study
After agreeing upon a budget, researchers then decide upon factors like the adequate sampling frame (which is a list of individuals or units in a population from which a sample can be drawn), sample size, and statistical power based on which they can then randomize treatment.
Procure a survey firm
Data Quality Assurance Plan
Obtain Ethical Approval
There are strict rules about acquiring approval from human subjects. Researchers must understand the ethics and rules for security of sensitive data, and should use proper tools for encryption and de-identification of personally identifiable information (PII).
After validating the programming of the questionnaire, the researchers train enumerators and monitor data quality to generate a final draft of the instrument. Monitoring can be done in the form of back checks, high frequency checks, as well as other methods.
- Oxfam, Brief on Planning Survey Research
- DIME (World Bank), Guide on Planning, Preparing & Monitoring Household Surveys
- DIME Analytics (World Bank), Guidelines on Preparing for Data Collection
- Oxfam, Case study on using electronic data collection (SurveyCTO) and Stata to improve data quality in the field