DataWork Folder
Since the DataWork folder is setup to be used throughout the impact evaluation project, it is important to set it up correctly. Setting the folder up correctly can help increase efficiency of the data work being done and also reduces the sources of errors in data work.
Inside the DataWork folder
Inside the DataWork folder there should only be folders and files that help navigating those folders. Each folder should correspond to a data source (survey-rounds, admin data, monitor data) or be a folder containing meta-data sets such as the master data set.
The most important file that help navigating all sub-folders of the DataWork is the main master do-file. This do-file calls the master do-file of each data source and meta data folder and re-run all code needed to generate all data sets and output in the DataWork folder. Another example of file that helps with the navigation of the folder could be a Word document or a PDF describing how to navigate the sub-folders. It is important that the number of files here are kept to an absolute minimum.
Survey Round
Each round of the survey should have it's own sub-folder inside the data folder. For example - Inside the main data folder, you can have sub-folders like baseline, follow up 1, follow up 2, midline, endline, etc. Each of these folders should have the folders described below.
DataSets folder
The DataSets folder inside a Survey Round folder should be further divided into three sub-folders. These folders are called Raw, Intermediate and Final. Raw and Final have strict rules on which datasets can be saved in those folders. All other datasets should be saved in Intermediate.
- Raw Folder
This folder should contain the datasets in exactly the same state as you got them. This includes data downloaded from the internet, data received from data collection, and data received from other projects. The data in this folder should be exactly as you got it and absolutely no changes should be made to it. Even simple changes like correcting obvious mistakes, changing variable names, changing format from csv to Stata or other formats, file name changes should not be done to the data in this folder. The only exception to this is if the file name needs to be changed to be imported, then the file name changes can be done in this folder.
If there are mistakes in your raw datasets that you know of, write a dofile that corrects that mistake and then save the corrected data in the intermediate folder. This is the only way corrections can be fully documented. If corrections to the datasets are not documented, then we will not fully understand the quality of our data, and that means that we will not fully understand the quality of our research.
- Intermediate Folder
This folder should contain all datasets that are not supposed to be either in the Raw folder (see above) or Final folder (see below). Raw datasets on which simple changes has been made as mentioned above should be put in the intermediate folder. Since this is a work-in-progress folder there are no specific rules how this folder should be organized. It still make sense to keep it organized in sub-folders. And
- Final folder
The Final data folder contains clean, and final constructed datasets.
DoFiles
Each survey round folder will also contain a dofile folder.
- Have a master do file that runs all other dofiles needed for this project. This is also your map to the data folder.
- Organize all other master files in sub folders.
Output Folder
The output folder should the raw and final tables output folders inside it.
Documentation
This folder will contain the documentation for the analysis done including any duplicate reports, survey logs, etc.
Back to Parent
This article is part of the topic Data Management
Additional Resources
- list here other articles related to this topic, with a brief description and link