DataWork Survey Round

This article describes the organization of a Survey Round folder inside the DataWork folder in DIME's standardized template for how to organize a the data work in project folder.

Subfolders in each Survey Round

These are the folders that iefolder creates inside the survey round folder:

• DataSets Folder
• Dofiles Folder
• Outputs Folder
• Documentation Folder
• Questionnaire Folder

DataSets Folder

Image 1. Example of a DataSets folder. (Click to enlarge.)

The DataSets folder contains three sub-folders: Deidentified, Intermediate and Final (Image 1).

Note that it is not recommended to create a folder for raw data in the DataSets folder: raw data almost always includes identifying information and the DataSets folder is not in the encrypted branch of the DataWork folder. Instead, save raw data in the DataWork Survey Encrypted Data folder. The Raw Data section describes best practices for raw data in more detail.

De-identified Folder

Once the raw data is de-identified, it can be saved in the De-identified folder. Working with the raw, de-identified data rather than raw, encrypted data will make the work flow easier for the entire research team.

Intermediate Folder

The Intermediate folder is a work-in-progress folder containing all datasets that belong neither in the Raw nor Final folder. Raw datasets on which simple changes have been made belong in the Intermediate folder. There are no specific rules how this folder should be organized. However, it is a good idea to keep it organized in sub-folders and to name files according to efficient naming conventions.

Final Folder

The Final folder contains the clean datasets with final variables constructed. This folder should only contain one version of each dataset. If your project has various datasets for different units of observation, (i.e. student dataset, school dataset, teacher dataset), then the Final folder should contain a sub-folder for each dataset. The Final folder is the folder most likely to be visited several years after the project has ended by someone with very little knowledge of the project. This folder should therefore be one of the folders organized to the most level of detail.

Dofiles Folder

Image 2. Example of a DataSets folder. (Click to enlarge.)

Each Survey Round folder contains a Dofiles folder. The top level of this folder should only contain a master do-file and sub-folders. The master do-file maps out all do-files and datasets relevant to this survey round.

The sub-folders in the Dofiles folder should be organized according to task. For example, import, cleaning, analysis. All do-files related to each of these tasks should be saved in those folders. Make sure to use naming conventions and version control rather than having multiple versions of the same dofile. Each sub-folder should have a task level master do-file.

Task-level do-files have very little technical purpose as the do-files can technically be called from the round master do-file. However, the task-level master do-files are critical for documentation for data work and make it significantly easier for someone unfamiliar with the code to follow the data work.

Outputs Folder

The output folder should also be organized in sub-folders. Which sub-folders thatare needed depends on what is outputted and the method to do so. If, for example, both tables and graphs are outputted then it probably makes sense to have separate folders for each of them. In these folders there should be sub-folders called raw and formatted if thatis relevant. Examples when that is relevant is if tables are exported in .csv format and the data is used as input in the formatted file, or for when single graphs are outputted to disk and later combined to one graph file with multiple graphs. See our articles on exporting analysis for many more recommendations in relation to this.

This folder is one of the few examples where it could be good practice to save multiple versions of the same file. The reason for this is that it is common to compare different versions of the analysis, and it is convenient to be able to do so without using any version control software and then re-generating an old output. But do not use the convention of calling it table_v1, table_v2 etc. Call them by date, for example table_Apr30, table_Jun6 etc. Although, do not do this if the outputted file size is significant in terms of disk space. Then the multiple version of the file will soon take up a lot of space on the disk, and will make it impossible for people with a slow internet connection to connect to the folder over syncing services like DropBox.

Documentation Folder

This folder will contain the documentation for the analysis done including any duplicate reports, survey logs, etc. There is no strict format for this folder. But it is a really good practice to save any documentation typed up for this survey round in this folder. It could be anything from formal documentation to email conversations etc. You never know what you and your project team will need to know about this round in the future to be able to conduct the analysis.

Questionnaire Folder

Most Impact Evaluation projects collect data using surveys, and all the documentation in relation to the data collection should be gathered in this folder. The most important thing to document is the questionnaire, hence the name of the folder, but it is also important to save any other material that documents the survey. That includes material from the enumerator training, survey manuals used in the field, contract with survey firm, and anything else that can be used for the project team to get a better qualitative understanding of their quantitative data.

Encrypted Round Folder

Image 5. Example of a DataSets folder. (Click to enlarge.)

The encrypted round folders are created by iefolder at the same time as a round folder is created, but they are created in the encrypted branch of DataWork instead of inside the round Folder. The reason the encrypted round folders are separated from the other round folders as they are likely to include identifying data and the encrypted branch should be encrypted.

Raw Data

Any data downloaded from the server used to collect data should be saved in the Raw Identified Data folder.

This folder should contain the datasets in exactly the same state as you got them. This includes data downloaded from the internet, data received from data collection, and data received from other projects. The data in this folder should be exactly as you got it and absolutely no changes should be made to it. Even simple changes like correcting known errors, changing variable names, changing format from csv to Stata or other formats, or file name changes should never be done to any files in this folder. The only exception to this rule is if the file name needs to be changed to be imported, then the file name changes can be done in this folder.

If there are known errors in your raw datasets, then you should import the raw data set, write a dofile that corrects the errors and then save the corrected data in the intermediate folder. The data set in the raw folder with known errors should remain unaltered in the raw folder. This is the only way corrections can be fully documented. If corrections to the datasets are not documented, then we will not fully understand the quality of our data, and that means that we will not fully understand the quality of our research.

Import Dofiles

The dofiles used to import, edit and de-identify the raw data is likely to include idenitfying information. Therefore we need to keep those files in the encryoted folder as well.

Running these files should import the data, address any immediate issues such as missing IDs or duplicates, de-identify it and save the data set in the Intermediate folder in the DataSets folder in the branch of the DataWork folder that is not encrypted.

Back to Parent

This article is part of the topic DataWork Folder.