Master Do-files

The master do-file is the main do file that is used to call upon all the other do files. By running this file, all files needed from importing raw data to cleaning, constructing, analysing and outputting results should be run. This file therefore also functions as a map to the data folder.

Read First

• The person creating the master do file should be able to run the do files from all stages(cleaning, construct, analysis, exporting tables, etc) from the master do-file and the someone else running the master do file should be able to run all of those just by changing the paths to their Dropbox/Box folders.

Purpose of a Master Do-File

A master do-file has the following three purposes:

• The first reason is that it possible to run all code related to a project by running only one dofile. This is incredible important for replicability.
• The second purpose is to set up globals. The most important type of global to be set up is folder paths that enables dynamic file paths that in turn allows multiple users run the same code, it shortens the file paths as well as making it possible to move files and folders with minimal updates to the code. Standardized conversion rates are another example of global.
• The third purpose is that this file is the main map to the DataWork folder.

Each of these purposes are described in more detail below.

Purpose 1. Run do-files needed for the data work

As projects grow large it is impractical to write all code in a single do-file. And even the code needed for a high level tasks such as cleaning or analysis is too long to be in one do-file. It would be impossible to run all of these files manually as the number of files grow, but a master do-file is a solution to that. Using Stata's do command a do-file can run other do-files. In a master do-file you will find a section where all do-files needed for the project are ran. Typically you will have multiple master do-files. For example, you have one for each data collection, baseline, endline etc. that runs the files needed for all the data work for that round. Then you will have a project master do-files that runs the round master do-files if you want to run all code related to the project with one click.

Purpose 2. Set up globals

In the master do-file we store a wide variety of information in globals that we want to make sure is used exactly the same across the project. Here are some of the most common types of globals set up in the master do-file, but any project could have unique types of globals.

File paths. Most projects use a shared folder service like DropBox, Box, OneDrive, Google Drive etc. All collaborators on a project are likely to have slightly different file path to these shared folders, for example, "C:\Users\JohnSmith\Dropbox" or "C:\Users\AnnDoe\Dropbox". We can use global to get dynamic file paths in order for collaborators being able to use the same do-file. We can also use globals to shorten the file paths used in the code. Also, if someone would move where they have their Dropbox, they would only have to update this one global instead of every file reference in all do-files.

Conversion rates. Another common usage of globals is to standardize conversion rates. If you, for example, need to convert amounts between currencies in your code, then you should have a global where you store that conversion rate, and each time you convert an amount you reference the global instead of typing it out each time you need the currency exchange rate. This practice reduce the risk for typos, and it reduces the risk of different conversion rates from different sources or points in time are used in the project. It also makes updates very easy, as you only need to update the global at one location and then run master do-file to update all the analysis. You should use this for standardized conversion for length, weight, volume etc., as well as for your best estimation for local non-standardized conversion rates.

Purpose 3. Map to DataWork Folder

Since all code can be run from this file, all important folders are pointed to with globals, and since all outputs are (indirectly) created by this file, this file is the starting point to find where any do-file, data set or output is located in the DataWork folder. Another examples of files that help with the navigation of the folder could be a Word document or a PDF describing how to navigate the sub-folders. Such files are not included in our folder template, but may sometimes be a good addition. However, those files needs to be updated in parallel which often does not happen even if that is the intention.

Components of a Master Do file

Since the master do file acts as a map to all the other do files in the project, it is important that the do file is organized and contains all the information necessary during the analysis. Some of the necessary components of a do file are as follows:

Intro Header

The intro header should contain some descriptive information about the do file such that somebody who doesn't know the do file can read it and understand what the do file does and what it produces. Some examples of information to put on the header are the purpose of the do file, the outline of the do file, the data files required to run the do file correctly, the data files created by the do file, the name variable that uniquely identifies the unit of observation in the datasets, etc.

Settings to Declare in the Master do-file

After the intro header, settings that are used throughout the project should also be declared in the master do-file. Some of the settings are as follows:

Version Settings

The version settings for Stata needs to be declared in the master do-file. Since, things like Stata's randomization algorithm sometimes changes across versions, it is important to declare Stata's version number to make sure that the analysis done using Stata is reproducible.

Example: version 12.0

Basic and Advanced Memory Limits

Memory limits on Stata affect various things like the maximum number of variables a dataset can have ( maxvar), number of variables than be used during Stata's estimation commands matsize, the amount and time Stata uses the system memory niceness / min_memory / max_memory , etc. Declaring the memory limits on the master do-files makes sure that the analysis runs smoothly with maximum efficiency.

Example: set maxvar 20000

Default Options

Default options like setting more off/on, pause on/off, and abbreviation should also be set in the master do file. Declaring options in the main file ensures that when the other do-files are run through the master do file, the settings do not have to be declared again.

Example: set varabbrev off

Standardization of Units and Assumptions

Conversion rates for standardization of units and assumptions that need to be defined should be defined as globals in the master do-files. Varlist commonly used across the projects are also defined using globals/locals in the master do file. Since, globals defined in one do file also work on other do files throughout a Stata session, it is important to declare all the global variables necessary during the project on the master do-file.

Installing any user written commands

User written commands that need to be installed for the do-file should also be declared in the master do-file. Since, every computer that runs the code will not have the commands installed, it is necessary to install those commands. For example to install the command outreg2 used for exporting regression results in LaTeX and text formats, you should declare ssc install outreg2, replace . The replace makes sure that the latest version of the command with updated functionalities is installed if any previous versions have already been installed on the computer.

Sub Master do-file(s)

Sub Master do-files are similar to a Master do-file except they perform a singular function, whereas the Master do-file runs all the necessary do-files from the raw data stage to the analysis and output stage. A sub Master-do file could be a do-file that runs all the do-files and commands used to generate all the graphs produced for a project. Instead of including each do-file that was used to produce the graphs needed for a project in the Master do-file, one could create a sub Master-do file for graphs outputs that will be called by the Master do-file. Following this technique one could have a sub Master do-file for graphs outputs, regressions, and data cleaning; all of which will be called upon by the Master do-file.

Implementation

Example of the settings declared in a master do file.

DIME's Stata command ieboilstart from the ietoolkit package declares all the necessary basic settings to standardize the code across multiple people working on the same project. This can be done adding the following 2 lines of code to every do-files.

ssc install ietoolkit, replace
ieboilstart, versionnumber(version_number) options
`r(version)'

Declaring these commands at the top of do file used by every member of the project ensures that the version settings are the same across all runs for the project. However, the globals and any extra commands installed should be declared as well.

Back to Parent

This article is part of the topic Data Management