Randomization in SurveyCTO

During surveys, you might often need to randomize various aspects of the questionnaire. While SurveyCTO has a random number generator, is is usually not recommended that you use it. This article will argue for doing the randomization in Stata, R, or similar software, before the start of the survey, and preload the results of the randomization as dummies or categorical variables.

Read first

Unless your survey falls into very rare exceptions, do not randomize in SurveyCTO, do it in StataR or similar and preload the results into SurveyCTO.

Why randomization is better to do before the Survey

During surveys, we often need to randomize various aspects of the questionnaire. For example – sometimes we need to randomize which household members to interview, and sometimes which set of questions to ask. While most CAPI software have random number generators, it is not the preferred option. Using, for example, Stata to randomize and then preloading the generated data file into the survey software is in almost all cases the better option among the two. The main advantages of using Stata over CAPI software during randomization are as follows:

• Randomization in Stata is transparent and reproducible which is important for [[Publishing Data|publishing research].
• Randomization results in Stata are dependent, guaranteeing that no disproportional large share of the results falls into any group. Randomization is always independent in SurveyCTO which means that it is possible no groups are assigned observations if the number of observation per groups is low.
• Randomization in Stata provides the option of ensuring that the result of the randomization is balanced over other variables, i.e. stratas. This means that we can, for example, guarantee that not all female respondents end up in a certain group.
• Randomization in Stata is done before the survey takes place. This provides an opportunity to double check the result of a randomization and fix bugs and typos in the randomization code before it is used in the field, as it then would be too late to fix.

How to randomize in Stata for a survey in SurveyCTO

This is a basic example on how to do randomize in Stata and preload the results to SurveyCTO. See randomization in Stata more details for how to implement more advanced randomization, but the procedure for how to preload the result will still be the same as described below.

In the example below we have a survey in which we want a random 30% to answer to a long survey and the rest to take a shorter survey. We also have a gender variable and we want the ratio of 30/70 to be as exact as possible within both genders.

Step by step - known respondents

• Let's say we a dataset with all the respondents and replacement respondents, and that we have the following variables: unique_id and gender. Make sure that all observations have uniquely and fully identifying ID in unique_id.
  • If you do not have ID for your respondents you should create it.
  • If you do not have a list of your respondents (for example, if you randomize the sample in the field by drawing lots) see the section below.
• Set version, set seed and sort the data to guarantee a replicable sort. See replicable randomization in Stata why all these three variables are needed.
• Generate a random number and create a dummy variable (long_survey in this example) that indicates for each observation if that observation should answer the long or short survey.
  • Note that it does not have to be a dummy, we could just as well have had randomized into three groups and saved the result into a categorical variable with the value 1, 2, 3.
• In the end we should have a dataset with the unique_id and the categorical variables with the result of the randomization, in this example only long_survey.
• In your SurveyCTO survey, have a question where the enumerator enters the ID for the respondent currently being interviewed. For this to be possible the enumerator needs a list with both the name and the ID.
  • It is very important that these lists are not publicly disclosed as that would allow anyone to identify observations even in datasets that have names and other identifying information removed. If the data collected is extra sensitive, consider using a different ID than the main ID for this purpose.
• Preload the data set you generated in Stata into your SurveyCTO survey using the ID entered by the enumerator.
• Restrict the relevant section of the survey in SurveyCTO using the value just preloaded.

Code example - known respondents

*Set version 
ieboilstart , version(12.1)
`r(version)'

*Open the data set after ieboilstart
use sample.dta

*Set seed
set seed 123456 //this is an example seed, replace this with another number

*Sort data set
sort unique_id

*Generate random number, rank that random number per gender, and assign 
* long survey if the rank is less than or equal the total number of observations 
* in that gender
gen rand = uniform()
bys gender : egen rank = rank(rand)
bys gender : gen long_survey = (rank/_N <= .3)

Preload randomization without IDs

It is still possible to preload randomized categories even if there is no way of knowing the respondents in the survey. For example, you might only have a list of villages to survey and intend to do a lottery to decide which respondents to interview. Or you might have randomized school classes and you will interview all students in those classes but you do not know exactly how many students each class has. In both these cases, we can not assign a randomized category to each respondent as we do not know who they are yet. Instead we will randomize a category to the first respondent in the village/class, randomize a category for the second respondent, etc. And in the field, the enumerators have lists for each village/class where they cross off each ID being used.

The following list explains the steps needed for this. We will keep using the example where we know which school classes we want to interview but not the size of those classes. Let's say that we want to collect basic demographic information on all students and want to do a short survey on 40% of the students and a long one on 10% of the students.

Step by step - unknown respondents

• Step 1: Create a dataset with one observation per class if you do not already have that. Make sure that all classes have unique IDs.
• Expand the dataset (code example below) with the maximum number of students you want to interview per class, or the maximum number of students there are in a class if you want to interview all of them. If you expect that the highest of number of students in a class is 50, expand the number of observations to 100 so that there is no risk your expectation was too low. We pick the number 99 so that no ID requires three digits.
• Create a unique ID for each student. It is easy to do in combination with the class ID.
• Set version, set seed and sort the data to guarantee a replicable sort. See replicable randomization in Stata for why all three variables are needed.
• Since we are generating 99 IDs but most classes will be considerably smaller, there is a risk that all or a large share of the 10% of the students that will take the long surveys are among the higher IDs 50-99 that we never expect to be used. To guarantee that that will never happen, we can create stratas for each group of 10 IDs, 1-10, 11-20, 21-30 etc. and randomize within each group. See the code example for how to do so.
  • There are many different ways to do this. We can make sure that any number of categories shows up among the first 10 or 20 numbers. Just make sure that the students are selected in random order so that the order in which they are assigned IDs is random. The way they sit in a class room might not be random as it might be differences in motivation or socioeconomic status between the students in the front or in the back.
• The example code below generates a variable surveyType with a randomized category for the first respondent, the second, the third etc., up to 99. This variable is preloaded the same way as explained above where we have IDs for all respondents.
• The final step is to create a system where the enumerators use each ID in correct order and only once. The best way to do that is to print out lists for each village where the enumerator crosses off the used IDs.
  • If there are multiple enumerators per class, then the lists can be split into IDs with odd or even numbers and the list is crossed of by the enumerators.
• Make sure that the randomization information is not on the lists available to the enumerators so that the enumerator has a change to influence the assignment of ID.
  • For the same reason we need to randomize differently for each class as enumerators quickly learn the randomization order if it would have been the same for each class.

Code example - unknown respondents

ieboilstart , version(12.1)
`r(version)'

*Open the data set with all classes you have randomized after ieboilstart
use class.dta

* Create one observation per each student ID. We use 99 as the highest number of students
expand 99

* Create unique ID
bys class_id : gen student_id = class_id * 100 + _n

* Generate strata for each group of 10 id, 1-10, 11-20, 21-30 etc.
bys class_id : gen id10strata = floor(_n/10)

*Set seed and sort
set seed 123456 //this is an example seed, replace this with another number
sort student_id

*Generate the result variable and the random number, rank that random number 
* per class_id and ID strata, and assign survey type to 2 if the rank is less than or 
* equal the total number of observations in that class and ID strata, and assign survey 
* type 3 if the rank is larger than 90% of the total number of observations in that 
* class and ID strata. The rest of the observations keep the survey type 1.
gen surveyType = 1
gen rand = uniform()
bys class_id id10strata : egen rank = rank(rand)
bys class_id id10strata : replace surveyType  = 2 if (rank/_N <= .5)
bys class_id id10strata : replace surveyType  = 3 if (rank/_N > .9)

Other things to consider- unknown respondents

'Test field system: The most obvious step where this could go wrong is when enumerators use the correct ID and do not use IDs more than once. This may not be done properly, or the system they are using is not designed well enough, but we can easily check that by looking into which IDs are used by each enumerator each day. We should already have a good understanding in the pilot if our system is good.

Stratification: If you want to stratify over, for example, gender, you can have two lists, one for male respondents and one for female respondents. However, this increases the complexity in the field and more than one layer of strata organized in this way immediately becomes very complicated. This is one of the few cases where randomization in SurveyCTO is preferred. However, make sure that there are enough observations in each strata so that even the independent randomization will have a balanced result.

Back to Parent

This article is part of the topic Randomized Control Trials

Additional Resources

• SurveyCTO Randomizing Survey Elements
• SurveyCTO Randomizing Form Elements
• SurveyCTO Randomizing Form Section Order