List Experiments

A list experiment is a questionnaire design technique used to mitigate respondent social desirability bias when eliciting information about sensitive topics. With a large enough sample size, list experiments can be used to estimate the proportion of people for whom a sensitive statement is true.

Read First

• A list experiment requires that you randomly divide the sample into two groups: the Direct Response Group and the Veiled Response Group.
• Downsides to list experiments include possible introduction of noise to the data, and possible influence of the RCT treatment on the distributions of responses.

Overview

List experiments present respondents with a list of items, experiences, or statements . Respondents must then report how many items in the list pertain to them. In a list experiment, the sample is randomly divided into two groups: the Direct Response Group and the Veiled Response Group. While the Direct Response Group is presented with a list of neutral and non-sensitive items (length=N), the Veiled Response Group is presented with an identical list plus the sensitive item (length=N+1). With a large enough sample, researchers can estimate the proportion of people to whom the sensitive item pertains by subtracting the average response of the Direct Response Group from the average response of the Veiled Response Group.

List experiments provide respondents with an additional level of privacy, as the researcher can never perfectly infer an individual’s answer to the sensitive item unless either 0 or N+1 items are true.

Issues with list experiments

List experiments require respondents to count/add, possibly introducing noise to the data – especially if the list is long. Further, unless the Direct Response Group questions are completely unrelated and have responses with known distributions, the RCT treatment may affect the response distributions. List experiments also reduce power. However, as they are individually randomized, you may not have to allocate half of your sample but rather only 5-10% of your sample to the Direct Response Group.

Examples

Example 1

Enumerator reads: “How many of the following statements are true for you?”

Direct Response Group List:

1. I remember where I was the day of the Challenger space shuttle disaster
2. I spent a lot of time playing video games as a kid
3. I would vote to legalize marijuana if there was a ballot question in my state
4. I have voted for a political candidate who was pro-life

Veiled Response Group List:

1. I remember where I was the day of the Challenger space shuttle disaster
2. I spent a lot of time playing video games as a kid
3. I would vote to legalize marijuana if there was a ballot question in my state
4. I have voted for a political candidate who was pro-life
5. I consider myself to be heterosexual

The above example comes from Coffman, Coffman, and Ericson 2016.

Example 2

Enumerator reads: “Now I am going to read you three things that sometimes make people angry or upset. After I read all, just tell me how many of them upset you. I don’t want to know which ones, just how many.”

Direct Response Group List:

1. The federal government increasing the tax on gasoline;
2. Professional athletes earning large salaries;
3. Requiring seat belts be used when driving;
4. Large corporations polluting the environment

Veiled Response Group A List:

1. The federal government increasing the tax on gasoline;
2. Professional athletes earning large salaries;
3. Requiring seat belts be used when driving;
4. Large corporations polluting the environment
5. Black leaders asking the government for affirmative action

Veiled Response Group B List:

1. The federal government increasing the tax on gasoline;
2. Professional athletes earning large salaries;
3. Requiring seat belts be used when driving;
4. Large corporations polluting the environment
5. Awarding college scholarships on the basis of race

The above example comes from Gilens, Sniderman, and Kuklinski 1998.

Back to Parent

This article is part of the topic Questionnaire Design

Additional Resources

• Ozler via The World Bank Development Impact blog discusses issues of data collection and measurement
• Ozler’s follow-up post via The World Development Impact blog discusses list experiments for sensitive experiments
• Ozler's Sex, Lies and Measurement: Do Indirect Response Survey Methods Work?
• Andrew Gelman provides reasons to question list experiments
• Blair, Imai and Lyall’s paper compares, combines and analyzes the effectiveness of list and endorsement experiments
• Blair and Imai develop a set of new statistical methods for list experiments