Dimewiki

Difference between revisions of "List Experiments"

Jump to: navigation, search
 
(5 intermediate revisions by 2 users not shown)
Line 1: Line 1:
<onlyinclude>
A list experiment is a [[Questionnaire Design|questionnaire design]] technique used to mitigate respondent social desirability bias when eliciting information about [[Sensitive Topics |sensitive topics]]. With a large enough [[Sample Size | sample size]], list experiments can be used to estimate the proportion of people for whom a sensitive statement is true.
A technique to get around [[Social Desirability Bias]], typically used when trying to measure [[Sensitive Topics]]


</onlyinclude>
== Read First ==
*A list experiment requires that you [[Randomization|randomly]] divide the [[Sampling|sample]] into two groups: the Direct Response Group and the Veiled Response Group.
*Downsides to list experiments include possible introduction of noise to the data, and possible influence of the [[Randomized Control Trials|RCT treatment]] on the distributions of responses.
 
==Overview==
 
List experiments present respondents with a list of items, experiences, or statements . Respondents must then report how many items in the list pertain to them.
In a list experiment, the [[Sampling|sample]] is randomly divided into two groups: the Direct Response Group and the Veiled Response Group. While the Direct Response Group is presented with a list of neutral and non-sensitive items (length=N), the Veiled Response Group is presented with an identical list plus the [[Sensitive Topics|sensitive item]] (length=N+1). With a large enough '''sample''', researchers can estimate the proportion of people to whom the '''sensitive item''' pertains by subtracting the average response of the Direct Response Group from the average response of the Veiled Response Group.


== Read First ==
List experiments provide respondents with an additional level of privacy, as the researcher can never perfectly infer an individual’s answer to the '''sensitive item''' unless either 0 or N+1 items are true.


== Issues with list experiments ==


== What is a list experiment? ==
List experiments require respondents to count/add, possibly introducing noise to the data – especially if the list is long. Further, unless the Direct Response Group questions are completely unrelated and have responses with known distributions, the [[Randomized Control Trials|RCT treatment]] may affect the response distributions. List experiments also reduce [[Power Calculations|power]]. However, as they are individually [[Randomization|randomized]], you may not have to allocate half of your [[Sampling|sample]] but rather only 5-10% of your '''sample''' to the Direct Response Group.


List experiments aggregate responses to sensitive questions with responses to non-sensitive questions, e.g. how many of the following statements do you agree with? This provides the respondent with an additional level of privacy, as the researcher can never perfectly infer an individual’s answer to the sensitive item (unless either 0 or N+1 items are true).
== Examples ==


Procedure: Randomly divide the sample into two groups
===Example 1===
* ''Direct response'': report how many of N items are true for themselves, where items are neutral and non-sensitive
* ''Veiled response'': report how many of N+1 items are true, with N items being identical to control group’s items, and the N+1st item being a sensitive item
With a large enough sample, estimate the population mean for the N+1st item (sensitive item), by differencing out the mean of the sum of N items estimated from the control. In other words, subtracting the average response in control group from the average response in the treatment group yields the proportion of people who say 'yes' to the sensitive statements.


== Guidelines ==
'''Enumerator reads: “How many of the following statements are true for you?” '''
=== Examples of list experiments ===
From ''Coffman, K. B., Coffman, L. C., & Ericson, K. M. M. (2016). The size of the lgbt population and the magnitude of antigay sentiment are substantially underestimated. Management Science.''


How many of the following statements are true for you?
'''Direct Response Group List: '''


Group A (direct response) gets:
# I remember where I was the day of the Challenger space shuttle disaster
# I remember where I was the day of the Challenger space shuttle disaster
# I spent a lot of time playing video games as a kid
# I spent a lot of time playing video games as a kid
Line 28: Line 29:
# I have voted for a political candidate who was pro-life
# I have voted for a political candidate who was pro-life


Group B (veiled response) gets:
'''Veiled Response Group List: '''
The same list as Group A, plus
 
# I remember where I was the day of the Challenger space shuttle disaster
# I spent a lot of time playing video games as a kid
# I would vote to legalize marijuana if there was a ballot question in my state
# I have voted for a political candidate who was pro-life
# I consider myself to be heterosexual


: 5. I consider myself to be heterosexual
The above example comes from [https://pdfs.semanticscholar.org/2c09/f0ab565539d5a0038486ef1048d427d4710a.pdf Coffman, Coffman, and Ericson 2016].  


===Example 2===


From ''Gilens, M., Sniderman, P. M., and Kuklinski, J. H. (1998). Affirmative action and the politics of realignment. British Journal of Political Science 28, 1, 159– 183.''
'''Enumerator reads: “Now I am going to read you three things that sometimes make people angry or upset. After I read all, just tell me ''how many'' of them upset you. I don’t want to know which ones, just ''how many''.” '''


"Now I am going to read you three things that sometimes make people angry or upset. After I read all three, just tell me HOW MANY of them upset you. I don’t want to know which ones, just HOW MANY."
'''Direct Response Group List: '''


Group A gets the list:
# The federal government increasing the tax on gasoline;
# The federal government increasing the tax on gasoline;
# Professional athletes earning large salaries;
# Professional athletes earning large salaries;
Line 44: Line 50:
# Large corporations polluting the environment
# Large corporations polluting the environment


Group B receives Group A's list, plus:
'''Veiled Response Group A List: '''


: 5. Black leaders asking the government for affirmative action
# The federal government increasing the tax on gasoline;
# Professional athletes earning large salaries;
# Requiring seat belts be used when driving;
# Large corporations polluting the environment
# Black leaders asking the government for affirmative action


Group C receives Group A's list, plus:
'''Veiled Response Group B List: '''
 
: 5. Awarding college scholarships on the basis of race
 
 
=== Issues with List Experiments ===
* Require people to count/add, possibly introducing noise to the data (especially if the list is long)
* Unless the “innocent” questions are completely unrelated and have a known distribution, there is a chance that the treatment in your RCT might have an effect on its distribution.
** Moreover, designing your common questions that way makes your sensitive ones stand out even more. 
* Reduces power
** However, as this is individually randomized, may not have to allocate half of your sample to direct response (instead could do 5-10%)


# The federal government increasing the tax on gasoline;
# Professional athletes earning large salaries;
# Requiring seat belts be used when driving;
# Large corporations polluting the environment
# Awarding college scholarships on the basis of race


The above example comes from [https://www.cambridge.org/core/journals/british-journal-of-political-science/article/affirmative-action-and-the-politics-of-realignment/932AD683804205C8D01CFD67EED3D339 Gilens, Sniderman, and Kuklinski 1998]. 
== Back to Parent ==
== Back to Parent ==
This article is part of the topic [[Questionnaire Design]]
This article is part of the topic [[Questionnaire Design]]


== Additional Resources ==
== Additional Resources ==
* Development Impact Blog posts by Berk Ozler.
*Ozler via The World Bank Development Impact blog discusses [https://blogs.worldbank.org/impactevaluations/issues-data-collection-and-measurement issues of data collection and measurement]
:: Issues of data collection and measurement" https://blogs.worldbank.org/impactevaluations/issues-data-collection-and-measurement
*Ozler’s follow-up post via The World Development Impact blog discusses [https://blogs.worldbank.org/impactevaluations/list-experiments-sensitive-questions-methods-bleg list experiments for sensitive experiments]
:: "List Experiments for Sensitive Questions – a Methods Bleg" https://blogs.worldbank.org/impactevaluations/list-experiments-sensitive-questions-methods-bleg
*Ozler's [https://blogs.worldbank.org/impactevaluations/sex-lies-and-measurement-do-indirect-response-survey-methods-work-no Sex, Lies and Measurement: Do Indirect Response Survey Methods Work?]
 
*Andrew Gelman provides [https://andrewgelman.com/2014/04/23/thinking-list-experiment-heres-list-reasons-think/ reasons] to question list experiments
 
* Blair, Imai and Lyall’s [http://www.jasonlyall.com/wp-content/uploads/2014/02/combine.pdf paper] compares, combines and analyzes the effectiveness of list and endorsement experiments
* Blair, Graeme, Kosuke Imai, and Jason Lyall. 2014. “Comparing And Combining List and Endorsement Experiments: Evidence from Afghanistan.” American Journal of Political Science 58(4): 1043–63.  
*Blair and Imai develop a set of [https://imai.fas.harvard.edu/research/files/listP.pdf new statistical methods] for list experiments
::''Abstract'': List and endorsement experiments are becoming increasingly popular among social scientists as indirect survey techniques for sensitive questions. When studying issues such as racial prejudice and support for militant groups, these survey methodologies may improve the validity of measurements by reducing non-response and social desirability biases. We develop a statistical test and multivariate regression models for comparing and combining the results from list and endorsement experiments. We demonstrate that when carefully designed and analyzed, the two survey experiments can produce substantively similar empirical findings. Such agreement is shown to be possible even when these experiments are applied to one of the most challenging research environments: contemporary Afghanistan. We find that both experiments uncover similar patterns of support for the International Security Assistance Force among Pashtun respondents. Our findings suggest that multiple measurement strategies can enhance the credibility of empirical conclusions. Open-source software is available for implementing the proposed methods.
 
 
* Blair, Graeme, and Kosuke Imai. 2012. “Statistical Analysis of List Experiments.” Political Analysis 20(1): 47–77.
::''Abstract'': The validity of empirical research often relies upon the accuracy of self-reported behavior and beliefs. Yet eliciting truthful answers in surveys is challenging, especially when studying sensitive issues such as racial prejudice, corruption, and support for militant groups. List experiments have attracted much attention recently as a potential solution to this measurement problem. Many researchers, however, have used a simple difference-in-means estimator, which prevents the efficient examination of multivariate relationships between respondents’ characteristics and their responses to sensitive items. Moreover, no systematic means exists to investigate the role of underlying assumptions. We fill these gaps by developing a set of new statistical methods for list experiments. We identify the commonly invoked assumptions, propose new multivariate regression estimators, and develop methods to detect and adjust for potential violations of key assumptions. For empirical illustration, we analyze list experiments concerning racial prejudice. Open-source software is made available to implement the proposed methodology.
 
 
* Reasons to be skeptical of list experiments, from Andrew Gelman, Macartan Humphreys, Lynn Vavreck, Cyrus Samii, Simon Jackman, Brendan Nyhan: http://andrewgelman.com/2014/04/23/thinking-list-experiment-heres-list-reasons-think/  
 


[[Category: Experimental Methods]]
[[Category: Experimental Methods]]

Latest revision as of 18:12, 9 August 2023

A list experiment is a questionnaire design technique used to mitigate respondent social desirability bias when eliciting information about sensitive topics. With a large enough sample size, list experiments can be used to estimate the proportion of people for whom a sensitive statement is true.

Read First

  • A list experiment requires that you randomly divide the sample into two groups: the Direct Response Group and the Veiled Response Group.
  • Downsides to list experiments include possible introduction of noise to the data, and possible influence of the RCT treatment on the distributions of responses.

Overview

List experiments present respondents with a list of items, experiences, or statements . Respondents must then report how many items in the list pertain to them. In a list experiment, the sample is randomly divided into two groups: the Direct Response Group and the Veiled Response Group. While the Direct Response Group is presented with a list of neutral and non-sensitive items (length=N), the Veiled Response Group is presented with an identical list plus the sensitive item (length=N+1). With a large enough sample, researchers can estimate the proportion of people to whom the sensitive item pertains by subtracting the average response of the Direct Response Group from the average response of the Veiled Response Group.

List experiments provide respondents with an additional level of privacy, as the researcher can never perfectly infer an individual’s answer to the sensitive item unless either 0 or N+1 items are true.

Issues with list experiments

List experiments require respondents to count/add, possibly introducing noise to the data – especially if the list is long. Further, unless the Direct Response Group questions are completely unrelated and have responses with known distributions, the RCT treatment may affect the response distributions. List experiments also reduce power. However, as they are individually randomized, you may not have to allocate half of your sample but rather only 5-10% of your sample to the Direct Response Group.

Examples

Example 1

Enumerator reads: “How many of the following statements are true for you?”

Direct Response Group List:

  1. I remember where I was the day of the Challenger space shuttle disaster
  2. I spent a lot of time playing video games as a kid
  3. I would vote to legalize marijuana if there was a ballot question in my state
  4. I have voted for a political candidate who was pro-life

Veiled Response Group List:

  1. I remember where I was the day of the Challenger space shuttle disaster
  2. I spent a lot of time playing video games as a kid
  3. I would vote to legalize marijuana if there was a ballot question in my state
  4. I have voted for a political candidate who was pro-life
  5. I consider myself to be heterosexual

The above example comes from Coffman, Coffman, and Ericson 2016.

Example 2

Enumerator reads: “Now I am going to read you three things that sometimes make people angry or upset. After I read all, just tell me how many of them upset you. I don’t want to know which ones, just how many.”

Direct Response Group List:

  1. The federal government increasing the tax on gasoline;
  2. Professional athletes earning large salaries;
  3. Requiring seat belts be used when driving;
  4. Large corporations polluting the environment

Veiled Response Group A List:

  1. The federal government increasing the tax on gasoline;
  2. Professional athletes earning large salaries;
  3. Requiring seat belts be used when driving;
  4. Large corporations polluting the environment
  5. Black leaders asking the government for affirmative action

Veiled Response Group B List:

  1. The federal government increasing the tax on gasoline;
  2. Professional athletes earning large salaries;
  3. Requiring seat belts be used when driving;
  4. Large corporations polluting the environment
  5. Awarding college scholarships on the basis of race

The above example comes from Gilens, Sniderman, and Kuklinski 1998.

Back to Parent

This article is part of the topic Questionnaire Design

Additional Resources

Retrieved from "https://dimewiki.worldbank.org/index.php?title=List_Experiments&oldid=9169"