Difference between revisions of "List Experiments"

Jump to: navigation, search
Line 10: Line 10:
  
 
=== What is a list experiment? ===
 
=== What is a list experiment? ===
 +
List experiments aggregate responses to sensitive questions with responses to non-sensitive questions, e.g. how many of the following statements do you agree with? For a control, the list of statements would not include any sensitive statements. The treatment group would choose from the same set of statements the control group had, plus additional sensitive statements. Subtracting the average response in control group from the average response in the treatment group yields the proportion of people who say 'yes' to the sensitive statements.
 +
 +
For example, from Gilens, M., Sniderman, P. M., and Kuklinski, J. H. (1998). Affirmative action and the politics of realignment. British Journal of Political Science 28, 1, 159– 183.
 +
The control group receives the list:
 +
# The federal government increasing the tax on gasoline;
 +
# Professional athletes earning large salaries;
 +
# Requiring seat belts be used when driving;
 +
# Large corporations polluting the environment
 +
 +
The treatment groups receive the above with either:
 +
# Black leaders asking the government for affirmative action
 +
or
 +
# Awarding college scholarships on the basis of race
 +
 
=== When should I use a list experiment? ===
 
=== When should I use a list experiment? ===
=== Should I do a list experiment? ===
+
 
* Thoughts from Andrew Gelman, Macartan Humphreys, Lynn Vavreck, Cyrus Samii, Simon Jackman, Brendan Nyhan on reasons to be sketpical: http://andrewgelman.com/2014/04/23/thinking-list-experiment-heres-list-reasons-think/
 
  
 
== Back to Parent ==
 
== Back to Parent ==
Line 24: Line 37:
 
* Blair, Graeme, and Kosuke Imai. 2012. “Statistical Analysis of List Experiments.” Political Analysis 20(1): 47–77.  
 
* Blair, Graeme, and Kosuke Imai. 2012. “Statistical Analysis of List Experiments.” Political Analysis 20(1): 47–77.  
 
''Abstract'': The validity of empirical research often relies upon the accuracy of self-reported behavior and beliefs. Yet eliciting truthful answers in surveys is challenging, especially when studying sensitive issues such as racial prejudice, corruption, and support for militant groups. List experiments have attracted much attention recently as a potential solution to this measurement problem. Many researchers, however, have used a simple difference-in-means estimator, which prevents the efficient examination of multivariate relationships between respondents’ characteristics and their responses to sensitive items. Moreover, no systematic means exists to investigate the role of underlying assumptions. We fill these gaps by developing a set of new statistical methods for list experiments. We identify the commonly invoked assumptions, propose new multivariate regression estimators, and develop methods to detect and adjust for potential violations of key assumptions. For empirical illustration, we analyze list experiments concerning racial prejudice. Open-source software is made available to implement the proposed methodology.
 
''Abstract'': The validity of empirical research often relies upon the accuracy of self-reported behavior and beliefs. Yet eliciting truthful answers in surveys is challenging, especially when studying sensitive issues such as racial prejudice, corruption, and support for militant groups. List experiments have attracted much attention recently as a potential solution to this measurement problem. Many researchers, however, have used a simple difference-in-means estimator, which prevents the efficient examination of multivariate relationships between respondents’ characteristics and their responses to sensitive items. Moreover, no systematic means exists to investigate the role of underlying assumptions. We fill these gaps by developing a set of new statistical methods for list experiments. We identify the commonly invoked assumptions, propose new multivariate regression estimators, and develop methods to detect and adjust for potential violations of key assumptions. For empirical illustration, we analyze list experiments concerning racial prejudice. Open-source software is made available to implement the proposed methodology.
 +
 +
* Reasons to be skeptical of list experiments, from Andrew Gelman, Macartan Humphreys, Lynn Vavreck, Cyrus Samii, Simon Jackman, Brendan Nyhan: http://andrewgelman.com/2014/04/23/thinking-list-experiment-heres-list-reasons-think/
  
 
[[Category: Questionnaire Design]]
 
[[Category: Questionnaire Design]]

Revision as of 18:21, 6 February 2017

A technique to get around Social Desirability Bias, typically used when trying to measure Sensitive Topics


Read First

Guidelines

What is a list experiment?

List experiments aggregate responses to sensitive questions with responses to non-sensitive questions, e.g. how many of the following statements do you agree with? For a control, the list of statements would not include any sensitive statements. The treatment group would choose from the same set of statements the control group had, plus additional sensitive statements. Subtracting the average response in control group from the average response in the treatment group yields the proportion of people who say 'yes' to the sensitive statements.

For example, from Gilens, M., Sniderman, P. M., and Kuklinski, J. H. (1998). Affirmative action and the politics of realignment. British Journal of Political Science 28, 1, 159– 183. The control group receives the list:

  1. The federal government increasing the tax on gasoline;
  2. Professional athletes earning large salaries;
  3. Requiring seat belts be used when driving;
  4. Large corporations polluting the environment

The treatment groups receive the above with either:

  1. Black leaders asking the government for affirmative action

or

  1. Awarding college scholarships on the basis of race

When should I use a list experiment?

Back to Parent

This article is part of the topic Questionnaire Design

Additional Resources

  • Blair, Graeme, Kosuke Imai, and Jason Lyall. 2014. “Comparing And Combining List and Endorsement Experiments: Evidence from Afghanistan.” American Journal of Political Science 58(4): 1043–63.

Abstract: List and endorsement experiments are becoming increasingly popular among social scientists as indirect survey techniques for sensitive questions. When studying issues such as racial prejudice and support for militant groups, these survey methodologies may improve the validity of measurements by reducing non-response and social desirability biases. We develop a statistical test and multivariate regression models for comparing and combining the results from list and endorsement experiments. We demonstrate that when carefully designed and analyzed, the two survey experiments can produce substantively similar empirical findings. Such agreement is shown to be possible even when these experiments are applied to one of the most challenging research environments: contemporary Afghanistan. We find that both experiments uncover similar patterns of support for the International Security Assistance Force among Pashtun respondents. Our findings suggest that multiple measurement strategies can enhance the credibility of empirical conclusions. Open-source software is available for implementing the proposed methods.

  • Blair, Graeme, and Kosuke Imai. 2012. “Statistical Analysis of List Experiments.” Political Analysis 20(1): 47–77.

Abstract: The validity of empirical research often relies upon the accuracy of self-reported behavior and beliefs. Yet eliciting truthful answers in surveys is challenging, especially when studying sensitive issues such as racial prejudice, corruption, and support for militant groups. List experiments have attracted much attention recently as a potential solution to this measurement problem. Many researchers, however, have used a simple difference-in-means estimator, which prevents the efficient examination of multivariate relationships between respondents’ characteristics and their responses to sensitive items. Moreover, no systematic means exists to investigate the role of underlying assumptions. We fill these gaps by developing a set of new statistical methods for list experiments. We identify the commonly invoked assumptions, propose new multivariate regression estimators, and develop methods to detect and adjust for potential violations of key assumptions. For empirical illustration, we analyze list experiments concerning racial prejudice. Open-source software is made available to implement the proposed methodology.