Difference between revisions of "Questionnaire Design"

Jump to: navigation, search
Line 1: Line 1:
A well-designed questionnaire results from careful planning, consideration of analysis and indicators, close review of existing questionnaires, [[Survey Pilot | survey pilots]], and team and partner review. Questionnaire design, which takes place after compiling the [[Survey Budget | survey budget]] and determining the sampling frame and before [[Enumerator Training | training enumerators]], plays a critical role in determining data quality. This page will outline the process and guidelines of designing a questionnaire and provide insight into how to address challenges to measurement.


== Read First ==<onlyinclude>
== Read First ==
'''Do''' start with a careful review of existing survey instruments that cover similar topics. '''Don't''' reinvent the wheel -- working from a high-quality, previously-piloted survey instrument will save time and improve the quality of your final output. </onlyinclude>
*Before designing the questionnaire, carefully review existing survey instruments that cover similar topics.  
*Don't reinvent the wheel: working from a high-quality, previously-piloted survey instrument will save time and improve the quality of your final output.  
* To avoid recall bias, use objective indicators as much as possible.
* Sometimes, seemingly simple survey questions are actually quite nuanced. Wording questions clearly, [[Survey Pilot | piloting]] questionnaires thoroughly, [[Enumerator Training | training]] enumerators well, and including definitions within the questionnaire all help to ensure that the questionnaire consistently elicits the same information across respondents.


 
== Process ==
== Guidelines ==
 
=== Questionnaire design process ===


When designing a survey instrument from scratch, follow these steps:
When designing a survey instrument from scratch, follow these steps:


# Review (or draft) a [[Theory of Change]] and [[Reproducible Research#Pre-Analysis Plan | Pre-Analysis Plan]].  
# Review (or draft) a [[Theory of Change | theory of change]] and [[Pre-Analysis Plan | pre-analysis plan]].
# Make a list of all intermediary and final outcomes of interest, as well as important covariates and sources of heterogeneity
# Make a list of all intermediary and final outcomes of interest, as well as important covariates and sources of heterogeneity.
# Prepare an outline of questionnaire modules, based on the above list. Get feedback from research team.  
# Prepare an outline of questionnaire modules, based on the above list. Get feedback from research team.  
# For each module, prepare a list of specific indicators to measure. Get feedback from research team and implementing partners.  
# For each module, prepare a list of specific indicators to measure. Get feedback from research team and implementing partners.  
# [[Literature Review for Questionnaire|Review existing questionnaires]] and compile databank of relevant questions for each module
# [[Literature Review for Questionnaire|Review existing questionnaires]] and compile a databank of relevant questions for each module.
# Draft questionnaire, noting source of each question (e.g. source: Uganda National Panel Survey (LSMS 2013-14), source: Uganda DHS 2011, source: Uganda Social Assistance Grants for Empowerment Programme 2013, Evaluation Follow-Up Survey [http://microdata.worldbank.org/index.php/catalog/2653], source: own design - extra attention required in pilot), and get feedback from research team and implementing partners
# Draft the questionnaire and note the source of each question (i.e. source: Uganda DHS 2011, source: Uganda Social Assistance Grants for Empowerment Programme 2013, Evaluation Follow-Up Survey [http://microdata.worldbank.org/index.php/catalog/2653], source: own design - extra attention required in pilot). Get feedback from research team and implementing partners.
# [[Survey Pilot#Guidelines for Survey Pilot| Content-based Pilot]] & resulting revisions
# Conduct a [[Survey Pilot#Guidelines for Survey Pilot| content-based pilot]] and make revisions accordingly.
# [[Questionnaire Translation]] & [[Questionnaire Programming]] (can happen concurrently)
# [[Questionnaire Translation | Translate]] & [[Questionnaire Programming | program]] the survey (can happen concurrently).


Designing a follow-up questionnaire is simpler. Try to keep as close to the baseline survey instrument as possible, to facilitate panel analysis. It is better to add/subtract questions than to modify existing ones.
Designing a follow-up questionnaire is simpler. Try to keep it as close to the baseline survey instrument as possible in order to facilitate panel analysis. It is better to add and/or subtract questions than to modify existing ones.


=== Basic Rules for Questionnaire Design ===
== Guidelines ==


#Begin the questionnaire with an [[Informed Consent | informed consent]] form
#Identify each survey respondent and each survey with [[ID_Variable_Properties| Unique IDs]]
# Group questions into modules
# Group questions into modules
#* Write an introductory script for each module, to guide the flow of the interview
#* Write an introductory script for each module, to guide the flow of the interview. For example: ''Now I would like to ask you some questions about your relationships. It’s not that I want to invade your privacy. We are trying to learn how to make young people’s lives safer and happier. Please be open because for our work to be useful to anyone, we need to understand the reality of young people’s lives. Remember that all your answers will be kept strictly confidential.''  
#** Example: ''Now I would like to ask you some questions about your relationships. It’s not that I want to invade your privacy. We are trying to learn how to make young people’s lives safer and happier. Please be open because for our work to be useful to anyone, we need to understand the reality of young people’s lives. Remember that all your answers will be kept strictly confidential.''  
# All questions should have pre-coded answer options. Answer options must:
# All questions should have pre-coded answer options. Answer options must be:
#* Be clear, simple, and mutually exclusive
#* Clear, simple, and mutually exclusive
#* Be exhaustive (tested and refined during the [[Survey Pilot]])
#* Exhaustive (tested and refined during the [[Survey Pilot]])
#*Include 'other' (but if >5% of respondents choose 'other', answer choices were insufficiently exhaustive)
#*Include 'other' (but if >5% of respondents choose 'other', answer choices were insufficiently exhaustive)
# Include hints to the enumerator as necessary, typically coded to appear in italics (not part of the question read to the respondent)
# Include hints to the enumerator as necessary. These hints are typically coded to appear in italics to clarify that they are not part of the question read to the respondent.
#* Example: "For how many months did you work in the last 12 months? ''Enumerator: if less than 1 month, round up to 1''
#*For example: "For how many months did you work in the last 12 months? ''Enumerator: if less than 1 month, round up to 1.''
 
==Challenges to Measurement ==
===Nuanced Definitions===
Sometimes, seemingly simple survey questions are actually quite nuanced. For example , while ''household size'' seems relatively straight-forward, it in fact depends entirely on the definition of ''household member.'' Is a household member anyone currently living in the household? Anyone who has lived more than 6 of the last 12 months in the household? Is a domestic worker a household member? Are students away at school who are economically dependent on the household considered household members? What about a household head who has migrated but sends remittances back to support the household? As a second example of nuanced data points, while it may seem that all respondents should know their age, ''age'' can be difficult if people are innumerate, do not have birth certificates, or do not know their birth year.


==== Key elements all questionnaires must have ====
Pay careful attention during the [[Survey Pilot]] for questions that are hard for the respondent; adjust the questionnaire and training accordingly. Wording questions clearly, [[Survey Pilot | piloting]] questionnaires thoroughly, [[Enumerator Training | training]] enumerators well, and including definitions within the questionnaire all help to ensure that the questionnaire consistently elicits the same information across respondents.
* [[ID_Variable_Properties| Unique ID]]
* [[Human_Subjects_Approval#Informed Consent | informed consent]]  
* Most surveys  also include identification of survey respondent


=== Measurement Issues ===
===Recall Bias and Estimations===
Keep in mind that even simple-seeming data points might not be simple to capture. For example, ''Household size'' will depend entirely on how 'household member' is defined (only those currently living in the household? those who have lived more than 6 of the last 12 months in the household? what about domestic servants? students away at school who are economically dependent on the household? Household head who has migrated but sends remittances back to support the household?)
When asking survey respondents to recall or estimate information (i.e. income in the last year, consumption last week, plot size, amount deposited in bank account last month), be aware of [[Recall Bias | recall bias]]. To avoid recall bias, use objective indicators as much as possible. For example, rather than asking a respondent the size of her agricultural plot, it is better to measure the plot area directly using GPS devices. Rather than asking a respondent how many times she deposited money in her bank account last month, it is better to acquire administrative bank data for accuracy. However, objective measures are often more expensive and may not always be possible. In these cases, make use of internal consistency checks, multiple measurements, and contextual references to ensure high quality data.


Types of data that are hard to measure in a questionnaire include:
=== Sensitive Topics ===
==== [[Difficult topics]] ====
* Things that are hard to estimate or hard to remember.
** Examples include: distance to grocery store, profits, plot size, income in the last year. See [[Recall Bias]].
* You should pay careful attention during the [[Survey Pilot]] for questions that are hard for the respondent. Questions that seem obvious to you may not be easy to answer, depending on the context. For example, ''Age'' can be difficult if people are innumerate, do not have birth certificates, or do not know their birth year.


==== [[Sensitive Topics|Sensitive and/or taboo topics]] ====
For [[Sensitive Topics| certain topics perceived as socially undesirable]] (i.e. drug/alcohol use, sexual practice,s violent behaviors, criminal activities), respondents may have incentives to conceal the truth due to taboos or social pressure. This can create bias, the size and direction of which can be hard to predict. To avoid this, enumerators should guarantee anonymity and confidentiality during the [[Informed Consent | informed consent]] section. Further, [[Survey Protocols | survey protocols]] should guarantee privacy and maximize trust. Consider asking the question in third person, framing the questions to avoid social desirability bias or even possibly allowing respondents to self-administer certain modules. Note that experimental methods such as [[Randomized Response Technique | randomized response technique]], [[List Experiments | list experiments]] and [[Endorsement Experiments | endorsement experiments]] can also help elicit accurate data on sensitive topics.  
* Includes any topic perceived as socially undesirable
* Examples include: drug/alcohol use, sexual practices, violent behaviors, criminal activities.


==== [[Abstract concepts]]====
=== Abstract concepts===
* May be defined differently across cultures or may not translate well
[[Abstract concepts]] such as [[Measuring Empowerment|empowerment]], risk aversion, social cohesion or trust may be defined differently across cultures and/or may not translate well. To measure abstract concepts, first define the concept, then choose the outcome you will use to measure that concept, and finally design a good measure for that outcome.  Pilot the question and measurement well.
* Examples include: [[Measuring Empowerment|empowerment]], [[risk aversion]], bargaining power, social cohesion


==== Outcomes that are not directly observable ====
=== Outcomes Not Directly Observable ===
*Examples include: corruption, quality of care.
* Strategies to use include: Audit Studies
* It is always best to directly measure outcomes when possible. For example, consider the following two measures of literacy:
** "Can you read?" ''Answer choices'': yes, no
** "Can you please read me this sentence?" [Enumerators holds up showcard with a sentence written in the local language]. ''Answer choices:'' read sentence correctly, read sentence with some errors, unable to read sentence


The second option, a more objective measure, is always preferable.
For outcomes not directly observable (i.e. corruption, quality of care), audit studies can help elicit accurate data. In general, it is always best to directly measure outcomes when possible. As a basic example, consider the following example of measuring literacy:
* "Can you read?" ''Answer choices'': yes, no
* "Can you please read me this sentence?" [Enumerators holds up card with a sentence written in the local language]. ''Answer choices:'' read sentence correctly, read sentence with some errors, unable to read sentence.
The second option, a more objective measure, is always preferable.  


== Back to Parent ==
== Back to Parent ==
Line 69: Line 63:
== Additional Resources ==
== Additional Resources ==


Comprehensive resources on survey design
* Grosh and Glewwe’s [http://documents.worldbank.org/curated/en/452741468778781879/pdf/multi-page.pdf Designing Household Survey Questionnaires for Developing Countries: Lessons from 15 Years of the Living Standards Measurement Study]
* Margaret Grosh and Paul Glewwe.  2000. Designing Household Survey Questionnaires for Developing Countries: Lessons from 15 Years of the Living Standards Measurement Study. Volumes 1, 2, and 3.  The World Bank.[http://documents.worldbank.org/curated/en/452741468778781879/pdf/multi-page.pdf]
* Dhar’s [https://www.povertyactionlab.org/sites/default/files/documents/Instrument%20Design_Diva_final.pdf Instrument Design 101] via Poverty Action Lab
* Dhar, Diva. Instrument Design 101 [Powerpoint Slides]. Retried from https://www.povertyactionlab.org/sites/default/files/documents/Instrument%20Design_Diva_final.pdf
* McKenzie’s [http://blogs.worldbank.org/impactevaluations/three-new-papers-measuring-stuff-difficult-measure Three New Papers Measuring Stuff that is Difficult to Measure] via The World Bank’s Development Impact Blog
 
* Lombardini’s [http://policy-practice.oxfam.org.uk/blog/2017/02/real-geek-faq-how-can-i-measure-household-income measuring household income] via Oxfam
* Development Impact Blog: Three New Papers Measuring Stuff that is Difficult to Measure [http://blogs.worldbank.org/impactevaluations/three-new-papers-measuring-stuff-difficult-measure]
* Zezza et al.’s [https://www.sciencedirect.com/science/article/pii/S0306919217306802?via%3Dihub Measuring food consumption and expenditures in household consumption and expenditure surveys (HCES)]
 
*DIME Analytics’ [https://github.com/worldbank/DIME-Resources/blob/master/survey-instruments.pdf Survey Instruments Design & Pilot]
Measurement Topics
*DIME Analytics’ [https://github.com/worldbank/DIME-Resources/blob/master/survey-preparing.pdf Preparing for Data Collection]
* Oxfam on measuring household income: http://policy-practice.oxfam.org.uk/blog/2017/02/real-geek-faq-how-can-i-measure-household-income
*DIME Analytics’ [https://github.com/worldbank/DIME-Resources/blob/master/survey-guidelines.pdf Survey Guidelines]
 
*DIME Analytics’ [https://github.com/worldbank/DIME-Resources/blob/master/survey-cto.pdf SurveyCTO] slides
* [https://www.sciencedirect.com/science/article/pii/S0306919217306802?via%3Dihub | Measuring food consumption and expenditures in household consumption and expenditure surveys (HCES)]
 
 
[[Category: Questionnaire Design]] [[Category: Primary Data Collection]]
[[Category: Questionnaire Design]] [[Category: Primary Data Collection]]

Revision as of 15:07, 28 May 2019

A well-designed questionnaire results from careful planning, consideration of analysis and indicators, close review of existing questionnaires, survey pilots, and team and partner review. Questionnaire design, which takes place after compiling the survey budget and determining the sampling frame and before training enumerators, plays a critical role in determining data quality. This page will outline the process and guidelines of designing a questionnaire and provide insight into how to address challenges to measurement.

Read First

  • Before designing the questionnaire, carefully review existing survey instruments that cover similar topics.
  • Don't reinvent the wheel: working from a high-quality, previously-piloted survey instrument will save time and improve the quality of your final output.
  • To avoid recall bias, use objective indicators as much as possible.
  • Sometimes, seemingly simple survey questions are actually quite nuanced. Wording questions clearly, piloting questionnaires thoroughly, training enumerators well, and including definitions within the questionnaire all help to ensure that the questionnaire consistently elicits the same information across respondents.

Process

When designing a survey instrument from scratch, follow these steps:

  1. Review (or draft) a theory of change and pre-analysis plan.
  2. Make a list of all intermediary and final outcomes of interest, as well as important covariates and sources of heterogeneity.
  3. Prepare an outline of questionnaire modules, based on the above list. Get feedback from research team.
  4. For each module, prepare a list of specific indicators to measure. Get feedback from research team and implementing partners.
  5. Review existing questionnaires and compile a databank of relevant questions for each module.
  6. Draft the questionnaire and note the source of each question (i.e. source: Uganda DHS 2011, source: Uganda Social Assistance Grants for Empowerment Programme 2013, Evaluation Follow-Up Survey [1], source: own design - extra attention required in pilot). Get feedback from research team and implementing partners.
  7. Conduct a content-based pilot and make revisions accordingly.
  8. Translate & program the survey (can happen concurrently).

Designing a follow-up questionnaire is simpler. Try to keep it as close to the baseline survey instrument as possible in order to facilitate panel analysis. It is better to add and/or subtract questions than to modify existing ones.

Guidelines

  1. Begin the questionnaire with an informed consent form
  2. Identify each survey respondent and each survey with Unique IDs
  3. Group questions into modules
    • Write an introductory script for each module, to guide the flow of the interview. For example: Now I would like to ask you some questions about your relationships. It’s not that I want to invade your privacy. We are trying to learn how to make young people’s lives safer and happier. Please be open because for our work to be useful to anyone, we need to understand the reality of young people’s lives. Remember that all your answers will be kept strictly confidential.
  4. All questions should have pre-coded answer options. Answer options must:
    • Be clear, simple, and mutually exclusive
    • Be exhaustive (tested and refined during the Survey Pilot)
    • Include 'other' (but if >5% of respondents choose 'other', answer choices were insufficiently exhaustive)
  5. Include hints to the enumerator as necessary. These hints are typically coded to appear in italics to clarify that they are not part of the question read to the respondent.
    • For example: "For how many months did you work in the last 12 months? Enumerator: if less than 1 month, round up to 1.

Challenges to Measurement

Nuanced Definitions

Sometimes, seemingly simple survey questions are actually quite nuanced. For example , while household size seems relatively straight-forward, it in fact depends entirely on the definition of household member. Is a household member anyone currently living in the household? Anyone who has lived more than 6 of the last 12 months in the household? Is a domestic worker a household member? Are students away at school who are economically dependent on the household considered household members? What about a household head who has migrated but sends remittances back to support the household? As a second example of nuanced data points, while it may seem that all respondents should know their age, age can be difficult if people are innumerate, do not have birth certificates, or do not know their birth year.

Pay careful attention during the Survey Pilot for questions that are hard for the respondent; adjust the questionnaire and training accordingly. Wording questions clearly, piloting questionnaires thoroughly, training enumerators well, and including definitions within the questionnaire all help to ensure that the questionnaire consistently elicits the same information across respondents.

Recall Bias and Estimations

When asking survey respondents to recall or estimate information (i.e. income in the last year, consumption last week, plot size, amount deposited in bank account last month), be aware of recall bias. To avoid recall bias, use objective indicators as much as possible. For example, rather than asking a respondent the size of her agricultural plot, it is better to measure the plot area directly using GPS devices. Rather than asking a respondent how many times she deposited money in her bank account last month, it is better to acquire administrative bank data for accuracy. However, objective measures are often more expensive and may not always be possible. In these cases, make use of internal consistency checks, multiple measurements, and contextual references to ensure high quality data.

Sensitive Topics

For certain topics perceived as socially undesirable (i.e. drug/alcohol use, sexual practice,s violent behaviors, criminal activities), respondents may have incentives to conceal the truth due to taboos or social pressure. This can create bias, the size and direction of which can be hard to predict. To avoid this, enumerators should guarantee anonymity and confidentiality during the informed consent section. Further, survey protocols should guarantee privacy and maximize trust. Consider asking the question in third person, framing the questions to avoid social desirability bias or even possibly allowing respondents to self-administer certain modules. Note that experimental methods such as randomized response technique, list experiments and endorsement experiments can also help elicit accurate data on sensitive topics.

Abstract concepts

Abstract concepts such as empowerment, risk aversion, social cohesion or trust may be defined differently across cultures and/or may not translate well. To measure abstract concepts, first define the concept, then choose the outcome you will use to measure that concept, and finally design a good measure for that outcome. Pilot the question and measurement well.

Outcomes Not Directly Observable

For outcomes not directly observable (i.e. corruption, quality of care), audit studies can help elicit accurate data. In general, it is always best to directly measure outcomes when possible. As a basic example, consider the following example of measuring literacy:

  • "Can you read?" Answer choices: yes, no
  • "Can you please read me this sentence?" [Enumerators holds up card with a sentence written in the local language]. Answer choices: read sentence correctly, read sentence with some errors, unable to read sentence.

The second option, a more objective measure, is always preferable.

Back to Parent

This article is part of the topic Questionnaire Design and Translation

Additional Resources