Questionnaire Design

This topic cover questionnaire design and measurement issues.

Read First

Do start with a careful review of existing survey instruments that cover similar topics. Don't reinvent the wheel -- working from a high-quality, previously-piloted survey instrument will save time and improve the quality of your final output.

Guidelines

Questionnaire design process

When designing a survey instrument from scratch, follow these steps:

1. Review (or draft) a Theory of Change and Pre-Analysis Plan.
2. Make a list of all intermediary and final outcomes of interest, as well as important covariates and sources of heterogeneity
3. Prepare an outline of questionnaire modules, based on the above list. Get feedback from research team.
4. For each module, prepare a list of specific indicators to measure. Get feedback from research team and implementing partners.
5. Review existing questionnaires and compile databank of relevant questions for each module
6. Draft questionnaire, noting source of each question (e.g. source: Uganda National Panel Survey (LSMS 2013-14), source: Uganda DHS 2011, source: Uganda Social Assistance Grants for Empowerment Programme 2013, Evaluation Follow-Up Survey [1], source: own design - extra attention required in pilot), and get feedback from research team and implementing partners
7. Content-based Pilot & resulting revisions
8. Questionnaire Translation & Questionnaire Programming (can happen concurrently)

Designing a follow-up questionnaire is simpler. Try to keep as close to the baseline survey instrument as possible, to facilitate panel analysis. It is better to add/subtract questions than to modify existing ones.

Key elements all questionnaires must have

Informed Consent

The Institutional Review Board that granted approval for the study typically provides a template for requesting Informed Consent.

Basic Rules for Questionnaire Design

• Group questions into modules
  • Write an introductory script for each module, to guide the flow of the interview
    • Example: Now I would like to ask you some questions about your relationships. It’s not that I want to invade your privacy. We are trying to learn how to make young people’s lives safer and happier. Please be open because for our work to be useful to anyone, we need to understand the reality of young people’s lives. Remember that all your answers will be kept strictly confidential.

• All questions should have pre-coded answer options. Answer options must be:
  • Clear, simple, and mutually exclusive
  • Exhaustive (tested and refined during the Survey Pilot)
  • Include 'other' (but if >5% of respondents choose 'other', answer choices were insufficiently exhaustive)

• Include hints to the enumerator as necessary, typically coded to appear in italics (not part of the question read to the respondent)
  • For example, "For how many months did you work in the last 12 months? Enumerator: if less than 1 month, round up to 1

Measurement Issues

Keep in mind that even simple-seeming data points might not be simple to capture. For example, Household size will depend entirely on how 'household member' is defined (only those currently living in the household? those who have lived more than 6 of the last 12 months in the household? what about domestic servants? students away at school who are economically dependent on the household? Household head who has migrated but sends remittances back to support the household?

Types of data that are hard to measure in a questionnaire include:

Difficult topics

• Things that are hard to estimate or hard to remember.
• Examples include: distance to grocery store, profits, plot size, income in the last year. See Recall Bias.
• You should pay careful attention during the Survey Pilot for questions that are hard for the respondent. Questions that seem obvious to you may not be easy to answer, depending on the context. For example, Age can be difficult if people are innumerate, do not have birth certificates, or do not know their birth year.

Sensitive and/or taboo topics

• Includes any topic perceived as socially undesirable
• Examples include: substance abuse, sexual practices, criminal behaviors.

Abstract concepts

• May be defined differently across cultures or may not translate well
• Examples include: empowerment, risk aversion, bargaining power, social cohesion

Outcomes that are not directly observable

• Examples include: corruption, quality of care.
• Strategies to use include: Audit Studies, Mystery Patients.

Note: It is always best to directly measure outcomes when possible. For example, consider the following two measures of literacy:

1. "Can you read?" Answer choices: yes, no
2. "Can you please read me this sentence?" [Enumerators holds up showcard with a sentence written in the local language]. Answer choices: read sentence correctly, read sentence with some errors, unable to read sentence

The second option, a more objective measure, is always preferable.

Back to Parent

This article is part of the topic Questionnaire Design

Additional Resources

Comprehensive resources on survey design

• Margaret Grosh and Paul Glewwe. 2000. Designing Household Survey Questionnaires for Developing Countries: Lessons from 15 Years of the Living Standards Measurement Study. Volumes 1, 2, and 3. The World Bank.[2]
• Dhar, Diva. Instrument Design 101 [Powerpoint Slides]. Retried from https://www.povertyactionlab.org/sites/default/files/documents/Instrument%20Design_Diva_final.pdf

• Development Impact Blog: Three New Papers Measuring Stuff that is Difficult to Measure [3]