Designing a Survey | Assessment and Research

Competencies

Ability to develop appropriate survey questions with consideration of purpose, wording, type, sequencing, and number.

Survey design involves the overall process of creating and implementing a survey intended to gather specific information. If we take into consideration the Assessment Cycle illustrated in Module 3 , survey design would fit within the step: “Determine methodology & build tools.” The six stages of survey design outlined below will assist you with developing a clear, well-built, robust tool that can be used to gather meaningful data.

Start with a clear purpose

Before writing your first survey question, it is critical to identify a purpose for the survey. Determining the “why” assists with prioritizing your goals and drives the direction of your survey. For example, the purpose of developing a survey could be to gather specific feedback from participants of a training that you facilitated. Some questions to answer:
• Why is a survey needed?
• What is the goal of the survey?
• What drives the need for the survey?
• What purpose will the results serve (potentially to prove or improve)?

Determine some key pieces of information

The next step in designing a survey is to determine what pieces of information you would like to obtain from your data. This influences the questions you will ask, and ultimately the variables in the dataset that will be generated from the survey. Using the example from stage one, desired information could include satisfaction, whether learning outcomes were met, or some demographic information about participants. Some questions to answer:
• What do I want to know?
• What variables do I need?

Identify a target population

A well-developed survey has a target audience in mind. Knowing who could potentially complete your survey helps you to tailor the questions to the respondent. For example, the target population for the training you facilitated could be the participants, or if you had small group facilitators, you may want to gather feedback from them. Some questions to answer:
• Do I want to survey everyone in the population, or just a sample?
• What kind of respondents do I need in my sample?
• Who needs to complete this survey?

Develop the Instrument

Developing the instrument involves outlining the questions needed for the survey. This step is the most involved and may take a few iterations to get it to a desired place. Asking good questions elicits a combination of art and science, and tremendous care should be taken to craft them thoughtfully. A detailed breakdown of this stage will follow the overview of the 6 stages of survey design. Some questions to answer:
• What questions do I need to ask?
• How many questions are needed?
• What question types would be most suitable?

Test the survey

One of the most critical stages of survey design is where testing occurs. Once you have developed a draft of the survey items, it is recommended that you recruit a small group of people to pilot the complete instrument. Piloting could involve colleagues, potential respondents, or content experts. Individuals who pilot should send you feedback about aspects of the survey such as flow, question type, wording, sequencing, clarity, comprehensiveness, etc. Some questions to answer:
• Are the questions well-written?
• Are the questions clear?
• Are the questions specific?
• Is the survey too long?
• Are there any grammatical errors?

Administer the survey

After the survey has been tested and adequately revised, you are then ready to administer your survey to the intended audience. Consider specifics (such as timing and method) of the administration to help optimize response rate. It might be helpful to determine some of the administration logistics earlier on the process. Some questions to answer:
• When should I send out the survey?
• Should it be electronic or paper-based?
• For how long should the survey be open?
• Am I aiming for a specific response rate?

Developing the Instrument

The survey should begin with an introductory statement. According to Rea & Parker, “A questionnaire is a conversation, and, like most other conversations, it builds on itself, commencing with an introduction” (p. 32). At the minimum, the introductory statement should include the organization or agency responsible for the survey; the purpose, objective, or goal of the survey; use of results; clause about confidentiality (anonymous vs. identified); and the approximate amount of time it will take to complete the survey.

When drafting survey questions, consider another quote from Rea & Parker: “The questionnaire must be easily understood, internally consistent, and must lend itself to appropriate and meaningful data analysis” (p. 26). It is important to note that a good question has a purpose, is specific, and is clear.

Components of instrument development

A. Sequencing - involves the way in which items are ordered

• Introductory questions – Should be relatively easy to answer and placed towards the beginning of the survey

• Question groups – Are related questions that should be placed together within the survey

• Filtered questions – Are branched questions that are based on previous responses

• Sensitive questions – Should be reviewed for necessity and appropriateness, and should be placed later in the survey

B. Question Format – identifies one of two main types of questions

Closed-Ended – items that provide a fixed list of alternative responses. Examples include multiple choice items, rankings, scaled items, and check-all items.
• Advantages of closed-ended items:
 Provides a uniform list of alternative answers
 Fixed list tends to make question clearer to respondents
 Aids with sensitive questions (provides a list of acceptable responses)
 Less burdensome to the respondent (choose a response instead of construct one)
• Disadvantages of closed-ended items:
 Uncertainty of best fit answer
 Increases inadvertent errors
 Misunderstood question may provide an erroneous response
 Forces respondent to choose the closest representation based on pre-established categories (can mitigate by adding “other: specify”)

Open-ended – permits the respondent an opportunity to openly respond using their own words/numbers (no fixed list)
• Advantages of open-ended items:
 Allows respondent to provide an open response
 Used when closed-ended presents a constraint
• Disadvantages of open-ended items:
 Difficult and/or time-consuming to analyze/interpret
 May get repetitive and or irrelevant responses

Tips
• Use open-ended items sparingly (only when needed)
• At the end of survey, it is acceptable (and beneficial) to include one or two open-ended items where respondents are asked to add any other information, comments, or opinions that pertain to the questionnaire subject matter (Rea & Parker, p. 46).

C. Be BOSS! – Be brief, objective, simple, and specific when writing the questions.

To be BOSS, avoid the following:
• Leading questions – “questions that – by their content, structure, or wording – push the respondent in the direction of a certain answer (Iarossi, p.32).”

• Double barreled questions – “those that might inadvertently confuse the respondent by introducing two or more issues with the expectation of a single response” (Rea &Parker, p.56)

• Jargon – “use language and terminology that exploits the simplest words and phrases” (Iarossi, p. 37)

• Ambiguous words/phrases – “can occur from the use of vague words or phrases” (Rea & Parker, p. 54)

• Emotionally charged words- words that could elicit extreme or strong emotions within the respondent, leading to potentially biased responses

Additional Tips:
• Consider survey length. Surveys that are too long may jeopardize obtaining your desired response rate.

• Provide adequate instructions. Some questions may need additional instructions or explanation to aid respondents with providing an answer to the question. For instance, if you ask for salary information, you may decide to clarify that you only want the annual rate, and that you want respondents to only provide numbers (e.g. 25000).

• Edit the instrument before sharing it with others to pilot.

References

Dillman, D. A., Smyth, J. D., Christian, L. M., & Dillman, D. A. (2009). Internet, mail, and mixed-mode surveys: the tailored design method. Wiley & Sons, c2009.

Henning, G., & Roberts, D. (2016). Student affairs assessment: theory to practice. Stylus Publishing.

Iarossi, G. (2006). The power of survey design: a user’s guide for managing surveys, interpreting results, and influencing respondents. World Bank, c2006.

Rea, L. M., & Parker, R. A. (2005). Designing and conducting survey research: a comprehensive guide. Jossey-Bass, c2005.

Next Module