Garbage In, Garbage Out: Mistakes to Avoid When Writing a Survey
February 28th, 2013 by Megan Crowley in Talent Systems and Processes“Garbage in, garbage out” was a phrase often used by my favorite professor in grad school. He told us that if we didn’t ask the right questions, we wouldn’t get the right data, and we wouldn’t make the right decisions. With the greater emphasis in talent management on using data to inform decisions and reinforce initiatives, survey writing skills are becoming increasingly more important (data geeks rejoice!).
In a previous blog I presented 10 questions you should ask before you develop a survey. Below are five common mistakes to avoid when writing a survey and my proposed solutions/better ideas:
| Mistake | Problem It Creates | Solution/Better Idea | |
| 1 | Response scales with too many options | If you have too many response options, it becomes difficult for the survey participant to make meaningful differentiations. (How different is “seldom” from “once in a while”?) | I recommend using a 4-to-7-point scale. Anything fewer than 4 points and you may be losing meaningful differentiation; anything more than 7 and you often complicate your ability to interpret the results. |
| 2 | Ambiguous response scales | If you don’t define what each number on your scale means, you’re leaving a lot open to interpretation. When 1 = Never and 5 = Always, does 2 = Sometimes or Rarely . . . or something else? | Define and label each number/response option. (Need help? Check out this document.) |
| 3 | Ambiguous items | Again, this leaves too much open to interpretation. How would you respond to: “I like my job”? Do you define your job in terms of your work tasks, position, industry, company, level, or your overall work experience? | Be specific. Define unique terms. Don’t use acronyms or jargon. Don’t be fancy. |
| 4 | Absolutes in items | “My manager is always prepared for meetings.” Maybe my manager is prepared 99 percent of the time, but she was unprepared once so I mark “disagree” on this item. Is my response really sending the company the right information? | Don’t use absolutes! Use a frequency response scale if you are interested in the frequency of a particular behavior. In this example, you might say, “My manager is prepared for meetings (1 = Rarely, 2 = Sometimes . . . )” |
| 5 | Double-barreled items* *This is a very common mistake! | “I am satisfied with my pay and benefits” is a double-barreled item. What if I am satisfied with my pay but not my benefits? If most participants disagreed with this statement, could you confidently say the results indicate that people are unhappy with their pay? What if just the benefits are dissatisfying? | Make this two separate questions. If you’re interested in assessing both together, use “I am satisfied with my total rewards package (including salary, bonus, benefits, etc.)” |
Once the survey is written, run a pilot test! Have someone else (preferably a small group of people) take the survey and give you feedback on the items and how they interpreted them.
These are just some of the common mistakes I see. What other mistakes have you seen? What are your tips for writing great surveys? How are you using surveys in your organization?
Megan Crowley is an associate consultant at FlashPoint. With a background in industrial/organizational psychology, Megan contributes a unique perspective based on some of the newest research and techniques in her field.
Image: Keerati
This post currently has no responses.




Recently my coworker Tam asked me if I had heard about the clerk who got fired from Harrods, the high-end department store in London. Harrods, Tam explained, has a strict dress code that requires female clerks to wear makeup. The woman refused and was let go.
One of my roles at