What is Needed in the Questionnaire?
To select the right mode for a survey, we have already considered both the audience and the sample size. To get the best output from the desired audience, the mode and the questionnaire must be thoughtfully aligned.
Ideally, the mode will be selected prior to designing the survey questionnaire, as there are different best practices on both design and programming by mode. We know from working with fast-paced clients that this isn’t always possible. At Coleman, we partner with our clients to balance those time considerations, either suggesting the mode that best fits the questionnaire or by tweaking the questionnaire to best fit the mode, depending on which solution better aligns to the overall project priorities.
There are three main aspects of a questionnaire that should be considered when aligning to a mode: the length, the structure, and the desired data type.
How long is the questionnaire?
Any market researcher would recommend keeping the survey as short as possible. However, a common fear is needing information that wasn’t asked in the survey. The typical thought process is that the risk of not collecting enough data is greater than the risk of drop-offs or quality issues due to survey length—and so surveys balloon in size and scope. We understand it’s a tricky balance, but it’s important to think carefully about the desired data so as not to overwhelm respondents.
When balancing survey length versus other factors, it may be helpful to keep a few basic benchmarks in mind. While different target audiences may react differently toward survey length, here’s a simple heuristic to gauge an appropriate length.
Regarding mode, questionnaire length is a double-edged sword. Figure that a phone interview will take about twice as long to administer as the same questionnaire taken online. Logically it follows that the longer the questionnaire the less suited it becomes for CATI—but that’s not the whole story.
With an online survey, there are few mechanisms to monitor a respondent’s attention. However, with CATI, we not only monitor but also actively engage each respondent. Many people don’t realize how difficult it is to provide undivided attention to an online survey for even 15 minutes — let alone 20, 30, or 40. So, even though that same survey might be closer to 45-60 minutes, engaging people actively on the phone might just be the way to hold on to the respondent all the way to the end.
Takeaway: CATI surveys increase the duration for the respondent but also keep the respondent more engaged from start to finish. It’s a trade-off.
How is the questionnaire structured?
Questionnaire structure and content can often determine the mode to choose. A picture is worth 1,000 words — that’s a lot to convey over the phone! Visuals, complex ideas, wordy descriptions, and large lists will work better online, or with a screen-sharing addition to CATI outreach.
For online, consider how the survey is designed and how it looks on a variety of devices. For instance, we know that many millennials are connected to their phones throughout the day. A survey targeting this age group could be thoughtfully designed to increase the mobile compatibility of the programmed link. For example, matrix questions can be broken into individual questions with the same answer scale to eliminate the need for horizontal scrolling.
For CATI, consider how the survey sounds. Reading the questionnaire aloud is a simple but effective method to make sure your design works and your language is clear. You can also leave instructions for moderators on how to approach certain questions.
Takeaway: Intuitive design is extremely important for online, as respondents will receive no instructions beyond what they see on the screen. Less so for CATI, as moderators can be instructed how to deliver the questions.
What kind of data will be collected?
Most of the surveys we run at Coleman are quantitative in nature (though we have successfully run many in-depth interviews, or IDIs). However, even quantitative surveys may contain qualitative elements. When considering which mode to employ, consider the nature and frequency of open-ended questions. Also ask yourself how important this information is to you. Some of our clients see qualitative comments as enrichment. Without making these questions mandatory, any data is better than nothing. However, other clients rely on the insight derived from qualitative questions.
There are some ways to boost open-ended responses within online surveys, such as forcing a character minimum, but respondents cannot be made to type thoughtful answers to all questions. Entering “NA” or “Not at this time” are valid responses, though they may be disappointing.
By nature, a CATI survey is more conversational and, therefore, facilitates qualitative responses. Additionally, moderators can be instructed to ask a simple “Is there anything else?” after key qualitative questions to probe for additional information.
Takeaway: The more open ends within a survey — and the more important they are — the riskier it becomes to pose these questions online. Questionnaires with more than two to three open ends push the bounds of online research.
In the next post on our series, we will discuss identifying project timelines when it comes to different modes.