When assessing a survey through the lens of length and timing, nearly all question types can be placed on a spectrum and bucketed into low, medium, and high engagement. Understanding how question types are bucketed can provide perspective on how long a survey may take to complete before going into the field. This advance knowledge of the survey length helps when assessing the feasibility and will save time in the long run by avoiding numerous revisions.
Here are some common question types and estimated response times.
Let’s go into each of these categories in more detail.
Low-Engagement Questions
Low-engagement questions can be answered quickly, requiring little mental thought or consideration, and we usually expect them to take no more than 10 or 15 seconds to answer. These are typically single- and multi-select question types.
Evaluating the ownership of a product or whether the respondent has experienced certain life events, for example, should be easy to answer. Sometimes these questions can be answered in simple, Yes or No terms, like ‘Do you own a vehicle?’ Sometimes the question lists multiple answers, such as ‘Which of the following medications have you taken in the past month?’
In the example below, we asked IT administrators the type of hosting model their company uses. These kinds of questions should not be difficult for the average respondent to recall and should take just a few moments to elicit a response.
Medium-Engagement Questions
Questions that take more than a couple moments to answer but not more than about 60 seconds to complete can be classified into a middle bucket. Some examples include numeric open-ended questions where respondents are asked to quantify an answer freely without selecting from a list of options. Respondents might be asked how many hours they spend per week playing videos games on a PC, for example. Numeric open-end questions require minor mental calculations, and we expect respondents to spend a few moments gathering his or her thoughts before answering this question.
Another example of a moderately time-consuming question is a matrix or grid question. The time it takes to answer these types of questions can vary depending on the size and number of attributes asked. A matrix with three attributes to assess will be answered more quickly than a matrix with 10 attributes. In any case, we expect most grid questions to take more than just a couple of moments to complete.
In a recent survey, we used a matrix to ask respondents to describe their company’s ability to achieve 10 different business goals, including ensuring cyber security and containing costs. This question was presented with a five-point scale and took just under 60 seconds to complete via phone. In an online version of the same survey, the question took 30-40 seconds to complete.
A note of caution for survey writers: while it may be tempting to use matrix questions to limit the number of questions while still collecting feedback on a wide variety of attributes, too many matrix questions can fatigue a respondent and risk collecting inconsistent or insincere data. Sometimes it’s better to break up those matrix questions!
High-Engagement Questions
In the third bucket are questions that require the most engagement and typically take more than a minute to answer. Examples of these questions include open-end questions, in which a respondent must answer a question freely without selecting from a list of answer options. We expect respondents to spend a few moments formulating their response. Additional time is taken for the respondent to supply an actual response, whether that be typing the answer in an online survey or communicating the answer to a moderator over the telephone. Time spent on open-end questions can vary depending on the content of the question asked, but in most cases, we expect completion to take over a minute.
Ranking questions, where respondents are tasked with ranking multiple attributes against each other, is another example of a high engagement question. The question requires the respondent to process each statement then mentally evaluate the importance of each attribute. In the example below, we asked pharmacists to rank factors important when recommending certain types of medication. This question took well over a minute to complete.
Because survey length is a key component to a successful quantitative study, the Coleman Research survey team implements a variety of methods to ensure projects run efficiently and achieve the desired research goals while being considerate of the audience’s time. We will advise on which questions types best serve the ultimate goals while weighing the survey length. Ultimately, we aim to be thought partners with our clients throughout the process, including during the survey design stage.