Whether you are measuring public opinion, evaluating programme impact, or estimating consumer markets, the first step towards implementing an effective survey is designing a good questionnaire. Often it takes longer to design and test thew questionnaire than it takes to collect and analyse the data. In this blog post, we share some insights based on our experiences with questionnaire design, specifically in the context of mobile-based surveys.

Ask good questions

The first thing you need to do is to define your objective. The more precise you are, the easier it is to develop a good questionnaire. For example, saying you want to understand your clients is too broad. Try to be more specific and define exactly what you want to know. Do you want to know who they are and where they live? Do you want to know what they think about your company or organization? Or do you want to map their behaviour? All questions can be answered using mobile-based surveys, but they require that we ask different questions.

Make sure you ask good questions. This may sound obvious but asking bad questions is probably the most common reason why surveys sometimes fail to meet their objectives. The following list provides some general guidance:

Keep questions simple

Make sure you use simple language. Avoid jargon and technical language. Remember you are communicating with people of all from all walks of life. Complex language may cause them to misunderstand the question or even exit the survey.

Keep questions neutral

Avoid loaded words that could lead your respondents towards a specific answer. For example, do not ask, ‘How much do you like the Mayor?’. Instead, ask, ‘How would you rate the Mayor on a scale from 1 to 5, where 1 means low and 5 means high?’. In that way you maintain the neutrality.

Avoid double-barrel questions

When you ask multiple questions at the same time you are likely to get poor responses. For example, do not ask, ‘Do you believe the Government should raise taxes and invest more in education?’. Instead, split the question into two and ask, ‘Do you believe the Government should raise taxes?’, and then, ‘Do you believe the Government should invest more in education?’

Avoid abstract measures

Terms like ‘always’, ‘often’, ‘sometimes’, and ‘rarely’ mean different things to different people, and this can complicate the analysis. When asking respondents how often they watch TV, give them concrete response options, like ‘every day’, ‘once or twice a week’, ‘never’ etc.

Avoid double-negatives

In much the same, terms like ‘not’ and ‘prevent’ can confuse the respondents, particularly when they are combined with ‘yes’ and ‘no’ responses. For example, asking respondents, ‘Do you agree that the Government should not raise taxes’, and giving them the option to answer, ‘yes’ and ‘no’, is likely to generate poor-quality data.

Consider the sequencing of questions

Once you have developed your questions, think about the order in which they asked. The following list provides some guidance:

Start with general questions that are easy to answer

‘How old are you?’. ‘Where do you live?’ ‘What is your level of education?’ These are questions that are (relatively) easy to answer and help warm up the respondent.

Ask most important questions first

Respondents tend to lose interest with each question we ask. Because of this, make sure you ask the most important questions first, without compromising the logic of the questionnaire.

Group questions by theme

Respondents might get annoyed if you jump back and forth between different themes. Try to group your questions by theme, and complete one theme before starting the next.

Ask sensitive questions at the end

Some respondents do not like to answer sensitive questions, for example, about politics, alcohol consumptions, or drug abuse, and they may decide to exit the survey rather than answering your question. If your questionnaire includes sensitive questions, ask them at the end. In that way, you collect as many responses as possible before you risk losing your respondent.

Consider the length of your questionnaire

Some respondents may decide to exit the survey before they complete the entire questionnaire. This is known as attrition and is common a problem in survey research, regardless of how data are collected. In our mobile-based surveys, attrition is typically around 1-2% per question. This means we lose around one fifth of our respondents every time we ask ten questions. To avoid attrition bias, we recommend that questionnaires are limited to 15-20 questions.

Test your questionnaire before you launch

A lot of things can go wrong when you design a questionnaire, particularly when it is converted into SMS and/USSD. The sequence can be changed, or response categories can be split across multiple screens. Because of that, we always advise our clients to test the questionnaire, across multiple phones and multiple mobile network operators, to make sure it surveys exactly as intended.