How to design questionnaire, B R M

1. HOW TO DESIGN QUESTIONNAIRE – BRM
REPORT
Questionnaire Design
Questionnaires are an inexpensive way to gather data from a potentially large number of
respondents. Often they are the only feasible way to reach a number of reviewers large enough
to allow statistically analysis of the results. A well-designed questionnaire that is used effectively
can gather information on both the overall performance of the test system as well as information
on specific components of the system. If the questionnaire includes demographic questions on
the participants, they can be used to correlate performance and satisfaction with the test system
among different groups of users.
It is important to remember that a questionnaire should be viewed as a multi-stage process
beginning with definition of the aspects to be examined and ending with interpretation of the
results. Every step needs to be designed carefully because the final results are only as good as
the weakest link in the questionnaire process. Although questionnaires may be cheap to
administer compared to other data collection methods, they are every bit as expensive in terms
of design time and interpretation.
The steps required to design and administer a questionnaire include:
1. Defining the Objectives of the survey
2. Determining the Sampling Group
3. Writing the Questionnaire
4. Administering the Questionnaire
5. Interpretation of the Results
This document will concentrate on how to formulate objectives and write the questionnaire.
Before these steps are examined in detail, it is good to consider what questionnaires are good at
measuring and when it is appropriate to use questionnaires.
What can questionnaires measure?
Questionnaires are quite flexible in what they can measure, however they are not equally suited
to measuring all types of data. We can classify data in two ways,Subjective vs.
Objective and Quantitative vs. Qualitative.
When a questionnaire is administered, the researchers control over the environment will be
somewhat limited. This is why questionnaires are inexpensive to administer. This loss of control
means the validity of the results are more reliant on the honesty of the respondent.

2. Consequently, it is more difficult to claim complete objectivity with questionnaire data then with
results of a tightly controlled lab test. For example, if a group of participants are asked on a
questionnaire how long it took them to learn a particular function on a piece of software, it is
likely that they will be biased towards themselves and answer, on average, with a lower than
actual time. A more objective usability test of the same function with a similar group of
participants may return a significantly higher learning time. More elaborate questionnaire
design or administration may provide slightly better objective data, but the cost of such a
questionnaire can be much higher and offset their economic advantage. In general,
questionnaires are better suited to gathering reliable subjective measures, such as user
satisfaction, of the system or interface in question.
Questions may be designed to gather either qualitative or quantitative data. By their very nature,
quantitative questions are more exact then qualitative. For example, the word "easy" and
"difficult" can mean radically different things to different people. Any question must be carefully
crafted, but in particular questions that assess a qualitative measure must be phrased to avoid
ambiguity. Qualitative questions may also require more thought on the part of the participant
and may cause them to become bored with the questionnaire sooner. In general, we can say that
questionnaires can measure both qualitative and quantitative data well, but that qualitative
questions require more care in design, administration, and interpretation.
When to use a questionnaire?
There is no all encompassing rule for when to use a questionnaire. The choice will be made
based on a variety of factors including the type of information to be gathered and the available
resources for the experiment. A questionnaire should be considered in the following
circumstances.
a. When resources and money are limited. A Questionnaire can be quite inexpensive to
administer. Although preparation may be costly, any data collection scheme will have similar
preparation expenses. The administration cost per person of a questionnaire can be as low as
postage and a few photocopies. Time is also an important resource that questionnaires can
maximize. If a questionnaire is self-administering, such as a e-mail questionnaire, potentially
several thousand people could respond in a few days. It would be impossible to get a similar
number of usability tests completed in the same short time.
b. When it is necessary to protect the privacy of the participants.Questionnaires are
easy to administer confidentially. Often confidentiality is the necessary to ensure participants
will respond honestly if at all. Examples of such cases would include studies that need to ask
embarrassing questions about private or personal behavior.
c. When corroborating other findings. In studies that have resources to pursue other data

3. collection strategies, questionnaires can be a useful confirmation tools. More costly schemes
may turn up interesting trends, but occasionally there will not be resources to run these other
tests on large enough participant groups to make the results statistically significant. A follow-up
large scale questionnaire may be necessary to corroborate these earlier results.
I. Defining the Objectives of the Survey
The importance of well-defined objectives can not be over emphasized. A questionnaire that is
written without a clear goal and purpose is inevitably going to overlook important issues and
waste participants' time by asking useless questions. The questionnaire may lack a logical flow
and thereby cause the participant to lose interest. Consequential, what useful data you may have
collected could be further compromised. The problems of a poorly defined questionnaire do not
end here, but continue on to the analysis stage. It is difficult to imagine identifying a problem
and its cause, let alone its solution, from responses to broad and generalizing questions. In other
words, how would it be possible to reach insightful conclusions if one didn't actually know what
they had been looking for or planning to observe.
A objective such as "to identify points of user dissatisfaction with the interface and how these
negatively affect the software's performance" may sound clear and to the point, but it is not. The
questionnaire designer must clarify what is meant by user dissatisfaction. Is this dissatisfaction
with the learning of the software, the power of the software, of the ease of learning the software?
Is it important for the users to learn the software quickly if they learn it well? What is meant by
the software's performance? How accurate must the measurements be? All of these issues must
be narrowed and focused before a single question is formulated. A good rule of thumb is that if
you are finding it difficult to write the questions, then you haven't spent enough time defining
the objectives of the questionnaire. Go back and do this step again. The questions should follow
quite naturally from the objectives.
II. Writing the Questionnaire
At this point, we assume that we have already decided what kind of data we are to measure,
formulated the objectives of the investigation, and decided on a participant group. Now we must
compose our questions.
If the preceding steps have been faithfully executed, most of the questions will be on obvious
topics. Most questionnaires, however, also gather demographic data on the participants. This is

4. used to correlate response sets between different groups of people. It is important to see
whether responses are consistent across groups. For example, if one group of participants is
noticeably less satisfied with the test interface, it is likely that the interface was designed without
fair consideration of this group's specific needs. This may signify the need for fundamental
redesign of the interface. In addition, certain questions simply may only be applicable to certain
kinds of users. For example, if one is asking the participants whether they find the new tutorial
helpful, we do not want to include in our final tally the responses of experienced users who
learned the system with an older tutorial. There is no accurate way to filter out these responses
without simply asking the users when they learned the interface.
Typically, demographic data is collected at the beginning of the questionnaire, but such
questions could be located anywhere or even scattered throughout the questionnaire. One
obvious argument in favor of the beginning of the questionnaire is that normally background
questions are easier to answer and can ease the respondent into the questionnaire. One does not
want to put off the participant by jumping in to the most difficult questions. We are all familiar
with such kinds of questions.
It is important to ask only those background questions that are necessary. Do not ask income of
the respondent unless there is at least some rational for suspecting a variance across income
levels. There is often only a fine line between background and personal information. You do not
want to cross over in to the personal realm unless absolutely necessary. If you need to solicit
personal information, phrase your questions as unobtrusively as possible to avoid ruffling your
participants and causing them to answer less than truthfully.
What kind of questions do we ask?
In general, there are two types of questions one will ask, open format or closed format.
Open format questions are those that ask for unprompted opinions. In other words, there are no
predetermined set of responses, and the participant is free to answer however he chooses. Open
format questions are good for soliciting subjective data or when the range of responses is not
tightly defined. An obvious advantage is that the variety of responses should be wider and more
truly reflect the opinions of the respondents. This increases the likelihood of you receiving
unexpected and insightful suggestions, for it is impossible to predict the full range of opinion. It
is common for a questionnaire to end with and open format question asking the respondent for
her unabashed ideas for changes or improvements.
Open format questions have several disadvantages. First, their very nature requires them to be
read individually. There is no way to automatically tabulate or perform statistical analysis on
them. This is obviously more costly in both time and money, and may not be practical for lower
budget or time sensitive evaluations. They are also open to the influence of the reader, for no

5. two people will interpret an answer in precisely the same way. This conflict can be eliminated by
using a single reader, but a large number of responses can make this impossible. Finally, open
format questions require more thought and time on the part of the respondent. Whenever more
is asked of the respondent, the chance of tiring or boring the respondent increases.
Closed format questions usually take the form of a multiple-choice question. They are easy for
the respondent, give
There is no clear consensus on the number of options that should be given in an closed format
question. Obviously, there needs to be sufficient choices to fully cover the range of answers but
not so many that the distinction between them becomes blurred. Usually this translates into five
to ten possible answers per questions. For questions that measure a single variable or opinion,
such as ease of use or liability, over a complete range (easy to difficult, like to dislike),
conventional wisdom says that there should be an odd number of alternatives. This allows a
neutral or no opinion response. Other schools of thought contend that an even number of
choices is best because it forces the respondent to get off the fence. This may induce the some
inaccuracies for often the respondent may actually have no opinion. However, it is equally
arguable that the neutral answer is over utilized, especially by bored questionnaire takers. For
larger questionnaires that test opinions on a very large number of items, such as a music test, it
may be best to use an even number of choices to prevent large numbers of no-thought neutral
answers.
Closed format questions offer many advantages in time and money. By restricting the answer
set, it is easy to calculate percentages and other hard statistical data over the whole group or
over any subgroup of participants. Modern scanners and computers make it possible to
administer, tabulate, and perform preliminary analysis in a matter of days. Closed format
questions also make it easier to track opinion over time by administering the same
questionnaire to different but similar participant groups at regular intervals. Finally closed
format questions allow the researcher to filter out useless or extreme answers that might occur
in an open format question.
Whether your questions are open or closed format, there are several points that must by
considered when writing and interpreting questionnaires:
1. Clarity: This is probably the area that causes the greatest source of mistakes in
questionnaires. Questions must be clear, succinct, and unambiguous. The goal is to
eliminate the chance that the question will mean different things to different people. If
the designers fails to do this, then essentially participants will be answering different
questions.

6. To this end, it is best to phrase your questions empirically if possible and to avoid the use of
necessary adjectives. For example, it asking a question about frequency, rather than supplying
choices that are open to interpretation such as:
1. Very Often
2. Often
3. Sometimes
4. Rarely
5. Never
It is better to quantify the choices, such as:
1. Every Day or More
2. 2-6 Times a Week
3. About Once a Week
4. About Once a Month
5. Never
There are other more subtle aspects to consider such as language and culture. Avoid the use of
colloquial or ethnic expressions that might not be equally used by all participants. Technical
terms that assume a certain background should also be avoided.
1. Leading Questions: A leading question is one that forces or implies a certain type of
answer. It is easy to make this mistake not in the question, but in the choice of answers.
A closed format question must supply answers that not only cover the whole range of
responses, but that are also equally distributed throughout the range. All answers should
be equally likely. An obvious, nearly comical, example would be a question that supplied
these answer choices:
1. Superb
2. Excellent
3. Great
4. Good
5. Fair
6. Not so Great
A less blatant example would be a Yes/No question that asked:

7. 1. Is this the best CAD interface you have every used?
In this case, even if the participant loved the interface, but had an favorite that was preferred,
she would be forced to answer No. Clearly, the negative response covers too wide a range of
opinions. A better way would be to ask the same question but supply the following choices:
1. Totally Agree
2. Partially Agree
3. Neither Agree or Disagree
4. Partially Disagree
5. Totally Agree
This example is also poor in the way it asks the question. It's choice of words makes it a leading
question and a good example for the next section on phrasing.
1. Phrasing: Most adjectives, verbs, and nouns in English have either a positive or
negative connotation. Two words may have equivalent meaning, yet one may be a
compliment and the other an insult. Consider the two words "child-like" and "childish",
which have virtually identical meaning. Child-like is an affectionate term that can be
applied to both men and women, and young and old, yet no one wishes to be thought of
as childish.
In the above example of "Is this the best CAD interface you have every used?" clearly "best" has
strong overtones that deny the participant an objective environment to consider the interface.
The signal sent the reader is that the designers surely think it is the best interface, and so should
everyone else. Though this may seem like an extreme example, this kind of superlative question
is common practice.
A more subtle, but no less troublesome, example can be made with verbs that have neither
strong negative or positive overtones. Consider the following two questions:
1. Do you agree with the Governor's plan to oppose increased development of
wetlands?
2. Do you agree with the Governor's plan to support curtailed development of
wetlands?
They both ask the same thing, but will likely produce different data. One asks in a positive way,
and the other in a negative. It is impossible to predict how the outcomes will vary, so one

8. method to counter this is to be aware of different ways to word questions and provide a mix in
your questionnaire. If the participant pool is very large, several versions may be prepared and
distributed to cancel out these effects.
4. Embarrassing Questions: Embarrassing questions dealing with personal or private
matters should be avoided. Your data is only as good as the trust and care that your respondents
give you. If you make them feel uncomfortable, you will lose their trust. Do not ask
embarrassing questions.
1. Hypothetical Questions Hypothetical are based, at best, on conjecture and, at worst,
on fantasy. I simple question such as:
1. If you were governor, what would you do to stop crime?
This forces the respondent to give thought to something he may have never considered. This
does not produce clear and consistent data representing real opinion. Do not ask hypothetical
questions.
1. Prestige Bias: Prestige bias is the tendency for respondents to answer in a way that
make them feel better. People may not lie directly, but may try to put a better light on
themselves. For example, it is not uncommon for people to respond to a political opinion
poll by saying they support Samaritan social programs, such as food stamps, but then go
on to vote for candidates who oppose those very programs. Data from other questions,
such as those that ask how long it takes to learn an interface, must be viewed with a little
skepticism. People tend to say they are faster learners than they are.
There is little that can be done to prevent prestige bias. Sometimes there just is no way to phrase
a question so that all the answers are noble. The best means to deal with prestige bias is to make
the questionnaire as private as possible. Telephone interviews are better than person-to-person
interviews, and written questionnaires mailed to participants are even better still. The farther
away the critical eye of the researcher is, the more honest the answers.
Now What?
Now that you've completed you questionnaire, you are still not ready to send it out. Just like any
manufactured product, your questionnaire needs to go through quality testing. The major
hurdle in questionnaire design is making it clear and understandable to all. Though you have
taken great care to be clear and concise, it is still unreasonable to think that any one person can

9. anticipate all the potential problems. Just as a usability test observes a test user with the actual
interface, you must observe a few test questionnaire takers. You will then review the
questionnaire with the test takers and discuss all points that were in any way confusing and
work together to solve the problems. You will then produce a new questionnaire. It is possible
that this step may need to be repeated more than once depending on resources and the need for
accuracy.
QUESTIONNAIRE DESIGN, SURVEY METHODS, AND THE ART OF
ASKING QUESTIONS
Questionnaires are used in connection with many modes of observation in social research. They
serve as an essential tool in public opinion survey research. The format of a questionnaire is as
important as the wording of the questions asked. An improperly designed questionnaire can
lead respondents to skip questions, confuse respondents as to the nature of the data desired, and
in the extreme, lead people to refuse to respond.
I. Standardized Questionnaires
A questionnaire is like a conversation with a purpose. It is important to ask initially, do I want to
use a standardized questionnaire? A standardized questionnaire is one in which every
respondent gets the same set of questions in the same order. Typically, if a large sample is
involved, the services of many interviewers required, and the data are to be analyzed
statistically. A standardized questionnaire is the preferred methodology.
II. Deciding on Content
Considerations in constructing a questionnaire:
1. Decide what information is required to produce independent, dependent and control
variables.
2. Draft some questions to gather that information.
3. Put them into meaningful order and format (e.g. open ended, closed ended, Likert)
4. Pretest the result on a sample of potential respondents.
5. Make necessary changes to the final form of the questionnaire.
III. Understanding the Dimensions of Opinion.
There are a variety of dimensions of public opinion according to George Gallup they are:
1. A person's knowledge or awareness of the issue is a major dimension. People can have
opinions that have no basis in facts.
2. A person's interest or the saliency of the issue to the person is an important dimension. Some
people are apathetic others highly concerned about a given issue; e.g. gun control.

10. 3. A person's attitude or the direction of his opinion is another dimension. (Typically, for or
against an issue).
4. A person's strength of believe- from very strong to slightly is another dimension.
5. Why a person believes as he or she does, this is the most difficult question to answer.
III. Mode of Administration
There are three major ways to administer questionnaire. Mail. Face to Face, Telephone.
A) Mail Respondents receive questionnaire, read it and administer it to them elves (fill it out).
Good for collecting data from specialized and highly motivated groups for example, planners.
Very cost effective since one of the main costs in survey research is locating the respondent and
interviewer costs. You can get both wide geographical spread and cost savings from mail
questionnaires, it costs no more to do a national than a statewide survey of the same size...a 29
cent stamp costs the same.
Disadvantages
1. You don't have rapport with respondent.
2. No probing for more specific responses from respondents is possible.
3. The timeliness of mail questions does not make them appropriate for dynamic issues, that is
things subject to rapid change.
4. Question order bias may emerge because respondent can see the entire questionnaire before
answering the first question. This is a real problem if the order in which the questions are to be
asked is important to the research design.
5. You are never sure if the respondent sampled is the one who filled out the questionnaire and
less educated people may have trouble following instructions and there is no one to explain
them.
6. If the question asks information questions and you want to measure knowledge the
respondent has the opportunity to look up the answers. It is difficult to measure knowledge.
7. There are no enumerated lists from which to sample of the general public.
8. You can get low response rates without inducements.
9. You can't display or present certain alternatives that you can in a face to face setting; e.g. this
is our product which of these three package designs do you prefer?
10. The size of the questionnaire is more of a limitation and you compete with junk mail or if you
are doing the survey online with Spam.
Personal Interviews Personal interviews are conducted in one of two ways face to face or by
phone. They are extremely expensive and require skilled interviewers who must be trained and
supervised. There are certain types of questions that can only be dealt with using this technique,
for example, teenage sexuality, which is a very controversial issue. This involves the interviewer

11. asking questions in the physical presence of the respondent. It is an effective tool when the
length of the interview exceeds 20 minutes. It is extremely effective when you want to display a
product or device; e.g. photographs of potential candidates or products. Some elements of the
population can be reached only by going to the door or in a given location; e.g. delegates to the
Republican convention. You have the rapport factor not possible by mail or phone. Increasing
the computer is used to administer and collect information about sensitive issues. Typically, a
multimedia approach is employed in which a questionnaire is programmed to ask and capture
the answers of thew respondent without a human being present. RTI has pioneered some of
these applications.
Disadvantages
1. Face to face interviewing allows for greater control over the respondents behavior, but can
result in interviewer respondent interactions like white interviewer asking black respondent
about L.A. riots get different responses than black interviewers.
2. Very costly you need to locate and interview respondents.
3. You can't control interviewer bias (what they say how they follow instructions) because they
are out of your span of control.
4. Many respondents are in hard to reach places, behind closed doors, or, in undesirable parts of
the city.
C) Telephone
Telephone interviewing is an excellent method for timely generalizations. With a CATI system,
your results are right there after the interview. Telephone interviews are excellent for general
audiences and have a large geographical reach. They are cheaper than a personal interview
conducted on a face to face basis and have completion rates that are similar. Studies show that
the quality of data collected by telephone surveys is comparable to other methods. Finally, in
terms of interviewer effects they are free of many of the biases that are possible in face to face
interviewing. Also, with random digit dialing and computer sampling of phone lists, you can
take random samples of households even those phone is unlisted.
Disadvantages
1. They are disruptive of home life and are easier to reject.
2. They compete with gimmicks that pretend to be surveys when they are really phone
solicitation. Many legitimate polls now start with the statement that we will not attempt to sell
you anything....
3. Like mail questionnaires certain forms of presentation are more difficult or impossible.
4. It under represents those without phones. About 95% of all households have them.

12. 5. Answering machines can act as a screen to prevent the interviewer from direct contact with
the respondent.
IV. Survey Process
Note: you need to have a good understanding of the following:
1. What are your independent variables? What do you believe is the variable that is influencing
other variables? If you believe college educated people are more likely than non college educated
people to buy your product or vote for your candidate, then education is the independent
variable.
2. What are your dependent variable(s)? This is the variable being effected, In the above
example it is the product or the candidate.
3. What are your control variables? These are variables that need to be held constant in order to
measure the effect of the IV on the DV. For example, a control, variable might be race, in which
case, you would want to look at the relationship between education and candidate choice (vote)
for both Afro-Americans and Whites separately so that you can control (hold constant) the effect
of race.
4. At what level of measurement is very variable in your study measured, nominal, and ordinal.
Interval, ratio. Why? All things being equal measure at the highest level of measurement
possible?
5. What are the hypotheses you are testing? Remember that a hypothesis is a relationship
between two or more variables. For example; mothers with children in this study are more like
to support anti pornography legislation than other women or the men in the sample.
6. Do you have single or multiple measurements (questions) of your concepts e.g. conservatism
versus liberalism, subjective or objective measures, one or multiple questions.
V. Questionnaire Format
A. Question Sequence.
Introduction
Main Body
Demographics
Introduction. It should be short, realistically worded, non-threatening, serious, neutral and
establish legitimacy. It does not necessarily have to ask permission. It helps today to add that
you will attempt to sell them nothing. Also, this is the point at which you will do any screening,
e.g. ASK REGISTERED VOTERS ONLY.
Main Body. This is typically the main body of IV's DV's Control Variables. There is a major
division of the types of question based on weather the question can be verified (fact or
empirical) These are behavioral or empirical questions and are verifiable. Opinion questions,

13. These ask the respondent to rank with some degree of intensity statements designed to place
them on a continuum with respect to a concept.
Demographics. These should be placed at the back of the questionnaire. They typically
constitute the major independent variables of the study.
B. Open and Closed Ended Questions
There are two options in the realm of asking questions: open-ended and closed-ended questions.
Whereas in open-ended questions respondents are asked to provide their own answers to the
question, in closed-ended questions they are asked to select their answer from among a list
provided to them. Closed-ended questions yield greater uniformity of responses and are more
easily processed, thus they are preferred by most researchers. However, in terms of reliability
and validity, closed-ended questions must provide response categories that are exhaustive and
mutually exclusive. To minimize form differences, investigators are advised to pretest their
questionnaire. They often begin with open questions on large samples of the target population
and use the responses to construct closed alternatives that reflect the substance and wording of
what people say spontaneously. Despite the obvious advantages of closed questions, open
questions cannot be eliminated from survey research.
Open-ended questions may be needed:
- to pretest closed categories
- to document the absence of a type of response
- to follow up on closed questions and determine why people answer the way they do
- to gather data where rapidly shifting external events can affect answers
- to avoid newly emerging categories (over a long time period)
- if the set of meaningful alternatives is too large or complex to present to respondents
C. Statements
Although survey research is commonly viewed as the asking of questions, researchers are often
interested in the extent to which respondents hold a particular attitude or position. Such a
position is often summarized in a statement, a declarative sentence which respondents are asked
to agree or disagree with. This procedure is formalized through the creation of a Likert scale, a
format in which respondents are asked to "strongly agree", "agree", "disagree", "strongly
disagree", or "strongly approve", "strongly disapprove", etc ...
D. Semantic Differential
This is a device in which an person, object, office, etc. is evaluated by list a series of bipolar
adjectives with points of gradations in between. The respondent can then rank the idea
congressman and then two real congressmen. The research can then compare differences.
E. Skipping and Filter Questions
Occasionally it is necessary to skip some questions and branch to other questions depending

14. upon the respondent's answer.
For example, ASK ONLY IF RESPONDENT IS REGISTERED TO VOTE, may be a filter for
questions about voting preferences.
VI. Importance of Response Rate
In general, using inferential statistics in connection with survey analysis assumes that all
members of the initial sample complete and return their questionnaire. Problems associated
with low response rates arise because the researcher intends to generalize survey information to
a group larger than that from which the data were obtained. It may be that nonrespondents
differ systematically from respondents and this difference is not reflected in the data obtained.
The purpose for indicating survey response rates is to help the reader decide to what extent the
sample data can be generalized to the whole population. Three main reasons for nonresponses
are:
(a) selected sample members never receive the questionnaire;
(b) they refuse to answer; and
(c) they forget to complete or return the survey.
The higher the response rate the better the more likely the survey is representative. As a rule of
thumb, a response rate of 50 percent is considered adequate, 60 percent good, and 70 percent or
more very good. There are techniques that can increase response rate.
Empirical research has demonstrated evidence that the following techniques can help to
improve response rates to mail questionnaires:
Most Effective:
- use of follow-ups (especially telephone reminders and special delivery letters)
- pre-contact with respondents
- type of postage (special delivery and airmail are superior to ordinary mail)
- cash rewards
- appearance of the sponsoring organization and title of the person signing the accompanying
letter (the more prestigious the better)
Somewhat Effective:
- explaining the place and importance of the respondent for the survey
- personalizing the questionnaire
- Anonymity
- Appeals to the respondent based on the social benefit of the study
VII. The Art of Asking Questions
Questionnaires are structured conservation with a purpose. The format of a questionnaire is as

15. important as the nature and wording of the questions asked. An improperly designed
questionnaire can lead respondents to miss questions, confuse them as to the nature of the data
desired, and in the extreme, lead them to refuse to respond. The purpose of asking questions in
a survey is to elicit information from the respondent. Some general rules should be followed in
the design and wording of questions and statements.
1. Making Items clear: Questionnaire items must be clear and unambiguous. They
should be precise so that the respondent knows exactly what question he/she is
expected to answer.
2. Avoiding Double-Barreled Questions: Never ask for a single answer to a
combination of questions. In general, whenever the words and or appear in a
question or questionnaire statement, check whether this is a double-barreled
question.
3. Asking Relevant Questions: Questions asked in a survey should be relevant to
most respondents. Disregard responses to fictitious issues.
4. Using short Items: Assume that respondents read items quickly and provide
quick answers. Therefore, provide clear, short items that will not be
misinterpreted under such conditions.
5. Avoiding negative Terms: Negations (e.g. the use of the word "not") will often is
read over, leading to misinterpretations.
6. Avoiding biased Items and Terms: The manner in which data are sought
determines the nature of the data received. The identification of an attitude or
position with a prestigious or popular person, for example, can bias responses
(e.g. President Clinton vs. Adolf Hitler).
7. Avoid the Ambiguity of Language When we say that there is an art to asking
questions that give unbiased answers we mean exactly that. There are rules that
you can follow, but there is an art that comes with practice and trial and error to
asking questions fairly. Here are some things to avoid.
VIII. Examples of Bias in Questions:
Unfair Alternatives
Bad: Some people say that the city is spending too much on building new public schools. Do you
agree or disagree?
Improved: Some people say that the city is spending too much on building public schools ... and
others say the city is not spending enough. With which opinion do you agree?
Maligning the Other Side
Bad: Do you think the government should spend more of our tax money on the slums?

16. Improved: Do you think the government should spend more ... or less money on replacing the
slum neighborhoods in the city with new housing projects?
Damning with Faint Praise
Bad: Some people say that the Mayor's plan is a poor plan to solve garbage removal problems in
the city. Others say it will do for now until a better solution is found. Do you think it is a good
plan or a poor plan?
Improved: Some people favor and some oppose the plan for combined garbage and trash
removal by the city.
Do you think the plan is a good solution ... or a poor solution to the garbage removal problem?
Deliberately Omitting Names
Bad: Hello ... I'm conducting a poll for Sam Snide, a candidate for mayor of the city. If the
election were held today, whom would you vote for ... Mr. Snide, or one of the other candidates?
Improved: I am conducting a survey on the mayoral election...
Inappropriate Use of Titles
Bad: State Attorney General Allen P. Mutt is running for governor this year against Tom L. Jeff.
Which man, Mutt or Jeff, is best qualified to be governor?
Improved: Allen P Mutt and Tom L. Jeff. Are running for governor this year. Which man, Mutt
or Jeff, is best qualified to be governor?
Personalities
Bad: Would you say that governor Hunt's energy program for promoting solar heating of private
homes has been very effective, fairly effective, not too effective, or not effective at all?
Improved: Would you say that the state energy program for promoting solar heating of private
homes has been very effective, fairly effective, not too effective, or not effective at all?
Emotionally Charged Words
Bad: Congressman Pork barrel has been accused of defrauding the voters of this district. Do you
agree or disagree with that charge?
Improvement: One of the issues in this campaign is how well Congressman Pork barrel has
carried out his campaign promises. Do you think that Pork barrel has done an excellent, good,
poor, or very poor job of doing what he said he would do?
Conditioned by Context
Q1: Tom Fetzer is mayor of the city. In your opinion is he doing a good ... or poor job as mayor?
Q2: As you understand it, what are the mayor's principal duties in office?
Note: the response to the second question is conditioned by your response to the first it would
be better to reverse the order or ask only one!
Embarrassing Questions
Bad: How much time did you spend reading the newspaper yesterday?

17. Improvement: Did you have a chance to read the newspaper yesterday? (IF YES: About how
much time did you spend reading the newspaper yesterday?)
Bad: What is your religion?
Improvement Do you happen to have a religious preference? (IF YES: What is your religious
preference?)
Bad: Did you vote in the city election last month?
Improvement Did you happen to vote in the city election last month, or didn't you have a chance
to vote?
Bad: How old are you?
Improvement: In what year were your born?
Illogical Sentence Construction
Bad: Some people say that Senator Helms is doing an excellent job in office, and some people
say he is doing a very poor job. What kind of job do you think Senator Helms is doing ...
excellent, good, poor, or very poor?
Improvement: Would you say that Helms is doing an excellent, good, poor, or very poor job as
United States Senator?
Two-part Questions
Bad: Do you think that Mayor Booth should run for re-election this year, or could the Democrats
find a stronger candidate?
Improvement: Q1: Do you think that Mayor Booth should or should not run for re-election this
year?
Q2: Do you think the Democrats could or could not find a stronger candidate than Mayor Booth
this year?
Ambiguous Questions
Bad: Did you vote in the last election?
Improvement: Did you vote in the city election for Mayor last June?
Bad: Are you in favor of a larger government role in housing and the environment?
Improvement: Q1: Are you in favor of a larger role for the federal government in the
environment? Q2: Are you in favor of a larger role for the federal government in housing?
Note: If you have "and" in the question with two possible responses, and the respondent has a
different view of the federal government's role in housing versus the environment he/she can't
answer the question.
Indefinite Persons or Places
Bad: Are there many voters living around here?
Improvement: Of the people you personally know living on Apple Street between 34th and 35th
Avenues, about how many do you know to be registered to vote ... Would you say nearly all,

18. maybe about three-quarters, about half ... or less than half?
Indefinite Concepts
Bad: Among you circle of friends, is there anyone whose opinions or advice you frequently ask
about the public affairs issues of the day?
Improvement: Among your circle of friends is there anyone whose opinions or advice you
frequently ask ... about such issues as the energy crisis?
Misperceived Questions
1. Outside Respondent's Experience
Interviewer asked: Generally, do you think that precinct captains get involved in politics for
egocentric or altruistic reasons?
Respondent said: "I don't care what their financial reasons are, policemen should keep out of
politics."
2. Technical words
Interviewer asked: Do you think we should change the present system of the Electoral College?
Respondent said: "No, the universities are all right. It's just one bunch of leftwing kids making
all the trouble."
Interviewer asked: Do you favor giving more power to election judges?
Respondent said: "No, they shouldn't have any more power than appointed judges."
Interviewer asked: What do you think of Red China?
Respondent said: "It looks very nice on a white table cloth."
3. Context
Interviewer asked: In challenging his opponent, the mayor demanded a dual meeting and debate
on the issues...
Respondent said: "Dueling ... in this day and age?"
Asking questions correctly is both an art and a science. Whenever you design a questionnaire,
pretest it. A pretest can help you isolate and avoid the ambiguity of language. Remember a
questionnaire is you operationalization of the concepts you will employ in the study. Your study
will be a good or as bad as your questionnaire...
Good & bad in Questionnaire Design, Written By B2B International
A catalogue of the commonest errors and how to construct a questionnaire that
avoids them
HOW OFTEN HAVE YOU STRUGGLED to respond to a questionnaire which just isn't quite
right for your answers? No doubt you became annoyed, frustrated and maybe even cynical about
market research. These problems arise because of poor questionnaire design. The most frequent
errors in questionnaires are:

19. 1. questions which don't quite mean what the researcher intended
2. questions which don't probe to find out what the respondent really meant
3. long and complicated questions
4. questions which repeat what has already been asked
5. questions which don't allow the respondent to answer in a way which is relevant
6. questions which are inappropriate to the research method
7. bad routing which leaves the interviewer wondering which question to ask next, or worse,
routing to the wrong one
8. too many questions
9. poorly laid out questions which are badly grouped in the questionnaire
10. Questions which have been missed out completely
11. pages of the questionnaire which are missing or out of order.
These errors can be eliminated or at least minimized if the researcher keeps two things clearly in
mind: who will be answering the questions and what the research is aiming to achieve.
The researcher must understand four fundamentals of questionnaire design:
- What types of question can be asked
- What types of response should be built in
- How to lay out the questionnaire
- How to test it.
WHAT TYPES OF QUESTION?
The researcher's tool box contains three different types of question: behavioral, attitudinal and
classificatory.
Answers to BEHAVIOURAL questions tell the researcher where a brand is now and where it
could be in the future. They show who is doing the buying and the pattern of purchasing
amongst companies using the product. They allow the researcher to position a company or
brand in terms of its market share. Here are some examples of the subjects which behavioral
questions can address.
Market size. Questions which need asking to assess the size of the market are: "Do you ever
buy?", "How often do you buy?", "How much do you buy?" To make it easier for the respondent
a time period may be built into the question: "Over the last month how much have you spent
on... ?"
Ownership patterns. Linked to market size questions are ownership questions. "Do you own a
certain product?", "How long have you owned it?", "How many do you own?"
Purchasing patterns. The researcher needs to understand the pattern of purchasing to work out
the best way to meet the demand. This understanding is acquired from questions such as "How
often do you buy?", "When did you last buy?", "How much do you usually buy?"

20. Behavioral patterns. Knowing the behaviors of a buyer helps the researcher provide the product
at the right time at the right place. Questions which provide an understanding of purchasing
behavior include "How often do you buy/use/visit.. ? " and "Who is it at your company who has
the final choice on the selection of a supplier?"
Market shares. The shares which companies or brands hold can be assessed by asking buyers,
"Which brands do you use?", "How much of this particular brand do you buy per month?"
Future trends. Being able to predict or simply obtain a feel for future trends is one of the most
important yet most difficult of the researcher's tasks. Pointers to the future can be obtained from
questions such as "Do you intend to buy/visit/use... over the next six months?", "if your
purchases this year were 100, what would you say they could be in five years' time?", "What are
your reasons for saying that you will buy less in the future?"
ATTITUDINAL questions establish respondents' opinions and their image or perception of a
subject. Their perceptions may, of course, be totally at variance with reality, but that may make
them all the more valuable as marketing data. Attitudinal research can help the researcher plan
a campaign to win sales by:
showing why a company buys a certain brand
showing whether or not a demand is being satisfied
establishing the views which people hold on certain products, services, prices, standards of
delivery, etc.
The third type of question in the researcher’s armoury is that which CLASSIFIES the
respondent. Such questions allow answers from different companies to be grouped and
compared. Classification questions enable the researcher to segment the market and decide
where marketing effort will be most rewarded. Common classifications used in industrial market
research are:
size of company (measured by number of employees or turnover)
the nature of the company’s business (usually determined by its Standard Industrial
Classification)
geographical location.
There may well be classifications which are special to a particular survey. Most industrial
surveys, for example, classify according to the size of the respondent company’s purchases, so
that the researcher can see the difference in attitudes or purchasing patterns between large and
small companies.
WHAT TYPES OF RESPONSE should be built in? Having selected the type of question
(attitudinal, behavioural or classificatory), the researcher must decide on the response option.
The questions can be either open ended or closed.
Open ended questions leave the respondent free to give any answer, and verbatim responses are

21. recorded. This type of question and answer is extremely flexible, as the researcher has not
committed to a list of pre-coded responses.
However, open ended questions are sometimes an easy option by researchers too lazy to think of
possible pre-codes. Open enders should only be asked where the response is truly unknown.
A closed question is one that has pre-coded answers. The simplest is the dichotomous question
to which the respondent must answer yes or no.
Closed response questions save the respondent having to think of possible replies. They also
make life easier for the interviewer who simply has to tick a box or circle a number. Moreover,
they spare the coding staff difficult judgements which, if wrong, can skew the findings.
A special type of pre-coded question is a scale. Scales can be of different types.
Verbal rating scales. Typical of these is the five point scale running from very good through to
very poor. Verbal rating scales can be applied to almost any adjective - good/poor,
happy/unhappy, satisfied/ dissatisfied, pleased/displeased, suitable/unsuitable, and so on.
Instead of words, the researcher can use a numerical weighting scale. This usually runs up to
five or 10. The respondent needs to know the direction of the scale, viz:
"I would like to ask your opinion on THE XYZ COMPANY. What score would you give it out of
10 where one is totally dissatisfied and ten is totally satisfied?"
Problems can occur if the researcher wants a rating on too many questions. Answers deteriorate
as the questions grind on and the respondent repeats the same score in an effort to speed the
completion of the interview.
Diagrammatic rating scales. These, as the name suggests, ask the respondent to mark an answer
on some continuum. This could be a line, a balance, or boxes. Diagrammatic scales are self
completed by the respondent and so they need to be clearly marked at each end of the scale.
They can be more fun and faster to complete than the verbal or numerical scale.
HOW TO LAY OUT THE QUESTIONNAIRE. The questions must now be organised
logically and attractively. This is for the benefit of the interviewers, the respondents and the
coders. The layout depends very much on who will be administering the questionnaire, who will
be answering it, and the type of questions which are being asked.
A well laid out questionnaire must have a sensible sequence of questions. The smooth flow of the
interview depends largely on the order in which questions are asked.
Questions on image, purchasing behaviour and classification should all be arranged together.
They should lead from one to another with the general subjects at the beginning and the more
particular later on. This allows the respondent to relax and loosen up.
For the same reasons, easy questions should precede the difficult ones. Sensitive questions
should be placed towards the end, so that the respondent is "warmed up" and in autonomic
response mode.

22. The questions and responses should be clear and easy to read. If the boxes to tick (or numbers to
circle) are out of line or placed too far from the pre-coded responses, the interviewer could mis-
cue an answer. There should be ample space for the interviewer to record verbatim responses.
A clear and attractive layout is even more important in a self-completion questionnaire, where it
has a significant effect on the response rate.
Good & bad in Questionnaire Design, Written By B2B International
A catalogue of the commonest errors and how to construct a questionnaire that
avoids them
HOW OFTEN HAVE YOU STRUGGLED to respond to a questionnaire which just isn't quite
right for your answers? No doubt you became annoyed, frustrated and maybe even cynical about
market research. These problems arise because of poor questionnaire design. The most frequent
errors in questionnaires are:
12. questions which don't quite mean what the researcher intended
13. questions which don't probe to find out what the respondent really meant
14. long and complicated questions
15. questions which repeat what has already been asked
16. questions which don't allow the respondent to answer in a way which is relevant
17. questions which are inappropriate to the research method
18. bad routing which leaves the interviewer wondering which question to ask next, or worse,
routing to the wrong one
19. too many questions
20. poorly laid out questions which are badly grouped in the questionnaire
21. Questions which have been missed out completely
22. pages of the questionnaire which are missing or out of order.
These errors can be eliminated or at least minimized if the researcher keeps two things clearly in
mind: who will be answering the questions and what the research is aiming to achieve.
The researcher must understand four fundamentals of questionnaire design:
- What types of question can be asked
- What types of response should be built in
- How to lay out the questionnaire
- How to test it.
WHAT TYPES OF QUESTION?
The researcher's tool box contains three different types of question: behavioral, attitudinal and
classificatory.
Answers to BEHAVIOURAL questions tell the researcher where a brand is now and where it
could be in the future. They show who is doing the buying and the pattern of purchasing
amongst companies using the product. They allow the researcher to position a company or
brand in terms of its market share. Here are some examples of the subjects which behavioral
questions can address.
Market size. Questions which need asking to assess the size of the market are: "Do you ever
buy?", "How often do you buy?", "How much do you buy?" To make it easier for the respondent
a time period may be built into the question: "Over the last month how much have you spent
on... ?"

23. Ownership patterns. Linked to market size questions are ownership questions. "Do you own a
certain product?", "How long have you owned it?", "How many do you own?"
Purchasing patterns. The researcher needs to understand the pattern of purchasing to work out
the best way to meet the demand. This understanding is acquired from questions such as "How
often do you buy?", "When did you last buy?", "How much do you usually buy?"
Behavioral patterns. Knowing the behaviors of a buyer helps the researcher provide the product
at the right time at the right place. Questions which provide an understanding of purchasing
behavior include "How often do you buy/use/visit.. ? " and "Who is it at your company who has
the final choice on the selection of a supplier?"
Market shares. The shares which companies or brands hold can be assessed by asking buyers,
"Which brands do you use?", "How much of this particular brand do you buy per month?"
Future trends. Being able to predict or simply obtain a feel for future trends is one of the most
important yet most difficult of the researcher's tasks. Pointers to the future can be obtained from
questions such as "Do you intend to buy/visit/use... over the next six months?", "if your
purchases this year were 100, what would you say they could be in five years' time?", "What are
your reasons for saying that you will buy less in the future?"
ATTITUDINAL questions establish respondents' opinions and their image or perception of a
subject. Their perceptions may, of course, be totally at variance with reality, but that may make
them all the more valuable as marketing data. Attitudinal research can help the researcher plan
a campaign to win sales by:
showing why a company buys a certain brand
showing whether or not a demand is being satisfied
establishing the views which people hold on certain products, services, prices, standards of
delivery, etc.
The third type of question in the researcher’s armoury is that which CLASSIFIES the
respondent. Such questions allow answers from different companies to be grouped and
compared. Classification questions enable the researcher to segment the market and decide
where marketing effort will be most rewarded. Common classifications used in industrial market
research are:
size of company (measured by number of employees or turnover)
the nature of the company’s business (usually determined by its Standard Industrial
Classification)
geographical location.
There may well be classifications which are special to a particular survey. Most industrial
surveys, for example, classify according to the size of the respondent company’s purchases, so
that the researcher can see the difference in attitudes or purchasing patterns between large and
small companies.
WHAT TYPES OF RESPONSE should be built in? Having selected the type of question
(attitudinal, behavioural or classificatory), the researcher must decide on the response option.
The questions can be either open ended or closed.
Open ended questions leave the respondent free to give any answer, and verbatim responses are
recorded. This type of question and answer is extremely flexible, as the researcher has not
committed to a list of pre-coded responses.
However, open ended questions are sometimes an easy option by researchers too lazy to think of
possible pre-codes. Open enders should only be asked where the response is truly unknown.

24. A closed question is one that has pre-coded answers. The simplest is the dichotomous question
to which the respondent must answer yes or no.
Closed response questions save the respondent having to think of possible replies. They also
make life easier for the interviewer who simply has to tick a box or circle a number. Moreover,
they spare the coding staff difficult judgements which, if wrong, can skew the findings.
A special type of pre-coded question is a scale. Scales can be of different types.
Verbal rating scales. Typical of these is the five point scale running from very good through to
very poor. Verbal rating scales can be applied to almost any adjective - good/poor,
happy/unhappy, satisfied/ dissatisfied, pleased/displeased, suitable/unsuitable, and so on.
Instead of words, the researcher can use a numerical weighting scale. This usually runs up to
five or 10. The respondent needs to know the direction of the scale, viz:
"I would like to ask your opinion on THE XYZ COMPANY. What score would you give it out of
10 where one is totally dissatisfied and ten is totally satisfied?"
Problems can occur if the researcher wants a rating on too many questions. Answers deteriorate
as the questions grind on and the respondent repeats the same score in an effort to speed the
completion of the interview.
Diagrammatic rating scales. These, as the name suggests, ask the respondent to mark an answer
on some continuum. This could be a line, a balance, or boxes. Diagrammatic scales are self
completed by the respondent and so they need to be clearly marked at each end of the scale.
They can be more fun and faster to complete than the verbal or numerical scale.
HOW TO LAY OUT THE QUESTIONNAIRE. The questions must now be organised
logically and attractively. This is for the benefit of the interviewers, the respondents and the
coders. The layout depends very much on who will be administering the questionnaire, who will
be answering it, and the type of questions which are being asked.
A well laid out questionnaire must have a sensible sequence of questions. The smooth flow of the
interview depends largely on the order in which questions are asked.
Questions on image, purchasing behaviour and classification should all be arranged together.
They should lead from one to another with the general subjects at the beginning and the more
particular later on. This allows the respondent to relax and loosen up.
For the same reasons, easy questions should precede the difficult ones. Sensitive questions
should be placed towards the end, so that the respondent is "warmed up" and in autonomic
response mode.
The questions and responses should be clear and easy to read. If the boxes to tick (or numbers to
circle) are out of line or placed too far from the pre-coded responses, the interviewer could mis-
cue an answer. There should be ample space for the interviewer to record verbatim responses.
A clear and attractive layout is even more important in a self-completion questionnaire, where it
has a significant effect on the response rate.
HOW TO TEST THE QUESTIONNAIRE. Questionnaires may have to be designed with only
a modicum of knowledge about the subject. In these circumstances mistakes are bound to occur
unless the questionnaire is tested. The test should be carried out using the interviewing medium
- ie telephone, face-to-face, postal – for which the questionnaire has been designed. The number
of interviews required to test a questionnaire could be as few as five to 10, though the more the
better. Cost and time are invariably the constraints. In industrial market research, the project
may use semi-structured questionnaires with a very small number of interviews. Piloting here is
a continuous process, and modifications are made from the moment the study gets under way

25. Conclusion
Interpretation. When these guidelines are followed, the questionnaire becomes a powerful and
economic evaluation tool.