Writing multiple choice questions Presentation for School of Animal & Veterinary Sciences - Charles Sturt University, Question Writing Workshop, Wagga Wagga, Australia, 6th February 2014 Liz Norman, Massey University, New Zealand

1. Writing MCQs
Liz Norman
Massey University

2. MCQs – why?, why not?

3. Sampling advantages
• Can test a wide breadth of subjects in a timeefficient manner
• Sampling across the subject may therefore be
more representative – validity
• Longer tests (more items) have better reliability

4. Applied vs fact recall
Fact recall:
Questions capable of being answered by reference to one
paragraph in a text or notes (or several paragraphs for
questions requiring recall of several facts)
Applied (higher order)
Questions that require the use of facts or concepts, the
solution of a diagnostic or physiologic problem, the
perception of a relationship, or other process beyond
recalling discrete fact
From: Peitzman et al. (1990). Academic Medicine, 65(9), S59-60.

5. Positive consequences of MCQs
on learning
• All examinations reinforce learning including
MCQ examinations
– MCQ formats may provide a feasible way to increase
formative assessment
– MCQs formats facilitate timely feedback when used
formatively
• MCQs foster a deep approach to learning

6. Other advantages
• Can construct examinations of known difficulty
(assuming psychometric analysis carried out)
• Established methods for setting cutpoints

7. Efficiency of MCQ
• Efficient and cost effective for large numbers of
candidates
• Possibility of automated development

9. Disadvantages of MCQ
Realising the advantages requires procedures
which makes them resource intensive and
expensive
– Creation of a large question bank
– Pretesting and statistical analysis of Qs
– Post examination statistical analysis

10. Problems that can occur with MCQs
• Fact recall Qs are easier to write therefore tend to
dominate
• Beginning writers will usually write fact recall Qs
• Some topics are particularly difficult to write
MCQs for

11. Problems that can occur with MCQs
• If the test is poorly crafted, then the validity of the
interpretation made of the resulting score is
questionable
• Candidates can’t indicate their interpretation of
the Q
• Identification of a correct response requires a
different type of thinking from candidates than
generation of a response
• Guessing can be rewarded

12. Other problems
• What is correct is still a subjective decision

13. D
Least
correct
E C
A
B
Most
correct

14. Other problems
• Circulating recall papers may reduce even higher
order Qs to recall Qs

16. Negative consequences of MCQs
on learning
• False data present in MCQ distractors can result
in learning of incorrect information

17. Item writing flaws
• 36-65% of questions
• 10-15% are serious enough to influence passfail decisions
• high-achieving students were more likely than
borderline students to be penalised by flawed
items.
• reducing item flaws does improve the
performance of MCQs

18. Avoiding irrelevant difficulty

19. Focus on one thing per Q
•
•
•
•
•
One thing
Something important – common or catastrophic
Application not recall
Something that would happen in real life
NOT something tricky or overly complex

20. Keep options short
Iris prolapse is a common sequel to penetrating corneal
wounds or ruptured corneal ulcers. Which of the following
steps is NOT appropriate for the treatment of iris prolapse?
A. primary closure of the corneal laceration with 8-0 vicryl
and treatment with topical antibiotics to control
infection.
B. placement of a nictitans flap and treatment with
systemic antibiotics to control infection.
C. placement of a corneal graft with an overlying
conjunctival pedicle graft and treatment with systemic
antibiotics to control infection.

21. A 7-year-old mare has had intermittent signs of moderately
severe colic for the past 48 hours. Heart rate is 56
beats/min. Hydration, acid-base balance, and electrolyte
parameters are near normal. On rectal examination, the left
dorsal and ventral colon feels distended and is felt coursing
in a dorsocranial direction. The spleen is displaced
caudomedially. Which of the following is the most likely
diagnosis?
A.
B.
C.
D.
E.
Cecocolic intussusception or cecal inversion
Displacement of the left colon over the nephrosplenic
ligament
Ileocecal intussusception
Infarction of the large colon
Volvulus of the large colon and the cecum

22. Pose a clear task
Chyle is :A. The semifluid mass which empties from the
stomach into the duodenum.
B. The lymph containing fat droplets found in the
lacteals of the small intestine.
C. The contents of the gall bladder.
D. The rounded piece of chewed food which passes
down the oesophagus when the animal swallows.

23. Make all distractors plausible and
homogenous
Which of the following statements regarding hepatic
encephalopathy is true?
A.
B.
C.
D.
E.
Patients typically present with asymmetrical neurological deficits
The most effective and appropriate anticonvulsant to use for a
patient that is seizuring due to hepatic encephalopathy is
phenobarbital
Abdominal radiographs of dogs with portosystemic shunts will
often show an enlarged liver
Cats with portosystemic shunts often exhibit ptyalism as a clinical
sign
An appropriate treatment for hepatic encephalopathy is
intravenous neomycin

24. Include only ONE best answer
Which of the following is true?
A. Bananas are green
B. Motorbikes are faster than cars
C. Boys are taller than girls

25. Include only ONE best answer
D
Least
correct
E C
A
B
Most
correct

26. Avoid none of the above
Which of the following is true regarding ligament injuries?
A. Ligament injuries are appropriately referred to as
“strains” or “sprains”.
B. Surgical intervention is indicated for treatment of
second-degree sprains with demonstrable instability.
C. The elastic nature of ligaments allows 30% elongation
before permanent deformation.
D. Following surgical repair of ligaments, immobilization
via ESF or external coaptation is contraindicated, as
range of motion is critical to successful repair.
E. None of the above.

27. Avoid negative framing
Which of the following statements is false regarding
arthrotomies?
A. When detachment of a ligament is necessary, this should be
performed by osteotomy of the bony origin rather than
transection of the ligament.
B. Complete closure of the synovium is necessary to prevent
synovial fluid leakage into subcutaneous tissue.
C. Surgical removal of osteophytes is often followed by their
relatively rapid regrowth, and has questionable value.
D. Monofilament absorbable suture material has a lower risk for
long term infection than does braided nonabsorbable suture.
E. None of the above.

28. Avoid double options
2-year-old male neutered Border Collie is presented
with the following history and neurologic signs: ……..
Which one of the following neuroanatomic localizations
and diagnoses is CORRECT?
A. Left C1-C5 myelopathy, intervertebral disk rupture
B. Right C6-T2 myelopathy, fibrocartilagenous
embolism
C. Right C1-C5 myelopathy, intervertebral disk rupture
D. Left C1-C5 myelopathy, fibrocartilagenous
embolism

29. How many options is enough?
The probability of selecting the correct answer by chance in a
three-option multiple choice question is:
A. 33%
B. 50%
C. 75%
The probability of achieving a score of 70% by chance in a test
comprising 30 three-option multiple choice questions is:
A. 0.00356%
B. 3.56%
C. 35.6%

30. 3 options is enough
– but sometimes you do need more
A horse suffering from an acute intestinal accident
is MOST likely to have
A.
B.
C.
D.
primary respiratory acidosis
primary respiratory alkalosis
primary metabolic alkalosis
primary metabolic acidosis

31. Avoiding irrelevant ease
“testwiseness”

32. Flaws to watch out for include
• Grammatical cues:
– one or more distractors don’t follow grammatically
from the stem
• Logical cues:
– a subset of the options are collectively exhaustive
• Absolute and vague terms:
– terms such as “always”, “never” or “all” used in
options.
– terms such as “usually” and “frequently” used in
options.

33. Flaws to watch out for include
• Long correct answer:
– correct answer is longer, more specific, or more
complete than other options
• Word repeats:
– a word or phrase is included in the stem and in the
correct answer
• Convergence:
– the correct answer includes the most elements in
common with the other options

34. Flaws to watch out for include
• Numeric data not stated consistently
• Options are in an illogical order
• Stems (lead-ins) are tricky or unnecessarily
complicated

35. A content free test
• From Phil Race who got this from Roger Lewis
who cant remember where it came from…..

36. Q1
The usual function of grunge prowkers is to
remove
A.
B.
C.
D.
grunges.
snarts.
trigs.
grods.

37. Q1
Don’t include a word from the
stem in one of the options, or it
will make people think it is the
correct option
The usual function of grunge prowkers is to
remove
A.
B.
C.
D.
grunges.
snarts.
trigs.
grods.

38. Q2
Antigrottification occurs
A. on spring mornings.
B. on summer evenings provided there is no rain
before dusk.
C. on autumn afternoons.
D. on winter nights

39. Q2
This option looks more probable,
because it is longer, and because
it is the only one which is
qualified
Antigrottification occurs
A. on spring mornings.
B. on summer evenings provided there is no rain
before dusk.
C. on autumn afternoons.
D. on winter nights

40. Q3
Lurkies suffer from trangitis because:
A.
B.
C.
D.
their prads are always underdeveloped.
all their brizes are horizontal.
their curnpieces are usually imperfect.
none of their dringoes can ever adapt.

41. Q3
Don’t mix indefinites with definites –
indefinites are more likely to be correct.
(Indefinites: usually, sometimes, often,
seldom…;
Definites: all, never, none, and so on)
Lurkies suffer from trangitis because:
A.
B.
C.
D.
their prads are always underdeveloped.
all their brizes are horizontal.
their curnpieces are usually imperfect.
none of their dringoes can ever adapt.

42. Q4
Non-responsive frattling is usually found in an:
A.
B.
C.
D.
gringle.
janket.
kloppie.
uckerpod

43. Q4
Don’t allow the indefinite article
(‘an’) to give away the right
answer
Non-responsive frattling is usually found in an:
A.
B.
C.
D.
gringle.
janket.
kloppie.
uckerpod

44. Q5
Which are exceptions to the law of lompicality?
A.
B.
C.
D.
The miltrip and the nattercup.
The bifid pantrip.
The common queeter.
The flanged ozzer

45. Don’t ask for something plural,
then make just one option plural
Q5
Which are exceptions to the law of lompicality?
A.
B.
C.
D.
The miltrip and the nattercup.
The bifid pantrip.
The common queeter.
The flanged ozzer

46. Q6
Which must be present for parbling to take place?
A.
B.
C.
D.
Phlot and runge.
Runge.
Stuke and runge.
Runge and trake

47. Q6
If it’s not this option, it can’t be
any of them (one way of
thinking about it!!)
Which must be present for parbling to take place?
A.
B.
C.
D.
Phlot and runge.
Runge.
Stuke and runge.
Runge and trake

48. Q7
One common disorder of an overspragged
uckerpod is:
A.
B.
C.
D.
copious vezzling.
intermittent weggerment.
non-responsive frattling.
uneven yerkation

49. Q7
Be careful about overlapping
questions! Candidates should
look at both questions, and get
maximum thinking
One common disorder of an overspragged
uckerpod is:
A.
B.
C.
D.
copious vezzling.
intermittent weggerment.
non-responsive frattling.
uneven yerkation

50. Q7
Be careful about overlapping
questions! Candidates should
look at both questions, and get
maximum thinking
One common disorder of an overspragged
uckerpod is:
A.
B.
C.
D.
copious vezzling.
intermittent weggerment.
non-responsive frattling.
uneven yerkation

51. Q8
Which of these is the correct answer?
A.
B.
C.
D.

52. Q8
There was a pattern. ‘D’ is due
to come up again
1: A; 2: B; 3: C; 4: D;
5: A; 6: B; 7: C; 8: D
Which of these is the correct answer?
A.
B.
C.
D.

53. Writing guidelines

54. Item writing guidelines
•
•
•
•
Test application of knowledge not recall
Don’t test “trivial” knowledge
Focus on real life problems
Use novel material - clinical vignettes are good
as long as they are realistic
• Use a blueprint

55. Item writing guidelines
• Minimize the amount of reading in each item
• Avoid unnecessary words
• Use correct grammar, punctuation,
capitalization, and spelling.
• Edit and proof items

56. Item writing guidelines
• Include the central idea in the stem instead of
the choices.
• Ensure that the directions in the stem are very
clear.
• Word the stem positively, avoid negatives such
as NOT or EXCEPT. If negative words are used,
use the word cautiously and always ensure that
the word appears capitalized and boldface

57. Item writing guidelines
• Place choices in logical or numerical order.
• Keep choices independent; choices should not
be overlapping.
• Keep choices homogeneous in content and
grammatical structure.
• Keep the length of choices about equal.
• Make sure that only one answer is right or
clearly better than any other
• Make all distractors plausible

58. Sample lead in and option lists for
clinical vignettes
• Which of the following is (abnormal)?
Options sets could include sites of lesions; list of nerves;
list of muscles; list of enzymes; list of hormones; types
of cells; list of neurotransmitters; list of toxins,
molecules, vessels, spinal segments.
• Which of the following findings is most likely?
Options sets could include list of laboratory results; list
of additional physical signs; autopsy results; results of
microscopic examination of fluids, muscle or joint tissue;
DNA analysis results; serum levels.
Constructing Written Test Questions For the Basic and Clinical Sciences, 3rd edn, 2001, National Board of Medical
Examiners (NBME), USA http://www.nbme.org/publications/item-writing-manual.html

59. Sample lead in and option lists for
clinical vignettes
• Which of the following is the most likely cause?
Options sets could include list of underlying mechanisms of the
disease; medications that might cause side effects; drugs or drug
classes; toxic agents; hemodynamic mechanisms, viruses,
metabolic defects.
• Which of the following should be administered?
Options sets could include drugs, vitamins, amino acids, enzymes,
hormones.
• Which of the following is
defective/deficient/nonfunctioning?
Options sets could include list of enzymes, feedback mechanisms,
endocrine structures, dietary elements, vitamins.
http://www.nbme.org/publications/item-writing-manual.html

60. Reviewing MCQs

61. Reviewing MCQs
• Peer review
– Both by subject experts and non-subject experts
• Evaluate Q performance after the exam
– Q difficulty – how many got the Q right
– Discrimination - How high performing and low
performing students answered each Q
– Utility of distracters

62. Item difficulty and the purpose of
testing
• If we are trying to rank students (norm-referenced) we
need to create items with a range of difficulty to try and
distinguish them, including some items we expect are
well above the standard required .
• If we are trying to determine competence or attainment
of a learning outcome (criterion-referenced) then items
need be no more difficult than required to determine the
difference between attainment and non-attainment.
• Remember that items can be difficult for the wrong
reasons

63. “Effective item writers are trained,
not born … “
Downing and Haladyna 2006, Handbook of test development ,p. 11

64. http://www.slideshare.net/liznorman