1. CONSTRUCT THE ULTIMATE
MULTIPLE CHOICE TEST!
STOP MEASURING RECALL
AND START MEASURING
APPLICATION
Presented by:
Ken Phillips
Phillips Associates
August 16, 2011
Phillips Associates

2. AGENDA
➤ Review Kirkpatrick/Phillips five level
evaluation model
➤ Examine Level 2 evaluation facts
➤ Analyze multiple choice test creation
errors in a sample Level 2 knowledge
test
➤ Learn 10 tips for creating valid,
scientifically sound multiple choice test
questions
Phillips Associates 2

3. KIRKPATRICK/PHILLIPS EVALUATION MODEL
Levels of Evaluation Measurement Focus Time Frame
Level 1: Reaction Participant reaction to a Conclusion of program
learning program
Level 2: Learning Degree to which participants Conclusion of program or
acquired new knowledge, within 6 to 8 weeks after
skills or attitudes
Level 3: Behavior Degree to which participants 2 to 12 months
applied back-on-the-job what
was learned
Level 4: Results Degree to which targeted 9 to 18 months
business outcomes were
achieved
Level 5: ROI Degree to which monetary 9 to 18 months
program benefits exceed
program costs
Phillips Associates 3

4. % LIVE CLASSROOM VS. TECH. BASED
PROGRAMS EVALUATED AT LEVELS 1 – 5*
Level
Live Classroom
Tech. Based
1-Reaction
81%
52%
2-Learning
50%
43%
3-Behavior
25%
15%
4-Results
14%
10%
5- ROI
6%
6%
*ASTD Research Study, “The Value of Evaluation: Making Training
Evaluations More Effective,” 2009
Phillips Associates 4

5. LEVEL 2 EVALUATION FACTS*
➤ 81% of organizations evaluate at least
some learning programs at Level 2
➤ 55% of organizations view data collected
as having high or very high value
*ASTD Research Study, “The Value of Evaluation: Making Training
Evaluations More Effective,” 2009
Phillips Associates 5

6. WHY THE DISCONNECT?
➤ Knowledge tests created are overly
simplistic
➤ Knowledge test administered too close to
learning event
➤ Test items focus on recall not application
➤ The data collected is not used
Phillips Associates 6

7. WHATʼS THE SOLUTION
➤ Create valid, scientifically sound Level 2
knowledge tests that measure job
application not just recall
➤ Include knowledge tests as part of
learning program design and use for both
evaluation and reinforcement
➤ Aggregate knowledge test score data and
use to demonstrate program value as well
as improve test item validity
Phillips Associates 7

8. SAMPLE LEVEL 2 MULTIPLE CHOICE
KNOWLEDGE
TEST
Phillips Associates 8

9. Whatʼs one thing that all 16 multiple
choice test questions have in
common?
Phillips Associates 9

10. They test for knowledge or recall – the
lowest level of cognitive ability
Phillips Associates 10

11. TEST CREATION TIP 1
➤ Test for application or understanding not
just knowledge
Phillips Associates 11

12. EXAMPLES
➤ What does TV stand for?
➤ What is the main function of a TV?
➤ What physical principle is used to display
images on a TV?
➤ Your TV is not working properly. What is
the most likely cause of the following set of
symptoms:…?
Phillips Associates 12

13. EXAMPLE
You sent a detailed program design document
to a group of SMEs to solicit their feedback,
and have incorporated a number of their
suggestions into the program design. This is
an example of what element in the ADDIE
model?
A. Analysis
D. Implement
B. Design
E. Evaluation
C. Development
Phillips Associates 13

14. WHATʼS WRONG WITH THIS?
10. Effective listening is defined as?
A. Attempting to understand the person
speaking from his or her perspective and
not your own.
B. Encouraging another person to talk.
C. Consolidating the important parts of a
discussion.
D. Creating a win-win communication
situation.
Phillips Associates 14

15. TEST CREATION TIP 2
➤ Keep response options equivalent in
length & writing style
Phillips Associates 15

16. WHATʼS WRONG WITH THIS?
4. A Collaborative Communication Style is
also known as a?
A. Win/Win communication style.
B. Low concern for self and high concern
for other communication style.
C. Engaging and outgoing communication
style.
D. Intermediate concern for self and high
concern for other communication style.
Phillips Associates 16

17. TEST CREATION TIP 3
➤ Question should not reveal correct
answer
Phillips Associates 17

18. EXAMPLE
4. A Collaborative Communication Style is
also known as a(n)?
A. Win/Win communication style.
B. Low concern for self and high concern
for other communication style.
C. Engaging and outgoing communication
style.
D. Intermediate concern for self and high
concern for other communication style.
Phillips Associates 18

19. WHATʼS WRONG WITH THIS?
14. What communication technique is used
to consolidate what has been previously
discussed and to move a conversation
from one topic to another?
A. Arguing
B. Interrupting
C. Summarizing
D. Initiating
Phillips Associates 19

20. TEST CREATION TIP 4
➤ All response choices must be plausible
Phillips Associates 20

21. WHATʼS WRONG WITH THIS?
6. A benefit of being an effective listener
is?
A. Facilitates getting to know another
person better.
B. Increases the likelihood of having a
mutual exchange.
C. Helps another person to get to know
himself or herself better.
D. All the above.
Phillips Associates 21

22. TEST CREATION TIP 5
➤ Avoid “All the above/None of the above”
Phillips Associates 22

23. WHATʼS WRONG WITH THIS?
12. Which is not one of the four
communication styles?
A. Intuitive
B. Avoiding
C. Aggressive
D. Collaborative
Phillips Associates 23

24. TEST CREATION TIP 6
➤ Write questions same way material taught
Phillips Associates 24

25. WHATʼS WRONG WITH THIS?
9. A perception check is a?
A. Technique for moving a conversation
from the objective, where the focus is on
facts, to the subjective, where the focus
is on feelings.
B. Technique for pulling together the
important parts of one long
conversation.
C. Technique for helping someone to
express his or her real self.
D. Technique for conveying interest in what
another person is saying.
Phillips Associates 25

26. TEST CREATION TIP 7
➤ Include central idea and most of phrasing in
stem
Phillips Associates 26

27. EXAMPLE
9. A Perception Check communication
technique is used to?
A. Move a conversation from the
objective to the subjective.
B. Pull together the important parts of
one long conversation.
C. Help someone express his or her real
self.
D. Convey interest in what another
person is saying.
Phillips Associates 27

28. WHATʼS WRONG WITH THIS?
2. ____________________ are used to report
specific, observable actions or details in
an objective, non-evaluative manner.
A. Behavior descriptions
B. Open probes
C. Evaluative statements
D. Direct statements
Phillips Associates 28

29. TEST CREATION TIP 8
➤ Write stem as a question – responses can
finish question, but donʼt use fill-in-the-
blank
Phillips Associates 29

30. WHATʼS WRONG WITH THIS?
1. Interpersonal communication is often
rendered ineffective due to the presence of
communication barriers. What are the three
categories of communication barriers?
A. Environmental, Interpersonal and
Intrapersonal
B. Environmental, Interpersonal and Cultural
C. Structural, Interpersonal and Cultural
D. Organizational, Intrapersonal and
Structural
Phillips Associates 30

31. TEST CREATION TIP 9
➤ Eliminate excessive verbiage or irrelevant
information from the stem
Phillips Associates 31

32. EXAMPLE
1. What are the three categories of
communication barriers?
A. Environmental, Interpersonal and
Intrapersonal
B. Environmental, Interpersonal and Cultural
C. Structural, Interpersonal and Cultural
D. Organizational, Intrapersonal and
Structural
Phillips Associates 32

33. TEST CREATION TIP 10
➤ Spread correct answers across A – D
Phillips Associates 33

34. OVERALL SUMMARY
➤ Develop Level 2 Multiple Choice Tests
that are:
• Valid
• Fair to the learner
• Fair to the organization &
• Test for understanding
Phillips Associates 34

35. FREE ARTICLES
Phillips, Ken, “Developing Valid Level 2 Evaluations”,
Training Today, Fall 2009, pps. 6-8.
Phillips, Ken, “Eight Tips on Developing Valid Level 1
Evaluation Forms”, Training Today, Fall 2007,
pps. 8 & 14.
Phillips Associates 35

36. Ken Phillips
Phillips Associates
34137 N. Wooded Glen Drive
Grayslake, Illinois 60030
(847) 231-6068
www.phillipsassociates.com
ken@phillipsassociates.com
Phillips Associates 36