1. The “How-To” Measurement & Evaluation Series
Increased
productivity
through
skill building
Construct the
Ultimate Multiple
Choice Test!
Stop Measuring
Recall and Start
Measuring
Application
Chicago e-Learning & Technology Showcase
August 16, 2011

2. Today’s Presentation
Do you feel like you know how questions that are too simple, that “Great speaker and subject matter.
much participants truly learn from are overly difficult, or that test Ken’s expertise really made the event
your elearning programs? If not, knowledge and not application. In worthwhile.”
–South Central Wisconsin ASTD
you’re not alone. According to a this session, you will learn the art
Chapter Member
2009 ASTD research study, 81% of and science of Level 2 multiple
organizations measure participants’ choice test creation. Together, we
“This was very helpful. Obviously
learning (Level 2 evaluation), yet will analyze a sample test and learn
many people write tests for adults that
only 55% of these organizations how to identify and correct common
can be flawed. This session provided
view the data collected to be of real mistakes. Whether you are new to
good guidelines for writing valid tests.”
value. The problem is that the elearning or an advanced
–South Central Wisconsin ASTD
validity of our results is only as professional, you will walk away Chapter Member
good as the quality of our test with skills to help you improve your
questions. Often, we develop online Level 2 multiple-choice tests.
About Your Presenter — KEN PHILLIPS
Ken Phillips is founder and CEO of Phillips Associates, a
consulting company with expertise in performance Ken is a frequent presenter at
management, measurement and evaluation of learning and ASTD chapter meetings and other
sales performance. Ken has more than 25 years experience industry events. Here’s a preview of
designing learning instruments and assessments and has where he’ll be speaking next.
authored more than a dozen published instruments. He
also regularly speaks to American Society for Training and
Development (ASTD) groups and university classes. For 16 Chicago e-Learning &
the past four years, he has spoken at the ASTD Technology Showcase
International Conference on topics related to measurement Topic: Construct the
and evaluation of learning. Ultimate Multiple-Choice
Prior to pursuing a Ph.D. in the combined fields of organizational behavior and Test! Stop Measuring Recall
educational administration at Northwestern University, Ken held management and Start Measuring
positions with two colleges and two national corporations. In addition, he has Application
written articles that have appeared in T+D magazine, Training Today and
HR.com.
Currently Ken is serving his seventh year on the board of directors of the 14 Training & Performance
Chicagoland Chapter of the American Society for Training and Development Management PDN
(CCASTD), and was chapter president in 2008. He earned the Certified (Chicago)Meeting
Professional in Learning and Performance (CPLP) credential from national
Topic: Learn 12 Surprising
ASTD in 2006 as a pilot pioneer and was recertified in 2009. Ken is also
Techniques for Obtaining
certified in Measuring and Evaluating Learning and Test Design and Delivery
Powerful Data from Level 1
by ASTD and in the Kirkpartick Four LevelsTM Evaluation. In January 2009,
Evaluations
he also was appointed to serve a three-year term (2009-2011) as a member of
the ASTD National Advisors for Chapters (NAC) committee.
Ken can be reached at (847) 231-6068 or ken@phillipsassociates.com.
Copyright ©2011 by Phillips Associates
All rights reserved. Printed in the United States of America. Except as permitted under the United States Copyright Act of 1976, no part
of the publication may be reproduced or distributed in an form or by any means, or stored in a database or retrieval system, without the
prior written permission from Phillips Associates

3. Sample Level 2 Multiple Choice Knowledge Test
Interpersonal communication is often rendered
ineffective due to the presence of communication
barriers. What are the three categories of
communication barriers? Correct Answer:
A.
A. Environmental, Interpersonal and Intrapersonal Environmental,
1.
Interpersonal
B. Environmental, Interpersonal and Cultural and
Intrapersonal
C. Structural, Interpersonal and Cultural
D. Organizational, Intrapersonal and Structural
_________________________ are used to report
specific, observable actions or details in an objective,
non-evaluative manner.
A. Behavior descriptions Correct Answer:
2. A. Behavior
B. Open probes descriptions
C. Evaluative statements
D. Direct statements
What are questions that begin with words such as is,
are, could, would, have and shall known as?
A. Clarifying questions Correct Answer:
3. C. Close-ended
B. Open-ended questions
questions
C. Close-ended questions
D. Bridging questions
A Collaborative Communication Style is also known as
a?
A. Win/Win communication style
Correct Answer:
B. Low concern for self and high concern for other A. Win/Win
4.
communication style communication
style
C. Engaging and outgoing communication style
D. Intermediate concern for self and high concern
for other communication style
1 www.phillipsassociates.com

4. What are “You” messages?
A. Non-ownership statements that convey feelings, Correct Answer:
but do not actually describe the feelings. A. Non-
ownership
B. Statements that contain the pronoun “you” two or statements that
5. more times. convey feelings,
but do not
C. Statements that point out what another person actually
has done or was supposed to do. describe the
D. Statements that describe behavior and begin with feelings.
the word “You.”
A benefit of being an effective listener is?
A. Facilitates getting to know another person better
B. Increases the likelihood of having a mutual
Correct Answer:
6. exchange
D. All the above
C. Helps another person to get to know himself or
herself better
D. All the above
What are Hidden Agenda Questions?
A. An indirect or “pseudo” form of communication.
Correct Answer:
B. A question used to clarify what someone has
A. An indirect or
7. said.
“pseudo” form of
C. A question used to direct a conversation in a communication.
different direction.
D. A technique for getting another person to talk.
What are questions that begin with words such as what,
where, when, how and why known as?
A. Bridging questions Correct Answer:
8. D. Open-ended
B. Clarifying questions
questions
C. Leading questions
D. Open-ended questions
2 www.phillipsassociates.com

5. A perception check is a?
A. Technique for moving a conversation from the Correct Answer:
objective, where the focus is on facts, to the A. Technique
subjective, where the focus is on feelings. for moving a
conversation
B. Technique for pulling together the important parts from the
9. of one long conversation or a series of objective, where
conversations. the focus is on
facts, to the
C. Technique for helping someone to express his or subjective,
her real self. where the focus
D. Technique for conveying interest in what another is on feelings.
person has said.
Effective listening is defined as?
Correct Answer:
A. Attempting to understand the person speaking A. Attempting to
from his or her perspective and not your own. understand the
10. person speaking
B. Encouraging another person to talk.
from his or her
C. Consolidating the important parts of a discussion. perspective and
not your own.
D. Creating a win-win communication situation.
A proven technique for improving interpersonal
communication effectiveness is?
A. Replace generalizations and evaluative
statements with behavior descriptions.
Correct Answer:
11. B. Seek first to understand before attempting to be D. All the
understood. above.
C. Replace hidden agenda or pseudo questions with
direct statements.
D. All the above.
Which is not one of the four communication styles?
A. Intuitive
Correct Answer:
12. B. Avoiding
A. Intuitive
C. Aggressive
D. Collaborative
3 www.phillipsassociates.com

6. Which is not a use for paraphrasing?
A. Pulling together the important parts of one long Correct Answer:
conversation or a series of conversations. A. Pulling
together the
B. Conveying interest in what another person has to
important parts
13. say.
of one long
C. Testing for understanding when accuracy and conversation or
clarity of communication is important. a series of
conversations.
D. Helping to achieve mutual understanding when
strong feelings are present.
What communication technique is used to consolidate
what has been previously discussed and to move a
conversation from one topic to another?
A. Arguing Correct Answer:
14.
C. Summarizing
B. Interrupting
C. Summarizing
D. Initiating
____________________ are non-verbal cues such as
head nods and brief verbal comments such as “right”,
“okay” and “umhum”.
A. Minimal encouragements Correct Answer:
15. A. Minimal
B. Communication fillers encouragements
C. Confirmations
D. Acknowledgements
According to Robert Merabian, what percentage of
interpersonal communication is transmitted non-verbally
through tone of voice and body language?
A. 90 percent. Correct Answer:
16.
A. 90 percent
B. 60 percent
C. 75 percent
D. 50 percent
4 www.phillipsassociates.com

7. Level 2 Multiple Choice Test Question Job Aid
 % of live classroom vs. technology based programs evaluated at Levels 1-5 *
Live Classroom Technology Based
Level 1 81% ______%
Level 2 50% ______%
Level 3 25% ______%
Level 4 14% ______%
Level 5 6% ______%
 Level 2 evaluation facts *
• 81% of organizations evaluate some learning programs at Level 2
• 55% of organizations view data collected as having high or very high value
 Multiple Choice Test Item Pros **
• Versatility in measuring all levels of cognitive ability
• Highly reliable
• Easy to score
• Minimizes guessing factor
 Multiple Choice Test Item Cons **
• Difficult and time consuming to create
• Lead to creation of test items that measure recall of simple facts
5 www.phillipsassociates.com

8.  Multiple choice test creation tips **
• Avoid use of “All the above” “None of the above”
• If administering electronically, use diminishing response technique
• Write stem as question – response can finish question but don’t use fill-in-the-
blank
• Test for understanding not just knowledge
• Keep response options equivalent in length and writing style
• Include central idea and most of phrasing in stem
• Question should not reveal correct answer
• Be sure there is only one correct response
• Spread correct answers across A – D
• All response choices must be plausible
• Eliminate excessive verbage or irrelevant information from stem
• Write questions same way material taught – avoid “null” questions
* ASTD research study, “The Value of Evaluation: Making Training Evaluations More
Effective,” 2009.
** Adapted from ASTD Test Design and Delivery Certificate program participant
workbook, 2006.
6 www.phillipsassociates.com

9. Training Resources Available from Phillips Associates
Insightful Learning Products
Turnkey training resources for organizations and individuals
Phillips Associates offers an array of learning instruments, multi-rater assessment systems, handbooks (job aids),
manuals and games designed to develop specific skills in the areas of performance management and sales
performance. Some of our best-sellers include:
No Pain—All Gain! Performance Coaching to Win: Coaching Meeting Inventory
Appraisal Meeting Inventory The Coaching Meeting Inventory assesses a manager’s
Help managers conduct performance strengths and development needs in eight skill areas and
appraisal meetings that improve outlines a logical step-by-step process for conducting
employee performance and are effective coaching discussions.
viewed as valuable and worthwhile
using the Performance Appraisal Superior Service Starts Here: Customer Service Skills
Meeting Inventory. The instrument Inventory (CSSI)
pinpoints a manager’s strengths and development needs CSSI consists of 32 statements divided into four keys of
in six skill areas and outlines a simple six-step process providing superior customer service: Developing a
for conducting effective performance appraisal meetings. relationship of Interpersonal Trust, Listening, Problem
Solving and Professionalism. Based on how they answer
Getting On the Same Page: Objective Setting the statements, customer service reps gain valuable
Handbooks insight into their greatest strengths and most pressing
Easy to read booklets (one for managers and one for development needs in delivering superior customer
direct reports) filled with proven tips and techniques on service.
transforming the objective setting process from little
more than an annual ritual into a valuable, worthwhile Taking Stock for Better Communication:
experience for everyone involved. Includes a set of Communication Style Inventory (CSI)
worksheets the reader can use to prepare a set of CSI consists of 24 sets of paired statements describing
objectives. the four most commonly used communication styles:
Acting the Part: Performance Management Role Plays Aggressive, Avoiding, Accommodating and Collaborative.
Ready-to-use role play scenarios designed to help By choosing the statement that most nearly describes
managers, supervisors and team leaders apply and how they would communicate in a particular situation,
practice the skills needed for coaching effectively and respondents generate a profile of their use of the four
conducting productive performance appraisals. communication styles.
Includes half-day workshop design and PowerPoint slides.
For descriptions of all Phillips Associates Learning Products, visit www.phillipsassociates.com
PERFORMANCE MANAGEMENT SALES PERFORMANCE PRODUCTS MULTI-RATER ASSESSMENT PRODUCTS
PRODUCTS • Face-to-Face Sales Skills Inventory • Performance Management Profile
• Performance Appraisal Meeting System
Inventory
• Cross-Selling Skills Inventory
• Territory Management Style • Selling Skills Profile System
• Managing Your Own Performance
Appraisal Inventory Inventory FOUNDATION SKILLS PRODUCTS
• Communication Style Inventory
• Coaching Meeting Inventory • Customer Service Skills Inventory
• Keys to Effective Coaching
• Objective Setting Inventory Browse and buy online
• Interpersonal Feedback Inventory
from the PHILLIPS ASSOCIATES
• Objective Setting Games
Training Resources Online Catalog
• Performance Management Role
Plays
www.phillipsassociates.com
• Objective Setting Handbooks Save 10% on your entire order
• Performance Appraisal Handbooks when you mention this code: CELTS811
(offer expires 9/16/11)

10. Performance Management and Sales Performance Programs
In addition to our full offering of learning products, • Exceeding Expectations: Managing the Performance
Phillips Associates delivers interactive programs to help Appraisal Process
employees achieve and maintain better on-the-job
• From Basics to Best in Class: Customer Service Skills
performance. Our highly participative approach teaches
more than just concepts and practices—it builds lasting • Coaching to Win: Coaching Employee Performance
skills. Some of our most popular programs include:
For a complete list of programs offered, please visit us at
www.phillipsassociates.com/seminars.
The “How-To” Measurement & Evaluation Series
Only Phillips Associates shows you how!
Presentations (75-90 minute learning programs with audience participation)
• Learn 12 Surprising Techniques for Obtaining Powerful Data from Level
1 Evaluations
• Crack the Code of Test Question Design and Take Your Level 2
Evaluations to 2.5!
• How to Capture Elusive Level 3 Data: The Secrets of Survey Design
Workshop (1 or 2 day presentation with participant interaction)
• Demonstrate Value and Get Your “Seat at the Table” 51 Tips and Best
Kept Secrets for Designing Level 1 – 4 Evaluations
For more information contact Ken Phillips at 847.231.6068 or ken@phillipsassociates.com
34137 N. Wooded Glen Drive | Grayslake, IL 60030
847.231.6068 | Info@phillipsassociates.com
www.phillipsassociates.com
I N T E R N AT I O N A L D I S T R I B U T O R S
CANADA UNITED KINGDOM
Training & Development Materials of Management Learning Resources, Ltd.
Canada PO Box 28, Carmarthen
372 Moonstone road east, PO Box 38 Dyfed SA31 1DT
Moonstone, ontario L0K 1N0 Wales, United Kingdom
800.283.4964 01267-281661