Caveon Webinar Series - Discrete Option Multiple Choice: A Revolution in Testing October 26 2016

Caveon Webinar Series:
Discrete Option Multiple Choice: A
Revolution in Testing
October 26, 2016

Today’s Presenter
October 26, 2016
Dr. David Foster, CEO
Caveon Test Security

Agenda for Today
• Overview of DOMC
• Why this revolutionary technology
is so important
• Barriers to using DOMC
• Immediate and Long-Term Benefits
• Super-DOMC
3

Poll
Please rate your familiarity with the DOMC item type.
I know very little about how DOMC works.
I am slightly familiar with how DOMC works.
I am very familiar with how DOMC works.
I know everything there is to know about how DOMC works.
©️ 2016, Caveon, LLC. All rights reserved. Do not copy or distribute without permission. 4

What is DOMC?
Description:
Options for the multiple choice question are presented
randomly one at a time accompanied by YES and NO
buttons.
Options are no longer needed when the test taker
answers the question correctly or incorrectly.
5

DOMC Test Drive
6
The best way to understand DOMC is to experience
it…
…going to trydomc.com

Foster Revolutionary Rants and Raves
• It greatly improves the security of my tests.
• It eliminates the unfairness found in traditional multiple
choice tests.
• It reduces my test administration costs.
• It reduces my test development costs.
• It increases the accuracy of my tests.
• It expands the types of skills that can be measured.
• It accommodates a broader range of test takers.
• It motivates students to prepare better for exams.

So, if it’s that good, why isn’t everyone
using it?
What are the biggest concerns?

Poll
What concerns me about DOMC?
List of 13 possible concerns on 3 polling screens. Pick the ones
that seem right for you.

Addressing these Concerns

I don’t know how DOMC works.
DOMC is unfamiliar to me. I’m used to normal multiple choice.
Foster Response:
DOMC seems more complicated on the surface than traditional
multiple-choice, but it’s not rocket science.
There is no underlying complex theory.
There are whitepapers, studies and websites that can provide
information.
Practice creating and using items will remove the mystery.

My test development software doesn’t
support DOMC.
I can’t create DOMC items on my development system.
Foster Response:
More and more test development systems support DOMC:
1. Kryterion’s Webassessor
2. Excelsoft’s SARAS
3. ITS’s Item Workshop
4. Caveon’s Scorpion
It is not difficult to build the capability into any test development
software.

My test administration software/vendor
doesn’t support DOMC.
I can’t find a test administration system that supports
DOMC.
Foster Response:
More and more test administration systems support
DOMC:
1. Kryterion’s Webassessor
2. Excelsoft’s SARAS
3. ITS’s Internet Test Driver
4. Caveon does not offer test administration, but has
an interface called SEI that supports DOMC

DOMC is too costly to implement.
I can’t afford to use DOMC.
Foster Response:
It’s true that DOMC is a patented technology and it costs a
nominal fee to use that patent. The average per-test fee ends
up being a few cents.
There may be some programming costs.
The DOMC cost-to-benefit ratio is lower than any other testing
innovation.

My examinees won’t like DOMC.
DOMC will cause a customer service issue. I don’t want to
respond to a lot of complaints.
Foster Response:
This is fairly accurate. Every single test taker wants to see all of
the options. They will complain and try to get you to go
backward.
But research has shown that the DOMC is more intuitive and
simpler cognitively.
Practice has shown that examinees quickly become familiar
with DOMC and find it acceptable, even a better method.
Tutorials and practice tests are helpful in familiarizing
examinees with DOMC.

Quote
"DOMC was a challenge! I have always been
a good test taker, in part because I
understand some of the tricks that can be
used to make educated guesses and, in turn,
elevate my score. DOMC tosses a significant
amount of those test taking tricks right out the
window and kept me on my toes.”
Patrick Watts, J.D., Exam Security Administrator, CFA
Institute

DOMC is too complicated.
DOMC is more complicated than what we are using today.
Foster Response:
DOMC is unfamiliar and seems more complicated.
DOMC items are actually easier to create, and for examinees,
they are easier to answer.
The complicated, often misunderstood item is the traditional
multiple choice. For example, it introduces complicated
behavior, such as comparing item choices.

DOMC takes longer to answer the
questions.
Because options are shown one at a time, it takes longer to
answer a DOMC item, and test takers will not like the longer time
limits.
Foster Response:
A DOMC-based test actually takes less time!
Despite options appearing one at a time, several research
studies have documented that overall testing times decrease,
sometimes by as much as 40%!!!

Nobody is using DOMC.
I’d feel better if other programs are using it.
Foster Response:
Along with other adoptions, at least two certification programs
use DOMC exclusively in their certification exams:
1. OKTA Certified Professional and Certified Administrator
2. Certified Exam Security Professional (Caveon)
3. Three other IT certification programs currently in
development
4. Medical education exams in Germany
5. Preschool assessments

It is more difficult to create a DOMC
item.
Item writers need to learn a new set of skills to create DOMC
items.
Foster Response:
Actually, DOMC uses a smaller number from the same set of
skills that trained item writers have.
Many test writing rules (such as keeping options similar in
length) are rendered unnecessary. It is easier to write DOMC
items, and writers are already trained.
Writers DO have to learn how to score the more flexible DOMC
item.

DOMC causes more security problems
than it solves.
The DOMC item type exposes all the item content eventually.
Foster Response:
Yes, eventually, all options will be exposed.
DOMC creates barriers to harvesting by reducing the rate of
item content exposure.
DOMC has several other security advantages.

DOMC is not a fair way to test.
It is more fair to be able to see all the options.
Foster Response:
This is simply untrue. Traditional MC shows all of the options,
giving those with better test taking skills an unfair advantage.
DOMC is able to remove sources of unfairness in standardized
testing contributing to a more inclusive testing experience. It
levels the playing field!

Converting to DOMC is not worth the
effort.
It’s a hassle to convert our existing exams, and change our
examinee experience. How do we know the change will be worth
it?
Foster Response:
How much is peace of mind worth? How much would you save
to stop or postpone breaches? What if you could make better
decisions from the test scores?
Return on Investment for DOMC is difficult to calculate as there
are many intangibles. But the cost of implementing DOMC is
easily returned 10-fold by the benefits.

DOMC is not psychometrically sound.
DOMC does not have decades of experience behind it. How do I
know it really works?
Foster Response:
Research conducted so far has shown that DOMC produces
accurate and valid results.
It performs better psychometrically than the time-honored
multiple-choice question.

Quote
“Based on the results of these
studies, there appear to be no
psychometric reasons for excluding
DOMC from testing programs.”
Neal Kingston, Ph.D., Director of Achievement
and Assessment Institute, University of Kansas

10 Major Benefits of DOMC for High-
Stakes Testing
Security (N=3)
Measurement (N=1)
Fairness (N=3)
Test Development (N=1)
Test Administration (N=2)
These are 10 benefits out of 23. The others may be just as
compelling.

Test Security
DOMC reduces item content exposure
with no loss of measurement precision.

Quote
“Okta's laser beam focus on security is what led
us to…adopt the DOMC format for our new
certification program. We are enthusiastic about
this modern innovation in item development and
its ability to prevent test fraud because our
priority is to protect the value of the credential for
those that earn it.”
Laura Brumage, Vice President, Training at Okta

Research: Reduced Content Exposure
Foster & Miller (2009) found that only
about 2.5 options, on the average, were
required in order to score an item as
correct or incorrect.

Test Security
DOMC prevents item harvesting and
deters thieves and cheaters.

RESEARCH: College Students were
asked:
32
“Compared to traditional multiple choice, using
DOMC, how difficult would it be to cheat on
tests?” (…steal test content?)

Quote
"Item and test creation is an art form that
combines good science, good business,
and user-friendliness, among other things.
I can see why the DOMC item type is said
to be the ultimate secure item since the
examinee never sees the stem and all the
answer options on the same screen.”
Kim Brunnert, Ph.D., Director of Education,
Elsevier Education

Quote
“DOMC testing may also considerably
reduce the danger of unwanted
copying and sharing of test items
because many answer options do not
have to be presented at all.”
Sonja Willing, Martin Ostapczuk, and Jochen Musch,
University of Dusseldorf, in Advances in Health Science
Education, 2014
34

Test Security
DOMC provides an inherent “marking” of a
test, almost as reliable as a fingerprint,
allowing for quick tracking an item
harvester.

Measurement
DOMC removes the influence of irrelevant
factors, making the test score more
accurate and therefore more useful.

Quote
"I think that the scoring and
analysis that comes from the
DOMC items will be a useful
addition to Psychometrics and
Measurement.”
Kim Brunnert, Ph.D., Director of
Psychometrics, Elsevier Education

Fairness
DOMC removes unfair advantages some
test takers have. Puts everyone on an
equal basis.

Fairness
DOMC reduces the impact of poor reading
ability.

Fairness
DOMC expands the reach of testing to
underserved populations (e.g., young
children; second language learners;
cognitively impaired individuals; people
unfamiliar with testing).

Test Development
DOMC reduces test development costs
because they are simpler to create and
have a longer “shelf-life”.

Quote: Jamie Mulkey, Ed.D, CESP
Program Director
“By using DOMC items for the CESP
Generalist exam we can extend our
item development timelines and limit
item exposure.”

Test Administration
DOMC is easier to use on mobile devices
as it requires less screen “real estate”.

Test Administration
DOMC generates significant time savings,
reducing test administration costs.

Research: Time Savings
Willing, Ostpczuk, &Musch (2014) found savings
of 39% in the time needed to administer medical
education tests.
Foster & Miller (2009) found savings of at least
10% for college students taking psychology
exams.

Learning and Motivation
Unable to use test taking skills, or to
obtain pre-knowledge of the test
questions, examinees will be required to
learn the proper knowledge or skills.

Quote
"The largest difference in my mind is removing
the ability to eliminate some of the answer
choices. I think this could have a profound
change on learning if implemented on a large
scale. Could it provide a stimulus for deeper
learning and understanding if students knew
they would not have the opportunity to choose
the correct answer but had to determine if it was
in fact the correct answer? Only time will tell.”
Joseph Wolfe, Director of State and Federal Testing,
Charlotte Mecklenburg Schools, North Carolina

All of these benefits lead to
greater…
…Validity of test scores

Quote
“Using DOMC items has allowed the
CESP program to feel confident about
candidate test results.”
Dr. Jamie Mulkey, CESP Program Director

What is Super-DOMC?
ALL of the benefits are boosted even more
if Super-DOMC is used.
Super-DOMC items are those where many
more incorrect and correct options are
produced.

Feasibility of DOMC
With research support, new tools, and
operational program use, DOMC remains
a solid option for testing programs

Thank you!
Follow Caveon on twitter @caveon
Check out our blog www.caveon.com/blog
LinkedIn Group “Caveon Test Security”
53

Caveon Webinar Series - Discrete Option Multiple Choice: A Revolution in Testing October 26 2016

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (14)

En vedette

En vedette (13)

Similaire à Caveon Webinar Series - Discrete Option Multiple Choice: A Revolution in Testing October 26 2016

Similaire à Caveon Webinar Series - Discrete Option Multiple Choice: A Revolution in Testing October 26 2016 (20)

Plus de Caveon Test Security

Plus de Caveon Test Security (20)

Dernier

Dernier (20)

Caveon Webinar Series - Discrete Option Multiple Choice: A Revolution in Testing October 26 2016

Notes de l'éditeur