Presented by Dr. John Fremer, Dennis Maynes and Steve Addicott, Caveon Test Security
Two important industry conferences have been held in the last couple of months, the European Association of Test Publishers (E-ATP) Conference and the Conference on Statistical Detection of Potential Test Fraud (CSDPTF). Caveon was at both of these events and wants to share some important information with you.
Join Dr. John Fremer, President of Caveon Consulting Services, Steve Addicott, Vice President, Sales and Marketing, and Dennis Maynes, Chief Scientist, Caveon Data Forensics, who attended both conferences and presented sessions. They will explore key takeaways and lessons learned on security. Stay updated on the latest and greatest industry security trends.
Caveon Webinar Series: Lessons Learned from EATP and CSDPTF November 2013
1. November 20, 2013
Caveon Webinar Series:
Lessons Learned from
The European Association of Test Publishers Conference and
The Conference on Statistical Detection of Potential Test Fraud
John Fremer, President Caveon Consulting Services
Steve Addicott, Vice President, Client Services
Dennis Maynes, Chief Scientist, Caveon Data Forensics
2. Agenda for Today
John Fremer, Observations from E-ATP
Handbook of Test Security
CCSSO TILSA Test Security Guidebook
The Emergence of Social Media
Steve Addicott, Observations from E-ATP and CSDPTF
ATP/CCSSO OBP ITC Guidelines
Electronic Badging
ATP Security Committee
CSDPTF Observations
Dennis Maynes, Observations from CSDPTF
Progress, Research and Direction
Techniques that work, and don’t work
Challenges in this field
4. European Association of Test
Publishers (E-ATP) Annual Conference
5th Annual E-ATP conference:
Growing Talent In Europe:
Gaining Advantage Through
Assessment
Brings together European and
other international test
publishers and related
organizations
Held in beautiful St.
Julian, Malta
5. HANDBOOK OF TEST SECURITY,
FIVE TAKEAWAYS
1. What security vulnerabilities exist
for all genres of testing?
2. The critical importance of security
planning
3. Practical and proven strategies for
preventing and detecting cheating
4. How security breaches have been
dealt with in specific programs
5. What lessons have we learned from
past instances of testing
misbehavior?
6. HANDBOOK OF TEST SECURITY,
FIVE PREDICTIONS
In many high stakes testing programs:
1. Cheating detection statistical analyses will be
performed routinely
2. Computer-based testing will increasingly become
the norm
3. Technology developments will be critically
important to test security
4. The internationalization of the program will
increase
5. “Test Security Expert” will become a recognized
and valued position
7. ADVICE FOR THE READER
Review the Table of Contents
“Skim read“ the chapters
Earmark chapters and sections for staff to read or for
you to return to
Read sections as you deal with the issues addressed
To get the Handbook,
http://www.psypress.com/books/details/978041
5816540/ use discount code HYJ82 for a 20%
discount.
8. TILSA GUIDEBOOK
Guidebook: Preventing,
Detecting, and
Following up on Testing
Irregularities
Center for Chief State
School Officers
Designed for State
Assessment Directors
11. TEST SECURITY INVESTIGATIONS
(Sample #1)
Areas Needing Attention
Level of evidence warranting an
investigation
Roles for state, district, and other staff
Time lines
Model investigations kit
12. TEST SECURITY INVESTIGATIONS (cont.)
(Sample #2)
Model Investigations Kit
Clear definition of responsibilities
Procedures for evaluating evidence
Planning and conducting interviews
Developing a report with
recommendations
13. TILSA GUIDEBOOK
To get a copy of the
Guidebook,
Download a PDF for free or
purchase your own copy
for only $3.59.
http://www.ccsso.org/Reso
urces/Publications
14. SOCIAL MEDIA
Importance of Social Media -- several presentations
Sessions suggest using “Glocal” approach
Social Media continue to gain users in testing
internationally
15. Ignite Session: Are You One Post
Away From a Social Media Crisis?
Speaker: Matthew Poyiadgi, Managing Director,
Pearson VUE
3 P’s to deal with a social media crisis:
Police
Proactive Stance
Proportional Response
Customers expect a 1-day response from you.
20% get it.
16. Ignite Session: Are You One Post Away
From a Social Media Crisis? (cont)
Summary
Choose your communication channels wisely
Write your guidelines – make them practical
Choose and train those representing your agency
Be proactive – put your guidelines in action
Be proportionate with resources and responses
Track what you do – watch for patterns
Be personable on social media! You’re dealing
with people!
17. Lessons Learned from
The European ATP Conference &
The Conference on Statistical Detection of
Potential Test Fraud
Steve Addicott
18. Important Test Security Resources
ATP/CCSSO Operational Best Practices for Statewide
Large Scale Assessment Programs
International Test Commission Guidelines for the
Security of Examinations, Tests and Other Assessments
19. ATP/CCSSO Operational Best Practices
Published in Spring 2010:
Enhance the quality, accuracy, and
timeliness of student test data
derived from large scale
assessments
Strengthen public confidence in the
accuracy and quality of testing
data and their uses
20. ATP/CCSSO Operational Best Practices
2013: Revised OBP for CBT and Online Testing
Just published September 2013!
May not be precisely applicable to “…(test programs)
used on an international basis”
Provides a solid framework from which others might
seek to define a set of practices tailored to their
testing programs
ATP and the CCSSO encourage others to use this
document
21. ATP/CCSSO Operational Best Practices
To get a copy of the OBP,
Purchase a copy for $29.99 at
http://www.ccsso.org/resourc
es/publications
22. International Test Commission (ITC)
Security Guidelines
International Test Commission Guidelines for the
Security of Examinations, Tests and Other
Assessments
23. ITC Guidelines: Purpose
With increasing security problems, the guidelines are:
intended to share key elements of security best
practices to promote better security, and
defend the value of scores produced through the
assessment process
24. ITC Guidelines: Scope
All high-stakes assessments, tests and exams
All stakeholders in the assessment process (publishers,
users, students, test developers, etc.)
The entire assessment process from development to
administration to results processing and scoring
Across all areas of testing (e.g., education, employment,
certification/licensure government, clinical psychology,
etc.)
25. ITC Guidelines: Value
The ITC Guidelines provide answers to these questions:
Threats. What are the dangers?
Test Fraud. How can we deal effectively with increasing
security problems?
Terminology. How can we best communicate about test
security
26. Electronic Badging
What is electronic badging?
Big industry players involved
Compelling value
Badging + accreditation + assessment = ???
Stay tuned
27. ATP Security Committee
Picking up from last February’s
conference
Annual Survey results released:
https://www.createspace.com/4418924
Hands-on demonstrations of new tech
innovations used by test thieves and
pirates
Harder and harder for the “good guys” to
keep up
28. About the Conference
The Conference on Statistical Detection of Potential
Test Fraud (CSDPTF)
2nd Annual Conference this year
Presents various methods of cheating detection and
offers an important forum to examine a variety of
methods
Huge thanks to University of Wisconsin at Madison’s Dr.
James Wollack and team!
29. Conference on Statistical Detection of
Potential Test Fraud
Last year’s event was groundbreaking
This year, growing momentum…
Premier academic & industry organizations
supporting/sponsoring
Attendance steadily increasing
Publishing of selected papers
Plans for future years already underway!
Rumors of a scholarly journal – stay tuned!!
30. Conference on Statistical Detection of
Potential Test Fraud
Compelling dynamics behind the conference:
Growth in testing
Stakes growing ever-higher
Focus on trustworthy test results
How could we NOT focus on statistical detection?
31. Conference on Statistical Detection of
Potential Test Fraud
Challenge:
Current research on the use of statistics.
What about the policies and procedures- the legal
defensibility- wrapping their use?
32. Lessons Learned from
The Conference on Statistical Detection
of Potential Test Fraud
Dennis Maynes
33. Summary of CSDPTF
There were three general sessions on Friday with
four or five presentations and a discussant.
About ten poster presentations were on display
Friday evening.
There were three concurrent sessions on Saturday
with five presentations in the first session, four in
the second, and a symposium/workshop in the
third.
The conference was attended by over
120 researchers!
34. Topics Covered at CSDPTF
Similarity and Person Misfit
Pre-knowledge and “Braindumps”
Unusual Gains
Robustness and Sensitivity Issues
Test Tampering
Computer-Based Testing
Policy
Test Fraud Detection Challenge
Security Planning
35. CSDPTF: 2012 versus 2013
The conference this year was more
varied in terms of topics and research
than last year. Attendees were from
all over the world. The conference
has expanded in its reach, focus, and
attractiveness.
36. Progress
For this field to progress, the following are needed:
A set of data sets that have known characteristics of security breaches.
Commonly accepted ways to compare the efficacy of methods.
Consensus on the utility and implementation of statistical methods.
Consensus on how to make inferences concerning potential test fraud.
Guidance to practitioners on how to use the statistics.
37. Research
The current state of research is that of individual or
organization-based efforts with low cross-fertilization
between those efforts.
39. Techniques that work well and/or
are gaining greater acceptance
Test Tampering aka “Erasure Analysis”
Similarity aka “Answer Copying”
40. Techniques that do not work well
Detection of item compromise in all its forms such as braindumps,
rogue review courses, and disclosure of live exam content by
teachers and/or instructors
Analysis of gain scores for detecting potential test compromise
Analysis of response times to detect potential test compromise
41. Challenges Practitioners in This Field Face
1. Using statistics to invalidate test scores because the scores
are not valid. The problem is that the public and
measurement professionals have focused on the behavior
not the scores.
2. Conducting investigations when the evidence that
misbehavior occurred is circumstantial. There are no
missing persons, broken doors, or misappropriated funds.
42. Challenges Practitioners in This Field Face
3. Demonstrating the science behind the
techniques that are being used:
Reproducible results
Sound reasoning
Quantitative measures
Defensible assumptions
Measurement of error
44. Caveon Online
Caveon Security Insights Blog
http://www.caveon.com/blog/
twitter
Follow @Caveon
LinkedIn
“Caveon Test Security” Group
Please contribute!
Dr. John’s Test Security Tip of the Day
Caveon Security Minute
Facebook
“Like” us!
45. Thank you for your attendance!
Dr. John Fremer
President, Consulting Services
John.fremer@caveon.com
@TestSecurityGuy
Steve Addicott
Vice President, Client Services
Steve.addicott@caveon.com
@SdAddicott
Dennis Maynes
Chief Scientist
Dennis.Maynes@caveon.com
@DennisMaynes
At Caveon, we fundamentally believe in quality
testing and trustworthy test results.