The webinar summarized key lessons from several summer testing conferences regarding test security. It discussed conferences focused on statistical detection of cheating, state assessment, and international testing. Top takeaways included a growing recognition of security importance, attention to standards, and using data to detect misbehavior. The webinar also summarized the National College Testing Association conference, which featured several sessions on detection, prevention, and responding to breaches. It concluded that awareness of security is high but cheating remains a challenge requiring innovative detection and prevention strategies.

1. Caveon Webinar Series
“The Endless Security Summer:
Testing Conference Lessons”
August 22, 2012

2. “The Endless Security Summer”
 Steve Addicott
• Vice President
• Caveon Test Security
 John Fremer, Ph.D.
• President
• Caveon Consulting Services
 Lou Woodruff, Ph.D.
• Senior Security Consultant, Caveon
• Former President, National College Testing Association
2

3. Agenda
• Conference on Statistical Detection of Potential
Test Fraud, May 23-24
• National Conference on Student
Assessment, June 26-29
• International Test Commission, July 3-5
• Dutch Exam Society, July 6
• National College Testing Association, August 1-4
• Closing Thoughts
3

4. Poll Question #1
4

5. Conference on Statistical Detection of
Potential Test Fraud
• University of Kansas’ CETE; Lawrence, KS
• First-ever Data Forensics Conference
• Gathering of measurement experts &
researchers
– Certification
– Education
– Vendors
– Test Publishers
– Academics
5

6. Conference on Statistical Detection of
Potential Test Fraud
• Most important takeaway
THIS CONFERENCE EXISTS!!
• Second most important take away
Protecting our tests grows ever harder, and
ever more important
6

7. National Conference on Student
Assessment (NCSA)
• June 26-29 in Minneapolis, MN
• Council of Chief State School Officers (CCSSO)
• 2012 Theme: Teaching, Learning, and
Assessment through the Next Decade
7

8. Top Three NCSA Key Takeaways
8

9. NCSA Key Takeaways
#3—Bombshell!
CCSSO is partnering with Center for
Assessment to “re-focus” the conference
– Reduced attendance?
– Budgets?
– Changing priorities?
9

10. NCSA Key Takeaways
#2—A number of dynamics in state assessments
impacting security
– Teacher Accountability
• Legislatures passing laws
• Using existing assessments to now grade teaching
performance
– Assessment Consortia
– Technology
• Classrooms
• Administration
• Learning
10 • Assessments

11. #1 NCSA Takeaway
• Despite all these changes,
THERE WAS NOT ONE SESSION FOCUSED
ON TEST SECURITY.
NOT ONE.
11

12. Topics That Could Have Covered…
• Thoughtful piloting of security strategies
• By 2014-2015, all state assessments will be CBT
– Colossal logistical challenge
– Too few PCs and test centers
• Blending individual test programs
– Dozens into two
– Exams will be shared across states and time zones
– Administration windows will last weeks
– More technology, more sharing?
12

13. International Test Commission
• July 3 – 5, 2012
• Group representing testing organizations and
national agencies world wide
• Amsterdam, The Netherlands
13

14. ITC – Major Takeaways
• Growing recognition throughout the world of the
importance of protecting IP and preventing
cheating
• Attention to standards – ATP, ITC, ANSI, Caveon
• More attention to how to use data to detect
misbehavior
• “Trust but verify”
14

15. CITO & Caveon Session
• CITO – very major provider of testing services
to Europe and many non-European countries
• Collaborating with other Dutch testing
agencies to promote security standards
• Caveon – evidence of importance of security
– Internationalization of testing
– Challenges of technologically savvy cheats
– Notorious US cases, especially Atlanta PS
15

16. Faking and Test Security
• Steve Addicott Contribution
• Dealing with Media
– Be prepared
– Draw on all parts of your organization –
communications, legal, membership, etc.
– Have a spokesman
– Tell the story from your perspective
– Don’t expect the story to go away
16

17. Dutch Exam Society (NVE) Security Summit
• NVE has 400+ members from dozens of test
programs in The Netherlands
• Eight leaders gathered to:
– Address Test Security concerns
– Consider adoption of Security Standards for
members
• Caveon invited to play a supporting role
17

18. Top NVE Security Summit Takeaways
#3—Times are changing
– Test Security was not, but is now, a front-burner
issue
#2—Security Risks are global
#1—Rising tide lifts all boats
– Organizations which are considered
“competitive” can collaborate and create positive
change
18

19. Poll Question #2
19

20. ncta-testing.org
National College Testing Association
11th Annual Conference
Minneapolis 2012
20

21. ncta-testing.org
College testing administrators and staff
representing over 600 colleges
More than 50 companies that do business
in the testing field
21

22. ncta-testing.org
Consortium of College Testing Centers
Listing of hundreds of college testing centers
offering proctoring services for individuals
needing to take a secure, proctored examination
Access the listing for free on the NCTA website
22

23. Six Sessions Devoted to Test Security:
The Keynote Address
The General Session
Four Breakout Sessions
23

24. Keynote Address
Mark Poole
Product Strategist and Security Expert
Pearson VUE
“The Future of Secure Testing in a Digital World”
24

25. Highlighted five key trends that are
shaping the next decade of testing:
o a global cheating culture
o high tech test fraud
o testing anywhere securely
o candidate authentication
o data forensics
25

26. Breakout Session:
“Demystifying Data Forensics: An Overview of the
Logic Underlying Cheating Detection Techniques”
Jim Wollack, Ph.D.
Director, Testing and Evaluation Services
University of Wisconsin, Madison
Presented the basic principles of applied data
forensics in a way that everyone could understand.
Reviewed some forensics approaches to identify
examinees who copy answers, had prior access to
test content, or hired surrogates to test for them
26

27. Breakout Session:
“Gimme Shelter: Weathering the Media Storm
of a Security Breach”
Steve Addicott
Vice President, Client Services
Caveon Test Security
Ray Nicosia
Executive Director of the Office of Testing Integrity
Educational Testing Services (ETS)
Faisel Alam
Test Security Specialist
Law School Admission Council (LSAC)
27

28. This panel of security experts shared key aspects of
media strategy for testing programs and techniques
for positioning incidents to various audiences
High profile security breaches were reviewed,
highlighting details of what went well and what did not
The use and effectiveness of various media tools was
analyzed, and a checklist of media “Do’s and Don'ts”
was presented
28

29. Breakout Session:
“Security Strategies for Every Stage of the Testing
Process”
DeDe Hedlund Jeff Place
Creighton University Questionmark
Looked at best practices for colleges to protect test
content and minimize cheating
Demonstrated the potential for using mobile test
centers with examinees testing on tablets such as
the Apple iPad
29

30. Breakout Session:
“Innovations in Online Proctoring”
Rick Beaudry Denis Gravel
CEO Director, Workforce Development
B Virtual Inc. School of Applied Technology
Presented a proctoring service that proctored test
takers in any location via web-cam and microphone,
with a proctor monitoring the session remotely
30

31. General Session
“Detection and Prevention of Test Fraud
The State of the Art”
John Fremer
President
Caveon Consulting Services
Lou Woodruff
Senior Security Consultant
Caveon Consulting Services
31

32. o Perspectives on the Perpetrators
o Test Security Anatomy
o Contemporary Test Security Responses
o The Technology of Detection
o Large Scale Security Investigations
o Future Test Security Challenges for Higher Ed
o Implications for College Testing Professionals
32

33. Survey data on the prevalence of cheating among the
nation’s youth was shared which showed an
alarming level of cheating
Cheating has grown from a strategy for increasing
test scores to the manifestation of a growing social
norm that accepts dishonesty in many areas of life
The cheating/detection arms race will continue with
advances in biometric, data forensic and test
delivery techniques enhancing prevention and
detection
33

34. Result of Using Data Forensics
in a State Assessment Program
Reduction in Data Forensic Analyses of
"Irregularities" school districts statewide
6.0%
5.0% Decisive actions taken on
the forensic results
4.0%
3.0% Two years of follow-up test
2.0% administration monitoring
1.0%
Found a 72% reduction in
0.0%
test security irregularities
Report 1
Report 2
Report 3
Report 4
Report 5
Report 6
Report 7
Report 8
Report 9
34

35. The Atlanta Public Schools
The Perfect
Test Security
Storm
35

36. Mandated assessments
tied to federal funding
The Perfect
Schools and teachers Test Security
evaluated on test scores
Storm
Administrators feeling
unjustly treated
Test security left in the hands of
the teachers and the principals
36

37. Atlanta Public Schools Test Security Case
Governor brought in the GA Bureau of Investigation
Sixty-one agents on the case with subpoena power
District Superintendent, all Deputy Superintendents,
and many Principals and Teachers were removed
Who do you trust? Sometimes – no one!
37

38. Take-Aways from the NCTA Conference
College testing professionals are highly aware of
and strongly interested in test security
NCTA is a valuable platform for promoting security
related initiatives and partnerships between
colleges the testing industry
Colleges are and will remain a rich resource for
implementing cheating detection and prevention
strategies
38

39. Questions?
39

40. Today’s Presenters
Steve Addicott
steve.addicott@caveon.com
John Fremer
john.fremer@caveon.com
Lou Woodruff
LCWoodruff@aol.com
40