Here are a few key points regarding the admissibility of expert evidence in court as it relates to data forensics:- Expert testimony must be based on sufficient facts or data, be the product of reliable principles and methods, and the expert must have reliably applied the principles and methods to the facts of the case. This establishes the scientific validity and reliability of the testimony.- Courts use the Daubert standard to determine if expert scientific testimony is admissible. Daubert established factors like testing, peer review, error rates, and general acceptance in the relevant scientific community. Data forensics methods would need to meet this standard. - Experts in data forensics would need to be qualified by knowledge, skill
This document provides an overview and agenda for a presentation on implementing a data forensics program to monitor test security. The presentation covers introducing data forensics and its uses in monitoring for security breaches and investigating potential breaches. It also discusses obtaining budget and stakeholder support, the legal foundation and issues around data forensics, and managing investigations. The agenda includes sections on introduction to data forensics, program implementation, obtaining budget and support, legal issues, and managing investigations.
Similar to Here are a few key points regarding the admissibility of expert evidence in court as it relates to data forensics:- Expert testimony must be based on sufficient facts or data, be the product of reliable principles and methods, and the expert must have reliably applied the principles and methods to the facts of the case. This establishes the scientific validity and reliability of the testimony.- Courts use the Daubert standard to determine if expert scientific testimony is admissible. Daubert established factors like testing, peer review, error rates, and general acceptance in the relevant scientific community. Data forensics methods would need to meet this standard. - Experts in data forensics would need to be qualified by knowledge, skill
Similar to Here are a few key points regarding the admissibility of expert evidence in court as it relates to data forensics:- Expert testimony must be based on sufficient facts or data, be the product of reliable principles and methods, and the expert must have reliably applied the principles and methods to the facts of the case. This establishes the scientific validity and reliability of the testimony.- Courts use the Daubert standard to determine if expert scientific testimony is admissible. Daubert established factors like testing, peer review, error rates, and general acceptance in the relevant scientific community. Data forensics methods would need to meet this standard. - Experts in data forensics would need to be qualified by knowledge, skill (20)
Here are a few key points regarding the admissibility of expert evidence in court as it relates to data forensics:- Expert testimony must be based on sufficient facts or data, be the product of reliable principles and methods, and the expert must have reliably applied the principles and methods to the facts of the case. This establishes the scientific validity and reliability of the testimony.- Courts use the Daubert standard to determine if expert scientific testimony is admissible. Daubert established factors like testing, peer review, error rates, and general acceptance in the relevant scientific community. Data forensics methods would need to meet this standard. - Experts in data forensics would need to be qualified by knowledge, skill
1. ―Everything You Need to Know to
Implement a Data Forensics Program‖
Presenters:
Dennis Maynes - Chief Scientist, Caveon Test Security
Jennifer Ancona Semko - Partner, Baker & McKenzie
Kerri Davis - Anti-Piracy Program Manager, Microsoft Learning
Presented September 27th, 2012
2. Agenda
Introduction to Data Forensics
Program Implementation
Obtaining Budget and Support
Legal Foundation of Data Forensics and
Navigating Legal Issues
Managing Investigations
002
4. Introduction
• Purpose of a data forensics program
– Measure and manage security risks
– Ensure fair and valid testing
– Use statistics to monitor and to investigate
• Purpose of security initiatives
– Mitigate losses and liability
• Illustrations of loss
– November 2007, Denver, de-icers
– July 2011, Atlanta, 178 educators
004
5. Data Forensics
• Science of examining data to find potential security risks
• There are clues in the data relating to:
– Collusion
– Use of recalled questions
– Rogue review courses
– Testing sites with poor security
– Exams and items that have become exposed
• ―We balance probabilities and choose the most likely. It is the
scientific use of the imagination.‖ – Sherlock Holmes, The Hound of
the Baskervilles
005
7. Test Security Threat Scale
Statistical Anomalies
Testing
Irregularities
Security
Violations
Security
Breaches
Test Fraud
008
8. Test Fraud Taxonomy
• Content Theft and/or Disclosure
• Collusion and/or Providing information during the
exam
• Violation of Proctoring and/or Administration Rules
• Tampering and/or Manipulating the Score
Distribution
• Based on Amrein-Beardsley, A., Berliner, D. C. &
Rideau, S. (2010). Cheating in the first, second,
and third degree: Educators' responses to high-
stakes testing.
009
9. Data Forensics Uses
• Two modes
– Monitor for security breaches
– Investigate potential breaches
• Inform investigations
• Take corrective actions
– Score invalidations
– Test site closures
– Replace test items
• Manage security health
• Monitor security risk levels
010
10. Data Forensics Monitoring
• Examine ALL of the data
• Must correct for multiple comparisons
– Bonferroni Correction
– Probability for threshold is /n
– Example: if n = 10,000 and is .05 – use .0000005
• Probabilities allow
– Objective measures
– Ensemble statistics
– Error rate control
011
11. Data Forensics Inference
• To invalidate scores, most psychometricians require
– An eye-witness account and
– Probability less than one in ten thousand.
• Hypothetical question: Suppose two individuals
submitted identical 500 word essays – would you act?
• Basis for action
– Strength of the evidence
– Is the score trustworthy?
If you accept statistics to determine candidate competence, why
would you reject statistics to determine score trustworthiness?
012
12. Circumstantial Evidence
• Requires an inference or deduction
• Seek ―disconfirming‖ or plausible explanations
• Value of multiple pieces of evidence
• Collect and document all the evidence
• Apply policy consistently for each case
―Circumstantial evidence is a very tricky thing. It may seem to point very
straight to one thing, but if you shift your own point of view a little, you
may find it pointing in an equally uncompromising manner to something
entirely different.‖ – Sherlock Holmes, The Boscombe Valley Mystery
013
16. Test Security is a Process
Data Forensics can be used to improve the process
Assess overall test security risks
Identify strengths and weaknesses in the process
Document the test security process
Take steps to improve the test security process
The goal of a Data Forensics program should be the improvement
of test security. The primary purpose of Data Forensics is NOT to
apprehend and punish potential cheaters.
015
17. General implementation approach
• Establish policy and precedent: scope of your
program, budget, legal, planned outcomes
• Obtain stake-holder support
• Create infrastructure
• Create agreements
• Reports
• Revise exam policies
• Implementation
– Conduct pilots and dry runs
– Train staff
– Perform data forensics analyses
– Review and revise
016
18. Stakeholder support is critical
• Explain how the statistical analysis works
• Present results from data forensics analyses
– How many test takers were cheating?
– How many locations had weak security?
• Outline overview of cheating evidence
• Explain work flow
• Outline costs associated with enforcement
• Address individual questions/concerns
017
19. Summary
• Think of a 3-tiered approach to your data forensics
implementation:
1. Establish scope
2. Propose a budget
3. Create an action plan
• Use this approach to formulate your program and
to gain stakeholder support
• Don’t be afraid to revise as you go; using data
forensics is a process itself.
21. It’s not ―IF‖… it’s ―WHEN‖
• Don’t wait for a breach to occur before you seriously
think about security.
• Don’t be in a position of explaining (to your
stakeholders, the public, or the press) why you are not
protecting the integrity of your exams.
Successful implementation of a Data Forensics program will
anticipate inquiries by the media and the public in order to
communicate that the program is pursuing a proper course for
ensuring the tests are administered fairly and securely.
021
22. Assemble the Security Team
• Identify key personnel from the affected departments:
– Exam Development
– Psychometrics
– Exam Administration
– Legal
– Risk Management
– Scoring/Grading
– Professional Conduct
022
23. Make the Case for Security
• Exam scores mean nothing if candidates can gain an
unfair advantage by cheating.
• Share highly-publicized examples of individuals gaining
an unfair advantage
– Prevention: Avoid being a news story
• Present the Cost of Security vs. Insecurity
– Cost of Development
– Reputational Harm
– Threat to the Public
023
24. Plan for Successful Implementation
• Propose a security budget
• Identify key individuals, their roles, and their time
commitment
• Develop and document process flows
• Anticipate and overcome obstacles
– Fear of statistics
• Don’t understand them
• Don’t understand how to use them
– Fear of what people might think
– Lack of familiarity with score review process
024
29. The First Brick:
the Candidate Agreement
• Contract: An agreement between two
or more persons which creates an
obligation to do or not to do a particular
thing. A legal relationship consisting of
the rights and duties of the contracting
parties. Black’s Law Dictionary, Sixth
Edition
• Your agreement with test takers
defines the relationship
• Memorializes your (and their) rights
and obligations
• If done properly, makes expectations
(and remedies) clear
030
30. What does your agreement say?
• Are candidates on notice that sharing items is a breach?
• Are candidates on notice that studying from recalled items is
improper?
• Did you reserve the right to invalidate scores? Suspend or
permanently ban access to the examination? To take other
action?
• Are candidates on notice of the possible use of data
forensics?
• What are the grounds for action? Is there a ―catch all‖?
• Do you regularly review your agreement language?
• Do you have uniform security procedures and policies in
place?
• Are candidates required to cooperate in investigations?
031
31. Successfully Using Data Forensics
•Can you defend your actions?
–Do you have to prove ―cheating‖?
•Contract law – ―good faith‖
–Language of agreements
–Documented policies & procedures
–Taking all steps to show ―good faith‖
•Will you need expert testimony?
•General deference to exam programs
–State actors: due process
032
32. Admissibility of Expert Evidence in Court
Federal Rule of Evidence 702, Testimony by Experts
If scientific, technical, or other specialized knowledge will assist
the trier of fact to understand the evidence or to determine a fact
in issue, a witness qualified as an expert by knowledge, skill,
experience, training, or education, may testify thereto in the form
of an opinion or otherwise, if:
(1) the testimony is based upon scientific facts or data,
(2) the testimony is the product of reliable principles and
methods, and
(3) the witness has applied the principles and methods
reliably to the facts of the case.
033
33. Admissibility of Expert Evidence in Court
• Daubert v. Merrell Dow Pharmaceuticals, Inc., 509 U.S.
579 (1993)
– ―[U]nder the Rules [of Evidence] the trial judge must
ensure that any and all scientific testimony or evidence
admitted is not only relevant, but reliable….‖
• Kumho Tire Co. v. Carmichael, 526 U.S. 137 (1999)
– ―The objective of [the trial court’s gatekeeper] requirement
is to ensure the reliability and relevancy of expert
testimony. It is to make certain that an expert … employs
in the courtroom the same level of intellectual rigor that
characterizes the practice of an expert in the relevant
field.‖
35. Deference to Exam Programs
Murray v. ETS, 170 F.3d 514 (5th Cir. 1999) (SAT
Exam)
• Louisiana basketball player; needed 820 on SAT
• Scored 700, then 1300
– Similarity to nearby student (3 in 100 million odds)
– Scored 800 on retake
• ―ETS’s contract with Murray clearly and explicitly
reserved to ETS the right to withhold any scores
ETS had reason to believe were not valid. The
only contractual duty ETS owed to Murray was to
investigate the validity of Murray’s scores in good
faith.‖
034
36. Deference to Exam Programs
Langston v. ACT, 890 F.2d 380 (11th Cir. 1989) (ACT
Exam)
• Alabama football player; scored 10 on ACT; then 20
• Inconsistent with GPA; unusual similarity to nearby
student
• ―Under the governing law, the outcome of plaintiff’s
case does not turn on whether or not plaintiff cheated
on his exam, but only on whether or not ACT carried
out its contractual obligations in good faith.‖
035
39. Managing Investigations
• Do your investigators have all they need to be effective?
– Corporate support (budget, effective legal counsel, training)
– Software/services
– Support from other departments (Psychometrics, Test
Development, Registration & Credentialing, etc.)
• What procedure is in place to select cases for
investigation?
– Is it prudent/efficient to investigate all matters?
• What metrics exist to determine the success/efficiency of
an investigation?
– Did the investigation glean the desired information?
– When does an organization ―close‖ an investigation?
027
40. Managing Investigations
• Who within (or outside) your organization conducts
investigations? When?
– May depend on investigation type: exam-day incidents,
ongoing copyright infringement, collusion, proxy testing
• At what point do you involve legal counsel, board
members, or other departments?
• Do your policies and procedures reflect what is
needed to manage investigations?
– Are candidates obligated to cooperate?
• How are results reported? Who makes sanctions
decisions?
41. Gather Evidence
• Similarity analysis, gains analysis, other statistics
• Reports of security incidents
• Seating charts and chain of custody of materials
• Review test taker associations and connections
• Review access logs to secured exam content
• Review score histories of test takers and locations
• Review test booklets for signs of ―work‖
• Responses by test center staff & test takers
• Adherence to security policies
018
42. Evaluate Evidence
• Do alternative explanations exist?
• Are candidate explanations/responses convincing?
• Could test fraud have occurred?
• Are the test results trustworthy?
– Evaluation depends upon trustworthiness of the scores, NOT
an inference of behavior.
The proper use of Data Forensics is to certify the trustworthiness of the test
results and the integrity of the test administration. It is NOT proper to use
these results to place a label, such as ―cheater,‖ on an individual.
019
43. Wrap up & key takeaways
• Data Forensics
– Measure and manage risks
– Ensure fair and valid testing
– Mitigate losses and liability
• Implementation
– Security is a process, not a state
– Policy—what will we do with the results?
– Breaking down organizational ―fiefdoms‖
• Support
– Not ―if‖, but ―when‖
– Nobody plans to fail, but…
44. A long and winding road….
Key Takeaways, cont.
• Legal
– Agreement is your foundation
• Even in K-12!
– Don’t be ―arbitrary and capricious‖
– Consistent and uniform
• Investigations
– ―Go/No Go‖ decision criteria
• Constrained resources
– Focus on the results, not the behaviors
• Statistically ―Indeterminate results‖ vs ―You’re a cheater!‖
47. Hope to see you soon…
• CCSSO TILSA SCASS
– Oct 1-5
– Indianapolis
• Next month’s webinar
– ―Do It Yourself Security Audits and Security Investigations‖
– Tuesday, Oct.16, noon EDT
• ICE (Institute for Credentialing Excellence)
– Nov 6-9
– Palm Springs
48. Want more?
LinkedIn group ―Caveon Test Security‖
twitter @caveon for updates, news, connect
Blogs!
Caveon Security Insights – www.caveon.com/blog
Cheating In The News – www.caveon.com/citn
www.caveon.com/resources/webinars - to see past sessions
Contact skyler.weisenburger@caveon.com for slides,
comments, and questions about this session
Editor's Notes
Introduction to Data Forensics - DennisProgram Implementation - KObtaining Budget and Support - KManaging Investigations - JLegal Foundation of Data Forensics and Navigating Legal Issues - J
Test Security is a ProcessWhetheryou already have a program in place, or are just starting out, it’s important to realize that data forensics will be one factor in your security plans. Data Forensics can be used to improve the processYou can take your program to the next level; data forensics can help you understand your gaps and needs in order to enhance any efforts you’re already doing. Assess overall test security risksIdentify strengths and weaknesses in the processData forensics provides that holistic view of what is going on in your testing program—the behaviors and trends that indicate where gaps lay.Document the test security processOnce you have the forensics piece in place, it can help you document the process overallTake steps to improve the test security process
First you will want to look at your program overall, if you have one, and identify its strengths and weaknesses. Then you can lay a foundation to stakeholders on how data forensics will help your program—whether its for the sole purpose of identifying security risks, for enforcement, for metrics, for etc. You may want to have a conversation with your legal team on your goals with data forensics, and get parameters around what you can do with the data. You might also talk to your exam delivery provider about any forensics they do, and how you can work with them. This is the stage where you set the scope of data forensics within your overall security program.Next, suggest a budget—will you do forensics on all your exams, part of your exams, part of your program, etc. This is important so that stakeholders know exactly what they are paying for and what metrics you are hoping to provide.At this point you should prepare your stakeholders for their support by having a clear scope, a legally-defensible plan for what to do with the data, a budget, and what kind of reporting or metrics you hope to provide. Once you have stakeholder support, you can begin to create infrastructure. Create your legal foundation by getting the right agreements in place. For instance, this may mean revising a candidate policy to include the use of forensics. Learn how to read reports and train any staff. Study the data and revise what you are measuring, if it is not helpful. At this point you may be fine-tuning your overall plan and policy as you learn how forensics work.Last, you’ll want to implement any changes to your program, or enforcement actions, that you planned to do as a result of forensics. These changes may not be immediate; you may want to have a period of “data collecting” only. But be sure to deliver on what you promised to stakeholders.
Also important to do your homework regarding legal limitations. You don’t want to make promises you can’t keep.
So probably the biggest concern you might get from a stakeholder is a question as to why you need data forensics. Can’t we just catch the cheaters without it? This is where you stress that the data forensics is much more than just “going after cheaters”—it’s about a holistic, independent look at your whole security program. The strength in data forensics lies in finding those hidden vulnerabilities, so that you can head them off before there are breaches. Think of it as preventative security spend rather than reactive. The other concern you will probably get is around the legal defensibility of such a program, and this is where having a clear scope and action plan, that you have cleared with your legal team, can help.
Again, this is where you really stress the importance of proactive planning vs reactive—prevention over enforcement. The cost of breaches are both tangible: rescoring, redeveloping exams, re-administering; and intangible: reputation, integrity
To sum:
Jennifer to edit this with new links/sources
Jennifer
[not sure I understand how this slide relates to data forensics]?
This slide speaks to the heart of “what are you going to do with your information?” could there be any other explanation for why test anomalies occurred? Depending on the size of your program, your data forensics will most likely lead to more follow-ups with schools, teachers, proctors, or exam delivery providers, rather than giving you a clear case to do xyz.