Caveon Webinar - Weathering the Perfect Test Security Storm May 2015 edited
Caveon Webinar - International Testing 11-15-2011
1. International Testing: Crossing Global
Borders While Protecting Your Own
Steve Addicott
Vice President, Caveon Test Security
Tom Gibbs
Senior Analyst, SHL Group
Aimee Hobby Rhodes, JD
Director of Exam Security, CFA Institute
2. Agenda
Intros
Security Threats to International Test Programs
Steve Addicott, Caveon
Employment Testing: The Stakes are High, and They’re
Global, Too
Tom Gibbs, SHL Group
Assessing Country Risks
Aimee Hobby Rhodes, CFA Institute
Q&A
4. About Caveon Test Security
Starting 9th year
Offering the “Three Ps”
Proven
Practical
Protection
Certification/Licensure & Education Clientele
Pre-eminence
Testing globally
We’ve seen a few things…
5. International Test Threats
Proxy Test Taking
Braindumps & “Rogue Review” Courses
Nefarious Test Centers
See above
Unusual practices
6. Nefarious Test Centers
NOT vendor-owned centers
Example of Unusual Test Taking
Test Center Hours 6 am to 6pm
130 tests taken at odd hours, ie 2am
Candidates unable to schedule tests when Test Center is
closed
WHAT IS GOING ON?
And, Why are these candidates scoring much higher than
others?
7. Braindumps and Rogue Review Courses
Test Prep Courses advertising & selling live test
items
Prevalence of stolen-answer key usage may exceed
20%
How are items stolen?
Infiltrate a top vendor’s premium security, or
Bribe a remote test center?
Evidence points to unauthorized decryption of exam after
downloading to test site servers
8. Proxies and “Gunmen”
Proxy Test Takers:
Make Big Money
Analysis uncovered a large proxy test taking service operating in
at least six testing sites: USA, Bangladesh, Turkey, UAE, and an
unknown country.
500 test instances @ $1000 per instance in 6 months
Own and operate test centers
Are branching out
15. Overview of presentation
The user model
It is 24/7/365 and its online
Why the stakes are high
The war for talent and the current economic conditions
Why the stakes are global
The war for talent is everywhere
Are we managing the cheats and the pirates?
The value of web patrols and a clear security framework
Summary and further information
We are publishing more about our security processes
17. 24/7/365
The user model is increasingly not proctored and not
delivered via group testing
Delivered online with a variety of security features
Linear On-the Fly Testing (LOFT) and Computer Adaptive
Testing (CAT)
Systems for managing item exposure and answer key exposure
Supported by a security framework including data forensics,
web patrols and incident response procedures (see next slide)
Initial version has been operating since 2002
Verify has been operating since 2006
Now available in 26 languages
18. Test security monitoring
Security Breach Process
•Collect Evidence
Web Patrols •Report to Legal
•Communications with
Suspected suspected offender
security breach
Data Forensics
Reporting •Run analysis on data specific
(through clients/business/other) to test, date range and client
of suspected breach
No
Continue Item Bank
monitoring compromised?
Yes
Replace
compromised
items
20. Further Information...
Burke, E. (2009). Preserving the integrity of online
testing. Industrial and Organizational Psychology, 2, 35-
38.
This is a special edition of I/O Psychology on this topic
and can be accessed via www3.interscience.wiley.com/
journal/118902544/home
23. Quotes from a well known graduate
recruitment portal
“Being the honest idiot in a sea full off successful liars
will get you nowhere”
“I put sweat, blood and tears into improving my CV and
am absolutely sickened that lazy, lying cheating
graduates are getting jobs they have done nothing to
deserve”
“Have to say that you shouldn't lie about what you can
get caught out on but anything else is fair game. Real
life is about communication, persuasion and
negotiation”
“It's important that you are able to describe in detail
your fake job just in case you are asked about it”
24. The war for talent is local and
global
Geography is becoming less of a
constraint
26. So, how do we know if we are managing
the cheats and the pirates?
27. Web patrols are telling us that there is a
threat but we can detect it and manage it
28. Web patrols are also telling us that the
applicants know there is security and
that there are consequences
29. Internet discussion from April 12th 2010
ttsstt “With all the psychometric testing for a number of jobs i have applied for
i am wondering how many people cheat and get help from family/friends while
doing the test. From a few people i've spoken too I get the sense that it is MORE
than common …Thanks guys”
underthepump: “I didn't for two reasons, well two reasons on top of being
honest :p … 1) They make you do a supervised one at ACs anyway … 2) Time
limit. You don't exactly have time to sit there and come to a consensus and
different people work at different paces. Personally having someone else sitting
there would annoy me more than it would help”
brendan58: “what underthepump said pretty much. they make you sit a
supervised test in the AC and it would be quiet a shame to get rejected based on
fluctuating test results … just need to have the confidence to complete them
yourself”
nebasil: “There is no point in cheating, how do you know the person doing the
test for you will do better than you? I'd rather trust myself than someone else”
Huynhzie: “lol cheat now, ur going to get screwed in there. there was a PWC guy
who went to his AC, he absolutely got owned at the verification test there and
he was pointed and told off by the HR. HR was like "how can u perform
absolutely so low", SHL is to test ur honesty as well, not just brilliance”
34. Coming soon …
Lievens, F., and Burke, E. (2011). Dealing with the threats inherent in
unproctored Internet testing of cognitive ability: Results from a large-
scale operational test program. Journal of Occupational and
Organizational Psychology (due for publication this year)
First scientific journal article describing features designed to promote
the security of online high stakes testing
Bartram, D., and Burke, E. (In press). Industrial/Organizational
Testing Case Studies. In Wollack, J. A. and Fremer, J. F. (Eds.).
Handbook of Test Security. New York, NY: Routledge.
This also includes a review of faking good on self-report (typical
performance) measures such as personality questionnaires
37. Due Diligence for International Testing
CFA Institute
CFA Institute awards the CFA Charter to candidates who successfully
complete 3 levels of testing and meet other education and work-
related criteria.
In 2010, approximately 200,000 candidates registered for CFA Exams
administered in 89 countries across the globe. Exams are
administered twice a year.
All three levels of the CFA Exam are paper and pencil.
CFA Institute also gives the CIPM designation and exam, which is CBT.
38. Country Risk Assessment
Two Different Assessments:
Focus on Legal, Regulatory & Tax
Focus on ability to protect rights and
intellectual property (including test items) in
a country. CFA Institute does this prior to
opening a test center in a new country.
39. Legal, Regulatory & Tax
What governmental agencies, if any, must we
interact with in order to administer our tests in the
way our program is designed?
What Taxes do we owe based on our program?
- keep in mind other activities in the country
- inquiries on a “no name” basis
40. Assessment to Open a New Test Center
Practice established in 2009
Multi-part assessment
Goal is to examine risks & exposures unique to each
country
41. Demographics and Political Landscape
Hi, we’re new here!
Understand the country to which you are going
Social and Economic history – informs how you
may interact with regulators, staff, and candidates
Propensity for unrest and how it generally unfolds.
42. Protection of Intellectual Property
Why is our logo on that guy’s test?
To which IP conventions is the country a signatory?
What protections are in place on a national basis?
How do you register your trademark? Is it a first to file
system? Automated or manual?
Are service marks recognized?
Is there institutional capacity to address IP issues?
Judicial resources and corruption should be
considered.
43. Corruption & Crime
Are you gonna lose your shirt and how will you get it back
when you do?
Transparency International Corruption Index – also consider
if the areas of corruption are related to your business.
Anti-corruption legislation and enforcement – is it robust?
Violent street crime in area of your test – does it endanger
candidates and staff?
Do the police have the ability and desire to be helpful?
Is this a location in which you would want to involve the
police?
44. Ability to Investigate
What can you do when you lose your stuff?
Is it safe for your home office staff to travel to the
location for investigation?
Is a visa required? How long to get it?
If home office staff cannot go – what is your plan?
Outside security vendor? Do you have one
identified?
45. First-Hand Experience of other Organizations
Hey, did you guys lose your shirt?
Reach out to other organizations that test in the
location you are evaluating.
Requires building relationships with other
organizations.
This can be invaluable. You compare what your
research reveals to real experiences.
46. Customs
Hey, our stuff doesn’t seem to be getting to our testing
staff!
Understand how to work with the customs officials.
U.S. Department of Commerce can advise and may
intervene if necessary – cultivate this relationship.
International Trade Office (part of Dept. of Commerce).
Talk to your delivery agent – what is their experience in
that country?
Know your timelines.
47. Current Test Centers
Do we really need a test center here?
Proximity of nearest centers in other countries?
Ease of travel for candidates to adjacent countries
– understand that not everyone can go
everywhere.
Does expense of travel keep potential examinees
out of your program?
48. Testing Partners and Locations
So you decided to take the plunge – with whom
are you swimming and in which city?
Is your testing partner in the country? If not, who
is?
Which city (or locations) are best for our test?