SlideShare une entreprise Scribd logo
1  sur  49
Télécharger pour lire hors ligne
International Testing: Crossing Global
Borders While Protecting Your Own


   Steve Addicott
   Vice President, Caveon Test Security

   Tom Gibbs
   Senior Analyst, SHL Group

   Aimee Hobby Rhodes, JD
   Director of Exam Security, CFA Institute
Agenda
 Intros
 Security Threats to International Test Programs
    Steve Addicott, Caveon
 Employment Testing: The Stakes are High, and They’re
  Global, Too
    Tom Gibbs, SHL Group
 Assessing Country Risks
    Aimee Hobby Rhodes, CFA Institute
 Q&A
Steve Addicott
Vice President, Caveon
About Caveon Test Security

 Starting 9th year
 Offering the “Three Ps”
    Proven
    Practical
    Protection
 Certification/Licensure & Education Clientele
    Pre-eminence
    Testing globally
 We’ve seen a few things…
International Test Threats


 Proxy Test Taking
 Braindumps & “Rogue Review” Courses
 Nefarious Test Centers
    See above
    Unusual practices
Nefarious Test Centers
 NOT vendor-owned centers
 Example of Unusual Test Taking
    Test Center Hours 6 am to 6pm
    130 tests taken at odd hours, ie 2am
    Candidates unable to schedule tests when Test Center is
     closed
    WHAT IS GOING ON?
    And, Why are these candidates scoring much higher than
     others?
Braindumps and Rogue Review Courses
 Test Prep Courses advertising & selling live test
  items
 Prevalence of stolen-answer key usage may exceed
  20%
 How are items stolen?
    Infiltrate a top vendor’s premium security, or
    Bribe a remote test center?
    Evidence points to unauthorized decryption of exam after
     downloading to test site servers
Proxies and “Gunmen”

Proxy Test Takers:
 Make Big Money
    Analysis uncovered a large proxy test taking service operating in
     at least six testing sites: USA, Bangladesh, Turkey, UAE, and an
     unknown country.
    500 test instances @ $1000 per instance in 6 months
 Own and operate test centers
 Are branching out
A Gunman for Hire


               September 10, 20xx

               Dear sir:

               We can take XTest exam service for you now in china,
               please fill out the attached file and send to me. We'll finish
               the exam within one week.

               The price now for XTest exam service is USD 1000, you
               can send half payment before exam, half payment after
               exam.

               Best regards,
               Mr. Zhao



9   © Caveon Test Security
Gunman


             September 12, 20xx

             Hello:

             We only accept WESTERN UNION payment, you can send
             half before exam, half after exam. If you do not have a
             passport, you can send me a copy of your driver's ID, you
             can scan it and send to me.

             Please fill out the attached file and send to me when you
             want to start the XTest exam.

             Cheers.
             Mr. Zhao



10   © Caveon Test Security
Gunman
            October 16, 20xx

            My answer for you question:
            1. Testing ID,when we register XTest for you, the test
               system will automatically give you the ID.

            2. After exam, XTest will send you the XTest
            certification directly from USA.

            3. When we finish XTest for you, you can check exam
            result by XTest official site,which can verify your exam
            result.

            For the payment, you can send half before exam, half
            after we finish XTest for you, XTest will send you
            certification within 4-8 weeks. When you want to start
            the exam, please let me know.

            Cheers.
            Mr.Zhao
11    © Caveon Test Security
Gunman



             October 27, 20xx

             Hello:

             Our X test centre is being checked until now, I'm afraid we should wait
             for a little more time to start the XTest. Do you need YTest or ZTest
             exam service? I can prepare for you next week. Please let me know
             your decision, thanks.

             Best regards.
             Mr. Zhao




12   © Caveon Test Security
Gunman

                November 10, 2007

                The payment is OK, thanks.
                Now I send you XTest info: [Link to Site] Test ID* or XTest Certification
                ID: XTest11327691
                Password: 123456

                ***Please keep your personal information updated always. You'd better
                login the XTest site each week. You can modify the password now.

                If you or your friend need YTest, ZTest exam in the future, you can
                contact me, there are XTest certification status in the XTest site,
                please pay attention to it.

                Contact me if you have any question.

                Best regards,
                Mr.Zhao




13   © Caveon Test Security
Tom Gibbs
Senior Analyst, SHL
Overview of presentation
 The user model
    It is 24/7/365 and its online
 Why the stakes are high
    The war for talent and the current economic conditions
 Why the stakes are global
    The war for talent is everywhere
 Are we managing the cheats and the pirates?
    The value of web patrols and a clear security framework
 Summary and further information
    We are publishing more about our security processes
The User Model
24/7/365
 The user model is increasingly not proctored and not
  delivered via group testing
 Delivered online with a variety of security features
    Linear On-the Fly Testing (LOFT) and Computer Adaptive
     Testing (CAT)
    Systems for managing item exposure and answer key exposure
    Supported by a security framework including data forensics,
     web patrols and incident response procedures (see next slide)
 Initial version has been operating since 2002
 Verify has been operating since 2006
 Now available in 26 languages
Test security monitoring
                                                          Security Breach Process
                                                          •Collect Evidence
       Web Patrols                                        •Report to Legal
                                                          •Communications with
                                     Suspected            suspected offender

                                   security breach
                                                          Data Forensics
         Reporting                                        •Run analysis on data specific
(through clients/business/other)                          to test, date range and client
                                                          of suspected breach



                                                     No
 Continue                                                            Item Bank
 monitoring                                                        compromised?



                                                                              Yes
                                                                      Replace
                                                                    compromised
                                                                       items
Further Information...
Further Information...
Burke, E. (2009). Preserving the integrity of online
testing. Industrial and Organizational Psychology, 2, 35-
38.
This is a special edition of I/O Psychology on this topic
and can be accessed via www3.interscience.wiley.com/
journal/118902544/home
Further Information...
The stakes in the employment market have
increased
Quotes from a well known graduate
recruitment portal

“Being the honest idiot in a sea full off successful liars
will get you nowhere”
 “I put sweat, blood and tears into improving my CV and
 am absolutely sickened that lazy, lying cheating
 graduates are getting jobs they have done nothing to
 deserve”
“Have to say that you shouldn't lie about what you can
get caught out on but anything else is fair game. Real
life is about communication, persuasion and
negotiation”
“It's important that you are able to describe in detail
your fake job just in case you are asked about it”
The war for talent is local and
           global

Geography is becoming less of a
         constraint
An applicant could be applying for a job
    located anywhere from anywhere
So, how do we know if we are managing
      the cheats and the pirates?
Web patrols are telling us that there is a
threat but we can detect it and manage it
Web patrols are also telling us that the
applicants know there is security and
    that there are consequences
Internet discussion from April 12th 2010
 ttsstt “With all the psychometric testing for a number of jobs i have applied for
  i am wondering how many people cheat and get help from family/friends while
  doing the test. From a few people i've spoken too I get the sense that it is MORE
  than common …Thanks guys”
 underthepump: “I didn't for two reasons, well two reasons on top of being
  honest :p … 1) They make you do a supervised one at ACs anyway … 2) Time
  limit. You don't exactly have time to sit there and come to a consensus and
  different people work at different paces. Personally having someone else sitting
  there would annoy me more than it would help”
 brendan58: “what underthepump said pretty much. they make you sit a
  supervised test in the AC and it would be quiet a shame to get rejected based on
  fluctuating test results … just need to have the confidence to complete them
  yourself”
 nebasil: “There is no point in cheating, how do you know the person doing the
  test for you will do better than you? I'd rather trust myself than someone else”
 Huynhzie: “lol cheat now, ur going to get screwed in there. there was a PWC guy
  who went to his AC, he absolutely got owned at the verification test there and
  he was pointed and told off by the HR. HR was like "how can u perform
  absolutely so low", SHL is to test ur honesty as well, not just brilliance”
Cheating and piracy is a fact
It is global!
It requires constant vigilance
It requires innovation and commitment
Coming soon …

Lievens, F., and Burke, E. (2011). Dealing with the threats inherent in
unproctored Internet testing of cognitive ability: Results from a large-
scale operational test program. Journal of Occupational and
Organizational Psychology (due for publication this year)
First scientific journal article describing features designed to promote
the security of online high stakes testing
Bartram, D., and Burke, E. (In press). Industrial/Organizational
Testing Case Studies. In Wollack, J. A. and Fremer, J. F. (Eds.).
Handbook of Test Security. New York, NY: Routledge.
This also includes a review of faking good on self-report (typical
performance) measures such as personality questionnaires
Thank You
Aimee Rhodes
Director of Examination Security
         CFA Institute
Due Diligence for International Testing
     CFA Institute
CFA Institute awards the CFA Charter to candidates who successfully
complete 3 levels of testing and meet other education and work-
related criteria.


In 2010, approximately 200,000 candidates registered for CFA Exams
administered in 89 countries across the globe. Exams are
administered twice a year.


All three levels of the CFA Exam are paper and pencil.


CFA Institute also gives the CIPM designation and exam, which is CBT.
Country Risk Assessment
Two Different Assessments:
 Focus on Legal, Regulatory & Tax
 Focus on ability to protect rights and
  intellectual property (including test items) in
  a country. CFA Institute does this prior to
  opening a test center in a new country.
Legal, Regulatory & Tax

 What governmental agencies, if any, must we
  interact with in order to administer our tests in the
  way our program is designed?

 What Taxes do we owe based on our program?

- keep in mind other activities in the country
- inquiries on a “no name” basis
Assessment to Open a New Test Center


Practice established in 2009

Multi-part assessment

Goal is to examine risks & exposures unique to each
 country
Demographics and Political Landscape


 Hi, we’re new here!
 Understand the country to which you are going
 Social and Economic history – informs how you
  may interact with regulators, staff, and candidates
 Propensity for unrest and how it generally unfolds.
Protection of Intellectual Property

 Why is our logo on that guy’s test?
 To which IP conventions is the country a signatory?
 What protections are in place on a national basis?
 How do you register your trademark? Is it a first to file
  system? Automated or manual?
 Are service marks recognized?
 Is there institutional capacity to address IP issues?
  Judicial resources and corruption should be
  considered.
Corruption & Crime

 Are you gonna lose your shirt and how will you get it back
  when you do?
 Transparency International Corruption Index – also consider
  if the areas of corruption are related to your business.
 Anti-corruption legislation and enforcement – is it robust?
 Violent street crime in area of your test – does it endanger
  candidates and staff?
 Do the police have the ability and desire to be helpful?
 Is this a location in which you would want to involve the
  police?
Ability to Investigate

 What can you do when you lose your stuff?
 Is it safe for your home office staff to travel to the
  location for investigation?
 Is a visa required? How long to get it?
 If home office staff cannot go – what is your plan?
  Outside security vendor? Do you have one
  identified?
First-Hand Experience of other Organizations

 Hey, did you guys lose your shirt?
 Reach out to other organizations that test in the
  location you are evaluating.
 Requires building relationships with other
  organizations.
 This can be invaluable. You compare what your
  research reveals to real experiences.
Customs
 Hey, our stuff doesn’t seem to be getting to our testing
  staff!
 Understand how to work with the customs officials.
 U.S. Department of Commerce can advise and may
  intervene if necessary – cultivate this relationship.
 International Trade Office (part of Dept. of Commerce).
 Talk to your delivery agent – what is their experience in
  that country?
 Know your timelines.
Current Test Centers

  Do we really need a test center here?
  Proximity of nearest centers in other countries?
  Ease of travel for candidates to adjacent countries
   – understand that not everyone can go
   everywhere.
  Does expense of travel keep potential examinees
   out of your program?
Testing Partners and Locations

 So you decided to take the plunge – with whom
  are you swimming and in which city?
 Is your testing partner in the country? If not, who
  is?
 Which city (or locations) are best for our test?
Questions?




             Thanks!

     steve.addicott@caveon.com

      tom.gibbs@shlgroup.com

    aimee.rhodes@cfainstitute.org

Contenu connexe

Similaire à Caveon Webinar - International Testing 11-15-2011

Free Report 16 Critical Questions You Must Ask Before Hiring Any IT Company -...
Free Report 16 Critical Questions You Must Ask Before Hiring Any IT Company -...Free Report 16 Critical Questions You Must Ask Before Hiring Any IT Company -...
Free Report 16 Critical Questions You Must Ask Before Hiring Any IT Company -...Ron Pierce
 
Anomaly Detection and You
Anomaly Detection and YouAnomaly Detection and You
Anomaly Detection and YouMary Kelly Rich
 
The Power of Benford's Law in Finding Fraud
The Power of Benford's Law in Finding FraudThe Power of Benford's Law in Finding Fraud
The Power of Benford's Law in Finding FraudFraudBusters
 
Cybersecurity Series - Cyber Defense for Internal Auditors
Cybersecurity Series - Cyber Defense for Internal AuditorsCybersecurity Series - Cyber Defense for Internal Auditors
Cybersecurity Series - Cyber Defense for Internal AuditorsJim Kaplan CIA CFE
 
Next generation pentest your company cannot buy
Next generation pentest your company cannot buyNext generation pentest your company cannot buy
Next generation pentest your company cannot buyVlad Styran
 
Владимир Стыран - Пентест следующего поколения, который ваша компания не може...
Владимир Стыран - Пентест следующего поколения, который ваша компания не може...Владимир Стыран - Пентест следующего поколения, который ваша компания не може...
Владимир Стыран - Пентест следующего поколения, который ваша компания не може...UISGCON
 
NH Bankers 10 08 07 Kamens
NH Bankers 10 08 07 KamensNH Bankers 10 08 07 Kamens
NH Bankers 10 08 07 Kamenskamensm02
 
'Conducing Security Investigations' Webinar 1-17-2012
'Conducing Security Investigations' Webinar 1-17-2012'Conducing Security Investigations' Webinar 1-17-2012
'Conducing Security Investigations' Webinar 1-17-2012SkylerWeisenburger
 
Fantastic Tests - The Crimes of Bad Test Design
Fantastic Tests - The Crimes of Bad Test DesignFantastic Tests - The Crimes of Bad Test Design
Fantastic Tests - The Crimes of Bad Test DesignWinston Laoh
 
How to prepare for your first anti fraud review
How to prepare for your first anti fraud reviewHow to prepare for your first anti fraud review
How to prepare for your first anti fraud reviewJim Kaplan CIA CFE
 
GTB - Protecting PII in the EU
GTB - Protecting PII in the EUGTB - Protecting PII in the EU
GTB - Protecting PII in the EUVCW Security Ltd
 
20141116_Roots of Trust IIC_Nist Version
20141116_Roots of Trust IIC_Nist Version20141116_Roots of Trust IIC_Nist Version
20141116_Roots of Trust IIC_Nist VersionMichael Mossbarger
 
Cybercrime and the Developer Java2Days 2016 Sofia
Cybercrime and the Developer Java2Days 2016 SofiaCybercrime and the Developer Java2Days 2016 Sofia
Cybercrime and the Developer Java2Days 2016 SofiaSteve Poole
 
Security human
Security humanSecurity human
Security humanJisc
 
Vulnerability Assessment and Penetration Testing Framework by Falgun Rathod
Vulnerability Assessment and Penetration Testing Framework by Falgun RathodVulnerability Assessment and Penetration Testing Framework by Falgun Rathod
Vulnerability Assessment and Penetration Testing Framework by Falgun RathodFalgun Rathod
 
Getting Started With WebAuthn
Getting Started With WebAuthnGetting Started With WebAuthn
Getting Started With WebAuthnFIDO Alliance
 

Similaire à Caveon Webinar - International Testing 11-15-2011 (20)

Free Report 16 Critical Questions You Must Ask Before Hiring Any IT Company -...
Free Report 16 Critical Questions You Must Ask Before Hiring Any IT Company -...Free Report 16 Critical Questions You Must Ask Before Hiring Any IT Company -...
Free Report 16 Critical Questions You Must Ask Before Hiring Any IT Company -...
 
Anomaly Detection and You
Anomaly Detection and YouAnomaly Detection and You
Anomaly Detection and You
 
The Power of Benford's Law in Finding Fraud
The Power of Benford's Law in Finding FraudThe Power of Benford's Law in Finding Fraud
The Power of Benford's Law in Finding Fraud
 
Orion Controls Case Essay
Orion Controls Case EssayOrion Controls Case Essay
Orion Controls Case Essay
 
Checkpoint ccsa r76
Checkpoint ccsa r76Checkpoint ccsa r76
Checkpoint ccsa r76
 
Tealeaf
TealeafTealeaf
Tealeaf
 
Cybersecurity Series - Cyber Defense for Internal Auditors
Cybersecurity Series - Cyber Defense for Internal AuditorsCybersecurity Series - Cyber Defense for Internal Auditors
Cybersecurity Series - Cyber Defense for Internal Auditors
 
Next generation pentest your company cannot buy
Next generation pentest your company cannot buyNext generation pentest your company cannot buy
Next generation pentest your company cannot buy
 
Владимир Стыран - Пентест следующего поколения, который ваша компания не може...
Владимир Стыран - Пентест следующего поколения, который ваша компания не може...Владимир Стыран - Пентест следующего поколения, который ваша компания не може...
Владимир Стыран - Пентест следующего поколения, который ваша компания не може...
 
NH Bankers 10 08 07 Kamens
NH Bankers 10 08 07 KamensNH Bankers 10 08 07 Kamens
NH Bankers 10 08 07 Kamens
 
'Conducing Security Investigations' Webinar 1-17-2012
'Conducing Security Investigations' Webinar 1-17-2012'Conducing Security Investigations' Webinar 1-17-2012
'Conducing Security Investigations' Webinar 1-17-2012
 
Fantastic Tests - The Crimes of Bad Test Design
Fantastic Tests - The Crimes of Bad Test DesignFantastic Tests - The Crimes of Bad Test Design
Fantastic Tests - The Crimes of Bad Test Design
 
How to prepare for your first anti fraud review
How to prepare for your first anti fraud reviewHow to prepare for your first anti fraud review
How to prepare for your first anti fraud review
 
GTB - Protecting PII in the EU
GTB - Protecting PII in the EUGTB - Protecting PII in the EU
GTB - Protecting PII in the EU
 
20141116_Roots of Trust IIC_Nist Version
20141116_Roots of Trust IIC_Nist Version20141116_Roots of Trust IIC_Nist Version
20141116_Roots of Trust IIC_Nist Version
 
Cybercrime and the Developer Java2Days 2016 Sofia
Cybercrime and the Developer Java2Days 2016 SofiaCybercrime and the Developer Java2Days 2016 Sofia
Cybercrime and the Developer Java2Days 2016 Sofia
 
Cyberhunter solutions penetration test
Cyberhunter solutions penetration testCyberhunter solutions penetration test
Cyberhunter solutions penetration test
 
Security human
Security humanSecurity human
Security human
 
Vulnerability Assessment and Penetration Testing Framework by Falgun Rathod
Vulnerability Assessment and Penetration Testing Framework by Falgun RathodVulnerability Assessment and Penetration Testing Framework by Falgun Rathod
Vulnerability Assessment and Penetration Testing Framework by Falgun Rathod
 
Getting Started With WebAuthn
Getting Started With WebAuthnGetting Started With WebAuthn
Getting Started With WebAuthn
 

Plus de Caveon Test Security

Unpublished study indicates high chance of fraud in thousands of tests of enem
Unpublished study indicates high chance of fraud in thousands of tests of enemUnpublished study indicates high chance of fraud in thousands of tests of enem
Unpublished study indicates high chance of fraud in thousands of tests of enemCaveon Test Security
 
Caveon webinar series - smart items- using innovative item design to make you...
Caveon webinar series - smart items- using innovative item design to make you...Caveon webinar series - smart items- using innovative item design to make you...
Caveon webinar series - smart items- using innovative item design to make you...Caveon Test Security
 
Caveon webinar series - smart items- using innovative item design to make you...
Caveon webinar series - smart items- using innovative item design to make you...Caveon webinar series - smart items- using innovative item design to make you...
Caveon webinar series - smart items- using innovative item design to make you...Caveon Test Security
 
Caveon Webinar Series - A Guide to Online Protection Strategies - March 28, ...
Caveon Webinar Series -  A Guide to Online Protection Strategies - March 28, ...Caveon Webinar Series -  A Guide to Online Protection Strategies - March 28, ...
Caveon Webinar Series - A Guide to Online Protection Strategies - March 28, ...Caveon Test Security
 
Caveon Webinar Series - Five Things You Can Do Now to Protect Your Assessment...
Caveon Webinar Series - Five Things You Can Do Now to Protect Your Assessment...Caveon Webinar Series - Five Things You Can Do Now to Protect Your Assessment...
Caveon Webinar Series - Five Things You Can Do Now to Protect Your Assessment...Caveon Test Security
 
The Do's and Dont's of Administering High Stakes Tests in Schools Final 121217
The Do's and Dont's of Administering High Stakes Tests in Schools Final 121217The Do's and Dont's of Administering High Stakes Tests in Schools Final 121217
The Do's and Dont's of Administering High Stakes Tests in Schools Final 121217Caveon Test Security
 
Caveon Webinar Series - The Art of Test Security - Know Thy Enemy - November ...
Caveon Webinar Series - The Art of Test Security - Know Thy Enemy - November ...Caveon Webinar Series - The Art of Test Security - Know Thy Enemy - November ...
Caveon Webinar Series - The Art of Test Security - Know Thy Enemy - November ...Caveon Test Security
 
Caveon Webinar Series - Four Steps to Effective Investigations in School Dis...
Caveon Webinar Series -  Four Steps to Effective Investigations in School Dis...Caveon Webinar Series -  Four Steps to Effective Investigations in School Dis...
Caveon Webinar Series - Four Steps to Effective Investigations in School Dis...Caveon Test Security
 
Caveon Webinar Series - On-site Monitoring in Districts 0317
Caveon Webinar Series - On-site Monitoring in Districts 0317Caveon Webinar Series - On-site Monitoring in Districts 0317
Caveon Webinar Series - On-site Monitoring in Districts 0317Caveon Test Security
 
CESP Study Session #1 October 2016
CESP Study Session #1 October 2016CESP Study Session #1 October 2016
CESP Study Session #1 October 2016Caveon Test Security
 
A Tale of Two Cities - School District Webinar #1 Jan 2017
A Tale of Two Cities - School District Webinar  #1 Jan 2017A Tale of Two Cities - School District Webinar  #1 Jan 2017
A Tale of Two Cities - School District Webinar #1 Jan 2017Caveon Test Security
 
Caveon Webinar Series - Discrete Option Multiple Choice: A Revolution in Te...
Caveon Webinar Series  - Discrete Option Multiple Choice:  A Revolution in Te...Caveon Webinar Series  - Discrete Option Multiple Choice:  A Revolution in Te...
Caveon Webinar Series - Discrete Option Multiple Choice: A Revolution in Te...Caveon Test Security
 
Caveon Webinar Series - Test Cheaters Say the Darnedest Things! - 072016
Caveon Webinar Series -  Test Cheaters Say the Darnedest Things! - 072016Caveon Webinar Series -  Test Cheaters Say the Darnedest Things! - 072016
Caveon Webinar Series - Test Cheaters Say the Darnedest Things! - 072016Caveon Test Security
 
Caveon Webinar Series - The Test Security Framework- Why Different Tests Nee...
Caveon Webinar Series -  The Test Security Framework- Why Different Tests Nee...Caveon Webinar Series -  The Test Security Framework- Why Different Tests Nee...
Caveon Webinar Series - The Test Security Framework- Why Different Tests Nee...Caveon Test Security
 
Caveon Webinar Series - Conducting Test Security Investigations in School Di...
Caveon Webinar Series -  Conducting Test Security Investigations in School Di...Caveon Webinar Series -  Conducting Test Security Investigations in School Di...
Caveon Webinar Series - Conducting Test Security Investigations in School Di...Caveon Test Security
 
Caveon Webinar Series - Creating Your Test Security Game Plan - March 2016
Caveon Webinar Series -  Creating Your Test Security Game Plan - March 2016Caveon Webinar Series -  Creating Your Test Security Game Plan - March 2016
Caveon Webinar Series - Creating Your Test Security Game Plan - March 2016Caveon Test Security
 
Caveon Webinar Series - Mastering the US DOE Test Security Requirements Janua...
Caveon Webinar Series - Mastering the US DOE Test Security Requirements Janua...Caveon Webinar Series - Mastering the US DOE Test Security Requirements Janua...
Caveon Webinar Series - Mastering the US DOE Test Security Requirements Janua...Caveon Test Security
 
Caveon Webinar Series - Will the Real Cloned Item Please Stand Up? final
Caveon Webinar Series -  Will the Real Cloned Item Please Stand Up? finalCaveon Webinar Series -  Will the Real Cloned Item Please Stand Up? final
Caveon Webinar Series - Will the Real Cloned Item Please Stand Up? finalCaveon Test Security
 
Caveon Webinar Series - Lessons Learned at the 2015 National Conference on S...
Caveon Webinar Series -  Lessons Learned at the 2015 National Conference on S...Caveon Webinar Series -  Lessons Learned at the 2015 National Conference on S...
Caveon Webinar Series - Lessons Learned at the 2015 National Conference on S...Caveon Test Security
 
Caveon Webinar - Weathering the Perfect Test Security Storm May 2015 edited
Caveon Webinar  - Weathering the Perfect Test Security Storm May 2015 editedCaveon Webinar  - Weathering the Perfect Test Security Storm May 2015 edited
Caveon Webinar - Weathering the Perfect Test Security Storm May 2015 editedCaveon Test Security
 

Plus de Caveon Test Security (20)

Unpublished study indicates high chance of fraud in thousands of tests of enem
Unpublished study indicates high chance of fraud in thousands of tests of enemUnpublished study indicates high chance of fraud in thousands of tests of enem
Unpublished study indicates high chance of fraud in thousands of tests of enem
 
Caveon webinar series - smart items- using innovative item design to make you...
Caveon webinar series - smart items- using innovative item design to make you...Caveon webinar series - smart items- using innovative item design to make you...
Caveon webinar series - smart items- using innovative item design to make you...
 
Caveon webinar series - smart items- using innovative item design to make you...
Caveon webinar series - smart items- using innovative item design to make you...Caveon webinar series - smart items- using innovative item design to make you...
Caveon webinar series - smart items- using innovative item design to make you...
 
Caveon Webinar Series - A Guide to Online Protection Strategies - March 28, ...
Caveon Webinar Series -  A Guide to Online Protection Strategies - March 28, ...Caveon Webinar Series -  A Guide to Online Protection Strategies - March 28, ...
Caveon Webinar Series - A Guide to Online Protection Strategies - March 28, ...
 
Caveon Webinar Series - Five Things You Can Do Now to Protect Your Assessment...
Caveon Webinar Series - Five Things You Can Do Now to Protect Your Assessment...Caveon Webinar Series - Five Things You Can Do Now to Protect Your Assessment...
Caveon Webinar Series - Five Things You Can Do Now to Protect Your Assessment...
 
The Do's and Dont's of Administering High Stakes Tests in Schools Final 121217
The Do's and Dont's of Administering High Stakes Tests in Schools Final 121217The Do's and Dont's of Administering High Stakes Tests in Schools Final 121217
The Do's and Dont's of Administering High Stakes Tests in Schools Final 121217
 
Caveon Webinar Series - The Art of Test Security - Know Thy Enemy - November ...
Caveon Webinar Series - The Art of Test Security - Know Thy Enemy - November ...Caveon Webinar Series - The Art of Test Security - Know Thy Enemy - November ...
Caveon Webinar Series - The Art of Test Security - Know Thy Enemy - November ...
 
Caveon Webinar Series - Four Steps to Effective Investigations in School Dis...
Caveon Webinar Series -  Four Steps to Effective Investigations in School Dis...Caveon Webinar Series -  Four Steps to Effective Investigations in School Dis...
Caveon Webinar Series - Four Steps to Effective Investigations in School Dis...
 
Caveon Webinar Series - On-site Monitoring in Districts 0317
Caveon Webinar Series - On-site Monitoring in Districts 0317Caveon Webinar Series - On-site Monitoring in Districts 0317
Caveon Webinar Series - On-site Monitoring in Districts 0317
 
CESP Study Session #1 October 2016
CESP Study Session #1 October 2016CESP Study Session #1 October 2016
CESP Study Session #1 October 2016
 
A Tale of Two Cities - School District Webinar #1 Jan 2017
A Tale of Two Cities - School District Webinar  #1 Jan 2017A Tale of Two Cities - School District Webinar  #1 Jan 2017
A Tale of Two Cities - School District Webinar #1 Jan 2017
 
Caveon Webinar Series - Discrete Option Multiple Choice: A Revolution in Te...
Caveon Webinar Series  - Discrete Option Multiple Choice:  A Revolution in Te...Caveon Webinar Series  - Discrete Option Multiple Choice:  A Revolution in Te...
Caveon Webinar Series - Discrete Option Multiple Choice: A Revolution in Te...
 
Caveon Webinar Series - Test Cheaters Say the Darnedest Things! - 072016
Caveon Webinar Series -  Test Cheaters Say the Darnedest Things! - 072016Caveon Webinar Series -  Test Cheaters Say the Darnedest Things! - 072016
Caveon Webinar Series - Test Cheaters Say the Darnedest Things! - 072016
 
Caveon Webinar Series - The Test Security Framework- Why Different Tests Nee...
Caveon Webinar Series -  The Test Security Framework- Why Different Tests Nee...Caveon Webinar Series -  The Test Security Framework- Why Different Tests Nee...
Caveon Webinar Series - The Test Security Framework- Why Different Tests Nee...
 
Caveon Webinar Series - Conducting Test Security Investigations in School Di...
Caveon Webinar Series -  Conducting Test Security Investigations in School Di...Caveon Webinar Series -  Conducting Test Security Investigations in School Di...
Caveon Webinar Series - Conducting Test Security Investigations in School Di...
 
Caveon Webinar Series - Creating Your Test Security Game Plan - March 2016
Caveon Webinar Series -  Creating Your Test Security Game Plan - March 2016Caveon Webinar Series -  Creating Your Test Security Game Plan - March 2016
Caveon Webinar Series - Creating Your Test Security Game Plan - March 2016
 
Caveon Webinar Series - Mastering the US DOE Test Security Requirements Janua...
Caveon Webinar Series - Mastering the US DOE Test Security Requirements Janua...Caveon Webinar Series - Mastering the US DOE Test Security Requirements Janua...
Caveon Webinar Series - Mastering the US DOE Test Security Requirements Janua...
 
Caveon Webinar Series - Will the Real Cloned Item Please Stand Up? final
Caveon Webinar Series -  Will the Real Cloned Item Please Stand Up? finalCaveon Webinar Series -  Will the Real Cloned Item Please Stand Up? final
Caveon Webinar Series - Will the Real Cloned Item Please Stand Up? final
 
Caveon Webinar Series - Lessons Learned at the 2015 National Conference on S...
Caveon Webinar Series -  Lessons Learned at the 2015 National Conference on S...Caveon Webinar Series -  Lessons Learned at the 2015 National Conference on S...
Caveon Webinar Series - Lessons Learned at the 2015 National Conference on S...
 
Caveon Webinar - Weathering the Perfect Test Security Storm May 2015 edited
Caveon Webinar  - Weathering the Perfect Test Security Storm May 2015 editedCaveon Webinar  - Weathering the Perfect Test Security Storm May 2015 edited
Caveon Webinar - Weathering the Perfect Test Security Storm May 2015 edited
 

Caveon Webinar - International Testing 11-15-2011

  • 1. International Testing: Crossing Global Borders While Protecting Your Own Steve Addicott Vice President, Caveon Test Security Tom Gibbs Senior Analyst, SHL Group Aimee Hobby Rhodes, JD Director of Exam Security, CFA Institute
  • 2. Agenda  Intros  Security Threats to International Test Programs  Steve Addicott, Caveon  Employment Testing: The Stakes are High, and They’re Global, Too  Tom Gibbs, SHL Group  Assessing Country Risks  Aimee Hobby Rhodes, CFA Institute  Q&A
  • 4. About Caveon Test Security  Starting 9th year  Offering the “Three Ps”  Proven  Practical  Protection  Certification/Licensure & Education Clientele  Pre-eminence  Testing globally  We’ve seen a few things…
  • 5. International Test Threats  Proxy Test Taking  Braindumps & “Rogue Review” Courses  Nefarious Test Centers  See above  Unusual practices
  • 6. Nefarious Test Centers  NOT vendor-owned centers  Example of Unusual Test Taking  Test Center Hours 6 am to 6pm  130 tests taken at odd hours, ie 2am  Candidates unable to schedule tests when Test Center is closed  WHAT IS GOING ON?  And, Why are these candidates scoring much higher than others?
  • 7. Braindumps and Rogue Review Courses  Test Prep Courses advertising & selling live test items  Prevalence of stolen-answer key usage may exceed 20%  How are items stolen?  Infiltrate a top vendor’s premium security, or  Bribe a remote test center?  Evidence points to unauthorized decryption of exam after downloading to test site servers
  • 8. Proxies and “Gunmen” Proxy Test Takers:  Make Big Money  Analysis uncovered a large proxy test taking service operating in at least six testing sites: USA, Bangladesh, Turkey, UAE, and an unknown country.  500 test instances @ $1000 per instance in 6 months  Own and operate test centers  Are branching out
  • 9. A Gunman for Hire September 10, 20xx Dear sir: We can take XTest exam service for you now in china, please fill out the attached file and send to me. We'll finish the exam within one week. The price now for XTest exam service is USD 1000, you can send half payment before exam, half payment after exam. Best regards, Mr. Zhao 9 © Caveon Test Security
  • 10. Gunman September 12, 20xx Hello: We only accept WESTERN UNION payment, you can send half before exam, half after exam. If you do not have a passport, you can send me a copy of your driver's ID, you can scan it and send to me. Please fill out the attached file and send to me when you want to start the XTest exam. Cheers. Mr. Zhao 10 © Caveon Test Security
  • 11. Gunman October 16, 20xx My answer for you question: 1. Testing ID,when we register XTest for you, the test system will automatically give you the ID. 2. After exam, XTest will send you the XTest certification directly from USA. 3. When we finish XTest for you, you can check exam result by XTest official site,which can verify your exam result. For the payment, you can send half before exam, half after we finish XTest for you, XTest will send you certification within 4-8 weeks. When you want to start the exam, please let me know. Cheers. Mr.Zhao 11 © Caveon Test Security
  • 12. Gunman October 27, 20xx Hello: Our X test centre is being checked until now, I'm afraid we should wait for a little more time to start the XTest. Do you need YTest or ZTest exam service? I can prepare for you next week. Please let me know your decision, thanks. Best regards. Mr. Zhao 12 © Caveon Test Security
  • 13. Gunman November 10, 2007 The payment is OK, thanks. Now I send you XTest info: [Link to Site] Test ID* or XTest Certification ID: XTest11327691 Password: 123456 ***Please keep your personal information updated always. You'd better login the XTest site each week. You can modify the password now. If you or your friend need YTest, ZTest exam in the future, you can contact me, there are XTest certification status in the XTest site, please pay attention to it. Contact me if you have any question. Best regards, Mr.Zhao 13 © Caveon Test Security
  • 15. Overview of presentation  The user model  It is 24/7/365 and its online  Why the stakes are high  The war for talent and the current economic conditions  Why the stakes are global  The war for talent is everywhere  Are we managing the cheats and the pirates?  The value of web patrols and a clear security framework  Summary and further information  We are publishing more about our security processes
  • 17. 24/7/365  The user model is increasingly not proctored and not delivered via group testing  Delivered online with a variety of security features  Linear On-the Fly Testing (LOFT) and Computer Adaptive Testing (CAT)  Systems for managing item exposure and answer key exposure  Supported by a security framework including data forensics, web patrols and incident response procedures (see next slide)  Initial version has been operating since 2002  Verify has been operating since 2006  Now available in 26 languages
  • 18. Test security monitoring Security Breach Process •Collect Evidence Web Patrols •Report to Legal •Communications with Suspected suspected offender security breach Data Forensics Reporting •Run analysis on data specific (through clients/business/other) to test, date range and client of suspected breach No Continue Item Bank monitoring compromised? Yes Replace compromised items
  • 20. Further Information... Burke, E. (2009). Preserving the integrity of online testing. Industrial and Organizational Psychology, 2, 35- 38. This is a special edition of I/O Psychology on this topic and can be accessed via www3.interscience.wiley.com/ journal/118902544/home
  • 22. The stakes in the employment market have increased
  • 23. Quotes from a well known graduate recruitment portal “Being the honest idiot in a sea full off successful liars will get you nowhere” “I put sweat, blood and tears into improving my CV and am absolutely sickened that lazy, lying cheating graduates are getting jobs they have done nothing to deserve” “Have to say that you shouldn't lie about what you can get caught out on but anything else is fair game. Real life is about communication, persuasion and negotiation” “It's important that you are able to describe in detail your fake job just in case you are asked about it”
  • 24. The war for talent is local and global Geography is becoming less of a constraint
  • 25. An applicant could be applying for a job located anywhere from anywhere
  • 26. So, how do we know if we are managing the cheats and the pirates?
  • 27. Web patrols are telling us that there is a threat but we can detect it and manage it
  • 28. Web patrols are also telling us that the applicants know there is security and that there are consequences
  • 29. Internet discussion from April 12th 2010  ttsstt “With all the psychometric testing for a number of jobs i have applied for i am wondering how many people cheat and get help from family/friends while doing the test. From a few people i've spoken too I get the sense that it is MORE than common …Thanks guys”  underthepump: “I didn't for two reasons, well two reasons on top of being honest :p … 1) They make you do a supervised one at ACs anyway … 2) Time limit. You don't exactly have time to sit there and come to a consensus and different people work at different paces. Personally having someone else sitting there would annoy me more than it would help”  brendan58: “what underthepump said pretty much. they make you sit a supervised test in the AC and it would be quiet a shame to get rejected based on fluctuating test results … just need to have the confidence to complete them yourself”  nebasil: “There is no point in cheating, how do you know the person doing the test for you will do better than you? I'd rather trust myself than someone else”  Huynhzie: “lol cheat now, ur going to get screwed in there. there was a PWC guy who went to his AC, he absolutely got owned at the verification test there and he was pointed and told off by the HR. HR was like "how can u perform absolutely so low", SHL is to test ur honesty as well, not just brilliance”
  • 30. Cheating and piracy is a fact
  • 32. It requires constant vigilance
  • 33. It requires innovation and commitment
  • 34. Coming soon … Lievens, F., and Burke, E. (2011). Dealing with the threats inherent in unproctored Internet testing of cognitive ability: Results from a large- scale operational test program. Journal of Occupational and Organizational Psychology (due for publication this year) First scientific journal article describing features designed to promote the security of online high stakes testing Bartram, D., and Burke, E. (In press). Industrial/Organizational Testing Case Studies. In Wollack, J. A. and Fremer, J. F. (Eds.). Handbook of Test Security. New York, NY: Routledge. This also includes a review of faking good on self-report (typical performance) measures such as personality questionnaires
  • 36. Aimee Rhodes Director of Examination Security CFA Institute
  • 37. Due Diligence for International Testing CFA Institute CFA Institute awards the CFA Charter to candidates who successfully complete 3 levels of testing and meet other education and work- related criteria. In 2010, approximately 200,000 candidates registered for CFA Exams administered in 89 countries across the globe. Exams are administered twice a year. All three levels of the CFA Exam are paper and pencil. CFA Institute also gives the CIPM designation and exam, which is CBT.
  • 38. Country Risk Assessment Two Different Assessments:  Focus on Legal, Regulatory & Tax  Focus on ability to protect rights and intellectual property (including test items) in a country. CFA Institute does this prior to opening a test center in a new country.
  • 39. Legal, Regulatory & Tax  What governmental agencies, if any, must we interact with in order to administer our tests in the way our program is designed?  What Taxes do we owe based on our program? - keep in mind other activities in the country - inquiries on a “no name” basis
  • 40. Assessment to Open a New Test Center Practice established in 2009 Multi-part assessment Goal is to examine risks & exposures unique to each country
  • 41. Demographics and Political Landscape  Hi, we’re new here!  Understand the country to which you are going  Social and Economic history – informs how you may interact with regulators, staff, and candidates  Propensity for unrest and how it generally unfolds.
  • 42. Protection of Intellectual Property  Why is our logo on that guy’s test?  To which IP conventions is the country a signatory?  What protections are in place on a national basis?  How do you register your trademark? Is it a first to file system? Automated or manual?  Are service marks recognized?  Is there institutional capacity to address IP issues? Judicial resources and corruption should be considered.
  • 43. Corruption & Crime  Are you gonna lose your shirt and how will you get it back when you do?  Transparency International Corruption Index – also consider if the areas of corruption are related to your business.  Anti-corruption legislation and enforcement – is it robust?  Violent street crime in area of your test – does it endanger candidates and staff?  Do the police have the ability and desire to be helpful?  Is this a location in which you would want to involve the police?
  • 44. Ability to Investigate  What can you do when you lose your stuff?  Is it safe for your home office staff to travel to the location for investigation?  Is a visa required? How long to get it?  If home office staff cannot go – what is your plan? Outside security vendor? Do you have one identified?
  • 45. First-Hand Experience of other Organizations  Hey, did you guys lose your shirt?  Reach out to other organizations that test in the location you are evaluating.  Requires building relationships with other organizations.  This can be invaluable. You compare what your research reveals to real experiences.
  • 46. Customs  Hey, our stuff doesn’t seem to be getting to our testing staff!  Understand how to work with the customs officials.  U.S. Department of Commerce can advise and may intervene if necessary – cultivate this relationship.  International Trade Office (part of Dept. of Commerce).  Talk to your delivery agent – what is their experience in that country?  Know your timelines.
  • 47. Current Test Centers  Do we really need a test center here?  Proximity of nearest centers in other countries?  Ease of travel for candidates to adjacent countries – understand that not everyone can go everywhere.  Does expense of travel keep potential examinees out of your program?
  • 48. Testing Partners and Locations  So you decided to take the plunge – with whom are you swimming and in which city?  Is your testing partner in the country? If not, who is?  Which city (or locations) are best for our test?
  • 49. Questions? Thanks! steve.addicott@caveon.com tom.gibbs@shlgroup.com aimee.rhodes@cfainstitute.org