everything you need to implement a data forensics program
Post on 05-Jul-2015
228 Views
Preview:
DESCRIPTION
TRANSCRIPT
―Everything You Need to Know to
Implement a Data Forensics Program‖
Dennis Maynes - Chief Scientist, Caveon Test Security
Jennifer Ancona Semko - Partner, Baker & McKenzie
Kerri Davis - Anti-Piracy Program Manager, Microsoft Learning
Presented September 27th, 2012
Presenters:
Agenda
Introduction to Data Forensics
Program Implementation
Obtaining Budget and Support
Legal Foundation of Data Forensics and
Navigating Legal Issues
Managing Investigations
002
INTRODUCTION TO DATA
FORENSICS
003
Presented by: Dennis Maynes
Introduction
• Purpose of a data forensics program
– Measure and manage security risks
– Ensure fair and valid testing
– Use statistics to monitor and to investigate
• Purpose of security initiatives
– Mitigate losses and liability
• Illustrations of loss
– November 2007, Denver, de-icers
– July 2011, Atlanta, 178 educators
004
Data Forensics
• Science of examining data to find potential security risks
• There are clues in the data relating to:
– Collusion
– Use of recalled questions
– Rogue review courses
– Testing sites with poor security
– Exams and items that have become exposed
• ―We balance probabilities and choose the most likely. It is the scientific use of the imagination.‖ – Sherlock Holmes, The Hound of the Baskervilles
005
Examples of Statistics
• Aberrance or person-fit – (pre-knowledge)
• Similarity – (collusion)
• Erasures – (tampering)
• Gains – (pre-knowledge)
• Shared e-mails – (improper coordination)
• Foreign tests – (extra ―help‖)
• Response time – (braindumps)
• Score differences – (pre-knowledge)
006
Test Security Threat Scale
Statistical Anomalies
Testing
Irregularities
Security
Violations
Security
Breaches
Test Fraud
008
Test Fraud Taxonomy
• Content Theft and/or Disclosure
• Collusion and/or Providing information during the
exam
• Violation of Proctoring and/or Administration Rules
• Tampering and/or Manipulating the Score
Distribution
• Based on Amrein-Beardsley, A., Berliner, D. C. &
Rideau, S. (2010). Cheating in the first, second,
and third degree: Educators' responses to high-
stakes testing.
009
Data Forensics Uses
• Two modes
– Monitor for security breaches
– Investigate potential breaches
• Inform investigations
• Take corrective actions
– Score invalidations
– Test site closures
– Replace test items
• Manage security health
• Monitor security risk levels
010
Data Forensics Monitoring
• Examine ALL of the data
• Must correct for multiple comparisons
– Bonferroni Correction
– Probability for threshold is /n
– Example: if n = 10,000 and is .05 – use .0000005
• Probabilities allow
– Objective measures
– Ensemble statistics
– Error rate control
011
Data Forensics Inference
• To invalidate scores, most psychometricians require
– An eye-witness account and
– Probability less than one in ten thousand.
• Hypothetical question: Suppose two individuals
submitted identical 500 word essays – would you act?
• Basis for action
– Strength of the evidence
– Is the score trustworthy?
If you accept statistics to determine candidate competence, why
would you reject statistics to determine score trustworthiness?
012
Circumstantial Evidence
• Requires an inference or deduction
• Seek ―disconfirming‖ or plausible explanations
• Value of multiple pieces of evidence
• Collect and document all the evidence
• Apply policy consistently for each case
―Circumstantial evidence is a very tricky thing. It may seem to point very
straight to one thing, but if you shift your own point of view a little, you
may find it pointing in an equally uncompromising manner to something
entirely different.‖ – Sherlock Holmes, The Boscombe Valley Mystery
013
Questions so far?
PROGRAM IMPLEMENTATION
014
Presented by: Kerri Davis
Test Security is a Process
Data Forensics can be used to improve the process
Assess overall test security risks
Identify strengths and weaknesses in the process
Document the test security process
Take steps to improve the test security process
015
The goal of a Data Forensics program should be the improvement
of test security. The primary purpose of Data Forensics is NOT to
apprehend and punish potential cheaters.
General implementation approach
• Establish policy and precedent: scope of your
program, budget, legal, planned outcomes
• Obtain stake-holder support
• Create infrastructure
• Create agreements
• Reports
• Revise exam policies
• Implementation
– Conduct pilots and dry runs
– Train staff
– Perform data forensics analyses
– Review and revise
016
Stakeholder support is critical
• Explain how the statistical analysis works
• Present results from data forensics analyses
– How many test takers were cheating?
– How many locations had weak security?
• Outline overview of cheating evidence
• Explain work flow
• Outline costs associated with enforcement
• Address individual questions/concerns
017
Summary
• Think of a 3-tiered approach to your data forensics
implementation:
1. Establish scope
2. Propose a budget
3. Create an action plan
• Use this approach to formulate your program and
to gain stakeholder support
• Don’t be afraid to revise as you go; using data
forensics is a process itself.
OBTAINING BUDGET
AND SUPPORT
020
It’s not ―IF‖… it’s ―WHEN‖
• Don’t wait for a breach to occur before you seriously
think about security.
• Don’t be in a position of explaining (to your
stakeholders, the public, or the press) why you are not
protecting the integrity of your exams.
021
Successful implementation of a Data Forensics program will
anticipate inquiries by the media and the public in order to
communicate that the program is pursuing a proper course for
ensuring the tests are administered fairly and securely.
Assemble the Security Team
• Identify key personnel from the affected departments:
– Exam Development
– Psychometrics
– Exam Administration
– Legal
– Risk Management
– Scoring/Grading
– Professional Conduct
022
Make the Case for Security
• Exam scores mean nothing if candidates can gain an
unfair advantage by cheating.
• Share highly-publicized examples of individuals gaining
an unfair advantage
– Prevention: Avoid being a news story
• Present the Cost of Security vs. Insecurity
– Cost of Development
– Reputational Harm
– Threat to the Public
023
Plan for Successful Implementation
• Propose a security budget
• Identify key individuals, their roles, and their time
commitment
• Develop and document process flows
• Anticipate and overcome obstacles
– Fear of statistics
• Don’t understand them
• Don’t understand how to use them
– Fear of what people might think
– Lack of familiarity with score review process
024
Questions at this point?
LEGAL FOUNDATION OF
DATA FORENSICS AND
NAVIGATING LEGAL ISSUES
028
Presented by: Jennifer Ancona Semko
Why is this important?
029
The First Brick:
the Candidate Agreement
• Contract: An agreement between two or more persons which creates an obligation to do or not to do a particular thing. A legal relationship consisting of the rights and duties of the contracting parties. Black’s Law Dictionary, Sixth Edition
• Your agreement with test takers defines the relationship
• Memorializes your (and their) rights and obligations
• If done properly, makes expectations (and remedies) clear
030
What does your agreement say?
• Are candidates on notice that sharing items is a breach?
• Are candidates on notice that studying from recalled items is improper?
• Did you reserve the right to invalidate scores? Suspend or permanently ban access to the examination? To take other action?
• Are candidates on notice of the possible use of data forensics?
• What are the grounds for action? Is there a ―catch all‖?
• Do you regularly review your agreement language?
• Do you have uniform security procedures and policies in place?
• Are candidates required to cooperate in investigations?
031
Successfully Using Data Forensics
•Can you defend your actions?
–Do you have to prove ―cheating‖?
•Contract law – ―good faith‖
–Language of agreements
–Documented policies & procedures
–Taking all steps to show ―good faith‖
•Will you need expert testimony?
•General deference to exam programs
–State actors: due process
032
Admissibility of Expert Evidence in Court
Federal Rule of Evidence 702, Testimony by Experts
If scientific, technical, or other specialized knowledge will assist the trier of fact to understand the evidence or to determine a fact in issue, a witness qualified as an expert by knowledge, skill, experience, training, or education, may testify thereto in the form of an opinion or otherwise, if:
(1) the testimony is based upon scientific facts or data,
(2) the testimony is the product of reliable principles and methods, and
(3) the witness has applied the principles and methods reliably to the facts of the case.
033
Admissibility of Expert Evidence in Court
• Daubert v. Merrell Dow Pharmaceuticals, Inc., 509 U.S.
579 (1993)
– ―[U]nder the Rules [of Evidence] the trial judge must
ensure that any and all scientific testimony or evidence
admitted is not only relevant, but reliable….‖
• Kumho Tire Co. v. Carmichael, 526 U.S. 137 (1999)
– ―The objective of [the trial court’s gatekeeper] requirement
is to ensure the reliability and relevancy of expert
testimony. It is to make certain that an expert … employs
in the courtroom the same level of intellectual rigor that
characterizes the practice of an expert in the relevant
field.‖
Deference . . . within limits
Deference to Exam Programs
Murray v. ETS, 170 F.3d 514 (5th Cir. 1999) (SATExam)
• Louisiana basketball player; needed 820 on SAT• Scored 700, then 1300
– Similarity to nearby student (3 in 100 million odds)– Scored 800 on retake
• ―ETS’s contract with Murray clearly and explicitlyreserved to ETS the right to withhold any scoresETS had reason to believe were not valid. Theonly contractual duty ETS owed to Murray was toinvestigate the validity of Murray’s scores in goodfaith.‖
034
Deference to Exam Programs
Langston v. ACT, 890 F.2d 380 (11th Cir. 1989) (ACTExam)
• Alabama football player; scored 10 on ACT; then 20
• Inconsistent with GPA; unusual similarity to nearbystudent
• ―Under the governing law, the outcome of plaintiff’scase does not turn on whether or not plaintiff cheatedon his exam, but only on whether or not ACT carriedout its contractual obligations in good faith.‖
035
MANAGING TEST SECURITY
INVESTIGATIONS
025
Managing Investigations
• Do your investigators have all they need to be effective?
– Corporate support (budget, effective legal counsel, training)
– Software/services
– Support from other departments (Psychometrics, Test
Development, Registration & Credentialing, etc.)
• What procedure is in place to select cases for
investigation?
– Is it prudent/efficient to investigate all matters?
• What metrics exist to determine the success/efficiency of
an investigation?
– Did the investigation glean the desired information?
– When does an organization ―close‖ an investigation?
027
Managing Investigations
• Who within (or outside) your organization conducts
investigations? When?
– May depend on investigation type: exam-day incidents,
ongoing copyright infringement, collusion, proxy testing
• At what point do you involve legal counsel, board
members, or other departments?
• Do your policies and procedures reflect what is
needed to manage investigations?
– Are candidates obligated to cooperate?
• How are results reported? Who makes sanctions
decisions?
Gather Evidence
• Similarity analysis, gains analysis, other statistics
• Reports of security incidents
• Seating charts and chain of custody of materials
• Review test taker associations and connections
• Review access logs to secured exam content
• Review score histories of test takers and locations
• Review test booklets for signs of ―work‖
• Responses by test center staff & test takers
• Adherence to security policies
018
Evaluate Evidence
• Do alternative explanations exist?
• Are candidate explanations/responses convincing?
• Could test fraud have occurred?
• Are the test results trustworthy?
– Evaluation depends upon trustworthiness of the scores, NOT
an inference of behavior.
019
The proper use of Data Forensics is to certify the trustworthiness of the test
results and the integrity of the test administration. It is NOT proper to use
these results to place a label, such as ―cheater,‖ on an individual.
Wrap up & key takeaways
• Data Forensics
– Measure and manage risks
– Ensure fair and valid testing
– Mitigate losses and liability
• Implementation
– Security is a process, not a state
– Policy—what will we do with the results?
– Breaking down organizational ―fiefdoms‖
• Support
– Not ―if‖, but ―when‖
– Nobody plans to fail, but…
A long and winding road….Key Takeaways, cont.
• Legal
– Agreement is your foundation
• Even in K-12!
– Don’t be ―arbitrary and capricious‖
– Consistent and uniform
• Investigations
– ―Go/No Go‖ decision criteria
• Constrained resources
– Focus on the results, not the behaviors
• Statistically ―Indeterminate results‖ vs ―You’re a cheater!‖
Got questions?
kerrid@microsoft.com
jennifer.semko@bakermckenzie.com
dennis.maynes@caveon.com Caveon Confidential.
Do not share without permission.
Hope to see you soon…• CCSSO TILSA SCASS
– Oct 1-5
– Indianapolis
• Next month’s webinar
– ―Do It Yourself Security Audits and Security Investigations‖
– Tuesday, Oct.16, noon EDT
• ICE (Institute for Credentialing Excellence)
– Nov 6-9
– Palm Springs
Want more?
LinkedIn group ―Caveon Test Security‖
twitter @caveon for updates, news, connect
Blogs!
Caveon Security Insights – www.caveon.com/blog
Cheating In The News – www.caveon.com/citn
www.caveon.com/resources/webinars - to see past sessions
Contact skyler.weisenburger@caveon.com for slides,
comments, and questions about this session
top related