―Everything You Need to Know to

Implement a Data Forensics Program‖

Dennis Maynes - Chief Scientist, Caveon Test Security

Jennifer Ancona Semko - Partner, Baker & McKenzie

Kerri Davis - Anti-Piracy Program Manager, Microsoft Learning

Presented September 27th, 2012

Presenters:

Agenda

Introduction to Data Forensics

Program Implementation

Obtaining Budget and Support

Legal Foundation of Data Forensics and

Navigating Legal Issues

Managing Investigations

002

INTRODUCTION TO DATA

FORENSICS

003

Presented by: Dennis Maynes

Introduction

• Purpose of a data forensics program

– Measure and manage security risks

– Ensure fair and valid testing

– Use statistics to monitor and to investigate

• Purpose of security initiatives

– Mitigate losses and liability

• Illustrations of loss

– November 2007, Denver, de-icers

– July 2011, Atlanta, 178 educators

004

Data Forensics

• Science of examining data to find potential security risks

• There are clues in the data relating to:

– Collusion

– Use of recalled questions

– Rogue review courses

– Testing sites with poor security

– Exams and items that have become exposed

• ―We balance probabilities and choose the most likely. It is the scientific use of the imagination.‖ – Sherlock Holmes, The Hound of the Baskervilles

005

Examples of Statistics

• Aberrance or person-fit – (pre-knowledge)

• Similarity – (collusion)

• Erasures – (tampering)

• Gains – (pre-knowledge)

• Shared e-mails – (improper coordination)

• Foreign tests – (extra ―help‖)

• Response time – (braindumps)

• Score differences – (pre-knowledge)

006

Test Security Threat Scale

Statistical Anomalies

Testing

Irregularities

Security

Violations

Security

Breaches

Test Fraud

008

Test Fraud Taxonomy

• Content Theft and/or Disclosure

• Collusion and/or Providing information during the

exam

• Violation of Proctoring and/or Administration Rules

• Tampering and/or Manipulating the Score

Distribution

• Based on Amrein-Beardsley, A., Berliner, D. C. &

Rideau, S. (2010). Cheating in the first, second,

and third degree: Educators' responses to high-

stakes testing.

009

Data Forensics Uses

• Two modes

– Monitor for security breaches

– Investigate potential breaches

• Inform investigations

• Take corrective actions

– Score invalidations

– Test site closures

– Replace test items

• Manage security health

• Monitor security risk levels

010

Data Forensics Monitoring

• Examine ALL of the data

• Must correct for multiple comparisons

– Bonferroni Correction

– Probability for threshold is /n

– Example: if n = 10,000 and is .05 – use .0000005

• Probabilities allow

– Objective measures

– Ensemble statistics

– Error rate control

011

Data Forensics Inference

• To invalidate scores, most psychometricians require

– An eye-witness account and

– Probability less than one in ten thousand.

• Hypothetical question: Suppose two individuals

submitted identical 500 word essays – would you act?

• Basis for action

– Strength of the evidence

– Is the score trustworthy?

If you accept statistics to determine candidate competence, why

would you reject statistics to determine score trustworthiness?

012

Circumstantial Evidence

• Requires an inference or deduction

• Seek ―disconfirming‖ or plausible explanations

• Value of multiple pieces of evidence

• Collect and document all the evidence

• Apply policy consistently for each case

―Circumstantial evidence is a very tricky thing. It may seem to point very

straight to one thing, but if you shift your own point of view a little, you

may find it pointing in an equally uncompromising manner to something

entirely different.‖ – Sherlock Holmes, The Boscombe Valley Mystery

013

Questions so far?

PROGRAM IMPLEMENTATION

014

Presented by: Kerri Davis

Test Security is a Process

Data Forensics can be used to improve the process

Assess overall test security risks

Identify strengths and weaknesses in the process

Document the test security process

Take steps to improve the test security process

015

The goal of a Data Forensics program should be the improvement

of test security. The primary purpose of Data Forensics is NOT to

apprehend and punish potential cheaters.

General implementation approach

• Establish policy and precedent: scope of your

program, budget, legal, planned outcomes

• Obtain stake-holder support

• Create infrastructure

• Create agreements

• Reports

• Revise exam policies

• Implementation

– Conduct pilots and dry runs

– Train staff

– Perform data forensics analyses

– Review and revise

016

Stakeholder support is critical

• Explain how the statistical analysis works

• Present results from data forensics analyses

– How many test takers were cheating?

– How many locations had weak security?

• Outline overview of cheating evidence

• Explain work flow

• Outline costs associated with enforcement

• Address individual questions/concerns

017

Summary

• Think of a 3-tiered approach to your data forensics

implementation:

1. Establish scope

2. Propose a budget

3. Create an action plan

• Use this approach to formulate your program and

to gain stakeholder support

• Don’t be afraid to revise as you go; using data

forensics is a process itself.

OBTAINING BUDGET

AND SUPPORT

020

It’s not ―IF‖… it’s ―WHEN‖

• Don’t wait for a breach to occur before you seriously

think about security.

• Don’t be in a position of explaining (to your

stakeholders, the public, or the press) why you are not

protecting the integrity of your exams.

021

Successful implementation of a Data Forensics program will

anticipate inquiries by the media and the public in order to

communicate that the program is pursuing a proper course for

ensuring the tests are administered fairly and securely.

Assemble the Security Team

• Identify key personnel from the affected departments:

– Exam Development

– Psychometrics

– Exam Administration

– Legal

– Risk Management

– Scoring/Grading

– Professional Conduct

022

Make the Case for Security

• Exam scores mean nothing if candidates can gain an

unfair advantage by cheating.

• Share highly-publicized examples of individuals gaining

an unfair advantage

– Prevention: Avoid being a news story

• Present the Cost of Security vs. Insecurity

– Cost of Development

– Reputational Harm

– Threat to the Public

023

Plan for Successful Implementation

• Propose a security budget

• Identify key individuals, their roles, and their time

commitment

• Develop and document process flows

• Anticipate and overcome obstacles

– Fear of statistics

• Don’t understand them

• Don’t understand how to use them

– Fear of what people might think

– Lack of familiarity with score review process

024

Questions at this point?

LEGAL FOUNDATION OF

DATA FORENSICS AND

NAVIGATING LEGAL ISSUES

028

Presented by: Jennifer Ancona Semko

Why is this important?

029

The First Brick:

the Candidate Agreement

• Contract: An agreement between two or more persons which creates an obligation to do or not to do a particular thing. A legal relationship consisting of the rights and duties of the contracting parties. Black’s Law Dictionary, Sixth Edition

• Your agreement with test takers defines the relationship

• Memorializes your (and their) rights and obligations

• If done properly, makes expectations (and remedies) clear

030

What does your agreement say?

• Are candidates on notice that sharing items is a breach?

• Are candidates on notice that studying from recalled items is improper?

• Did you reserve the right to invalidate scores? Suspend or permanently ban access to the examination? To take other action?

• Are candidates on notice of the possible use of data forensics?

• What are the grounds for action? Is there a ―catch all‖?

• Do you regularly review your agreement language?

• Do you have uniform security procedures and policies in place?

• Are candidates required to cooperate in investigations?

031

Successfully Using Data Forensics

•Can you defend your actions?

–Do you have to prove ―cheating‖?

•Contract law – ―good faith‖

–Language of agreements

–Documented policies & procedures

–Taking all steps to show ―good faith‖

•Will you need expert testimony?

•General deference to exam programs

–State actors: due process

032

Admissibility of Expert Evidence in Court

Federal Rule of Evidence 702, Testimony by Experts

If scientific, technical, or other specialized knowledge will assist the trier of fact to understand the evidence or to determine a fact in issue, a witness qualified as an expert by knowledge, skill, experience, training, or education, may testify thereto in the form of an opinion or otherwise, if:

(1) the testimony is based upon scientific facts or data,

(2) the testimony is the product of reliable principles and methods, and

(3) the witness has applied the principles and methods reliably to the facts of the case.

033

Admissibility of Expert Evidence in Court

• Daubert v. Merrell Dow Pharmaceuticals, Inc., 509 U.S.

579 (1993)

– ―[U]nder the Rules [of Evidence] the trial judge must

ensure that any and all scientific testimony or evidence

admitted is not only relevant, but reliable….‖

• Kumho Tire Co. v. Carmichael, 526 U.S. 137 (1999)

– ―The objective of [the trial court’s gatekeeper] requirement

is to ensure the reliability and relevancy of expert

testimony. It is to make certain that an expert … employs

in the courtroom the same level of intellectual rigor that

characterizes the practice of an expert in the relevant

field.‖

Deference . . . within limits

Deference to Exam Programs

Murray v. ETS, 170 F.3d 514 (5th Cir. 1999) (SATExam)

• Louisiana basketball player; needed 820 on SAT• Scored 700, then 1300

– Similarity to nearby student (3 in 100 million odds)– Scored 800 on retake

• ―ETS’s contract with Murray clearly and explicitlyreserved to ETS the right to withhold any scoresETS had reason to believe were not valid. Theonly contractual duty ETS owed to Murray was toinvestigate the validity of Murray’s scores in goodfaith.‖

034

Deference to Exam Programs

Langston v. ACT, 890 F.2d 380 (11th Cir. 1989) (ACTExam)

• Alabama football player; scored 10 on ACT; then 20

• Inconsistent with GPA; unusual similarity to nearbystudent

• ―Under the governing law, the outcome of plaintiff’scase does not turn on whether or not plaintiff cheatedon his exam, but only on whether or not ACT carriedout its contractual obligations in good faith.‖

035

MANAGING TEST SECURITY

INVESTIGATIONS

025

Managing Investigations

• Do your investigators have all they need to be effective?

– Corporate support (budget, effective legal counsel, training)

– Software/services

– Support from other departments (Psychometrics, Test

Development, Registration & Credentialing, etc.)

• What procedure is in place to select cases for

investigation?

– Is it prudent/efficient to investigate all matters?

• What metrics exist to determine the success/efficiency of

an investigation?

– Did the investigation glean the desired information?

– When does an organization ―close‖ an investigation?

027

Managing Investigations

• Who within (or outside) your organization conducts

investigations? When?

– May depend on investigation type: exam-day incidents,

ongoing copyright infringement, collusion, proxy testing

• At what point do you involve legal counsel, board

members, or other departments?

• Do your policies and procedures reflect what is

needed to manage investigations?

– Are candidates obligated to cooperate?

• How are results reported? Who makes sanctions

decisions?

Gather Evidence

• Similarity analysis, gains analysis, other statistics

• Reports of security incidents

• Seating charts and chain of custody of materials

• Review test taker associations and connections

• Review access logs to secured exam content

• Review score histories of test takers and locations

• Review test booklets for signs of ―work‖

• Responses by test center staff & test takers

• Adherence to security policies

018

Evaluate Evidence

• Do alternative explanations exist?

• Are candidate explanations/responses convincing?

• Could test fraud have occurred?

• Are the test results trustworthy?

– Evaluation depends upon trustworthiness of the scores, NOT

an inference of behavior.

019

The proper use of Data Forensics is to certify the trustworthiness of the test

results and the integrity of the test administration. It is NOT proper to use

these results to place a label, such as ―cheater,‖ on an individual.

Wrap up & key takeaways

• Data Forensics

– Measure and manage risks

– Ensure fair and valid testing

– Mitigate losses and liability

• Implementation

– Security is a process, not a state

– Policy—what will we do with the results?

– Breaking down organizational ―fiefdoms‖

• Support

– Not ―if‖, but ―when‖

– Nobody plans to fail, but…

A long and winding road….Key Takeaways, cont.

• Legal

– Agreement is your foundation

• Even in K-12!

– Don’t be ―arbitrary and capricious‖

– Consistent and uniform

• Investigations

– ―Go/No Go‖ decision criteria

• Constrained resources

– Focus on the results, not the behaviors

• Statistically ―Indeterminate results‖ vs ―You’re a cheater!‖

Got questions?

kerrid@microsoft.com

jennifer.semko@bakermckenzie.com

dennis.maynes@caveon.com Caveon Confidential.

Do not share without permission.

Hope to see you soon…• CCSSO TILSA SCASS

– Oct 1-5

– Indianapolis

• Next month’s webinar

– ―Do It Yourself Security Audits and Security Investigations‖

– Tuesday, Oct.16, noon EDT

• ICE (Institute for Credentialing Excellence)

– Nov 6-9

– Palm Springs

Want more?

LinkedIn group ―Caveon Test Security‖

twitter @caveon for updates, news, connect

Blogs!

Caveon Security Insights – www.caveon.com/blog

Cheating In The News – www.caveon.com/citn

www.caveon.com/resources/webinars - to see past sessions

Contact skyler.weisenburger@caveon.com for slides,

comments, and questions about this session