Ten Years of Test SecurityThe Road Traveled,

The Road Ahead

1

National College Testing Association2014 Conference - Mining for Success

September 4, 2014

Introductions

• Rachel SchoenigACT

• Jim Wollack University of Wisconsin-Madison

• Steve AddicottCaveon Test Security

2

Objectives

• Share some collective Test Security experiences & “lessons learned”

• Arm us with information and tools for improving Test Security

• Explore the role of Test Centers and Test Administrators in ensuring valid test results

3

Agenda

• A Quick Look Back– An Important Milestone

• The Future: Three Key Trends– Growth In Testing– Technology Innovations, The Good and the Bad – Your Evolving Role

4

Steve AddicottCaveon Test Security

5

www.caveon.com 6

What is Test Security?

Test Security = Test Results We Can Trust

No one gains unfair advantage!

www.caveon.com 7

Poll Question #1

Compared to 5 years ago, I currently contend with ______ Test Security Issues than I did then.

A. Significantly More

B. A little more

C. Approximately the same number of

D. A little fewer

E. Significantly fewer

F. I was not in testing 5 years ago

www.caveon.com 8

Poll Question #2

On average, I deal with a Test Security Issue or Incident on the following frequency:

A. Almost daily

B. A few times a week

C. A few times a month

D. Every couple of months

E. Once or twice a year

F. Less than once a year

www.caveon.com 9

10

The Past www.caveon.com

An Important Milestone

www.caveon.com 11

Caveon’s Tenth Anniversary

Caveon

www.caveon.com 12

“We fundamentally believe in quality

testing and the importance of

trustworthy test results.”

13www.caveon.com

Things Began to Change

• Stakes grow ever-higher

• Technology Innovations

• Public Awareness

www.caveon.com 14

A Real-Life Example

• Medical certification program

• Administration to 7,500 candidates on Saturday

• Anonymous email on Wednesday– “I thought you should know…”

ENTIRE ITEM BANK ATTACHED!!

www.caveon.com 15

16www.caveon.com

17

The Futurewww.caveon.com

Three Key Test Security Trends for the Future

• Growth in Testing

• Technology Innovations, The Good and The Bad

• Your Evolving Role

www.caveon.com 18

Growth in Testing

• Testing is GROWING…incredibly– International Expansion– Career Readiness– Testing “anywhere”– Privacy Issues– “Increased Scrutiny”

• Media• Government• Industry Organizations

www.caveon.com 19

International Expansion

• Good– New market opportunities– Larger student/candidate/employee pools– More $$

• Bad– Some regions of the globe are

“Wild, Wild West”– Cultures vary dramatically– Enforcement options

may be limited

www.caveon.com 20

Career Readiness

• Fascinating dynamic– Stagnant job growth– Global demand for qualified

(fill in the blank)

• Incredibly competitive job market

• Fear of loss

www.caveon.com 21

Testing “Anywhere”

• MOOCS

• Online Proctoring

• Technology-based Testing

www.caveon.com 22

Privacy Issues

Broad, complicated topic

• Test taker verification & biometric data

• Another interesting dynamic– Test takers want to share– Test publishers are challenged

• Openness– SOX– FIA

• Privacy– FERPA– HIPAA– Several others

– International is confounding

www.caveon.com 23

Increased Scrutiny

• Media• Government• Industry Organizations

www.caveon.com 24

Media Awareness

25www.caveon.com

Media

Plagiarism on standardized tests three times higher in New Orleans schools than rest of Louisiana

www.caveon.com 26

Two ex-educators in Philadelphia forced to surrender credentials

Students Disciplined in Harvard Scandal

Trial Set to Begin in Atlanta Cheating Case

Media

www.caveon.com 27

Types of Media

• Traditional – Newspaper– Television

• Online Media– Blogs– Forums

• Social Media– Twitter– Facebook– LinkedIn– SnapChat– Instagram

28www.caveon.com

Traditional Media

• Benefit vs. Threat• Politically Slanted• Antiquated

29www.caveon.com

Media Lessons from Atlanta

• Expect Media Attention• Get Out Front with Your Own Story• Cultivate Local Media

– Most do not understand test results– Looking for interesting story – a no problem

administration is not a good story

• Learn Who You Can Trust• Don’t Expect a Cheating Story to Go Away –

You Cannot Control It

30www.caveon.com

Online and Social Media

31www.caveon.com

Media Conversation on PARCC/SBAC

32www.caveon.com

Can Social Media Be a Security Threat?

• Are “Secure” Items on the Internet?– Yes, in many cases “secure” items are found on

the internet– Twitter, Facebook, SnapChat, Instagram

• Occurs in High Stakes Testing• Long Testing Windows Tend to Lead to

“Leakage”

33www.caveon.com

Government

• Past laws

• Current RTTT & common core

• Does test Security = National Security?

www.caveon.com 34

Government Involvement

www.caveon.com 35

Industry Organizations

www.caveon.com 36

An Amazing Convergence

www.caveon.com 37

Because…

www.caveon.com 38

Important Test Security Resources

• Handbook of Test Security

• CCSSO TILSA Test Security Guidebook

• International Test Commission Guidelines for the Security of Examinations, Tests and Other Assessments

• ATP/CCSSO Operational Best Practices for Statewide Large Scale Assessment Programs

• NCME Testing and Data Integrity in the Administration of Statewide Student Assessment Programs

www.caveon.com 39

Important Common Topics

• Best Practice/Standards

• Importance of an overarching Test Security Plan/Manual

www.caveon.com 40

Jim WollackUniversity of Wisconsin-Madison

41

Technology Testing

42

The Technology Challenge

• Advances in technology have made it much easier for examinees to compromise the integrity of an exam. – Increased availability and the quality of

technology, coupled with lower costs, all raise the threat level

• Vigilant proctoring is necessary, but is no longer sufficient– Cheating in plain view…and virtually undetectable

43

Oldies but Goodies

Assistive Clothing

45

Smaller, Sleeker . . .

46

And Smarter

Public Enemy #1: WWW

Web is a great way to disseminate/acquire information widely and anonymously

And to disseminate/acquire information on how to best disseminate information 49

Emerging Problems

Google Glass Research: “…to be successful, the device should be as physically unobtrusive as possible because humans, who learn to read faces for emotional information at a very early age, may be instinctively distrustful of anyone whose eyes or eyebrows are unusually obscured”

50

Emerging Problems

51

Emerging Problems

52

Anti-Fraud Check-In Process

1.

2.

3.

Technology can be our Friend

• Technology-Based Testing

– Improved delivery technologies

• Adaptive tests: Shorter, less overlapping tests

• Single item-at-a-time transmissions

– Ability to rotate/replace pools

– Less memorable and more cheat-resistant items

• Improved authentication

– Portable biometric devices

– Integration with Big Data Databases

54

Technology can be our Friend

• Electronic Monitoring

• HD Cameras in testing centers

• Remote proctoring

• Metal Detectors/Signal Detectors

• Software

– Lockdown browsers

– Logging keystrokes and programs/sites visited

– Automated prerequisite checks55

The Emergence of Data Forensics

56

Statistical Solutions to Test Fraud

• Approaches were very narrow in scope– Answer Copying– Person-Fit– Item Exposure Control

57

Cheating

on Tests

RESEARCH

58

Statistical Solutions to Test Fraud

Score Differencing

Mixture Models

RT Models

Diff. Test Function

Similarity

Analysis

Answer

Sharing Item

Preknowledge

and

Collusion

Test

Tampering

Answer Copying

Person-Fit

Cluster

Analysis,

Info

Theory

Erasure Distributions

Erasure Modeling

Hierarchical Linear Models

Gain Score Approaches

Prevention

MethodologiesItem Exposure Control

Item Harvesting

Research Trends in Cheating Detection

• Utilization of more data

• Empirical � Theoretical � Exact Dist’ns

• Applied � Simulated Validation Efforts

• Research � Operationalization– Proprietary � Dissemination� Collaborative

60

A Glimpse Ahead?

• Real-Time Data Forensics– Changes forensics from fishing expedition to

targeted analyses with fixed criteria– Continuous biometrics

• Consistency with regard to Methodologies– Standards?

• Data forensics driving test development

61

Data Forensics Paradigm Shift

• Industry undergoes paradigm shift and reformulates cheating problem as a validity issue

– Data forensics no longer regarded as a black eye on the testing industry

– Marked increase in attention

Cheating on

Tests

Public

Distrust of

Testing

Intensive

Data

Forensics

Recognition of

Recognition

that test

validity is

enhanced

Increased

Public Trust

of Testing

Intensive

Data

Forensics

62

Rachel SchoenigACT

63

Evolving Role of Test Administrator

Evolving Role of Test Administrator

Chippola College Testing Center

The “cheating problem”

Testing Process Silos

Test Development

Test Publication

Test Delivery

Score &Report

Holistic Testing Process

Development

Publication

AdministrationScoring

Reporting

Evolution of Test Security

Catching a cheater

Ensuring a fair and level playing field and valid test results

Test Administrator Evolving Role

Face of Testing Companies

Test Security Framework

DeterDetectDecide

Deter, Detect, Decide

Score challenge process

Test Taking Rules

Rules to secure materials

Irregularity Reports

Data

Forensics

Hotlines

New test forms / item pools

Encryption

Proctors

Examinee agreements

Incident Response Plans

Rules v. Principles

Test Security Principles

• Protect the integrity of tests & test scores throughout the entire testing life cycle.

• Promote conduct that enhances test security. Deter & detectconduct that materially & negatively affects Testing Assets.

• Ensure that a reported test score is an accurate & valid indicatorof the test taker’s own academic capabilities.

• Foster effective communication that enables prompt reporting & resolution of test security concerns.

• Ensure everyone in the testing process is aware of and supported in their roles .

• Avoid placing individuals in situations that may pose a conflict of interest or a safety concern .

• Build a sense of community, collaboration, and trust that engages and empowers people to act upon these principles.

Applying PrinciplesYou are a test administrator for a new high stakes exam for Advanced Certified Test Administration. You notice one test taker adjusting his tie in an odd way during the test and begin to strongly suspect he is recording the test. You check the manual, but it doesn’t say anything about cameras, much less camera ties. It does have the following principles:

• Protect the integrity of tests & test scores throughout the entire life cycle.

• Deter & detect conduct that materially & negatively affects testing assets.

• Ensure everyone in the testing process is aware of and supported in their roles.

• Avoid placing individuals in situations that may pose a conflict of interest or a safety concern.

Poll QuestionWhich action would be most appropriate?

A. Do nothingB. File irregularity reportC. Block him from leaving; demand tieD. Contact the police

Principles:– Protect the integrity of tests & test scores throughout the

entire life cycle.– Deter & detect conduct that materially & negatively affects

testing assets.– Ensure everyone in the testing process is aware of and

supported in their roles. – Avoid placing individuals in situations that may pose a

conflict of interest or a safety concern.

Incidents

Detection

Investigation

Resolution

You are Critical

Questions?

Thank You!

Steve Jim RachelAddicott Wollack Schoenig