caveon webinar series: what you should know about high stakes cheating in your schools
DESCRIPTION
This slide deck was presented during an informational webinar by industry experts Michael Stetter, former Delaware State Assessment Director and test security consultant, and Dr. John Fremer, President, Caveon Consulting Services. Fremer and Stetter talked about how to evaluate the fairness and accuracy of your state assessment results, and discussed strategies to keep your district from becoming the next Atlanta Public Schools scandal.TRANSCRIPT
Upcoming Caveon Events
• Caveon Webinar Series: Next session, May 17
“Using Decision Theory to Score Accurate Pass/Fail Decisions” Presented by Jamie Mulkey and Larry Rudner
• Caveon at NCSA – June 19-21, National Harbor, MD
Visit our booth and spend some time talking with us one-on-one!
Latest Publications
• Handbook of Test Security – Now available for purchase! We’ll share a discount code before end of session.
• TILSA Guidebook for State Assessment Directors on Data Forensics – coming soon!
Caveon Online
• Caveon Security Insights Blog– http://www.caveon.com/blog/
• twitter– Follow @Caveon
• LinkedIn– Caveon Company Page
– “Caveon Test Security” Group• Please contribute!
• Facebook– Will you be our “friend?”
– “Like” us!
www.caveon.com
“What You Should Know about High Stakes Cheating in Your Schools”
Mike Stetter EdDSenior Security ConsultantFormer State Assessment Director
May 2, 2013
Caveon Webinar Series:
John Fremer PhDPresidentCaveon Consulting Services@TestSecurityGuy
AGENDA
A. THE TEST SECURITY SITUATION
B. CHALLENGES FOR SCHOOL DISTRICT LEADERS
C. TOP TEN THINGS THAT YOU CAN DO TO
STRENGTHEN YOUR TEST SECURITY
D. GAINING A GREATER SECURITY ADVANTAGE
A. THE TEST SECURITY SITUATION
Who cheats?
How many of them are there?
How often do they cheat?
JOSEPHSON INSTITUTE SURVEY RESEARCH
• 95% admitted to cheating in high school
• 51% believed they must cheat to get ahead
• 90% were pleased with their own morality
40,000 College Students Surveyed in 2010
Under 17 Over 50
Believe cheating is necessary 50% 10%
OK to deceive your boss 31% 8%
OK to keep incorrect change 49% 15%
OK to lie to your spouse 45% 22%
7,000 Survey Respondents
SURVEY OF ADULTS OVER 50 AND YOUTH UNDER 17
29% felt pressure to cheat on standardized tests
34% felt pressure to change students’ scores for the better
21% knew educators that changed scores on students’ tests
8% admitted to changing students’ scores due to pressure
Detroit Free Press, July 26, 2011
MICHIGAN EDUCATOR SURVEY
School District Cheating Scandals
In The News
DC Public Schools
More than 100 schools
displayed suspicious
answer sheet erasures
So far, 2 teachers have been forced to resign
USA TODAY June 22, 2012
Houston Independent
School District
Teachers at 2 elementary
schools caught cheating
Changed student answer sheets on State exams
The Houston Chronicle February 25, 2013
Atlanta Public Schools
179 educators in
58 schools implicated
in dishonest test practices
41 have resigned or retired. 35 indicted by Grand Jury
Former Superintendent indicted on conspiracy
TIMENewsFeedApril 13, 2013
Analyzed test results from
69,000 public schools
Suspicious test scores, similar to those in Atlanta,
were found in 196 school districts
Dishonest test practices on State level assessments
appears to be a national problem
April 13, 2013
August 25, 2011
Duncan: No link between cheating and NCLB
Places the blame on school leaders
Regardless of who the perpetrators are . . .
School District Administrators will feel the heat!
B. CHALLENGES FOR SCHOOL DISTRICT LEADERS
• Proctors’ potential conflict of interest
• Non-Standardized environment
• Examinee familiarity
• Transporting and storing test materials
SCHOOL DISTRICT TEST SECURITY MATRIX
• District Superintendent
• District Assessment Coordinator
• School Test Coordinator
DISTRICT TEST ROLES SECURITY MATRIX
The full Matrix is posted on the Caveon website:
http://bit.ly/testroles
SCHOOL DISTRICT TEST SECURITY MATRIX
Essential Proactive
Steps
During Test Window
Before & After
Test Window
Superintendent
District Assessment
Director
Building Test Coordinator
District Superintendent
• Assign clear responsibility & roles for test security
• Maintain communications with all local stakeholders on assessment activities, results
• Advocate for test program best practices
SCHOOL DISTRICT TEST SECURITY MATRIX
District Assessment Director
• Supervise test training for all proctors
• Protect secure materials and test session integrity through detailed, accountable procedures
• Develop/Update district Test Security Plan
SCHOOL DISTRICT TEST SECURITY MATRIX
Building Test Coordinator
• Confirm every test proctor completes training, and understands Do’s & Don’ts during testing
• Observe test sessions to confirm procedural compliance
• Directly supervise all handling of secure test materials
• Report suspected test irregularities
SCHOOL DISTRICT TEST SECURITY MATRIX
C. TOP 10 THINGS YOU CAN DO
TO STRENGTHEN YOUR TEST SECURITY
#1 – COMMUNICATION
• Raise test security awareness
• Include security references in communications
• Issue periodic security focused communications
#2 – EXPECTATIONS
• Develop a system-wide Student Honor Code
• Emphasize the importance of honest test taking
• Issue clear expectations to test staff for professional behavior in test administration
#3 – PROCEDURES
• Review all procedures for security related content
• Emphasize procedures for handling test materials
• Review / update security procedures annually
#4 – TRAINING
• Review your test security training materials
• Track who completes the training
• Provide brush-up training annually
#5 – OBSERVATIONS
• Perform announced pre-test session site visits
• Perform unannounced test session observations
• Follow-up on all observation outcomes
#6 – FEEDBACK
• Seek feedback from test administrators / proctors
• Review procedures for reporting test irregularities
• Provide an anonymous tip line
#7 – STUDY YOUR TEST DATA CAREFULLY
• Very Unusual Gains
• Other Indices
• Similarity
• Erasures
• Individual Patterns
• Timing (In Computer Testing)
#8 – PREPARING FOR POSSIBLE SECURITY INCIDENTS
• Develop a “Security Incident Response Plan”
• Specify procedures, roles and responsibilities
• List potential outcomes for security infractions
#9 – LEARNING FROM OTHERS
• CCSSO TILSA SCASS “Guidebook on Test Security”
• Wollack & Fremer “Handbook of Test Security”
• Caveon Webinars and Security Insights Newsletter
Learn about developments in prevention and detection
#10 – IF YOU DEVELOP YOUR OWN TESTS
• Reduce exposure rate of test questions
• Randomize question locations
• Rotate answer options
DATA FORENSICS
D. GAINING A GREATER SECURITY ADVANTAGE
Data Forensics is the analysis of student test data
using sophisticated proprietary computer programs
to statistically identify instances likely representing
test fraud, collusion, and other security violations.
DATA FORENSICS
A statistical measure of similarity within test results
or test response patterns that highlight instances
with a very high likelihood of a test security violation.
Similarity Index
DATA FORENSICS
HOW CREDIBLE IS A FLAGGED SECURITY INDEX?
• Chance of being hit by lightning = 1 in 1 Million
• Chance of winning the lottery = 1 in 10 Million
• Chance of DNA false-positive = 1 in 30 Million
• Chance of students flagged for Similarity doing their own work = 1 in 1 Trillion
Caveon’s data forensic diagnostic screening service for
school districts – a cost-effective means to monitor for
• Answer copying
• Test coaching
• Proxy test taking
• Collusion and other security violations
THE CAVEON SECURITY SCREEN
• Custom forensic analyses of student test records
• Data output by subject, by school and by student
• Data interpretation with a Caveon test security specialist
• Advice on follow-up with highlighted security anomalies
• Consultation with test security planning and evaluation
• Security focused analysis of policies and procedures
THE CAVEON SECURITY SCREEN
SAMPLE FORENSICS OUTPUT DATA REPORT
Consultation
• Consultation with a Caveon Test Security Specialist
• Advice on Test Security planning and evaluation
• Interpretation of forensic data output reports
• Guidance on test irregularity follow-up
THE CAVEON SECURITY SCREEN SUMMARY
• Similarity Index analyses
• Perfect Test and Identical Test Analyses
• School level data reported by grade and by subject
• Flagged student output with cluster details for pairs and groups
Data Forensics
Michael W. Stetter, D.Ed.
Senior Security Consultant
Caveon Test Security
(302) 415-0375
For information on the Caveon Test Security Screen
HANDBOOK OF TEST SECURITY
Editors - James Wollack & John Fremer
Published March 2013
Preventing, Detecting, and Investigating Cheating
Testing in Many Domains
Certification/Licensure
Clinical
Educational
Industrial/Organizational
Don’t forget to order your copy at www.routledge.com
http://bit.ly/HandbookTS (Case Sensitive)
Save 20% - Enter discount code: HYJ82
- Follow Caveon on twitter @caveon- Check out our blog…www.caveon.com/blog- LinkedIn Group – “Caveon Test Security”
Mike StetterSenior Security ConsultantCaveon Test Security
John FremerPresidentCaveon Consulting Services@TestSecurityGuy
Thank You!