caveon webinar series: the good and bad of online proctoring
DESCRIPTION
These slides were shared during an informational Webinar presented by the Caveon Webinar Series with guest panelists David Foster, CEO, Caveon Test Security, and Harry Layman, Executive Director of Digital Assessment Planning, The College Board. Description: Online proctoring is rapidly garnering the attention of testing programs that want better security, a longer reach, convenience, and lower test administration costs. Today there are several vendors offering services in this area, and they differ widely in the type and quality of services they provide. What features of online proctoring boost test security and which actually hurt test security? These slides focus on the major security threats during test administration and how those threats can be managed by different features of online proctoring. The slides provided should help an organization make sense of the variety of offerings available, and to select online proctoring features that improve the integrity of their tests. Please join our LinkedIn group "Caveon Test Security" to join this and other discussions. Please contact [email protected] with any questions.TRANSCRIPT
![Page 1: Caveon Webinar Series: The Good and Bad of Online Proctoring](https://reader037.vdocument.in/reader037/viewer/2022102902/558fc2011a28abdc668b4705/html5/thumbnails/1.jpg)
Upcoming Caveon Events
• NCME Conference, April 26 – 30th, San Francisco Test Security I: Policy Issues: Technical/Statistical/Methodological Issues
– Presenters include: Dr. John Fremer, Caveon
• USDLA Conference, April 28- May 1, St. Louis
Will the real learner please stand up? – Presenters include Caveon’s Jamie Mulkey, Ed.D. and Patrick Martin
• Caveon Webinar Series: Next session, May 2
What You Need to Know about High Stakes Cheating in Your Schools– Presenters include former State Assessment Director, Dr. Mike Stetter,
and Dr. John Fremer, President, Caveon Consulting Services
![Page 2: Caveon Webinar Series: The Good and Bad of Online Proctoring](https://reader037.vdocument.in/reader037/viewer/2022102902/558fc2011a28abdc668b4705/html5/thumbnails/2.jpg)
Latest Publications
• Handbook of Test Security – Now available for purchase! We’ll share a discount code before end of session.
• TILSA Guidebook for State Assessment Directors on Data Forensics – soon to be released
![Page 3: Caveon Webinar Series: The Good and Bad of Online Proctoring](https://reader037.vdocument.in/reader037/viewer/2022102902/558fc2011a28abdc668b4705/html5/thumbnails/3.jpg)
Caveon Online
• Caveon Security Insights Blog– http://www.caveon.com/blog/
• twitter– Follow @Caveon
• LinkedIn– Caveon Company Page– “Caveon Test Security” Group
• Please contribute!
• Facebook– Will you be our “friend?”– “Like” us!
www.caveon.com
![Page 4: Caveon Webinar Series: The Good and Bad of Online Proctoring](https://reader037.vdocument.in/reader037/viewer/2022102902/558fc2011a28abdc668b4705/html5/thumbnails/4.jpg)
“The Good and Bad of Online Proctoring”
Harry LaymanExecutive DirectorDigital Assessment PlanningThe College Board
April 17, 2013
Caveon Webinar Series:
Dr. David FosterCEOCaveon Test Security
![Page 5: Caveon Webinar Series: The Good and Bad of Online Proctoring](https://reader037.vdocument.in/reader037/viewer/2022102902/558fc2011a28abdc668b4705/html5/thumbnails/5.jpg)
Agenda for today
• Recap of Structure and Content of recent paper “Online Proctoring Systems Compared”
• Introduction and Definitions for Online Proctoring Discussion• Test Security Threats Addressed by Online Proctoring• Online Proctoring Features and Threat Impacts• Seven Critical Online Proctoring Best Practices• Odds and Ends
– Research Support– Future Features– Trends
• Lessons for Test Program Users
![Page 6: Caveon Webinar Series: The Good and Bad of Online Proctoring](https://reader037.vdocument.in/reader037/viewer/2022102902/558fc2011a28abdc668b4705/html5/thumbnails/6.jpg)
Online Proctoring Systems Compared:Purpose of Paper
• Published by Dave Foster and Harry Layman– http://bit.ly/proctoring
• Background of paper• Recent paper provides a framework for describing different
offerings and several tables of feature comparisons• Lots of detail, not entirely complete – will remain a work in
progress• A good starting point as you begin to think about features /
benefits and your specific risk profile and threats of concern
![Page 7: Caveon Webinar Series: The Good and Bad of Online Proctoring](https://reader037.vdocument.in/reader037/viewer/2022102902/558fc2011a28abdc668b4705/html5/thumbnails/7.jpg)
Online Proctoring Systems Compared: Structure and Content
• Overview of Online Proctoring – what it is, what it is not, and primary value proposition
• An enumeration of 8 vendor products / services included in the scope of the review– Mix of capabilities and approaches; offerings not homogenous/equivalent and so
comparisons need to be more holistic– We have not yet investigated all options equally, so the “comparisons” are
informative but not complete; look for updates over time
• Seven groups of features or characteristics of Online Proctoring offerings organized into a series of comparison tables
• This presentation will “hit the highlights,” setting online proctoring in the general context of test security threats, and provide a high-level discussion of the attributes and features of Online Proctoring systems that can mitigate those threats.
![Page 8: Caveon Webinar Series: The Good and Bad of Online Proctoring](https://reader037.vdocument.in/reader037/viewer/2022102902/558fc2011a28abdc668b4705/html5/thumbnails/8.jpg)
Online Proctoring Is / Is not:
• Online Proctoring refers specifically to using an internet-based approach to remotely monitor individual test administrations, replacing traditional approaches that have relied on “eyes in the room”
– Not CCTV, roving personnel, or using volunteers in user-selected locations (libraries, etc.)
• Online Proctoring is NOT a “find-your-own-proctor” model as used for decades in distance education exams
![Page 9: Caveon Webinar Series: The Good and Bad of Online Proctoring](https://reader037.vdocument.in/reader037/viewer/2022102902/558fc2011a28abdc668b4705/html5/thumbnails/9.jpg)
Online Proctoring Provides:
• Key value proposition in a nutshell:
– Independent, bias-free proctors
– Professional proctors (training, supervision, logging, accountability, etc)
– Technology-assisted audio, visual and activity monitoring support (e.g., log files, video, lock-down, keystroke monitoring, test session control, real-time intervention, etc.)
![Page 10: Caveon Webinar Series: The Good and Bad of Online Proctoring](https://reader037.vdocument.in/reader037/viewer/2022102902/558fc2011a28abdc668b4705/html5/thumbnails/10.jpg)
General Proctoring Requirements
• Online Proctor During Entire Exam • Continuous Internet • Encryption for Data Transfer • Schedule Availability • Proctor Management
– Supervised – Training – Career Path – Certification
• Program Customization
![Page 11: Caveon Webinar Series: The Good and Bad of Online Proctoring](https://reader037.vdocument.in/reader037/viewer/2022102902/558fc2011a28abdc668b4705/html5/thumbnails/11.jpg)
Security Threats / Scope
• Online Proctoring is primarily concerned with
– Threats during test administration; and hence
– Attempts to either capture test content, or
– Get inappropriate assistance in any way.
![Page 12: Caveon Webinar Series: The Good and Bad of Online Proctoring](https://reader037.vdocument.in/reader037/viewer/2022102902/558fc2011a28abdc668b4705/html5/thumbnails/12.jpg)
Security Threats / Scope
• Online Proctoring is only concerned with a subset of the overall set of program vulnerabilities, threats and risks. What are they?
• The subset are those threats that are detectable by the online proctor (or onsite proctor) and over which he or she has some control
![Page 13: Caveon Webinar Series: The Good and Bad of Online Proctoring](https://reader037.vdocument.in/reader037/viewer/2022102902/558fc2011a28abdc668b4705/html5/thumbnails/13.jpg)
Test Security Threats: A Finite Universe
• Only Six Categories of Cheating Threats– Using pre-knowledge of test content– Colluding with others– Using cheating aids– Using a proxy test taker– Hacking into scoring system– Copying answers from other test takers
• Only Six Categories of Stealing Threats– Hacking into a system and stealing test files– Capturing content by digital photography devices– Capturing content by electronic recording of the screen– Memorizing content– Transcribing content verbally (on paper or recording device)– Getting content from test program insider
![Page 14: Caveon Webinar Series: The Good and Bad of Online Proctoring](https://reader037.vdocument.in/reader037/viewer/2022102902/558fc2011a28abdc668b4705/html5/thumbnails/14.jpg)
Test Security Threats: A Finite Universe
• Only Six Categories of Cheating Threats– Using pre-knowledge of test content– Colluding with others– Using cheating aids– Using a proxy test taker– Hacking into scoring system– Copying answers from other test takers
• Only Six Categories of Stealing Threats– Hacking into a system and stealing test files– Capturing content by digital photography devices– Capturing content by electronic recording of the screen– Memorizing content– Transcribing content verbally (on paper or recording device)– Getting content from test program insider
![Page 15: Caveon Webinar Series: The Good and Bad of Online Proctoring](https://reader037.vdocument.in/reader037/viewer/2022102902/558fc2011a28abdc668b4705/html5/thumbnails/15.jpg)
Down to Six! Test Security ThreatsRelevant for Online Proctors
• Cheating Threats– C1. Colluding with others– C2. Using cheating aids– C3. Using a proxy test taker– C4. Copying answers from other test takers
• Stealing Threats– S1. Capturing content by digital photography devices– S2. Transcribing questions verbally (on paper or
recording device)
![Page 16: Caveon Webinar Series: The Good and Bad of Online Proctoring](https://reader037.vdocument.in/reader037/viewer/2022102902/558fc2011a28abdc668b4705/html5/thumbnails/16.jpg)
C1. Colluding with Others
• How can a proctor effectively detect and deal with collusion?
– Camera that can see a large portion of the room– Camera that can pick up sounds and talking– Ability to pause test session, and perhaps suspend it– Ability to contact test taker, record/store
session and time-stamp incident– Proctors should be prohibited
from viewing test taker screen
![Page 17: Caveon Webinar Series: The Good and Bad of Online Proctoring](https://reader037.vdocument.in/reader037/viewer/2022102902/558fc2011a28abdc668b4705/html5/thumbnails/17.jpg)
C2. Using Cheating Aids
• How can a proctor effectively detect and deal with a test taker using a cheating aid?
– Camera that can see a large portion of the room, particularly desk, computer, keyboard, test taker head/arms/hands
– Strong lockdown program to prevent access to online aids– Ability to contact test taker, pause
test session, and perhaps suspend it– Ability to record/store session and
time-stamp incident
![Page 18: Caveon Webinar Series: The Good and Bad of Online Proctoring](https://reader037.vdocument.in/reader037/viewer/2022102902/558fc2011a28abdc668b4705/html5/thumbnails/18.jpg)
C3. Using a Proxy Test Taker
• How can a proctor effectively detect and deal with a proxy test taker?
– Strong initial authentication– Continuous authentication methods– Ability to contact test taker, pause test session, and
perhaps suspend it– Ability to record/store session and time-stamp incident
![Page 19: Caveon Webinar Series: The Good and Bad of Online Proctoring](https://reader037.vdocument.in/reader037/viewer/2022102902/558fc2011a28abdc668b4705/html5/thumbnails/19.jpg)
C4. Copying Answers from Another Test Taker
• How can a proctor effectively detect and deal with a test taker copying from another?
– Camera that can see a large portion of the room, including other test takers allowed in the room
– Ability to contact test taker, pause test session, and perhaps suspend it
– Ability to record/store session and time-stamp incident
![Page 20: Caveon Webinar Series: The Good and Bad of Online Proctoring](https://reader037.vdocument.in/reader037/viewer/2022102902/558fc2011a28abdc668b4705/html5/thumbnails/20.jpg)
S1. Capture Test Content with Digital Photography
• How can a proctor effectively detect and deal with the digital photography of the test questions?
– Camera that can see a large portion of the room, along with head/hands/torso of test taker
– Prevent proctor from viewing test screens– Ability to contact test taker, pause test session, and
perhaps suspend it– Ability to record/store session and time-stamp incident
![Page 21: Caveon Webinar Series: The Good and Bad of Online Proctoring](https://reader037.vdocument.in/reader037/viewer/2022102902/558fc2011a28abdc668b4705/html5/thumbnails/21.jpg)
S2. Transcribing Questions Verbally
• How can a proctor effectively detect and deal with the transcription of test questions?– Camera that can see a large portion of the room, along
with head/hands/torso of test taker– Prevent proctors from viewing the test taker screen– Ability to contact test taker, pause test session, and
perhaps suspend it– Ability to record/store session and
time-stamp incident
![Page 22: Caveon Webinar Series: The Good and Bad of Online Proctoring](https://reader037.vdocument.in/reader037/viewer/2022102902/558fc2011a28abdc668b4705/html5/thumbnails/22.jpg)
Seven Critical Online Proctoring Best Practices
1. Use wide-view camera with microphone
2. Require lockdown program
3. Prevent proctor view of test screens
4. Allow and require proctor to communicate with test taker
5. Allow proctor to control the test session
6. Use strong authentication methods
7. Record, store and time-stamp test session, along with proctor logs
![Page 23: Caveon Webinar Series: The Good and Bad of Online Proctoring](https://reader037.vdocument.in/reader037/viewer/2022102902/558fc2011a28abdc668b4705/html5/thumbnails/23.jpg)
First: Use a Capable Webcam
• Is external to the computer; connected by 2-4 feet of USB cable and set back
• Has large field of view (approx. 80° up to 360°)• Allows proctor to see testing workstation and head,
torso, arms, hands of test taker• Camera angle and placement
does not allow viewing the
workstation screen• Has integrated microphone
![Page 24: Caveon Webinar Series: The Good and Bad of Online Proctoring](https://reader037.vdocument.in/reader037/viewer/2022102902/558fc2011a28abdc668b4705/html5/thumbnails/24.jpg)
Second: Use a Capable Lockdown Program
Operating System/Computer/Browser• Prevent right-click • Prevent min/max windows
• Prevent printing • Prevent Copy/Paste
• Prevent function keys • Prevent running of applications
• Prevent important key combos • Prevent launch of applications
• Hide Taskbar and Desktop • Prevent communication tools
• Hide menus and icons • Prevent browser control/navigation
![Page 25: Caveon Webinar Series: The Good and Bad of Online Proctoring](https://reader037.vdocument.in/reader037/viewer/2022102902/558fc2011a28abdc668b4705/html5/thumbnails/25.jpg)
Third: Prevent Proctor View of Test Workstation Screen
• What are the reasons a proctor should not view the content of a test taker’s workstation screen?
– Security Reason #1: Viewing the screen may encourage collusion between proctor and test taker
– Security Reason #2: Seeing the items of a high-stakes exam on a proctor’s screen may encourage the theft of the test content
– Privacy Reason: It’s not appropriate for a proctor to view how a test taker answers test questions
![Page 26: Caveon Webinar Series: The Good and Bad of Online Proctoring](https://reader037.vdocument.in/reader037/viewer/2022102902/558fc2011a28abdc668b4705/html5/thumbnails/26.jpg)
Fourth: Allow Proctor to Communicate with Test Taker
• Chat• Canned Messages• Technical Support call if necessary
![Page 27: Caveon Webinar Series: The Good and Bad of Online Proctoring](https://reader037.vdocument.in/reader037/viewer/2022102902/558fc2011a28abdc668b4705/html5/thumbnails/27.jpg)
Fifth: Allow Proctor to Control Test Session
• Control allows proctor to deal effectively with detected threats– Pause the test
• cover or remove the current question• require acknowledgement and compliance from the test
taker• provide a warning
– Suspend the test (can restart at later time)– Cancel the test
• Proctor rules for these actions are subject to customization by the testing program
![Page 28: Caveon Webinar Series: The Good and Bad of Online Proctoring](https://reader037.vdocument.in/reader037/viewer/2022102902/558fc2011a28abdc668b4705/html5/thumbnails/28.jpg)
Sixth: Use Strong Authentication Methods
• Strong– Fingerprint readers– Facial recognition– Keystroke pattern recognition– Voice recognition
• Avoid weak authentication methods– Viewing of government-issued IDs through webcam
![Page 29: Caveon Webinar Series: The Good and Bad of Online Proctoring](https://reader037.vdocument.in/reader037/viewer/2022102902/558fc2011a28abdc668b4705/html5/thumbnails/29.jpg)
Seventh: Record and Store Sessions
• Record entire test session and proctor logs/communications
• Time stamp incidents to make them easy to locate
• Provide easy-to-use review application for both video of test session and proctor logs
• Store long-term or until no longer needed
![Page 30: Caveon Webinar Series: The Good and Bad of Online Proctoring](https://reader037.vdocument.in/reader037/viewer/2022102902/558fc2011a28abdc668b4705/html5/thumbnails/30.jpg)
Best Practices Effects on Security Incidents
• If these best practices are followed what benefits can be expected?
– Better detection of threats or attempted breaches– Professional, un-biased, and competent response by
proctors– Deterrence: Fewer incidents going forward– Stronger evidence basis for decision making and taking any
action, including legal action
![Page 31: Caveon Webinar Series: The Good and Bad of Online Proctoring](https://reader037.vdocument.in/reader037/viewer/2022102902/558fc2011a28abdc668b4705/html5/thumbnails/31.jpg)
Research Support
• Reliable data – on site or online – is hard to come by
• Limited data available for online proctoring; what is there is encouraging
• More reporting by participating organizations will increase the data pool
• Need studies also from academic researchers
![Page 32: Caveon Webinar Series: The Good and Bad of Online Proctoring](https://reader037.vdocument.in/reader037/viewer/2022102902/558fc2011a28abdc668b4705/html5/thumbnails/32.jpg)
It’s Only Going to Get Better: Future Proctoring Features
• Improvements (e.g., better cameras; better proctor training, etc.)
• New features: Automated proctoring technologies– Inappropriate keystroke detection – Audio level detection and display – Detection of suspicious response patterns with Real-Time
Data Forensics– Better facial, voice recognition software on the horizon
![Page 33: Caveon Webinar Series: The Good and Bad of Online Proctoring](https://reader037.vdocument.in/reader037/viewer/2022102902/558fc2011a28abdc668b4705/html5/thumbnails/33.jpg)
More Trends
• More providers, may see increasing stratification between cost and quality / professionalism (as with other security services)
• New technologies may allow better detection, or lower costs – or both– Augmenting human monitoring of online video with real-
time “artificial intelligence” could allow for greater proctor-to-examinee ratios with even better security
![Page 34: Caveon Webinar Series: The Good and Bad of Online Proctoring](https://reader037.vdocument.in/reader037/viewer/2022102902/558fc2011a28abdc668b4705/html5/thumbnails/34.jpg)
Lessons for Program Users
• Do it right or be an easy target• Strong security is always needed for tests where
cheating and theft occur, but not everyone has the same set threats or concerns
• Different audiences may have different levels of technical sophistication, risk, time sensitivity, etc.
• Online proctoring when done properly can be as capable as any other well-run proctoring method at reducing test security risks
![Page 35: Caveon Webinar Series: The Good and Bad of Online Proctoring](https://reader037.vdocument.in/reader037/viewer/2022102902/558fc2011a28abdc668b4705/html5/thumbnails/35.jpg)
HANDBOOK OF TEST SECURITY
• Editors - James Wollack & John Fremer• Published March 2013• Preventing, Detecting, and Investigating Cheating• Testing in Many Domains
– Certification/Licensure– Clinical– Educational– Industrial/Organizational
• Don’t forget to order your copy at www.routledge.com– http://bit.ly/HandbookTS (Case Sensitive)– Save 20% - Enter discount code: HYJ82
![Page 36: Caveon Webinar Series: The Good and Bad of Online Proctoring](https://reader037.vdocument.in/reader037/viewer/2022102902/558fc2011a28abdc668b4705/html5/thumbnails/36.jpg)
Questions?
Please join our “Caveon Test Security” LinkedIn group and submit questions into the discussion box or contact [email protected]
![Page 37: Caveon Webinar Series: The Good and Bad of Online Proctoring](https://reader037.vdocument.in/reader037/viewer/2022102902/558fc2011a28abdc668b4705/html5/thumbnails/37.jpg)
THANK YOU!
- Follow Caveon on twitter @caveon- Check out our blog…www.caveon.com/blog- LinkedIn Group – “Caveon Test Security”
Harry LaymanExecutive DirectorDigital Assessment PlanningThe College Board
Dr. David FosterCEOCaveon Test Security
Again, find our Online Proctoring Systems Compared paper at http://bit.ly/proctoring