Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Welcome
9 Risks to Test Security (and what to do about them)
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
John Kleeman MA MBCS C.Eng CIPP/E
Executive Director and Founder of Questionmark
30 years of experience in the assessment industry
Presenter today
Slide 2
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Create stimulus
Participants answer
Dataset created
Make a decision
Assessment Value Proposition
Slide 3
Can these results be trusted
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
• To make important decisions about people
• Decisions that matter to society, organizations and individuals
We need to be able to trust the
result dataset
• Assessments must be valid, reliable and fair
• Process and systems must be effective
To be able to make good decisions
• Make assessment results less trustable
• Make decisions taken on assessments less reliable
BUT Test security failures and
cheating
Slide 4
Why Test Security Matters
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Data Security
System Security
Slide 5
Trustable Results
Planning Assessment
Authoring Items
Assembling assessment
Pilot and Review
Delivery
Analyze Results
Trustable Results Rely on a Trustable & Secure Process
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Data Security
System Security
Slide 6
Trustable Results
Planning Assessment
Authoring Items
Assembling assessment
Pilot and Review
Delivery
Analyze Results
Content theft
Content theft
Test center disruption
Identity fraud
Unauthorized disclosure
Tampering
Some Risks to Trustable Results
Unauthorized aids / help
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
What keeps you up at night? ▪ Content theft / leakage
▪ Unauthorized aids for cheating during exams
▪ Identity fraud
▪ Results tampering
▪ Personal information breached
Poll Slide
Slide 7
Quick Poll
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Cheating on Exams a Problem Worldwide
Slide 8
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Meanwhile Data Breaches Rise …
Slide 9
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Confidentiality
IntegrityAvailability
Confidentiality
• Keep questions secure -avoid content theft
• Results only available to those who should know
Integrity
• Right person takes assessment
• Assessment process fair and robust
• No cheating
Availability
• Assessments can be taken when needed
• Results are stored safely
First Start with Your Security Objectives
Slide 10
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Formal process can identify all threats, less likely to miss key threats
Quantifying risk allows you to prioritize actions that will reduce risk
Recommended by all respected security authorities: ISO 27001, NIST & many more
Example▪ 2017 Verizon data breach report reported 81% of hacking-related
breaches involved weak or stolen passwords.▪ For most organizations, a risk assessment is going to highlight this
and suggest some mitigations
Why Assess Risks?
Slide 11
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
High probabilityLow impact
High probabilityHigh impact
Low probability.Low impact
Low probability.High impact
LOW HIGH
LOW
HIG
H
Impact
Pro
bab
ility
Then look at Risk Probability and Impact
Prioritize security mitigations based on risk impact and probability
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
After delivery
Tampering with results Unauthorized disclosure of results
During test delivery
Identity fraudContent theft /
harvestingCopy answers from
another Getting help from
othersUnauthorized test aids
Before delivery
Content theft from item bank Disruption at test center
9 Common Risks to Consider
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Risk:
Content theft from item bank
Authoring Items
Assembling assessment
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Content theft from item bank
Slide 15
Author leaks content deliberately to help their students
Author leaks content by mistake or by using poor security
Technical vulnerability in item bank
Potentially very high: requires rewriting all items
Could invalidate test for all test takers
Threats Risk impact Real world example
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
More real world examples
Slide 16
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Use a secure cloud service▪ Avoids local files▪ Avoids email of questions▪ ISO 27001 and good technical security
Good HR practices with authors▪ Training and education▪ Confidentiality agreements▪ Remove access when people leave project▪ Good passwords▪ Extend to translators too if test is translated
Use permissions and roles▪ Authors should only have access to the minimum they need for a project▪ Restrict access to questions/assessments they do not need▪ Single sign-on
Some mitigations to prevent item bank leakage
Slide 17
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Just enough capability to get the job done
In assessment management systems:▪ Use roles and permissions to give
people the minimum capabilities they need
▪ Restrict access to questions to those who need
▪ Have a separate user account for privileged (high capability) actions
One person
requests
Another person
approves
Activity authorized
Two important security principles help
Slide 18
Least privilege Segregation of duties
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Risk
Disruption at test center
Slide 19
Pilot and Review Delivery
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Disruption at a test center
Slide 20
Poor technical security at test center allows questions to be seen
Test center proctors corrupt
Test center proctors / teachers have incentive to help their students
Can result in a batch of test takers at the test center all being helped
Threats Risk impact Real world example
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
More real world examples
Slide 21
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Don’t allow download of questions to test center in advance
Consider not using test centers in pilot process
Data analysis / forensics
Using good technical measures▪ High quality TLS encryption
▪ Proctors unable to see questions
Use online proctoring▪ Remote proctors who do not know test takers and cannot receive a bribe less likely
to collude with test takers
Possible mitigations
Slide 22
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Traditional Proctoring•Proctor observes test taker physically in
person / in the same room Live Online Proctoring• Proctor observing test taker live via webcam
Copyright © 1995-2016 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Risk
Identity fraud
Slide 24
Delivery
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Identity fraud
Slide 25
Candidate gets a friend to take an exam for him/her
Candidate pays someone to take an exam for him/her
At least one result is meaningless
Can threaten acceptance of programme
Threats Risk impact Real world example
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Proctoring▪ Proctor checks government issued photo-ID▪ If concerned about proctors colluding with candidates, use online
proctoring
Frequent testing▪ Series of tests harder to get someone to take for you than single large test▪ With online assessment and online proctoring, much more realistic to have
more frequent, smaller tests
Use single sign-on (SAML)▪ People less keen to share credentials if they allow access to all their other
data
Reduce the motivation to cheat
Some ways to mitigate candidate impersonation
Slide 26
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Fraud needs:▪ Motivation▪ Opportunity▪ Rationalization
For a high stakes test, there may be Motivation
Anti-cheating measures seek to reduce Opportunity
How can we reduce Rationalization?
Fraud triangle
Slide 27
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Mitigating Rationalization
Slide 28
Rationalization
“Everyone else cheats so why shouldn't I?”
“I have no alternative but to cheat and I have to pass because …?”
“I can get away with this”
“I didn’t know I was doing anything wrong”
• Explain the facts:• Most people do not cheat• Consequences of cheating
Mitigation
• Provide environments for the candidate to be able to pass the exam honestly
• Explain the security measures in place and the consequences if they’re caught
• Explain and gain positive agreement to theo Honesty Codeo Code of ethicso Non-disclosure
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Risk
Content theft / harvesting
Slide 29
Pilot and Review Delivery
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Content theft / harvesting
Slide 30
Screenshot or other technical copy of test questions
Group harvesting, everyone remembers a few questions and shares online
Someone pays to take test just to see and remember questions
Slow degradation of integrity of test
Expensive as need to write new items
Threats Risk impact Real world example
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Use proctoring but don’t allow proctors to see content
Secure browsers like Questionmark Secure
Large and changing item banks with randomized test content
Reduce motivation to cheat / honor code
Use video/audio as part of stimulus to make harder to copy
Prevent harvesters from taking tests (pre-requisites, limit retakes)
Keep time limit as short as sensible
Possible mitigations
Slide 31
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
A secure browser reduces content theft and cheating
While using a secure browser, it is difficult for participants to▪ Make screenshots▪ Copy content▪ Run other programs▪ Search the Internet or use chat
Questionmark has a Questionmark Secure App and other vendors have other approaches
Secure browsers
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Risk
Copy answers from another
Slide 33
Delivery
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Copy answers from another
Slide 34
Candidate looks at the screen of the person next door to him/her and copies their answers
In a take-home exam, two people do it together
Organized answer sharing
Invalid test results for a candidate
Devalues credibility for others
Threats Risk impact Real world example
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Appropriate separation between workstations
Randomization of question order / question selections / choice order
How to mitigate copying answers
Slide 35
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Risk
Getting help from others
Slide 36
Delivery
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Getting help from others
Slide 37
Use a cellphone or instant messenging to get live help from a friend or coach
Someone sits by test taker to give coaching
Invalidates one result
If prevalent, degrades program
Threats Risk impact Real world example
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
More real world examples
Slide 38
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Use a secure browser (e.g. Questionmark Secure) for online tests
Proctoring (especially online)
Reduce motivation to cheat / honor code
Frequent testing
Forensics / analysis
Consider other approaches than essays
Some mitigations
Slide 39
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Risk
Unauthorized test aids
Slide 40
Delivery
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Unauthorized test aids
Slide 41
Cheat sheets / access to notes
Test taker Googles the answers
Calculators / other tools
Makes results less valid
Threats Risk impact Real world example
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
More real world examples
Slide 42
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Secure browsers
Proctoring
Make exam open book▪ In the real world, people have access to reference material, is it fair and valid to deny
it during an exam?
Ask higher level questions (e.g. Situational judgement)
Reduce motivation to cheat / honor code
Mitigations
Slide 43
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Risk
Tampering with results
Slide 44
DeliveryAnalyze Results
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Tampering with results
Slide 45
An insider changes someone’s score out of favouritism
Participant bribes someone to change score
Technical vulnerability exploited to adjust scores
Ranges from just one score awry to whole exam invalidated
Threats Risk impact Real world example
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Robust delivery and results platform
ISO 27001 or similar security management
Store results in an online cloud system
Permissions / roles / segregation of duty
Robust audit trail
Care with results when leave the assessment system
How to mitigate tampering with results
Slide 46
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Risk
Unauthorized results disclosure
Slide 47
Analyze Results
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Unauthorized results disclosure
Slide 48
Mistake by administrators discloses data
Malware or other technical vulnerability discloses data
Serious embarrassment and negative publicity
GDPR fines
Invasion of privacy
Threats Risk impact Real world example
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Ensure that your supplier(s) have strong security▪ ISO 27001 is good
▪ Check they remain committed to security
Train your personnel well
Remove access from personnel who leave organization/project
Ensure your systems are well secured if results go there
Strong passwords and single sign on
Permissions / roles / least privilege
Mitigations
Slide 49
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
After delivery
Tampering with results Unauthorized disclosure of results
During test delivery
Identity fraudContent theft /
harvestingCopy answers from
another Getting help from
othersUnauthorized test aids
Before delivery
Content theft from item bank Disruption at test center
The 9 risks we have looked at
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Identify goals of your assessment program
Look at risks that apply to you
Implement mitigation measures based on importance of those risks
Improved security gives better data for better decisions
Improved security also increases trust from stakeholders
Summary
Slide 51
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Data Security
System Security
Slide 52
Trustable Results
Planning Assessment
Authoring Items
Assembling assessment
Pilot and Review
Delivery
Analyze Results
Trustable Results Rely on a Trustable & Secure Process
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Your questions
Slide 53
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Wednesday, December 5th
• Tips for Writing Good Test Questions
Tuesday, December 18th
• 10 Quick Tips to Improve your Tests and Exams
Weekly
• Introduction to Questionmark (Live product demos)
Upcoming Webinars
Slide 54
Sign up today at www.questionmark.com/webinars
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
How to Evaluate Questionmark
• Request a one-on-one demoThe Questionmark team will contact you to arrange a demonstration tailored to your needs and questionswww.questionmark.com/go/request-demo-enus
• Request a 30-day trial of Questionmark OnDemandComplimentary technical support during trialwww.questionmark.com/go/od30us
Slide 55
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
REGISTER NOW!
Questionmark Conference 2019Assess for Success
Slide 56
www.questionmark.com/go/conference
February 26th-March 1stHard Rock Hotel
San Diego"My team always walks away with new knowledge -
and we have been using the product for over 12 years!"
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Follow us…
Thank you for attending!
We hope to see you at a future webinarKeep up to date at blog.questionmark.com!