outline teacher malpractice in assessment: the
TRANSCRIPT
30/03/2015
1
© UCLES 2015
Teacher Malpractice inAssessment: the InternationalContext
Ardeshir Geranpayeh, Ph.D.
Head of Psychometrics & Data Services
Oxford, 26 March 2015
Outline• Introduction
• Cheating (Malpractice):
- What, Where, How, Why, Who
- Consequences: Threat to Test Validity
• Standards and Regulatory Requirements
• How to avoid Cheating:– Prevention vs Detection
– Policies on punishment
• School Collusion: Teacher Malpractice
Cheating in the News
Driving test fraud:
More learner drivers
caught cheating,
BBC News, 26 Jan 2015
Cheating in the News
• 2,440 Chinese students caught cheatingin latest high-tech scam
Chinese state television reported on Sundaythat invigilators detected abnormal radiosignals from an illegal frequency duringnational licensing tests for pharmacists inShaanxi province. The independentWednesday, 28 October 2014
30/03/2015
2
Cheating in the News
According to The Harvard Crimson, a surveyof this year's incoming freshmen revealedthat 42 percent had cheated on high schoolhomework before arriving at the Ivy Leagueinstitution.
The Huffington Post,
18 February 2015
Cheating in the News
More than 50,000 English language teststaken by overseas students to extend theirBritish visas have been declared invalid orquestionable as a result of an officialinvestigation into cheating on a huge scale.
The Guardian, 24 June 2014
Cheating in the News
Cheating in Bihar:India arrests
Hundreds over Bihar
Blatant Cheating
In Schools
BBC News
21 March 2015
Cheating in the News
'Cheating teachers have lost morality':
Dirty tricks on
the rise to boost
exam grades
MailOnline
13 June 2014
30/03/2015
3
abcNews abcNews Poll1 in 3 theythemselveshave cheated.
Rising to 43%of older teens.
Most saycheaters don'tget caught.
What is Cheating?“Any action that violates the rules foradministering a test” Cizek, 1999:3
Where does it happen?
It can take a variety of forms
30/03/2015
4
Pen Scanners
There are YouTubevideos on How to Scan a
Document on the Go
Text Messaging
Examinee can askquestions and getanswers from friendduring test via textmessaging.
Toilet Breaks!
iPodSong names are renamedwith notes or test answersfor viewing on the screen.
Text files can be stored.
Audio notes can bestored.
Video notes can bestored.
Calculator
Notes are enteredinto calculators thathave memory forstoring notes.
30/03/2015
5
Camera Phone
Examinee takespictures of a test witha camera phone andsends picture toanother person whocan text messagecorrect answers back.
New Technology
Google Glass
Share your
adventures instantly
to Google+, Facebook,YouTube …
Apple Watch
Why does it happen?
–Material rewards such as
• Access to life chances
• Competitiveness
• Lack of self confidence
• Publication of league tables(Schools)
–Unintentional Cheating
• Students, material sharing
• Teachers and Schools
Who is involved?
• Maladministration
• Imposters
• Leaked paper / answers
• Prohibited material
• Copying
30/03/2015
6
What is at stake?– Threat to test validity
– Score obtained by fraudulent means is notvalid
– Has negative impact on the validity of scoresobtained by other candidates
– Denying opportunities to others
– Where cheating is seen to be widespread,even honestly obtained test results may losecredibility and certificates become devalued
What is at stake?
• Test Security is now seen as one of the mostimportant aspects of test Validity
• A Test may not be Valid if it is not secure
Standards for Prevention ofCheating
Explicit statements in the Standards for
Educational and Psychological Testing(1999)
Standards for Prevention ofCheating
1. Protect the security of tests (standard 11.7)
2. Ensure that individuals who administer the tests areproficient in administration procedures and understandthe importance of adhering to directions provided by thetest developer (standard 13.10)
3. Inform examinees that it is inappropriate for them tohave someone else take the test, for them to disclosesecure test materials, or engage in any other form ofcheating (standard 8.7)
30/03/2015
7
Standards for Prevention ofCheating
4. Ensure that test preparation activities and materialsprovided to students will not adversely affect the validityof test score inferences (standard 13.11) and
5. Maintain the integrity of test results by eliminatingpractices designed to raise test scores withoutimproving students, real knowledge, skills, or abilities inthe area tested (standard 15.9)
Cheating detection:Regulatory requirements
Ofqual’s conditions aroundmalpractice
• Ofqual is a risk-based regulator of qualifications(other than degrees) that sets the standardsthat awarding organisations need to meet whenthey design, deliver and award regulatedqualifications, via a number of conditions ofrecognition.
• Ofqual doesn’t tell AO’s how to managemalpractice. It tells AO’s what must result fromtheir behaviour.
Condition A8: Malpractice andmaladministration
Condition A8 of the General Conditions of Recognitioncovers:
Preventing malpractice and maladministration
Investigating and managing the effect of malpractice andmaladministration
Procedures relating to malpractice and maladministration
Dealing with malpractice and maladministration
30/03/2015
8
Examples of ‘Positive indicators’The awarding organisation: has ways of working that reduce risks and incidents
occurring makes sure that individuals involved in the delivery of its
qualifications understand and routinely follow these waysof working
takes all reasonable steps to provide information whichis not inaccurate or misleading
has policies, practices and/or procedures that reduce therisk of malpractice and maladministration
When Ofqual monitored Cambridge English LanguageAssessment they were happy with how we are managingmalpractice.
Examples of ‘Negative indicators’ Using an individual or Centre to assist with an
investigation when there is a suspicion or allegationthat they were connected to, or responsible for, thecase being investigated
Not demonstrating steps taken to prevent repeatincidents of previous cases
not following its own procedures during investigation not properly evaluating or validating evidence collected
during investigation not keeping records/documents relating to the incident
for an appropriate period not being able to enforce action taken against those
involved, and cannot demonstrate a good reason why ithas been unable to do so
How to avoid Cheating? How to avoid Cheating?
• The Best practice is “Prevention”
• No Statistical magic can replace goodinvigilation
• Tension between Faster/on demand resultsand proper data forensic processes
• Use variety of ID checks
– Candidate ID Verification
– Biomedical checks
– Hand writing
30/03/2015
9
Policies on punishment
• Once a cheating is detected, an action has tobe put in place to
• Stop fraudulent use of test results
• Deter future cheaters
• Punishment is dependent on the level ofcheating, which in turn can depend on 5 levelsof cheating detection
• Individual candidates, Group of candidates,Teacher Malpractice, Test Centre collusionand Widespread cheating
Level of punishment
• Withdrawing results/certificate (individual)
• Re-taking the exam (suspect results)
• Life Ban (if stakes is high or imposters)
• Informing stake holders (regulator)
• Legal action (insider)
What about School collusion:Teacher Malpractice
• Students implicated may not have beeninvolved in the cheating
• Whilst the candidate’s results may becancelled if school was to be blamed, nofurther action will normally be takenagainst candidates
Teacher Malpractice: the Atlanta case
• In 2009, Beverly Hall, Ex-Atlanta schoolsuperintendent, was named nationalsuperintendent of the year largely basedon her district’s improved standardisedtest scores
• She was once credited with turningaround some of the city’s lowest-performing schools
30/03/2015
10
Teacher Malpractice: the Atlanta case
• Biggest US cheating scandal in US History
• Cheating detected on a 2009 standardizedstate test involving 178 teachers andprinciples, 56 schools investigated cheated,43 people were indicted
• Prosecutors set out to reveal a culture of fearand pressure existed under Hall
• Georgia Governor determination to trace itssource
• Cheating traced back to 2001
Teacher Malpractice: the Atlanta case
• The scandal testifies that cheating is nolonger seen as an old-fashioned battlebetween teachers and students
• When the stakes are high, teachers wouldalso be willing to cheat
Final Remarks
• Use new psychometric techniques todetect school collusion
• Erasure analysis detection
• Multifaceted approach to monitor teachers’assessment
• Malpractice awareness among teachers
• Collaborative dialogue with teachers asthe guardians of students’ achievement