caveon webinar series - lessons learned from using statistics to invalidate scores june 2014

1

Liz Burns, Juniper NetworksJennifer Semko, Baker & McKenzie

Jack Terry, NBEORachel Schoenig, ACT

Dennis Maynes, Caveon Test Security

Lessons Learned from Using Statistics to Invalidate Scores

June 19, 2014

Caveon Webinar Series presents:

2

Agenda for Today

• Question/Answer Format

• Please use the chat window to ask questions

• Conclusions

3

To Kick Off the Discussion…

Let’s chat about why an organization would use statistics to invalidate scores.

5

Q1

If a candidate hires legal counsel and contests a score invalidation, what are the three most important things you should do?

6

Q1 - Answer

1. Don’t second guess yourself. Remain confident.

2. Be prepared to provide legal counsel with a high level summary.

3. Bring your counsel on board in a timely manner.

If a candidate hires legal counsel and contests a score invalidation, what are the three most important things you

should do?

7

Q2

In your opinion, which statistics provide the most credible information concerning potential test fraud?

8

Q2 - Answer

• Old/New or EVT* Items• Trojan Horse Items• Value• Consistent measurement• Identify candidates who benefit• Can be automated• Easy to explain/understand

In your opinion, which statistics provide the most credible information concerning potential test fraud?

*EVT = Embedded Verification Test

9

Q3

When and how do you believe one should recommend score invalidations to your board or governing body?

10

Q3 - Answer

When? When you have a good faith basis for questioning the validity of a score

How? Provide a full and frank summary; give your Board the tools they need

When and how do you believe one should recommend score invalidations to your board or governing body?

11

Q4

Describe at least three things that should be avoided when using statistics to invalidate scores?

12

Q4 - Answer

1. Using complicated statistical explanations as rationale

2. Using complex charts or graphs to illustrate your point

3. Using the word ‘cheat’, ‘cheater’, or other emotionally-charged, accusatory language

Describe at least three things that should be avoided when using statistics to invalidate scores?

13

Q5

Based on your experience, what are pros and cons to using statistics to invalidate scores?

14

Q5 - AnswerBased on your experience, what are pros and cons to

using statistics to invalidate scores?

Pros• Statistics are powerful and

illustrative.• Statistics can be applied

consistently.• Statistics can provide you

with a “targeted action” approach.

15

Q5 - AnswerBased on your experience, what are pro’s and con’s

to using statistics to invalidate scores?

Cons• Staying ahead of breaches to

your program is challenging.• Statistics are not a “silver

bullet.”• Exam security requires a

comprehensive plan that is constantly evolving.

16

Q6

Describe a time when you were successful in using statistics to invalidate a test score (or scores). What factors contributed to yoursuccess? Have you ever used only statistics to successfully invalidate a score (i.e., there was no physical or other corroborating evidence of cheating besides the statistical results)?

17

Q6 - AnswerDescribe a time when you were successful in using

statistics to invalidate a test score.

• Studied similarity data, performed analysis

• Reviewed seating charts• Identified index values >7.5• Created a report and referred

to board

18

Q7

Describe a time when you had statistical evidence of aberration, but were unsuccessful in using that evidence to invalidate the score (e.g., you lost a legal dispute or you decided not to pursue the case further for some reason). Explain why you were not successful.

19

Q7 - AnswerDescribe a time when you had statistical evidence of

aberration, but were unsuccessful in using that evidence to invalidate the score. What happened?

• Lack of corroborating evidence

• Unable to convey the compelling nature of the statistical evidence

20

Q8

If a program has a solid testing agreement in place with proctors and candidates, how much time, effort, and money do you estimate is required to invalidate one test score based on statistics?

21

Q8 - AnswerIf a program has a solid testing agreement in place with proctors

and candidates, how much time, effort, and money do you estimate is required to invalidate one test score based on statistics?

This varies based on your organization and processes followed.

• Two examples from our panelists• Example 1: Three primary areas of

expense• Example 2: Internal, automated

process

22

Q9

What internal challenges and obstacles did you confront in instituting your invalidation program? Were they operational? Legal? Political? Please describe.

23

Q9 - AnswerWhat internal challenges and obstacles did you confront in instituting your invalidation program? Were they operational? Legal? Political? Please describe.

• Political – Invalidated scores belonged to customers and partners. Stakeholders had to know what to expect and why this was beneficial.

• Legal – Is process legally defensible? Do we have all the right documentation in place?

• Operational – Continually evolving and staying on top of technologies is key.

26

Q10

Have you instituted any KPIs or other measurements around your use of statistical analyses and invalidations to measure program impact and success? What are they?

27

Q10 - AnswerHave you instituted any KPIs or other measurements around your use of statistical analyses and invalidations to measure

program impact and success? • Metrics re: cases opened, closed,

cancel rates• Metrics re: resolution options

selected by examinee• KPIs to measure effectiveness of

program

28

Q11

What legal agreements or legal foundation need to be in place in order to consider invalidating based on statistics?

29

Q11 - AnswerWhat legal agreements or legal foundation need to be in place in order to consider invalidating based on statistics?

• Clear, comprehensive, signed candidate agreements

• Ethics policies in place• Have a plan, be

comprehensive• Court will look to your agreements

30

Audience Questions

31

Audience Questions

“We see a lot of exam takers who score very high in very short amounts of time. We know that it would be impossible to even read the questions in the short time spans, but it’s hard to prove cheating.

Is there a guideline on the amount of time taken per question that we can use to take these cheaters down?”

32

Audience Question - Answer

• Some reading rates exceed 1,000 wpm (trained speed reading)

• Average adult reading rates are 250-300 wpm on non-technical content

• Reading rates should be verified• High-speed, erratic reading

strongly indicates braindump usage

Is there a guideline on the amount of time taken per question that we can use to take these cheaters down?

33

Conclusions

“I would urge us to reframe our concerns about test data integrity not as cheating concerns, but as a validity issue.”

“I would first strongly caution everyone NOT to do any analyses at all until first developing and adopting policies and procedures about what to DO with the results of any analyses.”

– Gregory Cizek, NCSA, 2013

http://www.caveon.com/tilsa-test-security-guidebook-next-steps/

34

Conclusions“Perhaps the worst situation one could be in would be a situation where analyses have been conducted and it must be admitted publically that nothing has been done with the results.”

“It is important to treat each similarly situated case the same way, and a coherent, comprehensive set of policies and procedures, uniformly applied, is essential.”

– Gregory Cizek, NCSA, 2013

http://www.caveon.com/tilsa-test-security-guidebook-next-steps/

35

Thank you!

- Follow Caveon on twitter @caveon- Check out our blog…www.caveon.com/blog- LinkedIn Group – “Caveon Test Security”

Liz Burns, Juniper NetworksRachel Schoenig, ACT

Jennifer Semko, Baker & McKenzieDr. Jack Terry, NBEO

Dennis Maynes, Caveon Test Security