confidence-based assessment of two-alternative format tests ahmed a. belal (1) & diala f.ammar...
DESCRIPTION
Confidence-Based Assessment of Two-Alternative Format Tests AHMED A. BELAL (1) & DIALA F.AMMAR (2) ( 1)Computer Engineering and Informatics- Beirut Arab University, Lebanon (2)Psychology department, Lebanese American University, Lebanon [email protected]. Computer Aided Assessment - PowerPoint PPT PresentationTRANSCRIPT
Confidence-Based Assessment of
Two-Alternative Format Tests
AHMED A. BELAL (1) & DIALA F.AMMAR (2)
(1)Computer Engineering and Informatics- Beirut ArabUniversity, Lebanon
(2)Psychology department, Lebanese American University,Lebanon
•Computer Aided Assessment
•Multiple Choice Exams :
No Partial Credit
Penalty for Wrong Answers
•Designing good distracters
•Multiple Select Exams
Penalty for Wrong Answers
•Multiple T/F Exams
Question may have no correct answersNo partial credit
• Individual T/F questions Weak resistance to guessing
Correct =+ XNo Answer = 0Wrong Answer= -Y
Poor discrimination
•Confidence-based Assessment
Correct 0 1 2 3 Grade 0 10 20 30
Correct 0 1 2 3 Grade -15 0 15 30 Scaled 0 15 30 45
Correct 0 1 2 3 Grade -30 -10 10 30 Scaled 0 20 40 60
+100
+10-5
+10-10
9 Different Grades
7 Different Grades
Correct 0 1 2 3 0 1 2 1 0 0 Wrong 3 2 1 0 2 1 0 0 1 0 No answer 0 0 0 0 1 1 1 2 2 3
Grade -15 0 15 30 -10 5 20 10 -5 0 Scaled 0 15 30 45 5 20 35 25 10 15
Grade -30 -10 10 30 -20 0 20 10 -10 0 Scaled 0 20 40 60 10 30 50 40 20 30
Correct +10 Wrong -5 No answer 0
Correct +10 Wrong -10No answer
0
Hard to correctly estimate one’s confidence level
Humans are more oriented to judge their uncertainty in things relative to each other
Leclereq [1,2]
Confidence Level Correct Answer Wrong Answer
0 0 0
1 +3 -1
2 +4 -2
3 +5 -5
• Student ranks questions relative to each other.
• Value of correct answer decreases as level of confidence decreases.
• Wrong answers change the confidence level.
• Use a step size and a reduction function to assign values for a correct answer.
Answer Vector
000 001 010 011 100 101 110 111
R=5 0 0 5 10 10 15 20 30
R=10 0 -10/0 0 0 10 10 20 30
R=15 0 -20/0 -5/0 -10/0 10 5 20 30
S=1
Confidence Level C1 C2 C3 C4 C5
Reduction Value 2 10 8 6 4 2
Reduction Value 5 10 5 0 -5 -10
10 Questions : answer vector 1101001100
Step size = 1R =2 Score = 10+10+0+8+0+0+4+4+0+0 = 36
R = 5 Score = 10+10+5 -5-5 = 15
Step size = 2R =2 Score = 10+10+10+8+8 = 46
R = 5 score = 10+10+10+5+5 = 40
Absolute (10-2p) (10-p) (10-5p) R=1;s=2 R=2;S=2 Answer Vector
16 24 32 10 37 34 100101100038 44 47 35 50 50 110111000025 40 50 15 56 52 101011101024 30 40 10 47 44 011010110062 64 72 40 76 72 111100111180 90 90 90 90 90 111111111062 66 73 45 80 80 101111111037 42 46 25 50 50 101111000018 32 36 20 39 38 101101000051 64 72 40 80 80 011111111030 50 60 30 66 62 111001101186 78 84 60 90 90 111011111146 42 56 20 65 60 110010111130 36 38 30 40 40 110110000058 64 67 55 70 70 111101110049 62 66 50 70 70 111011110070 58 64 40 70 70 101111110038 42 46 30 50 50 101111000016 22 36 5 45 40 010011101041 40 45 25 50 50 011111000016 24 32 5 37 34 01011010000 24 32 0 27 24 00111000001 22 26 10 29 28 0110100000
29 30 35 20 39 38 10110010004 34 37 30 39 38 1110001000
28 30 40 5 46 42 010111010054 62 66 50 68 66 111110011040 52 61 30 66 62 1110011101
Table 1
Strategy Average Standard Deviation
Absolute 37.46428571 22.57084672
10-2P 45.28571429 18.21854396
10-P 51.75 17.3901313
10-5P 29.46429 20.19989
R=1 S=2 56.14286 17.98912
R=2 S=2 54.42857 18.44597
Table 2