creating an in-house computerized adaptive testing (cat) program with concerto
DESCRIPTION
Presentation at JLTA (Japan Language Testing Association) 2013 National ConferenceTRANSCRIPT
![Page 1: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/1.jpg)
Creating an in-house computerized adaptive testing (CAT) program with Concerto
Atsushi, MIZUMOTO(Kansai University)
2013/09/20JLTA at Waseda University
![Page 2: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/2.jpg)
Computerized Adaptive Testing
![Page 3: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/3.jpg)
CAT needsItem Response Theory
![Page 4: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/4.jpg)
CTT vs. IRTAspect CTT IRT
Test score Ordinal scale Interval scale
Ability estimate Test-dependent Test-independent
Test result Person-dependent Person-independent
Measurement target (Precision) All test-takers Individuals
Equating/CAT Difficult Easy
Ohtomo (2009)
![Page 5: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/5.jpg)
CAT Needs IRT
CAT
IRT
IRT
IRT
![Page 6: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/6.jpg)
History of CAT Research
40 years (Thomson & Weiss, 2011))
30 in LT (Koyama, 2010))
![Page 7: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/7.jpg)
Example of CAT
![Page 8: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/8.jpg)
Example of CAT
![Page 9: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/9.jpg)
CBT ≠ CAT
![Page 10: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/10.jpg)
How CAT Works
http://www.j-cat.org/page/interpret
![Page 11: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/11.jpg)
Advantages of CAT
• Tailored for individual test-takers
• Shorter test time
• More precision (= SE smaller)
• No need for random sampling
www.geocities.jp/kosugitti/labo/irtnote.pdf
![Page 12: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/12.jpg)
Purposes
•Creating a CAT program
•Evaluation
![Page 13: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/13.jpg)
Creating a CAT Program
•Choosing the CAT System
•Constructing an Item Bank (Pretest)
•Calibrating the Item Bank
•Determine Specifications & Feedback
•Administering the CAT
![Page 14: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/14.jpg)
Creating a CAT Program
•Choosing the CAT System
•Constructing an Item Bank (Pretest)
•Calibrating the Item Bank
•Determine Specifications & Feedback
•Administering the CAT
![Page 15: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/15.jpg)
Moodle Plugin
http://moodle2x.info
![Page 16: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/16.jpg)
![Page 17: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/17.jpg)
![Page 18: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/18.jpg)
![Page 19: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/19.jpg)
1. Free account(150 test takers/month)
2. Amazon Machine Images(Free for a year)
3. Installing it on your own server
![Page 20: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/20.jpg)
• Open-source
• Running R on a server (catR, RMySQL)
• HTML-based
![Page 21: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/21.jpg)
Installation on a server
https://code.google.com/p/concerto-platform/wiki/installation4
![Page 22: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/22.jpg)
Wiki (Resources)
https://code.google.com/p/concerto-platform/wiki/Resources?tm=6
![Page 23: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/23.jpg)
Creating a CAT Program
•Choosing the CAT System
•Constructing an Item Bank (Pretest)
•Calibrating the Item Bank
•Determine Specifications & Feedback
•Administering the CAT
![Page 24: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/24.jpg)
Creating a CAT Program
•Choosing the CAT System
•Constructing an Item Bank (Pretest)
•Calibrating the Item Bank
•Determine Specifications & Feedback
•Administering the CAT
![Page 25: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/25.jpg)
Constructing an Item Bank (Pretest)
•Vocabulary Test (Mizumoto, 2006) http://www.mizumot.com/files/VocSizeMeasure.pdf
•Based on SVL 12,000 (Up to 8,000 level; 30 items for each level)
•716 university EFL learners
![Page 26: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/26.jpg)
Sample Question
(1) 心の, 精神の
A. essential
B. creative
C. loose
D. mental
![Page 27: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/27.jpg)
Creating a CAT Program
•Choosing the CAT System
•Constructing an Item Bank (Pretest)
•Calibrating the Item Bank
•Determine Specifications & Feedback
•Administering the CAT
![Page 28: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/28.jpg)
Creating a CAT Program
•Choosing the CAT System
•Constructing an Item Bank (Pretest)
•Calibrating the Item Bank
•Determine Specifications & Feedback
•Administering the CAT
![Page 29: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/29.jpg)
Calibrating the Item Bank
•240 items analyzed (Rasch model)
•150 items left for the item bank
•Calibrated with two parameter logistic model (item difficulty & discrimination)
•Update the csv file to Concerto
![Page 30: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/30.jpg)
Creating a CAT Program
•Choosing the CAT System
•Constructing an Item Bank (Pretest)
•Calibrating the Item Bank
•Determine Specifications & Feedback
•Administering the CAT
![Page 31: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/31.jpg)
Creating a CAT Program
•Choosing the CAT System
•Constructing an Item Bank (Pretest)
•Calibrating the Item Bank
•Determine Specifications & Feedback
•Administering the CAT
![Page 32: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/32.jpg)
Specifications of CAT
•Starting point (parameters, initial ability, randmized/fixed)
•Ability estimation method (empirical Bayes and others)
•Stopping rule (Number of items/Standard error)
•Final ability estimation
![Page 33: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/33.jpg)
Magis and Raîche (2012, p. 7)
![Page 34: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/34.jpg)
How many items for what SE?
• Simulation with catR package
Magis, D., & Raîche, G. (2012). http://www.jstatsoft.org/v48/i08
![Page 35: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/35.jpg)
True Theta = 1, SE = 0.3
Stopping rule = 30 items
![Page 36: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/36.jpg)
Concerto
![Page 37: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/37.jpg)
http://langtest.jp/concerto/?tid=20
![Page 38: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/38.jpg)
![Page 39: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/39.jpg)
Feedback Page
![Page 40: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/40.jpg)
Creating a CAT Program
•Choosing the CAT System
•Constructing an Item Bank (Pretest)
•Calibrating the Item Bank
•Determine Specifications & Feedback
•Administering the CAT
![Page 41: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/41.jpg)
Creating a CAT Program
•Choosing the CAT System
•Constructing an Item Bank (Pretest)
•Calibrating the Item Bank
•Determine Specifications & Feedback
•Administering the CAT
![Page 42: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/42.jpg)
268 test takers(university first year)
(1) CAT(2) Paper-pencil version (68 items) common person linking
(3) Questionnaire“What did you think of the CAT result?”
![Page 43: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/43.jpg)
Evaluation
CAT vs. Paper-pencil
![Page 44: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/44.jpg)
CAT Theta
0 1 2 3 4
-10
12
3
0.92
-1 0 1 2 3
01
23
4
Paper-pencil Theta
n = 268
Random30Qs
Fixed68Qs
![Page 45: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/45.jpg)
CAT Theta
0 1 2 3 4
-10
12
3
0.92
-1 0 1 2 3
01
23
4Paper-pencil Theta
n = 268
CAT (30Qs)M = 1.71SD = 1.13
P-P (68Qs)M = 1.72SD = 0.95
![Page 46: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/46.jpg)
CAT Theta
0 1 2 3 4
-10
12
3
0.92
-1 0 1 2 3
01
23
4Paper-pencil Theta
n = 268
CAT (30Qs)M = 1.71SD = 1.13
P-P (68Qs)M = 1.72SD = 0.95
Mean diff. = -0.0295% CI [-0.07, 0.04]
d = 0.01
Power = .06
![Page 47: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/47.jpg)
CAT Theta
0 1 2 3 4
-10
12
3
0.92
-1 0 1 2 3
01
23
4Paper-pencil Theta
n = 268
CAT SE (30Qs)M = 0.39SD = 0.11
P-P SE (68Qs)M = 1.71SD = 1.13
![Page 48: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/48.jpg)
CAT Theta
0 1 2 3 4
-10
12
3
0.92
-1 0 1 2 3
01
23
4Paper-pencil Theta
n = 268
CAT SE (30Qs)M = 0.39SD = 0.11
P-P SE (68Qs)M = 1.71SD = 1.13
Mean diff. of SE = -1.32
95% CI [-1.44, -1.19]
d = 1.65
Power = 0.99
![Page 49: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/49.jpg)
EvaluationCAT vs. Paper-pencil
Means: CAT = Paper-pencilSEs: CAT < Paper-pencil
CAT measures the same ability with much more precision
(with fewer items).
![Page 50: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/50.jpg)
Evaluation
Questionnaire
![Page 51: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/51.jpg)
Result of the Questionnaire
Frequency
Response
150 100 50 0 50 100 150
Very inaccurate Inaccurate Rather Inaccurate Rather accurate Accurate Very accurate
![Page 52: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/52.jpg)
Feedback Page
![Page 53: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/53.jpg)
Future Research
•More items in the item bank
•Better formula for predicting other test scores
• Improved feedback
•Collaboration
![Page 54: Creating an in-house computerized adaptive testing (CAT) program with Concerto](https://reader033.vdocument.in/reader033/viewer/2022052622/5590c5641a28abaa718b469b/html5/thumbnails/54.jpg)
Summary
•Created a CAT program
•Evaluation (1) CAT better than Paper-pencil (2) Feedback needs improvement.