thinking skills tests for university admission
DESCRIPTION
Thinking Skills Tests for University Admission. Assisting Admissions Decisions Robert Harding Director TSAT Project and ITAL Unit Interactive Technologies in Assessment and Learning http://tsa.ucles.org.uk/. Thinking Skills Assessment : Recent history in Cambridge. - PowerPoint PPT PresentationTRANSCRIPT
CAA 2004 1
Thinking Skills Tests forUniversity Admission
Assisting Admissions Decisions
Robert HardingDirector
TSAT Project and ITAL UnitInteractive Technologies in Assessment and Learning
http://tsa.ucles.org.uk/
CAA 2004 2
Thinking Skills Assessment :Recent history in Cambridge
Need for additional information to select applicants BMAT: BioMedical Admissions Test:
now used also by Oxford and UCL, Royal Vet College About 4000 candidates in 2003, expecting 6000 in 2004
TSA numbers (used in some Cambridge Colleges): 289 in 2001 478 in 2002 Over 1500 in 2003
There is evidence that in the Cambridge context the test is a better predictor of performance than either A-Levels or interview scores.
CAA 2004 3
Some admissions tests in ROW
The United States Medical Licensing Examination (USMLE)http://www.nbme.org/about/about.asp
The Law School Admission Test (LSAT)http://www.lsac.org/
Graduate Medical School Admissions Test (GAMSAT)http://www.acer.edu.au/tests/university/gamsatuk/intro.html
“The SAT is a three-hour test that measures verbal and mathematical reasoning skills students have developed over time and skills they need to be successful academically.”http://www.collegeboard.com/student/testing/sat/about/SATI.html
CAA 2004 4
BMAT :what is tested?
Section 1 Thinking Skills: 40 multiple choice questions, 1 hour
Section 2: Science and Maths core knowledge:30 questions, 30 mins
Section 3 essay: capacity to develop ideas and to communicate them effectively in writing30 mins.
CAA 2004 5
Thinking Skills Assessment :what is tested?
Critical Thinking and Problem Solving 50 multiple choice questions, online or on paper One ‘paper’: test lasts 90 minutes Origins in academic research around mid 1980’s MENO 1993/4 tested six ‘thinking skills’ Today:
OCR’s AS Level in Critical Thinking CIE’s AS Level in Thinking Skills
Evidence that these skills are highly relevant to successful study.
CAA 2004 6
Thinking Skills Assessment :Is it useful?
“The TSA Project has been running on a pilot basis since 2001 in Cambridge. Its results are used carefully and conditionally by Colleges as additional information and never replace any traditional criteria on which admission decisions are based. More research and evaluation is being done in this pilot project to establish its value for admissions purposes. Although this research is not yet complete, the use of the TSA test within Cambridge has grown each year as confidence in its usefulness grows.”
CAA 2004 7
Thinking Skills Assessment :what does this do for ‘access’?
Provides a uniform measure of readiness for Higher Education
Can be self-tested; encourage applicantsLow cost support materials widely
available, eg via wwwWorth learning these skillsFacilitates access research.
R = 0.33
0
50
100
150
200
250
300
350
0 10 20 30 40 50 60 70 80 90
TSA score
Trip
os m
ark
Data for 48 candidates who took the TSA in December 2001 and their first University examination (the Tripos) in the summer of 2003, all Computer Science.
2001/2003 data for 48 candidates, showing the top 12 TSA scorers and the bottom 12 TSA scorers.
R = 0.33
0
50
100
150
200
250
300
350
0 10 20 30 40 50 60 70 80 90
TSA score
Trip
os m
ark
2001/2003 data for 48 candidates, but Tripos marks replaced by randomised values.
R = -0.002
0
50
100
150
200
250
300
350
0 10 20 30 40 50 60 70 80 90
TSA score
Trip
os m
ark
Scatter plot of 2002 TSA score vs 2004 Tripos score:478 candidates took the TSA, 83 went on to 1st year exams in Natural Sciences, Computer Science, and Mathematics courses.
0 10 20 30 40 50 60 70 80 90 100
Overall Mark
0
50
100
150
200Fr
eque
ncy
Mean = 60.62Std. Dev. = 8.216N = 1,551
TSA 2003 total scores - distribution
0 10 20 30 40 50 60 70 80 90 100
Critical Thinking
0
50
100
150
200
Freq
uenc
y
Mean = 59.23Std. Dev. = 10.005N = 1,551
0 10 20 30 40 50 60 70 80 90 100
Problem Solving
0
50
100
150
Freq
uenc
y
Mean = 62.8Std. Dev. = 10.227N = 1,551
TSA 2003 scores distribution
Critical Thinking Problem solving
CAA 2004 14
What’s the CAA perspective?
Whole process is computer-assisted: Item banking & test construction Administration and registration Answer sheet scanning (ICR) Results return and reporting System developed by Cambridge’s CARET
(Centre for Applied Research in Educational Technology) To be made Open Source, QTI compliant
MCQ’s, so can be taken on paper or on-line: Can compare on-line & paper performance.
CAA 2004 19
What’s the CAA perspective?
Whole process is computer-assisted: Item banking & test construction Administration and registration Answer sheet scanning (ICR) Results return and reporting System developed by Cambridge’s CARET
(Centre for Applied Research in Educational Technology) To be made Open Source, QTI compliant
MCQ’s, so can be taken on paper or on-line: Can compare on-line & paper performance.
Delivery Mode
Score type N Mean Std Dev
Paper Total 1114 60.4 8.3
Prob. Solv. 1114 62.5 10.2
Crit. Think. 1114 59.0 10.1
Online Total 437 61.0 8.1
Prob. Solv. 437 63.4 10.2
Crit. Think. 437 59.5 9.6
CAA 2004 21
TSAT in 2004 and Beyond? UCLES is planning to continue pilot testing the TSA in Cambridge, on
a similar basis to 2003
The BMAT continues in 2004 with additional universities We will continue to consult interested parties and stakeholders
We will continue to develop the technology
We may look for interest from other Universities, and will consider
overseas testing We will look at issues of timing and location We will continue to research admissions issues.
CAA 2004 22
Some URL’s:
http://www.ucles.org.uk/http://tsa.ucles.org.uk/
http://bmat.org.u/http://www.caret.cam.ac.uk/
Thinking Skills Admissions Testing
Acknowledgements and thanks …
To many colleagues in UCLES and to our partners in CARET who played very full parts in this work.