![Page 1: Changing issues, changing questions, changing approaches: An overview of research in the Alternative Admission Research Project](https://reader033.vdocument.in/reader033/viewer/2022052700/55a1b7a01a28abda398b4696/html5/thumbnails/1.jpg)
Changing issues, changing questions, changing approaches
An overview of research in the Alternative Admissions Research Project
![Page 2: Changing issues, changing questions, changing approaches: An overview of research in the Alternative Admission Research Project](https://reader033.vdocument.in/reader033/viewer/2022052700/55a1b7a01a28abda398b4696/html5/thumbnails/2.jpg)
The late 1980s to about 2001
The main questions were about:
• Access• Finding ways to identify talented students
from poor schools (focus on the DET)
![Page 3: Changing issues, changing questions, changing approaches: An overview of research in the Alternative Admission Research Project](https://reader033.vdocument.in/reader033/viewer/2022052700/55a1b7a01a28abda398b4696/html5/thumbnails/3.jpg)
(extract: Placement Test in English for Education Purposes construct)
• aim to predict the performance of candidates in future situations in which language will be an important, but not sole, variable (language-as-vehicle rather than language-as-target);
• acknowledge the effects on cognitive functioning of the quantity and quality of prior mediated learning opportunities experienced by an individual (i.e. take seriously the impact of educational disadvantage by building in mediated learning opportunities).
• be based on a notion of knowing and learning which views learners as actively involved as individuals and in collaboration with others, in creating and negotiating meaning in a wide variety of settings. This process of conceptual development is seen as highly dependent on specific areas of expertise involving knowledge and information, and on the connections between these.
• be based on a componential model of language ability, which comprises topical knowledge and language knowledge, mediated by strategic competence (metacognitive strategy use) and affective schemata.
![Page 4: Changing issues, changing questions, changing approaches: An overview of research in the Alternative Admission Research Project](https://reader033.vdocument.in/reader033/viewer/2022052700/55a1b7a01a28abda398b4696/html5/thumbnails/4.jpg)
The main assessment related challenges were:
• producing a greater range of scores (spread), producing a greater range of scores (spread), so that capable students could be more so that capable students could be more clearly differentiated from weaker studentsclearly differentiated from weaker students
• improving (raising) the level of stronger improving (raising) the level of stronger students’ scoresstudents’ scores
• increasing the predictive validity of the test increasing the predictive validity of the test (that is, did the test correctly distinguish (that is, did the test correctly distinguish between weaker and stronger students)between weaker and stronger students)
![Page 5: Changing issues, changing questions, changing approaches: An overview of research in the Alternative Admission Research Project](https://reader033.vdocument.in/reader033/viewer/2022052700/55a1b7a01a28abda398b4696/html5/thumbnails/5.jpg)
Towards ‘dynamic’ testing:the old ELPT and the new PTEEP
•The same three major writing tasks (summary, description and contrast/comparison)
•The same additional text types (graph and diagram)
The PTEEP included structured and sequenced tasks designed to act as mediation for the writing assignments. The ELPT did not include these ‘scaffolding’ tasks.
•The same prose texts
DifferencesSimilarities
![Page 6: Changing issues, changing questions, changing approaches: An overview of research in the Alternative Admission Research Project](https://reader033.vdocument.in/reader033/viewer/2022052700/55a1b7a01a28abda398b4696/html5/thumbnails/6.jpg)
ASSESSING THE IMPACT OF THE SCAFFOLDING APPROACH
Questionnaire evidence from pilot groups suggests that candidates found the new test to be more ‘user-friendly’.
Correlations between first-year academic (UCT) performance and the selection tests strengthened.
Stronger candidates use preparatory tasks whereas weaker candidates do not see the connections between tasks (such tasks therefore help to widen the gap).
Based on scores from the same tasks (summary, short description and one page essay), the range of scores increased.
Candidate perceptions
Predictive validity
Task preparation
study
Range of scores
![Page 7: Changing issues, changing questions, changing approaches: An overview of research in the Alternative Admission Research Project](https://reader033.vdocument.in/reader033/viewer/2022052700/55a1b7a01a28abda398b4696/html5/thumbnails/7.jpg)
ELPT and PTEEP Scores
0
100
200
300
400
500
600
No
. o
f C
an
did
ate
s
10 20 30 40 50 60 70 80 90 100
Test Results (%)
ELPT
PTEEP
![Page 8: Changing issues, changing questions, changing approaches: An overview of research in the Alternative Admission Research Project](https://reader033.vdocument.in/reader033/viewer/2022052700/55a1b7a01a28abda398b4696/html5/thumbnails/8.jpg)
Quantitative approaches used at this stage and in early 2000s
• Problems with correlational analysis• A move to ‘survival analysis’• Classical item analysis• Psychometrically ‘naïve’ tests relying on
ranking – limitations for score use• Reliance on group performance for
‘standards’
![Page 9: Changing issues, changing questions, changing approaches: An overview of research in the Alternative Admission Research Project](https://reader033.vdocument.in/reader033/viewer/2022052700/55a1b7a01a28abda398b4696/html5/thumbnails/9.jpg)
The early 2000s
• Expansion of “AARP” testing• Introduction of Quantitative Literacy test• Major innovations in the area of test
delivery (use of innovative technology etc)• Pressure on tests reduced ‘potential’
aspect• Development of Health Science
Consortium group of tests
![Page 10: Changing issues, changing questions, changing approaches: An overview of research in the Alternative Admission Research Project](https://reader033.vdocument.in/reader033/viewer/2022052700/55a1b7a01a28abda398b4696/html5/thumbnails/10.jpg)
2005 →the main questions …
• Concerns largely shifted from access to throughput
• Placement function emphasised – implications for curriculum (focus mainly on extended progs)
• Concerns about the new school curriculum and qualification – target now all applicants (mass test)
• Increasing tensions with Department of Education (with stronger FET curriculum links)
![Page 11: Changing issues, changing questions, changing approaches: An overview of research in the Alternative Admission Research Project](https://reader033.vdocument.in/reader033/viewer/2022052700/55a1b7a01a28abda398b4696/html5/thumbnails/11.jpg)
Issues and challenges
• Academic literacy/ies construct widened to include greater focus on quantitative literacy (and reading not writing the focus)
• For the first time direct testing in the school curriculum (Mathematics & Maths Literacy)
• Radically different development trajectories, research traditions for the new NBTs (achievement tests) and TAPs (re-designing the old PTEEPs)
![Page 12: Changing issues, changing questions, changing approaches: An overview of research in the Alternative Admission Research Project](https://reader033.vdocument.in/reader033/viewer/2022052700/55a1b7a01a28abda398b4696/html5/thumbnails/12.jpg)
Research approaches:quantitative
• From Classical Test Theory to Item Response Theory
• Psychometric test development approaches
![Page 13: Changing issues, changing questions, changing approaches: An overview of research in the Alternative Admission Research Project](https://reader033.vdocument.in/reader033/viewer/2022052700/55a1b7a01a28abda398b4696/html5/thumbnails/13.jpg)
CTT vs IRT (Hambleton and Jones, Comparison of CTT and IRT and their applications to test
development)
![Page 14: Changing issues, changing questions, changing approaches: An overview of research in the Alternative Admission Research Project](https://reader033.vdocument.in/reader033/viewer/2022052700/55a1b7a01a28abda398b4696/html5/thumbnails/14.jpg)
Item and Test Characteristic Curves; Test Information Functions
and Equating
0
2
4
6
8
10
12
14
16
18
20
-3 -2.5 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 2.5 3
Information
Theta
Test Information Functions for Forms QL1 and QL2
Form QL1 TIF
Form QL2 TIF
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
-3.0 -2.5 -2.0 -1.5 -1.0 -0.5 0.0 0.5 1.0 1.5 2.0 2.5 3.0
P(theta)
Theta
Test Characteristic Curves for Forms QL1 and QL2
Form QL2 TCC
Form QL1 TCC
0.00
0.20
0.40
0.60
0.80
1.00
1.20
-3 -2.5 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 2.5 3
P(U=1|THETA)
THETA
ICC Plots for 3-PL Form QL1 Series2
Series3
Series6
Series19
Series20
Series23
Series24
Series26
Series28
Series50
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
-3 -2.5 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 2.5 3
SUM of P(u=1 | THETA)
THETA
Test Characteristic Curve for 3-PL Form QL1
TCC form QL1
![Page 15: Changing issues, changing questions, changing approaches: An overview of research in the Alternative Admission Research Project](https://reader033.vdocument.in/reader033/viewer/2022052700/55a1b7a01a28abda398b4696/html5/thumbnails/15.jpg)
THE NATIONAL BENCHMARK
TESTS PROJECT
![Page 16: Changing issues, changing questions, changing approaches: An overview of research in the Alternative Admission Research Project](https://reader033.vdocument.in/reader033/viewer/2022052700/55a1b7a01a28abda398b4696/html5/thumbnails/16.jpg)
Why did HESA commission the NBTP?
2. Demonstrable inefficiencies in Higher Education itself (low throughput etc.)
• Difficulties in identifying students’ educational needs• Lack of appropriate curriculum flexibility at entry to meet these needs
2. Concerns about how to interpret the new NSC
![Page 17: Changing issues, changing questions, changing approaches: An overview of research in the Alternative Admission Research Project](https://reader033.vdocument.in/reader033/viewer/2022052700/55a1b7a01a28abda398b4696/html5/thumbnails/17.jpg)
In summary ….
• The NBTP is about higher education getting its own act in order – it is not about pointing fingers at the school system
• It sets out to do this by providing information about the competence of entering students in terms of 3 core domains of knowledge / skills
• it is important to note that higher education’s ‘take’ on what these core sets are, and at what level they should be mastered, will in all probability differ somewhat from those deemed most salient by the school-leaving system.
![Page 18: Changing issues, changing questions, changing approaches: An overview of research in the Alternative Admission Research Project](https://reader033.vdocument.in/reader033/viewer/2022052700/55a1b7a01a28abda398b4696/html5/thumbnails/18.jpg)
What do the NBTs aim to do?
– Provide additional information about performance in core, underlying areas (additional to NSC information)
– The core (domain) areas are:
• Academic literacy one 3-hr test • Quantitative literacy• Mathematics one 3-hr test
![Page 19: Changing issues, changing questions, changing approaches: An overview of research in the Alternative Admission Research Project](https://reader033.vdocument.in/reader033/viewer/2022052700/55a1b7a01a28abda398b4696/html5/thumbnails/19.jpg)
Performance in domain areas suggests that academic performance will not be adversely affected.
If admitted, students should be placed on regular programmes of study.
Challenges in domain areas identified such that it is predicted that academic progress will be affected.
If admitted, students’ educational needs should be met in a way deemed appropriate by the institution (eg extended or augmented programmes, special skills provision).
Serious learning challenges identified: it is predicted that students will not cope with degree level study without extensive & long-term support, perhaps best provided through bridging programmes or FET. Institutions registering students performing at this level would need to provide such support.
0%
100%
Proficient
Basic
Intermediate
![Page 20: Changing issues, changing questions, changing approaches: An overview of research in the Alternative Admission Research Project](https://reader033.vdocument.in/reader033/viewer/2022052700/55a1b7a01a28abda398b4696/html5/thumbnails/20.jpg)
Example - Competency Specification Proportions NBT Quantitative Literacy
Competence area
KnowingApplying routine
procedures in familiar contexts
Applying multistep procedures in a
variety of contexts
Reasoning and
reflecting
1+ 1- 2+ 2- 3+ 3- 4+ 4- n %
Comprehending: identifying or locating
Vocabulary5 0 7 7 2 4 1 5 31
19.0
15-20
Representations of numbers and operations
1 0 1 2 1 1 1 0 7 4.3 5-10
Conventions for visual representations
4 0 10 6 1 5 1 5 3219.6
20-25
Acting, interpreting, communicating
Using representations of data
2 0 6 3 1 4 1 4 2112.9
20-25
Computing (and estimating?)
0 0 4 10 2 4 0 2 2213.5
15-20
Conjecturing 0 0 0 0 0 0 0 1 1 0.6 0-5
Interpreting5 0 10 5 3 5 2 5 35
21.5
10-15
Reasoning 0 0 1 0 0 0 1 4 6 3.7 5-10
Representing quantitative information
0 0 3 1 0 0 2 0 6 3.7 5-10
Describing quantitative relationships
0 0 1 0 1 0 0 0 2 1.2 0-5
n 17 0 43 34 11 23 9 26 163
% 10.4 0.0 26.4 20.9 6.7 14.1 5.5 16.0
% 10.4 47.2 20.9 21.5
![Page 21: Changing issues, changing questions, changing approaches: An overview of research in the Alternative Admission Research Project](https://reader033.vdocument.in/reader033/viewer/2022052700/55a1b7a01a28abda398b4696/html5/thumbnails/21.jpg)
Mathematical and statistical ideas
Quantity, number and operations (LO1) 1 0 6 9 3 4 1 2 26
31.0
25-30
Shape, dimension and space (LO3) 3 0 4 3 0 2 1 1 14
16.7
10-15
Relationships, pattern, permutation (LO2a) 0 0 5 2 1 0 2 1 11
13.1
10-15
Change and rates (LO2b)0 0 4 2 1 0 1 1 9
10.7
10-15
Data representation and analysis (LO4a) 2 0 5 3 1 4 0 6 21
25.0
20-25
Chance and uncertainty (LO4b) 0 0 1 0 0 2 0 0 3 3.6 5-10
n
6 0 25 19 6 12 5 11 84
%
7.1 0.0 29.8 22.6 7.1 14.3 6.013.1
% 7.1 52.4 21.4 19.0
15-20 25-30 25-30 25-30
![Page 22: Changing issues, changing questions, changing approaches: An overview of research in the Alternative Admission Research Project](https://reader033.vdocument.in/reader033/viewer/2022052700/55a1b7a01a28abda398b4696/html5/thumbnails/22.jpg)
How are the benchmark (cut-off points) derived?
• The process is fundamentally different to the examination paper design procedures, and the norm referenced standardising and resulting processes of the NSC
• All items need to have been through rigorous review (fairness, content etc) and be statistically robust
• All items need to have been piloted
• The benchmark setting process is NOT about whether students can pass an item or not - the process is based on a set of probability assessments made by first year lecturers, with the core questions being:
“if a student can’t pass this item / do this, will s/he experience academic difficulties – and if so, how severe?”
![Page 23: Changing issues, changing questions, changing approaches: An overview of research in the Alternative Admission Research Project](https://reader033.vdocument.in/reader033/viewer/2022052700/55a1b7a01a28abda398b4696/html5/thumbnails/23.jpg)
NBT information
INDIVIDUAL LEVEL• Benchmark level (Basic, Intermediate, Proficient)• Description of what this means for each domain (ie what does being in the
‘Basic’ category mean a student knows and can do in Mathematics)• Clear recommendations about the type and extent of support needed
GROUP LEVEL• At the level of a faculty, or qualification, or institution ….• Give clear indication of the needs and strengths of entering cohorts, either
before entry, or at registration: useful for placement into existing courses, and/or with course design or modification.
![Page 24: Changing issues, changing questions, changing approaches: An overview of research in the Alternative Admission Research Project](https://reader033.vdocument.in/reader033/viewer/2022052700/55a1b7a01a28abda398b4696/html5/thumbnails/24.jpg)
DATA BASED ON
FEB 2009 PILOTS
![Page 25: Changing issues, changing questions, changing approaches: An overview of research in the Alternative Admission Research Project](https://reader033.vdocument.in/reader033/viewer/2022052700/55a1b7a01a28abda398b4696/html5/thumbnails/25.jpg)
ACADEMIC LITERACY (overall)
N = 12,202
Participating institutions: Mangosuthu, Rhodes, Stellenbosch, UCT, UKZN, UWC, Wits.
Serious learning challenges – long term, pre-tertiary intervention needed.
Challenges identified such that it is predicted that academic progress will be adversely affected. If admitted, students’ educational needs should be met in a way deemed appropriate by the institution (eg extended or augmented programmes, special skills provision)
Performance such that academic performance will not be affected.
If admitted, students should be placed on regular programmes of study.
851
55715780
0
1000
2000
3000
4000
5000
6000
7000
Total
ACADEMIC LITERACYNBT Benchmark Levels, February 2009
Basic
Intermediate
Proficient
![Page 26: Changing issues, changing questions, changing approaches: An overview of research in the Alternative Admission Research Project](https://reader033.vdocument.in/reader033/viewer/2022052700/55a1b7a01a28abda398b4696/html5/thumbnails/26.jpg)
ACADEMIC LITERACY by Faculty
250
63
289
22141
31
317
1256
119
1067
187
668
210
1179
2289
35
1393
340
575
142
1007
0
500
1000
1500
2000
2500
Commerce Education Engineering Health Humanities Law Science
ACADEMIC LITERACYNBT Benchmark Levels, February 2009
Basic
Intermediate
Proficient
![Page 27: Changing issues, changing questions, changing approaches: An overview of research in the Alternative Admission Research Project](https://reader033.vdocument.in/reader033/viewer/2022052700/55a1b7a01a28abda398b4696/html5/thumbnails/27.jpg)
Quantitative LiteracyN = 12,202
3055
6125
3022
0
1000
2000
3000
4000
5000
6000
7000
Total
QUANTITATIVE LITERACYNBT Benchmark Levels, February 2009
Basic
Intermediate
Proficient
![Page 28: Changing issues, changing questions, changing approaches: An overview of research in the Alternative Admission Research Project](https://reader033.vdocument.in/reader033/viewer/2022052700/55a1b7a01a28abda398b4696/html5/thumbnails/28.jpg)
Quantitative Literacy by Faculty
596
136
563
78
590
142
599
1807
77
1265
292
673
212
1235
1392
4
921
179121
29
669
0
200
400
600
800
1000
1200
1400
1600
1800
2000
Commerce Education Engineering Health Humanities Law Science
QUANTITATIVE LITERACYNBT Benchmark Levels, February 2009
Basic
Intermediate
Proficient
![Page 29: Changing issues, changing questions, changing approaches: An overview of research in the Alternative Admission Research Project](https://reader033.vdocument.in/reader033/viewer/2022052700/55a1b7a01a28abda398b4696/html5/thumbnails/29.jpg)
Mathematics (overall)
1644
7390
737
0
1000
2000
3000
4000
5000
6000
7000
8000
Total
MATHEMATICSNBT Benchmark Levels, February 2009
Basic
Intermediate
Proficient
![Page 30: Changing issues, changing questions, changing approaches: An overview of research in the Alternative Admission Research Project](https://reader033.vdocument.in/reader033/viewer/2022052700/55a1b7a01a28abda398b4696/html5/thumbnails/30.jpg)
Mathematics overall[Intermediate ‘Top’ and ‘Bottom’]
5233
2157
0
1000
2000
3000
4000
5000
6000
Total
MATHEMATICSNBT Intermediate Benchmark Level, February 2009
Intermediate Bottom
Intermediate Top
![Page 31: Changing issues, changing questions, changing approaches: An overview of research in the Alternative Admission Research Project](https://reader033.vdocument.in/reader033/viewer/2022052700/55a1b7a01a28abda398b4696/html5/thumbnails/31.jpg)
Mathematics by Faculty
910
23
289
43 50 30
455
2765
58
1956
411
68 80
1658
372 387
821
189
0
500
1000
1500
2000
2500
3000
Commerce Education Engineering Health Humanities Law Science
MATHEMATICSNBT Benchmark Levels, February 2009
Basic
Intermediate
Proficient