the kimep english entrance test (keet) validity … argument‐based validation for the kimep...
TRANSCRIPT
![Page 1: The KIMEP English Entrance Test (KEET) Validity … Argument‐Based Validation for the KIMEP English Entrance Test (KEET) Marina Gerassimenko, University of Leicester Kazakhstan Institute](https://reader034.vdocument.in/reader034/viewer/2022051509/5ae2f94b7f8b9ae74a8d1e39/html5/thumbnails/1.jpg)
An Argument‐Based Validation for the KIMEP English Entrance Test (KEET)
Marina Gerassimenko,University
of
Leicester
Kazakhstan Institute of Management, Economics and Strategic Research (KIMEP)
![Page 2: The KIMEP English Entrance Test (KEET) Validity … Argument‐Based Validation for the KIMEP English Entrance Test (KEET) Marina Gerassimenko, University of Leicester Kazakhstan Institute](https://reader034.vdocument.in/reader034/viewer/2022051509/5ae2f94b7f8b9ae74a8d1e39/html5/thumbnails/2.jpg)
Outline
BackgroundValidity as an argument
Toulmin’s structure of an argumentKane (1992, 2006) and Chapelle et al.
(2008)The KEET’s validity argumentSummary of the KEET’s validity argumentLessons learned
![Page 3: The KIMEP English Entrance Test (KEET) Validity … Argument‐Based Validation for the KIMEP English Entrance Test (KEET) Marina Gerassimenko, University of Leicester Kazakhstan Institute](https://reader034.vdocument.in/reader034/viewer/2022051509/5ae2f94b7f8b9ae74a8d1e39/html5/thumbnails/3.jpg)
Background
KIMEP: Kazakhstan Institute of Management, Economics and Strategic Research, Almaty, Kazakhstan, Central Asia
An English‐medium university
Degrees in economics, management, finance
87% undergraduate students; 13% graduate; 3.6% international
3000‐3500 applicants annually
![Page 4: The KIMEP English Entrance Test (KEET) Validity … Argument‐Based Validation for the KIMEP English Entrance Test (KEET) Marina Gerassimenko, University of Leicester Kazakhstan Institute](https://reader034.vdocument.in/reader034/viewer/2022051509/5ae2f94b7f8b9ae74a8d1e39/html5/thumbnails/4.jpg)
The KEET: paper‐based 100 items
multiple‐choice three sections – Grammar, Vocabulary in Use, Reading
Comprehension parallel forms
![Page 5: The KIMEP English Entrance Test (KEET) Validity … Argument‐Based Validation for the KIMEP English Entrance Test (KEET) Marina Gerassimenko, University of Leicester Kazakhstan Institute](https://reader034.vdocument.in/reader034/viewer/2022051509/5ae2f94b7f8b9ae74a8d1e39/html5/thumbnails/5.jpg)
A validity argument
“A validity argument should present and integrate evidence and rationales from which
validity conclusion can be drawn pertaining to particular score‐based inferences and uses of
a test.”
Chapelle (1999:263)
![Page 6: The KIMEP English Entrance Test (KEET) Validity … Argument‐Based Validation for the KIMEP English Entrance Test (KEET) Marina Gerassimenko, University of Leicester Kazakhstan Institute](https://reader034.vdocument.in/reader034/viewer/2022051509/5ae2f94b7f8b9ae74a8d1e39/html5/thumbnails/6.jpg)
An Argument Structure (Toulmin, 1958, 2003) :
WARRANT
Backing 1
Backing 2
Assumption
1
Assumption
2 Assumption
3
Backing 3
Grounds 1
Claim (Grounds 2)
Rebuttals
![Page 7: The KIMEP English Entrance Test (KEET) Validity … Argument‐Based Validation for the KIMEP English Entrance Test (KEET) Marina Gerassimenko, University of Leicester Kazakhstan Institute](https://reader034.vdocument.in/reader034/viewer/2022051509/5ae2f94b7f8b9ae74a8d1e39/html5/thumbnails/7.jpg)
Kane (1992, 2006):
Two Stages of Arguments
• An interpretative argument
specifies the proposed interpretations and uses of test results by laying out the network of inferences
and assumptions leading from the observed performances to the conclusions and decisions
based on the performances
• The validity argument
provides an evaluation of the interpretative argument
![Page 8: The KIMEP English Entrance Test (KEET) Validity … Argument‐Based Validation for the KIMEP English Entrance Test (KEET) Marina Gerassimenko, University of Leicester Kazakhstan Institute](https://reader034.vdocument.in/reader034/viewer/2022051509/5ae2f94b7f8b9ae74a8d1e39/html5/thumbnails/8.jpg)
Chapelle et al. (2008)Test use
Construct
Target
score
Expected
score
Observation
Observed
score
Target
domain
UTILIZATION (6)
EXPLANATION (4)
EXTRAPOLATION
(5)
GENERALIZATION (3)
EVALUATION (2)
DOMAIN DESCRIPTION (1)
![Page 9: The KIMEP English Entrance Test (KEET) Validity … Argument‐Based Validation for the KIMEP English Entrance Test (KEET) Marina Gerassimenko, University of Leicester Kazakhstan Institute](https://reader034.vdocument.in/reader034/viewer/2022051509/5ae2f94b7f8b9ae74a8d1e39/html5/thumbnails/9.jpg)
Step 1: The Domain Description Inference
WARRANT
1: The KEET is a reliable and valid test for
admission purposes; the test reveals important
English language skills and abilities
B 1.1 Tasks are
selected as
representative
of General
English domain
B 1.2 Important
skills were
selected based
on format and
administration
constraints
A 1.1 Important tasks
are identified
A 1.2 Performance on the
test reveals abilities and
skills relevant at English‐
medium university
A 1.3 Scores are not
affected by the test format
(multiple‐choice)
B 1.3 Scoring is
objective
Target
Domain
Observations
![Page 10: The KIMEP English Entrance Test (KEET) Validity … Argument‐Based Validation for the KIMEP English Entrance Test (KEET) Marina Gerassimenko, University of Leicester Kazakhstan Institute](https://reader034.vdocument.in/reader034/viewer/2022051509/5ae2f94b7f8b9ae74a8d1e39/html5/thumbnails/10.jpg)
Step 2: The evaluation InferenceWARRANT 2: The KEET’s observed scores
reflect targeted language abilities
A 2.1 Scoring
rubrics are
appropriate
as evidenceA 2.2 Task
administration
conditions are
appropriate evidence
for targeted abilitiesA 2.3 The KEET has
appropriate
psychometric
properties
Observations
Observed
score
![Page 11: The KIMEP English Entrance Test (KEET) Validity … Argument‐Based Validation for the KIMEP English Entrance Test (KEET) Marina Gerassimenko, University of Leicester Kazakhstan Institute](https://reader034.vdocument.in/reader034/viewer/2022051509/5ae2f94b7f8b9ae74a8d1e39/html5/thumbnails/11.jpg)
B2.1
Rubrics: grammar, use
of English, reading
comprehension provide
evidence for targeted
abilities
B 2.2 Task administration
conditions are standardized
by Admission Office
protocol
B 2.3 Acceptable for standardized tests
coefficients Alfa:
form A .935; form B .922
Both forms have distribution
close to normal
![Page 12: The KIMEP English Entrance Test (KEET) Validity … Argument‐Based Validation for the KIMEP English Entrance Test (KEET) Marina Gerassimenko, University of Leicester Kazakhstan Institute](https://reader034.vdocument.in/reader034/viewer/2022051509/5ae2f94b7f8b9ae74a8d1e39/html5/thumbnails/12.jpg)
Backing 3.3: The KEET psychometric data for forms A and B:
Scale Form A Form B
N of items 100 100
Sample size 232 241
Mean 58.375 54.095
Variance 279.467 232.584
Median 58.000 55.000
Std. Dev. 16.717 15.251
SEM 4.235 4.273
Mean p 0.584 0.541
Alfa 0.935 0.922
ITEMAN software Research Questions: How significant is the difference?Why do parallel forms have different means, variances
and SD?Do parallel forms have items of similar difficulty level?
![Page 13: The KIMEP English Entrance Test (KEET) Validity … Argument‐Based Validation for the KIMEP English Entrance Test (KEET) Marina Gerassimenko, University of Leicester Kazakhstan Institute](https://reader034.vdocument.in/reader034/viewer/2022051509/5ae2f94b7f8b9ae74a8d1e39/html5/thumbnails/13.jpg)
How parallel is parallel?
Research methods: (1) parallel test theory as part of Classical Test Theory(Kline, 2005):
•Modes•Means•Medians•Variances(2) t‐test: how significant is the difference?(3) Cohen d‐test: what is the effect‐size?
Findings:(2) T = 2.9108, df (degrees of freedom) is 471, 2‐tail confidence at99.62% the difference in means
is significant;
(3) Cohen’s d = 0.27 Cohen (1988) – not significant; Wolf (1986) –
practically significant
![Page 14: The KIMEP English Entrance Test (KEET) Validity … Argument‐Based Validation for the KIMEP English Entrance Test (KEET) Marina Gerassimenko, University of Leicester Kazakhstan Institute](https://reader034.vdocument.in/reader034/viewer/2022051509/5ae2f94b7f8b9ae74a8d1e39/html5/thumbnails/14.jpg)
Step 3: The Generalization Inference
WARRANT 3: Observed scores are indicative
of expected scores over parallel versions of
tasks and test forms
B 3.1 Secondary
school EFL
programme
B 3.2 The KEET’s
specifications:Three fixed rubrics;100 free items
A 3.1 Test domain is
sampled from
general English
proficiency area:
Problematic
Grammar Structures,
vocabulary, reading
A 3.2 Test
specifications as
test template
Observed score
Expected score
![Page 15: The KIMEP English Entrance Test (KEET) Validity … Argument‐Based Validation for the KIMEP English Entrance Test (KEET) Marina Gerassimenko, University of Leicester Kazakhstan Institute](https://reader034.vdocument.in/reader034/viewer/2022051509/5ae2f94b7f8b9ae74a8d1e39/html5/thumbnails/15.jpg)
R 3.2Test Specifications : fixed and
free elements
Backing for assumption 3.2Test Specifications
METHODOLOGY:
could free elements
in test specifications
cause test SD,
means vary?
RESEARCH
QUESTION:
p‐values compared
across sections of
grammar, vocabulary in
context and reading
comprehension and
across two parallel forms
![Page 16: The KIMEP English Entrance Test (KEET) Validity … Argument‐Based Validation for the KIMEP English Entrance Test (KEET) Marina Gerassimenko, University of Leicester Kazakhstan Institute](https://reader034.vdocument.in/reader034/viewer/2022051509/5ae2f94b7f8b9ae74a8d1e39/html5/thumbnails/16.jpg)
FINDINGS:
Grammar Form A Form B
Mean p‐value .66 .59
P‐value range .11 ‐
.91 .18 ‐
.97
Vocabulary Form A Form B
Mean p‐value .49 .57
P‐value range .09 ‐
.87 .17 ‐
.89
Reading
comprehensionForm A Form B
Mean p‐values .52 .34
P‐values range .38 ‐
.76 . 12 ‐
.60
![Page 17: The KIMEP English Entrance Test (KEET) Validity … Argument‐Based Validation for the KIMEP English Entrance Test (KEET) Marina Gerassimenko, University of Leicester Kazakhstan Institute](https://reader034.vdocument.in/reader034/viewer/2022051509/5ae2f94b7f8b9ae74a8d1e39/html5/thumbnails/17.jpg)
The KEET example
Form A
14. This time next week, we.....the chemistry exam.A.
had finished
B.
have been finishingC.
will have finished
D.
will have been finishing
Prop. Disc. Point
Correct Index Biser.0.61 0.31 0.24
Form B
14. Don’t worry – I ..... writing my report by Friday next week. I’ll give it to you by then.
A.
finishB.
will be finished
C.
am finishingD.
will have finished
Prop. Disc. PointCorrect Index Biser.0.72 0.56 0.50
![Page 18: The KIMEP English Entrance Test (KEET) Validity … Argument‐Based Validation for the KIMEP English Entrance Test (KEET) Marina Gerassimenko, University of Leicester Kazakhstan Institute](https://reader034.vdocument.in/reader034/viewer/2022051509/5ae2f94b7f8b9ae74a8d1e39/html5/thumbnails/18.jpg)
Reading ComprehensionForm A Form B
Length 600 700
Topics General interest Specific interest
Discourse features
Narrative Narrative
imbedded with
descriptions
Vocabulary General Specific
Question types Asking for explicit
informationAsking for
inferences
![Page 19: The KIMEP English Entrance Test (KEET) Validity … Argument‐Based Validation for the KIMEP English Entrance Test (KEET) Marina Gerassimenko, University of Leicester Kazakhstan Institute](https://reader034.vdocument.in/reader034/viewer/2022051509/5ae2f94b7f8b9ae74a8d1e39/html5/thumbnails/19.jpg)
Summary of the claims about the KEET
1st claim: the test purpose
2nd claim: the test construct
3d claim: the test specifications
4th claim: the item analysis
5th claim: the use of the test scores
![Page 20: The KIMEP English Entrance Test (KEET) Validity … Argument‐Based Validation for the KIMEP English Entrance Test (KEET) Marina Gerassimenko, University of Leicester Kazakhstan Institute](https://reader034.vdocument.in/reader034/viewer/2022051509/5ae2f94b7f8b9ae74a8d1e39/html5/thumbnails/20.jpg)
Conclusion: Lessons Learned
• The KEET’s construct• The test specifications• Analysis of parallel forms
• Item analysis
![Page 21: The KIMEP English Entrance Test (KEET) Validity … Argument‐Based Validation for the KIMEP English Entrance Test (KEET) Marina Gerassimenko, University of Leicester Kazakhstan Institute](https://reader034.vdocument.in/reader034/viewer/2022051509/5ae2f94b7f8b9ae74a8d1e39/html5/thumbnails/21.jpg)
Thank you for your attention!