study at a glance
DESCRIPTION
Assessing higher education learning outcomes globally Professor Hamish Coates [email protected]. Are valid and reliable comparisons of learning outcomes scientifically and practically feasible?. Study at a glance. An international feasibility study – not a ‘pilot’. - PowerPoint PPT PresentationTRANSCRIPT
Study at a glance
Are valid and reliable comparisons of learning outcomes scientifically and practically feasible?
Conceived ~2005, announced 06, scoping 06-09, work 10-13
An international feasibility study – not a ‘pilot’
Testing 23000 ‘bachelor degree’ students, with data from 5000 faculty, 25 national managers, 250 institutional coordinators, 17 policymakers, and hundreds of advisors and stakeholders
Designed, developed, validated and evaluated frameworks, tests, infrastructure and processes, etc.
HEI reports (not for systems), along with international reports and materials, and recommendations for a main study
Linguistically diverse…
Arabic Dutch EnglishFinishFlemish Italian Japanese
KoreanNorwegian Russian SlovakSpanish
Is it feasible to develop frameworks and instruments to test discipline-specific learning outcomes?
Test fields: Economics, Engineering
Contextual Dimension
Generic Skills Economics Engineering
StudentFaculty
InstitutionNational
Instrument architecture
• Outcome specification
• Document analysis• Consultation• Synthesis, review
Framework creation
• Gather existing materials
• Item workshops• Technical review• Framework mapping• Adaptation,
translation• Verification
Item creation
• Qualitative testing
• Quantitative testing
• OperationalisationInstrument validation
Framework and item
development Tuning AHELO / QAA frameworks Curriculum documents Accreditation systems Discipline research
Authentic, hybrid item types ‘Above content’ reasoning
Source version
Translation 1
Translation 2
Final check
National review
Reconciliation
Verification: verifier (linguist) and domain
specialist
Designed to maintain cross-national comparability of assessment materials
A holistic, robust and flexible approach, linked with item production and validation
Adaptations managed as a continuous processNative speakers of target languagetrained to detect specific pitfalls
Economists or engineers who are speakers of target language
Translation, adaptation
and verification
Psychometric reviews
3
6
14
8
17
19
19
6
22
3
14
3
8
25
22
19
22
28
14
22
28
33
6
11
53
36
31
36
25
36
42
28
22
44
31
42
33
28
31
33
33
17
25
39
28
17
40
39
0% 20% 40% 60% 80% 100%
The task made me apply knowledge and skills in real-world ways
The task required me to apply capability gained in my program
The task challenged me to think
The assessment materials were easy to understand
The task was relevant to future professional practice
The materials stimulated my interest in the task
There was good linkage between the questions in each task
The task was relevant to my program of study
The task assessed an appropriate range of knowledge and skills
The task was relevant to the content being assessed
The time available was sufficient for me to complete this task
The task covered topics relevant to my program
Per cent
Strongly disagree
Disagree
Undecided
Agree
Strongly agree
Student feedback for a task
Project implementati
on
Leadership: International project management and supporting national teams
Operationalisation: Preparing tests and context instruments for secure online delivery, and training makers
Sampling: Engaging institutions, sampling faculty and students, and quality assurance
Assessment: Supporting national training, managing testing, and managing marking quality
Reporting: Compiling data products, psychometric and statistical analysis, and system, institution and stakeholder reports
Evaluating: Scientific and practical feasibility, recommendations for full-scale study
0102030405060708090
100
50 150 250 350 450 550 650 750 850 950
Stud
ents
(cum
ulat
ive p
er ce
nt)
Generic Skills Score
This institution Mean of all institutions All institutions
Country communicatio
ns
Defined engagement cycle established to support systems, institutions and students… but not faculty
Designed sustainable business models for such work… but other modelsAssessment frameworks and test instruments developed to support multidimensional test/context instrumentationEstablished methods for test design, development, translation/adaptation and validationDefined operational workflow and quality control procedures required to support global testing
Forming awareness of how such work is positioned globally
Broad insights
Change horizons
AHELO
Institutional
positioning
Faculty improvemen
t
Policy research
Learner feedback
Stakeholder
engagement
System monitoring
