OECD feasibility study for an international Assessment of Higher Education Learning Outcomes (AHELO)

2012 Asia-Pacific Quality Network ConferenceSiem Reap Angkor, 29 February – 2 March 2012

Karine Tremblay, OECD

2

AHELO Rationale

Feasibility study overview

Emerging insights from Phase 1

Looking ahead

3

Higher education today…

Mass access

Concern for drop out(3 out of 10 students across the OECD)

Equity remains an issue

Internationalisation of HE and high-skilled labour markets

4

AHELO rationale

Despite huge progress in quality assurance, institutional quality remains largely unknown No perfect proxy Reputation race Rankings biased Satisfaction culturally sensitive Labour market outcomes problematic

Shared vision of OECD Education Ministers (2006) Shift from quantity to quality

So what?

Information vacuum filled by available informationLearning outcomes need to be taken into account Defining them (Tuning) Incorporating them in quality assurance Measuring them (AHELO)

Quality information needed for all sorts of reasons

5

Why a feasibility study?

Assess whether we can measure what undergrad. students know and can do upon graduation …

… across diverse countries, languages, cultures and types of institutions

Test the science of the assessment

Test the practicality of implementation

6

AHELO Rationale

Feasibility study overview

Emerging insights from Phase 1

Looking ahead

7

AHELO: 4 strands of work

Discipline strand in Engineering

Discipline strand in Economics

Several perspectives to explore the issue of value-

added (conceptually, psychometrics), building on similar work at school level.

Research-based “Value-added” or “Learning gain”

measurement strandGeneric skills strand

Exploring the feasibility of measuring learning outcomes in 2 contrasted

disciplines to prove concept

Critical to strive in 21st Century knowledge societies

• •

Observers: Bahrein Brazil Saudi Arabia Singapore

• Generic skills

• Economics

• Engineering

•

•

• •

• • •

•

• •

•

• • •

•

•

• •

•

•

• •

A range of geographic, linguistic and cultural backgrounds involved

•

9

Work to be undertaken in 2 phases

Generic Skills

Framework

EconomicsFramework

EngineeringFramework

Project management,survey operations and

analyses of results

Contextual dimension surveys

Frameworks

Instrument development & small-scale

validation

Generic Skills

Instrument

EconomicsInstrument

EngineeringInstrument

Implementation

Phase 1 -Initial proof of concept

Phase 2 -Scientific feasibility & proof of

practicality

Jan 2010-June 2011

Mar 2011-Dec 2012

Where we are now

10

Where do we stand now?

11

AHELO Rationale

Feasibility study overview

Emerging insights from Phase 1

Looking ahead

12

Feasibility of instrumentation (criteria)

Generic skills

Economics and Engineering

Framework development Instrument development and mapping to framework

Framework development Instrument development and mapping to framework

Contextual dimension Framework development Instrument development and mapping to framework

OK

OK

OK

OK

OK

ONGOING

13

Feasibility of implementation (criteria)

Overall Valid and feasible assessment design

Practically feasible study

Methodologically rigorous implementation

Engagement of faculties and students

Successful instrument delivery

Successful survey operations in countries and HEIs

Successful scorer training for comparability

Institution reports valuable for participating HEIs and countries

ONGOING

ONGOING

ONGOING

ONGOING

ONGOING

ONGOING

ONGOING

ONGOING

14

AHELO Rationale

Feasibility study overview

Emerging insights from Phase 1

Looking ahead

15

Beyond the feasibility study

Assuming positive outcomes of the feasibility study…

No preset outcome, but reflection on AHELO longer term development motivated by…

Qualitative proof of concept Scientific feasibility (quantitative/psychometric focus) Feasibility of implementation

Encouraging interim assessment to date Growing interest in the work worldwide, not only from Governments but from institutions and other stakeholders

16

Imagine 2017…

… From diagnosis to treatment/improvement at institutional level

Better information on student learning outcomes is the first step to improve teaching and learning for all: Provide evidence for national and institutional policy and practice Equip institutions with the method and tools to improve teaching

… Learning outcomes data feeding into external QA evaluations

Policy recommendations on Quality Assurance from the OECD Thematic Review of Tertiary Education (2008): Improve quality information base Increase focus on student outcomes

17

Enhance quality assurance

… Importance of contextual data and subsequent analyses

AHELO to go beyond snapshot diagnosis

To enable analysis of what is distinctive of high-performing institutions and identification of best practice

Context survey questionnaires from students, faculties and institution leaders

Towards identifying what works, for which students and in which contexts. Huge potential in terms of equity and responsiveness

18

Foster transparency and mobility

… Importance of international LO assessments for credit transfer, degree recognition and mobility

Towards better portability of HE credentials.

… Strengthen attractiveness and competitiveness of HE system

Policy recommendations on internationalisation from the OECD Thematic Review of Tertiary Education (2008): Enhance the international comparability of higher education Develop alternatives to current global rankings Improve information to prospective international students

19

Current sponsors

20

For more information, visitwww.oecd.org/edu/ahelo

Karine.Tremblay@oecd.org