how to measure internationalisation – the european im(p)i project · 2010-06-21 · performance...

How to measure internationalisation –the European IM(P)I Projectthe European IM(P)I Project

Uwe Brandenburg

NAFSA conference, Los Angeles 2009

Situation in many EU countries until 21st century

� IOs with traditional tasks (mobility, int. Student enrolment,

counselling)

� International recruiting and marketing not an issue

� No tuition fees

of 37of 382

� No tuition fees

� EU programmes small and inefficient (80s and early 90s)

Uwe Brandenburg, Los Angeles, May 2009

The situation today

� Market pressure:– More exchange mobility

– international experience (study or internship) gained importance in career planning (from a nice add-on in the 80s and 90s to a must-have)

– Market growth produces pressure on HEIs to move into international recruitment and marketing and to enhance support services (notably in IOs)

� Reputation / public accountability:

of 37of 383

– Shanghai and THES ranking produce

– National rankings

� accountability:– Governments and board of trustees look for return on investment

– Target agreements are developed, increasingly including aspects of internationalisation

– IOs ill prepared for such a step (small budgets, few staff, low-profile, no int. strategies, no accountability)

� BUT: no coherent set of indicators for internal and external use!


Definitions

Internationality describes either an institution’s current status or the status discernible at the date of data acquisition with respect to international activities.

Internationalisation describes a process in which an institution moves, in a more or less steered process, from an actual status of internationality at time X

of 37of 384

from an actual status of internationality at time X towards a modified actual status of extended internationality at time X+N. In this instance, in the event of proper planning, the actual status is set against an expected target status. The result is then the difference between the actual situation after expiration of the period n and the desired situation after expiration of the period n.


If you look for indicators…

Availability is no reason to choose an indicator!

Availability is not a given fact but a result of priority-setting

The strategic outlook of an HEI has to overrule short- or medium-term improvement of indicators

Indicator and Key figure: the difference

of 37of 385

Focus: input and output indicatorsOutcomes are yet rather difficult or impossible to measure; however they should be in the focus on a later stage

Three possible levels:ResearchTeaching and StudiesOverall Aspects


Types of indicators

Which indicator types describe what?

Yes/no indicators: usually internationality

Time-series eligible indicators (e.g. numbers): internationality and internationalisation

of 37of 386

Usually snapshots and thus „internationality“ prevails

But: only process orientation guarantees quality enhancement (even simple quality assurance needs time series)

How to find out what to measure and why?


Methodological Considerations 3

Indicators are a very last step in the process

A useful approach:

1. definition of internationalisation targets

2. development of a coherent internationalisation strategy

3. short-, medium- and long-term measures ensuring the implementation and realisation of the internationalisation

of 37of 387

implementation and realisation of the internationalisation strategy

4. development of a quality management system that:

accompanying implementation of measures and their adjustment where applicable

documenting and analysing its influence on strategy targets

Focus on internal processes


Results from a German indicator project

Overall 186 key figures and indicators

170 eligible for time series

162 in input and process

24 output indicators186

indicators 170 162 Input

69Overall

24output 45

research

of 37of 388

24 output indicators

69 for „overall aspects“, 45 for „research“ and 72 for „teaching and studies“

Main effort for user: select the relevant indicators for the individual situation

indicatorsin

total

170 Time-lineEligible

indicators

Inputindicators

Overallaspects

72Teaching

Andstudies


Only for internal use or also system-wide?Internal use

Issues:has to consider cultural differences between departmentsSome departments are more international than others

Facilitators:Framework conditions in the institution are comparableInstitution-wide strategy dominates departmental strategy

External, system-wide use:

of 37of 389

External, system-wide use:Issues:

Diversity between institutions (size, conditions, types, strategies)No common understanding of internationalisation

Facilitators:Public pressureCommon national framework conditions (in Germany only partly)


General Use: Overall and Teaching/Study

Number of professors appointed from abroad relative to the total number of professors Number of international visiting researchers (minimum duration 1 week) Proportion of international students with non-German education relative to the total number of students Proportion of incoming international exchange students relative to the total number of students Proportion of outgoing exchange students and students with an

of 37of 3810

Proportion of outgoing exchange students and students with an international internship relative to the total number of students Number of lecture stays abroad undertaken by lecturers (Teaching Staff (TS) Mobilities) in relation to the total number of lecturers (Outgoing=Incoming) Proportion of graduates with joint or double/multiple degree relative to the total number of graduatesProportion of graduates of foreign nationality (international graduates with a non-German education) relative to the total number of graduates


General Use: Research

Number of professors appointed from abroad relative to the total number of professors

Amount of procured third-party funding from international sponsors in relation to the total sum of third-party funds per

of 37of 3811

sponsors in relation to the total sum of third-party funds per annum

Amount of third-party funding for international projects with international cooperation partners in relation to the total amount of third-party funding per annum


General Use: Others

Number of posts (full time equivalent=FTE) in the international core business (international strategy and basic questions, scientific cooperation, counselling and tutoring of students, alumni, admission) in relation to the total number of administrative posts

of 37of 3812

of administrative posts

Number of posts (full time equivalent=FTE) in the international areas of counselling and tutoring of students and admission in relation to the total number of students


However…

The responsibility for selecting the right indicators lies with the user

Different needs and interests to be considered (newspapers, ministries, companies, university leadership,

of 37of 3813

(newspapers, ministries, companies, university leadership, academics, students)

There is no one set of ultimately valid indicators!


Stipulations

Even a toolbox cannot be applied without the necessary institutional framework

IOs act in an institutional setting with individual players and their (known or

of 37of 3814

individual players and their (known or unknown) set of preferences (rational choice)


Stakeholders

Board of Trustees

University Leadership

Government / Ministries

Academics

of 37of 3815

Academics

Administration

Students

Business and Industry / Society


Expectations and perceptions

What is internationality / internationalisation?

Why is it important? Is it important?

What does it do for MY interests?

!?

! !

? ?!

?!

!?!

?

!

?

?!

of 37of 3816

What does it do for MY interests?

What is quality in internationalisation?

How does measuring it helps me in achieving MY goals?

…

?

! ?!

?! ??!

??!

?!?


Pressure of Internationalisation

BoardOf

Trustees

GovernmentMinistry

Accountability

Com

parability

Resultsdata

Academics

of 37of 3817

UniversityLeadership

BusinessAnd

Industry

Students

Opp

ortu

nitie

sre

puta

tion Administration

Resultsdata


The day-to-day dilemma…

Counter-productive: Internationalisation as lip service

Every university is a world-class universityBeing international and recruiting the best…

ad-hoc internationalisation:

Monday morning indicators

of 37of 3818

Monday morning indicatorsThe fast and furious…Administration: Give to the Emperor what the Emperor demands

Why is „more“ equivalent to „better“?

High international student numbersMany outgoingsMany…


What is needed…

Internationalisation Strategy based on a holistic approach

Strategy-based confident decisions: what to do and what NOT to do.

Profile building on own interests and perceptions

Be realistic:

of 37of 3819

Be realistic:

What is necessary, what a nice-to-have?Where are we? What are we?Priorities…

Good Examples:

TU BerlinInstitut des Études Politiques Paris


Once you have data… what now?

Rating

• often external benchmarks

• groupings• Goal: trust

building (investment)

Ranking

• indicators with general relevance

• League tables (exception

Benchmarking

• Internal and/or external benchmarks (depending on availability)

• No ranks, best

3 ways to go…

of 37of 3820

(investment) • famous in

business world and banking ( e.g. Standard and Poor‘s)

• Comparison against standards

(exception CHE ranking)

• Goal: transparency

• competition

• No ranks, best practices, possibly groupings

• Goal: enhancement (process or status)

• Cooperation, comparison


Why these principles?

Problem 1: whole universities

University X

Physics Political Sciences

of 37of 3821

Economy Medicine



Problem 2: weighted and accumulated indicators

• weights are entirely arbitrary and thus political

• A change of weights possibly changes the entire performance of an HEI (explains the jumps of some universities in rankings such as Shanghai, THES or

of 37of 3822

regional ones like USNews)

• If indicator results are accumulated to one numerical value this creates an artificial perception of accountability; however, this is an oversimplification

• Small differences in value in each indicator might create huge differences in overall value


Distribution of Results (SocSci and University

in THES 2008)

100

120


Problem 3: league tables

A difference of 4,3 points on a 100 point scale between place 33 and 43!

of 37of 3823

0

20

40

60

80

100

1 13 25 37 49 61 73 85 97 109 121 133 145 157 169 181 193

SocScie University

scale between place 33 and 43!

In the overall university ranking even just 6,7 points on a 100 point scale between place 21 and 41 and just 15,3 points between No.100 and No.200!


Limits of measurability

Not everything that can be counted, counts.

Not everything that counts can be counted.

of 37of 3824

Albert Einstein


Measurable or not that is here the question…

Not everything is measurable and needs measuring:

The famous „world-class university“:

reputation can NOT be measured but felt

Nobody is excellent in everything

Indirect means: in which networks do you act, which

of 37of 3825

Indirect means: in which networks do you act, which university approaches you, where do the students come from and why…

But most achievements are measurable:

E.g.: Goal „increase internationalisation at home“ –Possible measurements „international faculty / international students time series-based“


The eternal quest for quality

What is quality in internationalisation?

Question of perspective…

In general:

The level of quality in internationalisation is equivalent to the congruence between stated goals, created

of 37of 3826

to the congruence between stated goals, created expectations, experienced conditions and remaining outcomes as perceived by the providers (leadership, academics, administration) and the recipients (students, academics, ministries, etc.).


The volcano of relative and absolute quality relative quality:

Perceived „gap of quality“

of 37of 3827

absolute quality:

threshold of minimum

quality


Some „golden“ rules for benchmarking based on results from the EBI project

1. Benchmarking partners should

• Be selected based on a shared understanding of the benchmarking goals, fields and comparisons, which may or may not rely on existing inter-institutional contacts

• Have a clear and communicated understanding of the expected degree of involvement (time, human and financial resources) from the start

• Ensure a high level of trust within benchmarking networks, as

of 37of 3828

• Ensure a high level of trust within benchmarking networks, as sensitive data will be exchanged

• Obtain commitment from senior management of all partner higher education institutions

Internationalisation specifics:

Consider cultural differences between partner countries

Knowledge about the partner is more crucial than in national

settings


2. Purpose and goals should

• Be explicit, both internally and externally with benchmarking partners

• Link to the national/European context of quality assurance and measurements of performance

• Connect closely to the institution’s strategies and to the development of a benchmarking and quality culture


of 37of 3829

• As a result, the benchmarking exercise must have a clear focus


Communication might be very different between countries

Intercultural aspects should be considered, particularly with new unfamiliar partners


3. Indicators should

• Not only include inputs but also outputs and/or processes

• Be quantitative as well as qualitative, as most issues are best compared by using a mix of quantitative and qualitative methods

• Be selected according to the relevance for the purpose, not solely on existing data


of 37of 3830

existing data

• Measure outcomes in relation to inputsInternationalisation specifics:

Many data does not work between countries, such as university budgets or salaries (Experience from ExcellenceRanking)

Some data on internationalisation is not collected


4. Procedures/operations should

• Be clearly documented using a transparent methodology which is communicated both inside the institution and among benchmarking partners

• Be supported with adequate human, financial and other resources to carry out the benchmarking exercise

• Where appropriate, be implemented by carefully selected and trained experts in assessment/evaluation work

• Be established so as to process data in a well-structured way


of 37of 3831

• Be established so as to process data in a well-structured way

• Ensure a periodical monitoring/review of the effectiveness of the benchmarking process and its value in implementing changes at the relevant level within the participating institutions (measuring outcomes


Different communication mentalities (between and within HEIs)

Some areas such as ERASMUS are rather standardised, others such as individual exchange or cooperation not and need different approaches


5. Reporting results should

• Be carried out in an effective way (internal/external)

• Produce well-structured, transparent and comparable information (qualitative/quantitative) with a view to identifying good practices and gaps in performance which can lead to future target-setting

• Apply measures to enhance the credibility and the visibility of the benchmarking exercise



of 37of 3832

Internationalisation specifics:Internationalisation is a cross-sectoral activity and implementation needs much more procedural preparation and commitment on various levels than uni-sectoral benchmarking (e.g. budgeting)Internationalisation is increasingly political and under change, i.e. needs more re-adjustmentsBenchmarking results can be used very effectively against national bodies/ministries as well as for marketing


Finally…

In internationalisation, benchmarking seems to be more necessary than widely believed

Due to the subject any benchmarking in this area should be international by nature (members)

Due to the complex and shifting reference

of 37of 3833

Due to the complex and shifting reference system, benchmarking initiatives in this area should be started with considerable time allocations and patience

Special aspect: internationalisation of systems could also be compared (Bologna stocktaking report an example)


Thank you!

of 37of 3834

[email protected]


More information

1. Target country analysis: http://www.che.de/downloads/Analysing_the_Future_Market_Target_Countries_for_German_HEIs_AP107.pdf

2. Indicator development project: http://www.che.de/downloads/How_to_measure_internationality_AP_92.pdf

3. Higher Education in China:

of 37of 38

3. Higher Education in China: http://www.che.de/downloads/Higher_Education_in_China_AP97.pdf

4. Developments in recruitment: http://www.international.ac.uk/resources/International%20Focus%20issue%2024.10.09.08.pdf

5. ExcellenceRanking Methodology Round 1: http://www.che.de/downloads/CHE_ExcellenceRanking_2007_AP99.pdf


How to measure internationalisation –

the European Indicators for Mapping

and Profiling Internationalisation

IM(P)I Project

Uwe Brandenburg (CHE Consult)

Adinda van Gaalen (NUFFIC)

NAFSA conference, Los Angeles 2009

of 37of 382

Starting point of IM(P)I

The German partners felt a need to internationalise the

project

NUFFIC and SIU were also active in the field and joint in

the very beginning

In discussions with various European HEIs the need for a

European approach was becoming obvious


of 37of 383

How to select partners?

It should be collaborative partners of the first initiators

Whereas the first approaches in Germany (HEIs plus a

research centre), and the Netherlands and Norway (national

agencies) were rather mono-type-based, there should be

different types of partners (national agencies, transnational

agencies, consultancy, HEIs, network organisation)

Regional coverage of Europe

The HEI partners should mirror the diversity approach in

Germany (private-public, small-large, technical-full university,

university of applied sciences)

All partners should be known as serious workers and strongly

engaged in the issue


of 37of 384

Current group:

Transnational: ACA

National Agencies: CampusFrance, DAAD, NUFFIC,

Perspektywy Foundation, SIU

HE Consultancy: CHE Consult

HEI network: Coimbra group

Individual HEIs:

Denmark: Copenhagen University; Finland: TAMK University of Applied

Sciences; Germany: Berlin Institute of Technology, Catholic University

Eichstätt, LMU Munich, University of Applied Sciences

Oldenburg/Ostfriesland/Wilhelmshaven; Hungary: Budapest Technical

University, Italy: Università Cattolica del Sacro Cuore Milano; The

Netherlands: Hanze University of Applied Sciences, Leiden University;

Poland: Warsaw University of Technology; Portugal: Instituto Europeu

da Faculdade de Direito de Lisboa; UK: University of Southampton


of 37of 385

Aims of IM(P)I:

To compile an indicator list inclusive of most existing lists

To provide a toolbox for medium and large group internal

comparisons or check of internationalisation

To provide ideas for a structured strategy approach

(objectives, activities, indicators)

To develop three examples of application:

Individual institutional

Small benchmarking group (3-5)

Large benchmarking group (>10)

Find examples of good practice for processes

Define ways to use objectives to choose activities, services,

QA measures


of 37of 386

Next steps:

Application to the EU launched under ERASMUS

modernisation

Develop a functional matrix combining objectives, activities,

and indicators on the one hand with the core action fields of

HEIs (research, teaching/learning, services)


of 37of 387

Thank you!

[email protected]


how to measure internationalisation – the european im(p)i project · 2010-06-21 · performance...

Documents