searching for globally feasible indicators from domestic rankings

Searching for Globally Feasible Indicators from Domestic Rankings

Ya Lan Xie and Ying ChengGraduate School of Education,

Shanghai Jiao Tong University, China

The Fourth Meeting of International Ranking Expert Group14 - 16 June 2009, Astana, Kazakhstan

Background

• The new global rankings have been published one after another in recent years

• Rankings have problems and global rankings have additional problems, because“Not all nations or systems share the same values and beliefs

about what constitutes ‘quality’ in tertiary institutions, …”5th Item of Berlin Principles

• Consequently, very limited indicators have been used in global rankings

Aim of the study

• Are there some indictors used by domestic rankings that can also be used for global rankings?– Analysis of their International comparability– Examination of their relevance to institutions’

quality

Samples: 11 rankings in 8 countries Title of Ranking Country

Melbourne Institute Index of the International Standing of Australian Universities 2005 Australia

Sunday Times University League Tables 2008 UK

Guardian University Guide 2008 UK

U.S. News America’s Best Colleges 2008 USA

The Top American Research Universities 2007 USA

Maclean's Rankings 2007 CanadaComprehensive Competitiveness Ranking of Chinese Key Universities 2007 China

Netbig Ranking 2008 China

Perspektywy University Ranking 2002 Poland

Slovak Ranking 2007 Slovakia

Romanian University Ranking 2007 Romania

Assignation of indicators into 63 categories

Classification of indicator categories according to their international comparability and universality

Group 1: Comparable at a global level

• Those indicators – EITHER can show universities’ global competence,

e.g.• Publications and citations in international indexes• Globally selected top scholars, such as winners of

International awards, highly cited researchers

– OR are measurements of internationalization• Proportion of international students• Research income from international source

Group 2: Fairly comparable at a global level

• The actual meaning of an indicator is relatively similar in different higher education systems, e.g.– Degrees granted (Bachelor, Master’s, Doctor)– Employment rate of graduates– Spending per student– Domestic publications and patents– Total research income– Proportion of faculties with the highest degree– Students/staff ratio– Student evaluation/satisfaction– ……

Group 3: Hardly comparable at a global level

• The effectiveness of an indicator is heavily affected by different higher education systems, e.g.– Acceptance rate– Number of Master’s or Doctoral programs– Alumni giving rate– Graduation rate– Retention rate– Proportion of full-time faculty– Assessment by domestic administrators/scholars

Group 4: Uncomparable at a global level

• Not universities in all countries have such indicator, e.g.– National Academy Membership– Faculty who won major national awards– Excellent course/textbooks, nationally selected– Excellent programs, nationally selected– Excellent research labs/centers/units, nationally selected– Excellent research products, nationally selected– Honors awarded to graduates– Performance on nationally standardized tests or benchmarks – Research income, national competitive

Number of Indicator Categories in each Group No. of Indicator

Categories %

Total 63 100%

Group 1Comparable 8

59%Group 2

Fairly comparable 29

Group 3Hardly comparable 12

41%Group 4

Uncomparable 14

Frequency of indicators by groups in 11 rankings

Weight of indicators by groups in 11 rankings

The design of indicators for global rankings: TWO PATHS

Can these indicators be actually used in global rankings?

• Group 1: Comparable at a global level– Number of international projects– Books published at international editing houses

• Group 2: Fairly Comparable at a global level– Number of Bachelors granted– Average age of professors

Group 1 & Group 2 indicators that are used by rankings in Three or More Countries

Ranking CountryMelbourne Institute Australia

Sunday Times UK

Guardian UK

U.S. News USA

Top American Res. U. USA

Maclean's Canada

RCCSE China

Netbig China

Perspektywy Poland

ARRA Slovakia

CNCSIS Romania

Number of Countries

Number of Rankings

Students/faculty ratio√√√√

√√√√√

7

9

Spending per student√

√√

√√

√

6

6

Pub. & Cit. in int’l indexes√

√√

√√4

5

Total research income

√√√

√

4

4

Number of Doctors granted√

√

√

√4

4

Employment rate of graduates

√√

√

√

3

4

Proportion of postgraduates

√√√√

3

4

Proportion of professors

√√√√

3

4

Library resources

√√√√

3

4

Proportion of Ph. D. faculty

√

√√

3

3

Proportion of Int’l students

√

√√

3

3

Commonly used indicators

Indicator No. of Countries

No. of Rankings

Also used by

Students/faculty ratio 7 9 THES-QS

Spending per student 6 6

Pub. & cit. in int’l indexes 4 5 ARWU; Taiwan; THES

Total research income 4 4

Number of Doctors granted 4 4

Employment rate of graduates 3 4

Proportion of postgraduates 3 4

Proportion of professors 3 4

Library resources 3 4 Newsweek

Proportion of Ph. D. faculty 3 3

Proportion of int’l students 3 3 THES-QS

Should some indicators be normalized to be more comparable at a global level and how?

• Employment rate of graduates – Normalized by national employment rates or not?

• Spending (of an university) per student – Normalized by GDP per capita or not?

• Total research income – Normalized by Purchasing Power Parity instead of

exchange rate?

Underlying questions

• Does a decision need to be made across countries?

• To what extent a university is a global university?

• Global rankings: To measure universities’ performance as comprehensive as possible OR only to measure their “globalized part”?

Thanks for your attention!