impact assessment in the humanities: problems and prospects carl dolan vienna, 15 december 2008...
TRANSCRIPT
Impact assessment in the Humanities: problems and prospects
Carl DolanVienna, 15 December 2008
Presentation Roadmap
IntroductionsHERA study Basic data issues, bibliometrics etcA framework for evaluation and
benchmarking in the humanitiesFeasible benchmarking: a case
study
Introductions
• My role - disclaimer #1• Humanities in the European Research
Area (HERA)– Network of 18 humanities funding agencies
(www.heranet.info)
• Address common challenges e.g.– ex-ante peer review– joint programming and double jeopardy– benchmarking research quality
HERA report on the evaluation and benchmarking of humanities research
• HERA work-package (12 mths)• Coordinated by AHRC• Survey of evaluation practices in
agencies (questionnaire)• Survey of ex-post evaluation systems in
Europe (desktop research)• Review of developments in bibliometrics
and peer review• 2 workshops in London (Mar 06/Jan 07)
HERA report on the evaluation and benchmarking of humanities research
• Disclaimer #2 – feasibility study, not new research
• Response to immediate policy concerns – devise an informed, common position
HERA report on the evaluation and benchmarking of humanities research
• Short answer – credible benchmarking system not feasible (in short term)
• Basic input, activity & output data just isn’t there (or is patchy)
• Four main data sources– National Statistics Agencies, OECD, Eurostat– Universities– Funding agencies– Commercial bibliometric databases
International availability of data (OECD)
• HERD for humanities 1981-2002– 11 of 30 countries had no data– only 2-11 per year, lots of gaps– Germany, Denmark 14 yrs; Czech 4; France
0
• R&D personnel– No data before 1995– 7 of 30 had patchy data
• Research postgraduate ………..
Country Field of study 1998 1999 2000 2001Australia Humanities (ISC 22) 376 440 436 443Austria Humanities (ISC 22) 0 228 188 207Belgium Humanities (ISC 22) 127 148Belgium (Flemish Community) Humanities (ISC 22) 61 58Canada Humanities (ISC 22) 454 466 530Czech Republic Humanities (ISC 22) 33 52 57 90Denmark Humanities (ISC 22) 46 0 0Finland Humanities (ISC 22) 199 197 225 215France Humanities (ISC 22) 0 909 1,389 1,472Germany Humanities (ISC 22) 1,645 1,635 1,963 1,793Greece Humanities (ISC 22) 334 322 152 187Hungary Humanities (ISC 22) 0 0 0 0Iceland Humanities (ISC 22) 74 92 36 62Ireland Humanities (ISC 22) 96 135Italy Humanities (ISC 22) 568 568 543 534Japan Humanities (ISC 22) 920 988 1,116 1,190Korea Humanities (ISC 22) 548 530 581 11Luxembourg Humanities (ISC 22) 0 0 0Mexico Humanities (ISC 22) 0 54 63 61Netherlands Humanities (ISC 22) 240 241 0 222New Zealand Humanities (ISC 22) 0 1 0 1Norway Humanities (ISC 22) 30 67 57 79Poland Humanities (ISC 22) 0 0 0 0Portugal Humanities (ISC 22) 146Slovak Republic Humanities (ISC 22) 50 29 30Spain Humanities (ISC 22) 764 702 673 770Sweden Humanities (ISC 22) 185 180 166 194Switzerland Humanities (ISC 22) 179 146 130 132Turkey Humanities (ISC 22) 258 352 266 237United Kingdom Humanities (ISC 22) 0 0 1,287 1,691United States Humanities (ISC 22) 5,188 5,419 5,456 5,200
ISCED level 6 data = PhD equivalent
Universities
• Collect large amounts of data on outputs, staff numbers, grant income etc
• Data not easily accessible in all countries• Avoid imposing additional administrative burdens on
universities where possible• Few European countries where there is systematic
collection and collation of university data across all fields – UK (RAE)– Netherlands (VSNU)– Slovenia– Poland – Norway (to a lesser extent)
Funding agencies
• Survey reveals common practice (EoA reporting forms) and large amounts of output data
• But…– Data difficult to manipulate– Under-reporting of project outcomes– No significant agencies in some countries– Representative? - Relative share of total humanities
research output varies and is typically small• Trend toward creation of research information
systems/ CV repositories (DFG pilot, RCUK in pipeline)
Commercial bibliometric databases
• Poor coverage of output due to:– National/regional focus– US bias – Language issues
• 70% English in AHCI– Journals only
• 61% references to monographs in Hist & Phil Sci– Specialist journals only (non-academic audiences
excluded)– Different citation practices /windows– Team oriented v. single scholar – Differences in academic culture?
HISTORY in Thomson journal databases
0
200
400
600
800
1000
1200
1400
1600
1800
1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004
Num
bers
of j
ourn
al a
rtic
les
CANADA CZECH REPUBLIC FRANCE GERMANY UK USA
Response to criticisms
• Most criticisms of ISI AHCI• Expand coverage: combine databases (Google
scholar, Scopus, CSA illumina)• Possible to ‘mine’ AHCI for more ‘raw citations’• Evidence of growing ‘journal culture’• Evidence that AHCI has good coverage of
‘important international journals in fields with a dominant international research frontier’ (Nederhof)
• US bias not clear (in terms of performance)
Database developments
• Avoid citation databases• Use book and journal weights• Construct weights through peer-review
and consultation• E.g. Norwegian Ministry
– National Library and ISI give 90% coverage – Common documentation system– Classified according to publication type and
publication channel
Weighting of publications in Norwegian Model
Table 1. The weighting of publications
Channels at normal level
Channels at high level
Articles in ISSN-titles
1 3
Articles in ISBN-titles
0,7 1
Books (ISBN-titles)
5 8
Could Norwegian model be expanded?
• 2 necessary elementsi. Common documentation systemii. Agreement on weights
i. SI (SICRIS), NL (DAREnet), FI (KOTA), CA (Common CV), UK?
- Link and standardise?
ii. European Reference Index (ERIH)
Problems with this approach
• Need for constant revision and consultation
• Risk of perverse incentives and stifling emerging research
• Measuring impact of channels, not publications
A framework for evaluation in the humanities: common principles
• Importance of better data collection
• Importance of quantifiable evidence (objectivity & transparency)
• Importance of peer judgement• Proxies for peer judgement are
available
A framework for evaluation in the humanities: common principles
• Importance of disciplinary variation (within humanities)
• Assessment at disciplinary level – data not fine-grained enough
• A holistic approach –suite of indicators
• Applies to other disciplines too – no A&H exceptionalism
A feasible framework for international benchmarking? Case study
• Dicennial Assessment of PhD programmes in US (NRC)– Every 10 years since 1960s– 200 US institutions surveyed– All fields inc. humanities– Ranked list based on quality assessment
• Evaluation methodology revised– Pre-2007: Reputational method– Now: ‘implicit method’
Implicit method
Obtain reputation measures for a sample of programs in each field (anchor study)
Use multiple regression to construct a model of predictors of reputation ratings
Construct separate indicators for each field using weights obtained from questionnaire survey and statistical models
Use model to impute reputation measures for non-sampled programs with appropriate error bands (not simple ranking)
Implicit method
• Constructs indicators by weighting the following categories (faculty quality)Number of publications per faculty memberNumber of citations per faculty memberReceipt of extramural grants for research Involvement in interdisciplinary workRacial/ethnic diversity of program facultyGender diversity of program facultyReception by peers of a faculty member’s
work, as measured by honors and awards
A feasible framework for international benchmarking? Implicit method
Peer review at its heart (anchoring study + consultation on weightings)
Guided by appropriate quantitative data Sensitive to disciplinary variation Holistic -broad range of quantitative data
considered Effectively same method for all disciplines Avoids problems of bias and non-replicable
nature of peer review Feasible to apply to large quantities of data
Thank you for your attention