research fortnight - 08/07/2015

Every new opportunity for research funding from every sponsor in the UK, EU, US & beyond Every discipline Every fortnight Issue No. 8 July 2015 Updated daily at www.ResearchProfessional.co m Founded by William Cullerne Bown ELSEVIER’S ATTEMPTS to show a correlation between how universities performed in the 2014 Research Excellence Framework (REF) and how they would have done in a metrics-based system are flawed, according to the lead author of an independent report on metrics. There is an emerging consensus that both metrics and human judgement are needed in research management and evaluation—this view is held by Elsevier and other metrics providers, and will also be put forth in James Wilsdon’s report, due out on 9 July. However, Wilsdon, a professor of science and democracy at the University of Sussex, has his doubts about the quality of metrics and their ability to replace or do the same job as humans. Earlier this year, Elsevier released an analysis show- ing a strong correlation between the quality-related (QR) funding received by a university after the 2014 REF and the proportion of the institution’s outputs that were in the top 5 per cent of most-cited articles worldwide . Howeve r, that analysis also show ed tha t, for the eight UK institutions that receive more than £35 million a year, the correlation was weak. There was nothing factually wrong with Elsevier’s analysis, but it should still be viewed with caution, says Bill Browne, a professor of biostatistics at the University of Bristol. “Elsevier is looking at relationships at the institution level and should be careful of what is known as the ecological fallacy, where a relationship at an aggregated level (institution) may not be the same as that at the observation level (REF outputs).” In contrast, the Higher Education Funding Council for England, which commissioned Wilsdon’s review, has tested the ability of 15 metrics to reach the same result as the panels did for 150,000 individual outputs submitted to the 2014 REF. HEFCE was able to carry out this observation-level analysi s because it had access to these REF data before they were destroyed as part of the standard process. This analysis, which will be pub- lished alongside the report tomorrow, shows a weak link between metrics-based and human judgement. Wilsdon says t hat HEFCE’s analysis is “clea rly more robust” than Elsevier’s, “as looking closely at output- level scores and metrics allows for a far more accurate judgement to be made about any correlative link”. However, Nick Fowler, the managing director of research management at Elsevier, says the company is simply running lots of analyses to test the capabil- ity of various metrics in combination. For instance, an Elsevier analysis comparing QR funding with field- weighte d ci tation impact shows a strong correl ation for the eight institutions receiving more than £35m a year—despite the weak correlation overall. “We believe no single metric should be used,” says Fowler. “The notion of a weak correlation is misguid- ing. If you used multiple metrics then it would be possible to come up with an algorithm that comes close to the result from the current process.” Since its inception in April 2014, Wilsdon’s review has inspired an energetic debate between academics, metrics providers, funders and research managers. Judith Petts, the University of Southampton’s pro vice-c hance llor for resear ch, said in her submiss ion that metrics could help to create a “fairer and more objective method of assessment”. But many academics, including the pharmacologist David Colquhoun and Meera Sabaratnam, a lecturer in international relations, have cautioned against a growing role for metrics in research assessment. Although Wilsdon’s report will not recommend that peer review be replaced by metrics, it will say that REF panels should be handed more data and given the choice of which, if any, to use. Sabaratnam is uneasy about this: “There is already an incredible amount of power in the panels. I don’t want to disparage the work they do, which is very hard, but it’s already highly unaccountable.” The report is also expected to recommend a greater use of metrics in the impact and environment sections of the REF, and that bibliometric data, which Elsevier provided to 11 panels in 2014, continue to be used. by Adam Smith [email protected] Scientists push government for drugs bill exemption – p5 British Museum Impact on our terms – p6 Tag team How taxonomy could improve science funding and policy – p21 Tensions bubble below metrics consensus Influential review to favour human judgement over data in REF Every new opportunity for research funding from every sponsor in the UK, EU, US & beyond Every discipline Every fortnight Issue No. 460

research fortnight - 08/07/2015

Documents