Peer review of digital resources for the arts and humanities

David Bates and Jane Winters

What do we mean by peer review and evaluation?

Peer review

• Formal assessment of proposed research, undertaken at a sufficiently early stage to influence the course of that research and the nature of its outputs

Evaluation

• Evaluation during or at the end of a research project as part of a formal process of assessment

• Evaluation by end users, whether through informal feedback or in a published review

Institutional affiliation of survey respondents

UK HEI

Non-UK HEI

MLA sector

Other

What is important in determining the value of a particular digital resource for your own research?

0 50 100 150 200 250 300 350 400

Content

Comprehensiveness

Usability

Presentation

Authority

Permanence

Searchability

Unavailability of original

Transformative impact

How important is peer evaluation or recommendation in your selection of

resources for personal research?

Extremely important

Important

Somewhat important

Not important

Respondents’ comments

• ‘Peer review and provenance are key for me – I can get non-peer reviewed material any time through Google and evaluate its usefulness myself. It is no substitute for the academic resources’

• ‘A proper peer review mechanism, with recommendations made according to a specified and easily available set of criteria’

• ‘A review or reviews by experts in my broad fields of study/interest who know the interesting questions to ask’

• An open forum on digital resources with signed commentary … would be ideal’

Cultural change

• Greater recognition of the value of collaborative research

• Addressing the skills gap – greater investment in the training of researchers

• Publication of reviews of digital resources in leading scholarly journals

• Clear citation guidelines

Peer review

• The AHRC should consider developing an alternative to the ‘technical appendix’ as a means of ensuring robust project planning and methodologies

• Research councils should consider developing a two-stage application process

• Peer reviewers should continue to be selected primarily on the basis of subject expertise, but their ability to assess the technical elements of a proposal should also be taken into account

Evaluation

• Post-completion assessment of research projects with digital outputs – with reports and responses attributed and published

• Guidelines for reviewers

• Check-list for basic technical standards

• Levels of usage should not be a key indicator of scholarly value

• Kite-marking, or any ‘pass/fail’ system of assessment, should not be adopted

• Safeguard subjectivity

Applications for a framework of peer review and evaluation

• Facilitates the formal assessment of digital resources, e.g. in the Research Assessment Exercise

• Allows users to make decisions about appropriate resources for use in their research

• Assists librarians in making purchasing decisions

• Allows funding bodies to decide which projects to fund, in both the short and long term, and provides a mechanism for assessing their ‘success’

Sustainability

• Financial – largely beyond the scope of the project, although recognised as a key concern

• Technical – responsibility of resource creators to take account of technical sustainability when devising their projects

• Academic sustainability – requires investment of money from a central body, and time from the academic community

www.history.ac.uk/digit/peer/