Research evaluation requirements

José Manuel BarruecoUniversitat de València (SPAIN)

Servei de Biblioteques i DocumentacióMay, 2011

BackgroundLibrarian at the University of Valencia (Spain)

Manager of the institutional repository

Co-Founder of the RePEc digital library

Running an autonomous citation index for Economics (CitEc) since 2001

Contents1. The current situation2. What has been done?3. A case study: CitEc4. The challenge5. The requirements

The current situationOAR are a new medium to distribute research results

More than 1800 registered in ROARAim: increase the visibility and impact of the scientific literature

Citation analysis used for research evaluation purposes

Challenge: to apply citation analysis to contents distributed in repositories It will demonstrate the open access usefulness It will be an important added value for authors It will contribute to increase OAR contents

What has been done?Many research projects, not so many services

Usually working with OA journals, not OARSubject based repositories:

CiteSeerarXiv: referes to + cited byRePEc: referes to + cited by + author/institutions rankings

Institutional repositoriesNothing has been done yet

A case study: CitEcIt is a RePEc (Research Papers in Economics) serviceRePEc describes the whole discipline, not just research outputs:

Institutions (departments, research institutes …)AuthorsPublication channels (journals, working papers series, books …)Research outputs (articles, papers)

Every player in RePEc has a persistent identifierRePEc in numbers:

+1,000,000 working papers + journal articles+27,000 author contact and publication listings +12,000 institutional contact listingsComing from more than 1,300 data providers

A citation dataset of:6,222,142 references 2,517,198 citations 287,216 documents processed (about 30% of the available

documents)CitEc produces open citation data which is used by other

RePEc services to: add refence linking supportCreate rankings (authors, institutions … )

A case study: CitEc

Some examples:CitEc home page:

http://citec.repec.orgCitEc data used for reference linking:

http://econpapers.repec.org/RePEc:ags:arerjl:31417CitEc data used for research evaluation

http://ideas.repec.org/top/

A case study: CitEc

Can citation analysis be applied to contents distributed in OAR?

YES, WE CAN!!

But …. It is not easy! If technology is not a problem (ParsCit, Xpdf),

then what … ?

The challenge

OAR are quite empty43% of OAR listed in BASE have less than 1,000 itemsThey are not limited to research outputs but any digital

content (learning objects, cultural heritage … )The size is not a problem

We still have the subject repositoriesCitEc started with no more than 25,000 documentsWe may see the citation analysis as an opportunity to

increase OAR contents.

The challenge

Metadata issues No way to differenciate research outputs of other digital

content in OARsFew OARs are providing list of references for each

document E-LIS or MPRA

Limited use of persistent identifiers (DOIs)No direct access to the documents full texts Documents in OAR are described as independent entities

instead of representing relationship (isVersionOf, isPartOf … ). Research context gets lost

The challenge

Researchers identification issuesResearchers are the core of our effortsWe need to provide them aggregated citation data from

different OARBut OAR pay little attention to the identification of

researchers (authors represented as not standarized strings)

The big problem: author disambiguation!!Here we have an opportunity for CRIS+OAR integration

The challenge

1. Improve the interoperability of research OAR. 2. Improve the amount and quality of metadata3. Guidelines for providers. But, don’t we have DRIVER?

Not enought They haven’t been widely adopted, because there are not

enought incentives to implement them.

4. Use of persistent identifiers for researchers in CRIS+OAR OpenID, ORCID A working alternative: AuthorClaim by Thomas Krichel

The (basic) requirements

5. Develop citation service providers!! Services like BASE should move forward to provide new added

value services such as citation analysis There is ground for new players too The snake biting its tail!! They should provide open citation data so that it could be

reused by research managers and administrators Open data != API

But, such services should be expensive?! Not really, CitEc budget … 0 €

The requirements

Thanks for your attentionbarrueco@uv.es