November 2005 Anne AssersonResearch Administration Department,UiB

How can you assess your organisation through a

CRIS?

The Norwegian national CRIS, FRIDA, and evaluation by means of research

output

November 2005 Anne AssersonResearch Administration Department,UiB

Structure of Presentation

• Ministry Model for Assessing Research Output• Research Output – Publications• Requirements of a CRIS in this context• System Solution• Evaluation of Output• Ensuring Quality• Conclusion

November 2005 Anne AssersonResearch Administration Department,UiB

Aim of Ministry for Assessment

Model for assessment based on indicators to Stimulate research and Allocate resources

• The best researchers (by allocating money based on publications)

• The best ideas (allocating money to the competition-based financing of projects)

• Allocate money to special thematic areas

November 2005 Anne AssersonResearch Administration Department,UiB

New efforts: to use result indicators for academic/scientific publishing as a basis to budget the research at Universities and State Colleges. The Publications from 2004 will be included in the budgets for 2006

Main issues: 1. Provide better data on Academic Publishing2. Take in consideration different practices – thus avoiding

unfortunate effect on the publication pattern.3. Divide in Quality levels to avoid reduction in quality of the

publications pattern and avoid volume of low publications4. Develop indicators for activities within dissemination

St.prp.nr.1, UFD, pg. 115: ”Financing of Research” 2003

November 2005 Anne AssersonResearch Administration Department,UiB

• Ministry of Education and Research wish to improve research results in the universities

• Measure through indicators that say something of quality of results

• Results will trigger resource allocation to the universities

• 14% of total budget allocated based on research output indicators

November 2005 Anne AssersonResearch Administration Department,UiB

• Result Indicators that have been chosen – Doctoral theses– EU funding– National Research Council Funding– Publications

• These are quality indicators; all subject to peer review

November 2005 Anne AssersonResearch Administration Department,UiB

• Institution receives an allocation from the ministry

• The model is NOT recommended for use internally

• The ministry stresses the need for creating internal models to reward / stimulate

• Necessary to think complementarily in order to support particular subject areas

November 2005 Anne AssersonResearch Administration Department,UiB

Structure of Presentation

• Ministry Model for Assessing Research Output• Research Output – Publications• Requirements of a CRIS in this context• System Solution• Evaluation of Output• Ensuring Quality• Conclusion

November 2005 Anne AssersonResearch Administration Department,UiB

Distribution of published research by academic field and type of publication. Percentage of overall scientific production, University

of Bergen 1997-1998*

0,0

10,0

20,0

30,0

40,0

50,0

60,0

70,0

Biology Biomedicine Medicine Chemistry Physics Earthsciences

Mathematicsand

informatics

Socialsciences

Humanities

%

Int. jour. art. Scand./Norw. jour. art. Proceedings

*) Classification by institute assignments. Unpublished works, abstracts and posters are not included in the countings

Source: FORSKDOK, University of Bergen

November 2005 Anne AssersonResearch Administration Department,UiB

Definition of an academic Publication

The Publication must:• Present new insight/knowledge• Be in a form that make it possible to validate or

useful in new research • Be in a language and have a distribution that

can make the publication available for the researchers of whom it may be of interest

• Be in a publication channel ( journal, series, publisher, website) with routines for Peer review.

November 2005 Anne AssersonResearch Administration Department,UiB

• Publications– Peer reviewed articles– Academic monographs– Chapters / parts of anthologies– (Proceedings)

• Only a few classes of publications used in assessment

November 2005 Anne AssersonResearch Administration Department,UiB

Structure of Presentation

• Ministry Model for Assessing Research Output• Research Output – Publications• Requirements of a CRIS in this context• System Solution• Evaluation of Output• Ensuring Quality• Conclusion

November 2005 Anne AssersonResearch Administration Department,UiB

Functionality Required

• Must be able to express :– Person belonging to one or more

organisational unit– The output produced at one or more

organisational units– from one year or a temporal interval– From one or several projects

• And these in any combination

November 2005 Anne AssersonResearch Administration Department,UiB

Functionality Required 2

– Person must be linked to publication, organisational unit and project

– Reference format

• Vancouver, APA etc

– Person must be linkable to two or more organisational units

November 2005 Anne AssersonResearch Administration Department,UiB

Structure of Presentation

• Ministry Model for Assessing Research Output• Research Output – Publications• Requirements of a CRIS in this context• System Solution• Evaluation of Output• Ensuring Quality• Conclusion

November 2005 Anne AssersonResearch Administration Department,UiB

Structure Required

• relational model as basis with dynamic role-based temporal links between base entities

• Extended with elements of publication– Journal name, ISSN, shortname, acronym,

pagefrom, pageto, volume, issue

November 2005 Anne AssersonResearch Administration Department,UiB

PROJECT

ORGUNITPERSON

Skills

CV

GeneralFacility

ParticularEquipment

Contact

ResultsPublication

ResultsPatent

ResultsProduct

Service

FundingProgramme

Event

Classification

The Whole Thing

Prize/Award

November 2005 Anne AssersonResearch Administration Department,UiB

PROJECT

ORGUNITPERSON

Result_Publication

Can Express: (where DT-date/time)Person A (DT1 - DT2) (is author of) Publication XOrgunit O (DT1 - DT2) (is owner of IPR in) Publication XPerson A (DT1 - DT2) (is employee of ) Orgunit OPerson A (DT1 - DT2) (is project leader of) Project PPerson A (DT1-DT2) (is member of) Orgunit MPerson A (DT1-DT2) (is member of) Orgunit NOrgunit M (DT1-DT2) (is part of) Orgunit OOrgunit N (DT1-DT2) (is part of) Orgunit O

Secondary Base Entities: example: RESULT_PUBLICATION

November 2005 Anne AssersonResearch Administration Department,UiB

• National system for the four universities

• Funded and owned by the universities

• Central database

• Based on CERIF (with extensions)

• 300,000 records in 2006

• In production use for 1 year

November 2005 Anne AssersonResearch Administration Department,UiB

Structure of Presentation

• Ministry Model for Assessing Research Output• Research Output – Publications• Requirements of a CRIS in this context• System Solution• Evaluation of Output• Ensuring Quality• Conclusion

November 2005 Anne AssersonResearch Administration Department,UiB

A publications is divided in author and institutional parts within co-authoring

Publication

Author 1 Author 2 Author 3

Institution 1 Institution 1 Institution 2 Institution 3

November 2005 Anne AssersonResearch Administration Department,UiB

Channel

Form

Eexample : scientific article in Nature or scientific monografi at Aschehoug forlag

We find Academic Publications in the intersection between Publications Channel and Publication

Form.

November 2005 Anne AssersonResearch Administration Department,UiB

Level 1 and Level 2

• At level 2 there are specified certain publication channels within all subjects.Level 2 are expressing the desirable direction for the publication pattern.

• Level 1 is the ”normal” and includes all publishing according to the definition.

November 2005 Anne AssersonResearch Administration Department,UiB

Level 2: ca. 1/5 of the publications

Level 1: ca. 4/5 of the publications

November 2005 Anne AssersonResearch Administration Department,UiB

Guidelines for nomination at level 2

In general the publication channel must be

• Considered as leading within broader scientific community

• Publish the most important publications for researchers from different countries

November 2005 Anne AssersonResearch Administration Department,UiB

Structure of Presentation

• Ministry Model for Assessing Research Output• Research Output – Publications• Requirements of a CRIS in this context• System Solution• Evaluation of Output• Ensuring Quality• Conclusion

November 2005 Anne AssersonResearch Administration Department,UiB

Quality

• Import of ISI records– Problem: person and organisational unit not

connected

• Routines and responsibility for quality assurance– Researchers– Organisational unit

• Researcher as end-user validates records

November 2005 Anne AssersonResearch Administration Department,UiB

Bibliographical sources

• ISI-indexes will cover 62 % of all journal articles.

• Norart can cover further 32% of the articles by adjusting the application and by Nordic cooperation.

• Bibsys could possible cover 70-80% of book publishing

NORART: Norske tidsskriftartikler

November 2005 Anne AssersonResearch Administration Department,UiB

November 2005 Anne AssersonResearch Administration Department,UiB

Structure of Presentation

• Ministry Model for Assessing Research Output• Research Output – Publications• Requirements of a CRIS in this context• System Solution• Evaluation of Output• Ensuring Quality• Conclusion

November 2005 Anne AssersonResearch Administration Department,UiB

• Quality– Using selected publication channels, not only

publication volume, gives quality

• Requirements of System– Cannot be done by a library system– Requires a flexible structured CRIS