CRISs as a tool for evaluative purposes – Activities and future plans at the German Research Foundation (DFG)

Presented at EuroCRIS Members Meeting Tartu, Estonia, 18-20 May 2005

Deutsche Forschungsgemeinschaft

Juergen Guedler, German Research Foundation (DFG), Department of Information Management

Topics

The DFG as a knowledge agency

Present use of information storage in DFG-CRIS

Future possibilities: The „Institute for Research Information and Qualitity Assurance (IFQ)“

The German Research Foundation

promote academic excellence on a

competitive basis

serve science and the humanities in all fields

advise parliaments and public authorities on

research questions

support the advancement and education of

young researchers

encourage international research cooperation

Who We Are

What We Do

central public funding organisation

for academic research in Germany

Third party funding income of German universities by source (1999/2000)

Growth of DFG-budget: 1972 to 2003

Mio €

0,0

200,0

400,0

600,0

800,0

1.000,0

1.200,0

1.400,0

1972 1977 1982 1987 1992 1997 2002

Jahr

Source: DFG-reports 1972 to 2003 (Basis not including infrastructure funding)

Consequences of growth I

Public interest:

Grown public interest in questions of research and research-funding as well as on the results of DFG-funded research

Accountability:

Stakeholders (funds providers and taxpayers) ask for transparency in the use of funds provided

Base for planning purposes:

The DFG ist asked more and more to legitimate strategic decisions on data-based informations

Success control:

demand for data that inform about the success of funding programmes and funded projects

Political advice:

demand for data as a base for political decisions (e.g.: „Centers of Excellence“-Initiative)

Consequences of growth II

For these and other reasons the DFG today is asked increasingly

to inform about ist own activities and the activities of funded researchers (=„research information“) and

to collect data that allow a fact-driven evaluation of these activities (=„research and funding evaluation“)

This job is actually done by the department for information management (18 persons)

Consequences of growth III

Applicants

Reviewers

Departmentsof the DFG

rd. 20.000Persons /

year

rd. 7.000Persons /

year

more than 350 people part. on data based

proposal management

Abt. IISFBGRKWAW

IZ*LIS*

Budget Dept.**

sendreviews

ElektrA-DB

ask for reviews

send proposals

andreports

put informationin database

securequality

copy ofdata

IM

Evaluative reports / Ranking /statistics information

services

CRISs online (e.g.IDB, GEPRIS)

further databased information services(e.g. WWW of DFG)

* Starting in the end of 2005** Own data base (MACH)

QV

Base of information services: Process produced data of the DFG-funding database

time span: 23 years (since 1982 (in parts since 1976))

Persons: More than 120.000 persons with detailed personal information (date of birth, sex, adress etc.)

Roles: around 20 roles (applicant, partner, referee, conference partic., memb. o. inst. etc.)

Coordinated programmes: data on more than 1.600 programmes

Projects: around 300.000

Relaunch of the database1999-2005

The new database ELEKTRA

contains more than100 tables and 1.000 columns

that inform in a new depthabout the DFG-funding activities

Actual content of the DFG-database

Services on base of the DFG-Project database

Documentation (e.g. within (electronical) yearbook

Statistical Information

Evaluative Studies

Information Systems about experts and current research

Internet and Intranet Services

Research Information Services

GEPRIS (German Project Information System)

Database on Research Institutes in Germany (still without a name ;-) )

Principal purpose is the transfer of information

a) about expert knowledge

b) about the aims of current research projects

c) about potential research partners

GEPRIS – German Project Information System

GEPRIS-Homepage

[Ruth/Michael]

List format of GEPRIS

Project description in GEPRIS

[Ruth/Michael]

Aim: To support data manag. (e.g. adressing) and to serve statistical services

DFG- internal use

Outside

DFG

Every entity (e.g. persons, projects) can be matched with institutional data

support the process of data management (selection of adress vs. purely typing them)

possibility of new statistical and research information services

Better possibilities of data change out by standardization

new common (!) online information services (Research Institutes in Germany)

Konzeption

Bedarfsspezifikation

1999 2002 ff20012000

EL

EK

TR

A

first concept

discussion and contracts with other funding

agencies

First data collections with external partner

Actualization of data

The DFG-institute database: Aims

2005

Vorarbeiten

IDB

All Institutes and faculties from the 80 biggest universities

Main adress and a selection of institutes of all other universities

(in cooperation with HRK)

All member institutes of the big research org. in Germany

MPG, FhG, WGL, HGF and other

1

2

3

Universities non univ. bodies

3.700

All together around20.000 entities

1.500

Faculties etc.

Institutes

Sub-Institutes etc.

7.100

7.300

350

Coverage of the IDB

Name Name engl. Kurz-Name Fach_ID Fach_Name INST_ID Teil von Bundesland Land Einr.-art Inst.-Typ Postalische Bezeichnung

Straße PLZ Ort Telefon Telefax E-Mail Internet ...

Institut für Stromrichtertechnik & Elektrische AntriebeInstitute for Power Electronics & Electrical DrivesISEA7100Elektrotechnik68345Fakultät für Elektro- & Informationstechnik Nordrhein-WestfalenDEUUniversitätInstitutInstitut für Stromrichtertechnik & ElektrischeAntriebeRWTH Aachen Jägerstr. 17/1952066 Aachen0241/80-69200241/67505dedoncker@rwth-aachen.dehttp://www.isea.rwth-aachen.de...

Entity

Name (german/engl) Adress Homepage E-Mail-Contact Research Area (class.) Type of institute Type of organizationand other

Content of the IDB

More than 40.000 Persons (Applicants, Reviewers a.o.) matched1

planned: IDB online (late 2005)(in cooperation with DAAD)

Around 200-300 actualizations monthly

Around 80 – 120 new institutes monthly

More than 30.000 Projects machted2

3

4

6

ElektrA

IDB

+

planned: integration of data on foreign institutes (source: Alex.-v.Humboldt-Foundation) (late 2005)5

Actual Status of the IDB

Evaluation of processes: e.g. duration of decision process on approvals

Market analysis: e.g. acceptance of new funding programmes

Research policy: Research activity of certain groups of clients (e.g. male vs. female clients)

Rankings: Research activity of german universities and other research agencies

Common statistical services: Between 200 and 250 statistics yearly „on demand“

Questionaires: The database is more and more used as an adress-source for questionaires on scientific and science-policies questions (e.g. Open Access)

Data use for evaluative purposes

The most visual project: the DFG-“Funding-Ranking“

Idea: To give information about research activity of

german universities and other research bodies

To inform about the disciplinary research profile of these institutions and of the regions they are active

To inform about networks in science

„History“: First Ranking in 1997

(database: only DFG-funded money) as a reaction to a demand of the ten biggest german universities

Second Ranking in 2000 (new: regional aggregation of data)

Third Ranking in 2003 (new: networks in science, several other indicators)

DFG approvals to universitiesand non-university institutions 1999 to 2001by district and scientificdiscipline (in millions of €):

Total

Cooperative relationships in DFG-funded coordinated programmes: Biology

AvH visiting researchers 1997 to 2001 by DFG approval ranking group in relation to the number of professors/scientists and academics in total at universities

Visitor’s Statistics: www.dfg.de (1.7.2003 to 20.6.2004)

Full paper in german and (soon) in english language

Download-Area: Different tables and diagrammes

Special Analyses

Catalogue of interesting „links“

Adress: http://www.dfg.de/en/ranking/

Funding Ranking online

With respect to the DFG

Good progress with respect to quantitative and qualitative information about DFG-grantsbut

scarce information about the process as well as about the results and success of DFG funded research and

No working unit within or outside the DFG, which could satisfy the described needs in an appropriate way.

With respect to actors outside DFG

Low co-operation between different producers and users of research information

Large difficulty to link information from different sources due to missing standards

Absence of a central „network-point“ that offers and coordinates appropriate services

Current situation

Solution

Construction of a scientific facility, which supports the DFG and other actors involved in science policy with above described services for the purpose of programme and research planning as well as a service of information to the general public.

Institute for Research Information and Quality Assurance (IFQ)

Three purposes

Research information

Monitoring of funding and research activities

Quality assurance and program development

Supply of information about participants, projects and results of German research to wide public, e.g. with the help of data base-supported online information-systems.

Sustained monitoring of developments and trends in publicly funded research by use of a broad set of established and newly developed indicators.

Use of those data and indicators for evaluative studies that inform about the degree of goal reaching of the DFG-funded programmes, about the success of the funded bodies and about the needs for new funding activities.

Examples of Future Service

Qualifying information (who is doing research with whom and with which results?) (= General Research Information)

Quantifying information (Input- and output-data on DFG-funded projects) (=monitoring/evaluation)

Empirical based information about process and results of DFG-funded projects (e.g. career of DFG-Fellows, Internationality and Interdisciplinarity of projects, „output“(publications, patents and others)).

E.g. construction of an information system that publishes the final reports of DFG-funded projects.

Actual Situation and next steps

In May 2004 the joint committee of the DFG gave the „starting signal“ for the IFQ

In december 2004 the recruitment of the scientific head of the institute started

In march 2005 the finding committee spoke out its recommendation

In july the scientific director of the institute will be announced

In october 2005 the institute will start work

In 2008 the institute itself will be evaluated – after that it (hopefully) will offer its services to a broader public outside DFG

Thank you for your attention!

More Infos at www.dfg.de/

Dr. Juergen Guedler, Department of Information Management, juergen.guedler@dfg.de

Deutsche Forschungsgemeinschaft