information retrieval to informed action with research metrics thomson scientific research services...

Information Retrieval to Informed Action with Research Metrics

Thomson Scientific Research Services Group2007

• Science is very dynamic and growing exponentially

• Research is happening in a global and multidisciplinary context

• Emerging fields make it difficult to determine relevancy

• Selectivity becomes more challenging, e.g. how to find the right journals

Challenge of Managing and Evaluating Research

• Keep up to date on new and developing science

• Nurture new research through direction and funding

• Measure the impact of scientific efforts

• Access timely, objective and informed information to make policy and budget decisions

Responsibilities of leaders in research

Metrics – How to measure science?

• Scientists communicate through the literature in academic journals• Citations are considered the “coin of the realm” in science

Bibliometrics uses statistical analysis to draw a landscape of scientific communication and productivity.

• Citation analysis:• examines the elements of bibliographic records (citations) • studies relationships between citations• measures the amount of scientific activity in a subject category,

journal, country, topic or other areas of interest.

Why Metrics?

• Uses readily available data

• Employs transparent and reproducible methodology

• Provides quantitative information that policy makers can apply to their decision process

“If you can measure that of which you speak, and can express it by a number, you know something of your subject; but if you cannot measure it, your knowledge is meager and unsatisfactory”

- Lord Kelvin (William Thomson)

“If you can measure that of which you speak, and can express it by a number, you know something of your subject; but if you cannot measure it, your knowledge is meager and unsatisfactory”

- Lord Kelvin (William Thomson)

Elements of Evaluation using Bibliometric Methods• Theory of citation:

– Citations are a form of peer recognition

• Basic measures: – Papers, citations, citations per paper (impact)

• Relative measures: – impact relative to baselines;

– expected citation rates for articles;

– percentiles

• Units of analysis: – papers, authors, institutions, countries, journals

• Measures of interaction: – research fronts, interdisciplinarity, collaboration

A National view of Turkish Research

Economics informs but citation analysis provides more detail

Turkish research motivation is seen in the OECD records on percentage of GDP dedicated to Research & Development. Turkey is surging and soon to exceed Portugal.

Source: OECD Factbook 2008

Science and technology - Research and Development (R&D) - Expenditure on R&D

0.0

0.2

0.4

0.6

0.8

1.0

1.2

2000 2001 2002 2003 2004 2005

Greece Hungary Poland Portugal

Slovak Republic Turkey Estonia

Turkey

Research Growth and Impact in Turkey, 1981-2007

Turkish research has grown steadily over the last 27 years. On average, Turkey produced over 14,000 scientific papers per year from 2003-2007, with large movement in aggregate citations between 2003 and 2007.

0

20000

40000

60000

80000

100000

120000

140000

160000

Papers Citations

Source: National Science Indicators, 1981-2007 – WOS version

Annual Research Growth in Turkey, 1981-2007

In 2003, Turkey broke through to go above 10,000 research papers produced in a single year.


Annual Growth in Papers, 1981-2007

0

2000

4000

6000

8000

10000

12000

14000

16000

18000

1981

1982

1983

1984

1985

1986

1987

1988

1989

1990

1991

1992

1993

1994

1995

1996

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

2007

Total Papers

Research Growth in Turkey, 1981-2007, compared to Greece

Among the countries in the region, Greece is the closest in publications but continues to lag behind Turkey’s strong growth trend.


Annual Growth in Papers, 1981-2007

0

2000

4000

6000

8000

10000

12000

14000

16000

18000

1981

1982

1983

1984

1985

1986

1987

1988

1989

1990

1991

1992

1993

1994

1995

1996

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

2007

Turkey Greece

Total Papers

Percentage of papers in the world, 1981-2007


Turkey also exceeds Greece in overall contribution to research in the world.

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

1981

1982

1983

1984

1985

1986

1987

1988

1989

1990

1991

1992

1993

1994

1995

1996

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

2007

Turkey Greece

%Papers InWorld

Research in Turkey – Citation ImpactCitation impact is the average citation rate: total cites/total papers.

Turkey’s average citation rate has risen consistently since the mid-1990’s and is trending to continue its steep movement into the world scientific community.


00.20.40.60.8

11.21.41.61.8

22.22.4

1981

-198

5

1982

-198

6

1983

-198

7

1984

-198

8

1985

-198

9

1986

-199

0

1987

-199

1

1988

-199

2

1989

-199

3

1990

-199

4

1991

-199

5

1992

-199

6

1993

-199

7

1994

-199

8

1995

-199

9

1996

-200

0

1997

-200

1

1998

-200

2

1999

-200

3

2000

-200

4

2001

-200

5

2002

-200

6

2003

-200

7

Baseline=1

Avg.Cite Per Paper

Research in Turkey – Percentage Cited

While publication growth is strong, Turkey is still working to exceed Greece in percentage of papers cited.


0

10

20

30

40

50

60

70

19

81

-19

85

19

82

-19

86

19

83

-19

87

19

84

-19

88

19

85

-19

89

19

86

-19

90

19

87

-19

91

19

88

-19

92

19

89

-19

93

19

90

-19

94

19

91

-19

95

19

92

-19

96

19

93

-19

97

19

94

-19

98

19

95

-19

99

19

96

-20

00

19

97

-20

01

19

98

-20

02

19

99

-20

03

20

00

-20

04

20

01

-20

05

20

02

-20

06

20

03

-20

07

Greece Turkey

% of Papers Cited

Chemistry, 11%

Materials Sci 4%

Ecol/Envir3%

Biology & Biochem, 4%

Plant & Animal Sci, 6%

Physics6%

Engineering, 10%

Pharmacology2%

Clinical Medicine, 38%

Agricult. Sci4%

Turkey: Field Specialization

The following 10 fields make up over 85% of Turkish scientific research

Source: National Citation Report on the nation of Turkey (ESI), 1981-2007

Turkey: Field Specialization

Generally Turkey contributes 1.48% of all world papers in 2003-2007.

The following fields exceed that level and indicate a national strength for Turkey in world contribution

Field % papers in

field in the world

Agricultural Sciences 2.76

Clinical Medicine 2.62

Engineering 1.94

Plant & Animal Science 1.89

Field % papers in

field in the world

Ecology/Environment 1.59

Pharmacology 1.54

Materials Science 1.50

Source: National Science Indicators, 1981-2007, ESI version

Turkey: International Collaboration

Turkish research is written more often within the country. Here are the top 20 countries represented in Turkish collaborations since 2000.

Source: National Citation Report on the nation of Turkey, 1981-2007

Key Universities in Turkish Research

0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000

SELCUK UNIV

ONDOKUZ MAYIS UNIV

BASKENT UNIV

AKDENIZ UNIV

MARMARA UNIV

ERCIYES UNIV

CUKUROVA UNIV

FIRAT UNIV

ISTANBUL TEKNIK UNIV

DOKUZ EYLUL UNIV

ATATURK UNIV

ORTA DOGU TEKNIK UNIV

GAZI UNIV

EGE UNIV

ANKARA UNIV

HACETTEPE UNIV

ISTANBUL UNIV

Number of Papers, 2002-2006

University Publications

Source: National Science Indicators customized for Turkish Institutions, 1981-2006

These 17 universities published 1000 or more papers over the period 2002-2006. This illustrates academic contribution to the national performance

Growth of Publications in Specific Universities


In the 2002-2006 period, these 5 leading Turkish universities fit into the mid-range of publication counts for the US, UK, Australia, and Canada university indicators.

0

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

Ankara University

Hacettepe University

Instanbul University

Ege University

Gazi University

Relation of University Strengths to National Strengths


Most active fields for top 5 schools

# of papers Turkey overall % of country contribution

Agricultural Sciences 472 2266 21%

Biology & Biochemistry 806 2519 32%

Chemistry 2086 6754 31%

Clinical Medicine 7807 22909 34%

Ecology/Environment 163 1771 9%

Engineering 819 5807 14%

Geosciences 256 1358 19%

Materials Science 394 2234 18%

Neurosciences & Behavior 312 1247 25%

Pharmacology 560 1401 40%

Physics 586 3833 15%

Field Specialization, 2002-2006


0 100 200 300 400 500 600 700 800

ORTA DOGU TEKNIK UNIV (METU)

ISTANBUL TEKNIK UNIV

GAZI UNIV

ISTANBUL UNIV

FIRAT UNIV

ATATURK UNIV

BOGAZICI UNIV

YILDIZ TEKNIK UNIV

ERCIYES UNIV

BILKENT UNIV

Papers Citations

Other fields of Turkish strength like Engineering are represented by a unique set of universities.

Growth of Publications in Leading Technical Universities, 1981-2007


Highlight comparisons between Middle East Technical University and Instanbul Tecknick

0

50

100

150

200

250

300

350

400

450

500

1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007

ISTANBUL TEKNIK UNIV MIDDLE EAST TECH UNIV

Engineering subfields at METU


Cites Field Papers Avg. Cites/Paper

2,845 ENGINEERING, CHEMICAL 640 4.45

1,426 ENGINEERING, BIOMEDICAL 139 10.26

1,168 ENGINEERING, ELECTRICAL & ELECTRONIC

328 3.56

920 MATERIALS SCIENCE, BIOMATERIALS 65 14.15

706 ENGINEERING, MECHANICAL 194 3.64

Engineering at METU: ChemE journals


Journal Cites Papers Avg. Cites/Paper

2006 JournalImpact factor

ENERGY SOURCE 55 54 1.02 0.43

TURKISH JOURNAL OF CHEMISTRY 90 54 1.67 0.65

CHEMICAL ENGINEERING COMMUNICATIONS

61 38 1.61 0.35

FUEL 151 29 5.21 1.36

INDUSTRIAL & ENGINEERING CHEMISTRY REESARCH

173 26 6.65 1.52

CHEMICAL ENGINEERING SCIENCE 119 25 4.76 1.63

ENERGY & FUEL 105 25 4.2 1.52

JOURNAL OF FOOD ENGINEERING 109 25 4.36 1.7

AIChE JOURNAL 256 22 11.64 2.15

MINERALS ENGINEERING 37 22 1.68 0.94

Engineering at METU: Authors in ChemE


Cites Author Papers Avg. Cites/Paper

Cites2 H index

C index Average Percentile

548 DOGU, T 66 8.3 3,747 14 0.86 43.41

318 DOGU, G 36 8.83 1,864 11 0.91 37.66

258 YILMAZ, L 27 9.56 2,929 10 1.43 54.13

139 KOK, MV 49 2.84 500 6 0.65 62.87

69 TOSUN, I 28 2.46 572 3 0.34 58.74

52 HICYILMAZ C 25 2.08 82 4 0.58 71.52

18 BAGCI, S 23 0.78 18 1 0.2 71.68

18 TURKER, L 20 0.9 115 2 0.14 78.11

Engineering at METU: A well cited ChemE paper from Dr. T. Dogu


Comments

• Because science is such a dynamic enterprise, no one metric is adequate for evaluation.

• Evaluation is the culmination of quantitative and qualitative considerations, within a particular environment.

• Citation analysis can help highlight researchers and the work that impacts the overall impact of an organization, group, or nation.– Examines the elements of publıcatıon records

– Studies relationships between citations

– Allows for the measurement of the amount of scientific activity in varıous areas, e.g. Organızatıons, nıdıvıduals, journals, subject areas

Thank you!

information retrieval to informed action with research metrics thomson scientific research services...

Documents

research papers

research development

research fronts

technology research

annual research growth

turkish research motivation

science nurture new

rd turkey slide