www.gu.se

Bo Jarneving

Bibliometrifunktionen

GUB

2011

Evaluative Bibliometrics: Citation based Research Indicators

www.gu.se

Variables for citation based performance indicators

1. Document type

articles, reviews, letters, notes

2. Observation period

Citation interval

3. Self-citations

With or without

4. Subject classification

Generally based on ISI ”Journal Subject Categories”

Classification of papers published in multidisciplinary journals

5. Counting method decisions

How to count co-authored papers?

How to count when there are multiple author affiliations?

How to count when there are multiple subject categories?

www.gu.se

Observation period

• Observation period– 2004-2009– Variable or fixed citation interval (window)

PY 2004 2005 2006 2007 2008 2009

Number of years

6 5 4 3 2 1

Interval 2009-2004

2009-2005

2009-2006

2009-2007

2009-2008

2009-2009

www.gu.se

What is a self citation?

Schubert A, Glanzel W, Braun T (1989) Scientometric Datafiles - A Comprehensive Set of Indicators 0n 2649 Journals and 96 Countries In All Major Science Fields and Subfields 1981-1985.

Glanzel W, Thijs B, Schubert A, Debackere K (2009) Subfield-specific normalized relative indicators and a new generation of relational charts: Methodological foundations illustrated on the assessment of institutional research performance.

www.gu.se

Journal Subject Category

• A proxy for subject field• Sometimes multiple assignments of journals to categories• Based on citation data and ”manual methods”• A total of 173 categories in the science volume of Journal Citation Report (JCR)• Alphabetical order – no hierarchy

ISI Journal Subject CategoryACOUSTICSAGRICULTURAL ECONOMICS & POLICYAGRICULTURAL ENGINEERINGAGRICULTURE, DAIRY & ANIMAL SCIENCE···VETERINARY SCIENCESVIROLOGYWATER RESOURCESZOOLOGY

www.gu.se

Classification of multidisciplinary papers

• Reversing pathological neural activity using targeted plasticity• NATURE, 2011, V470, N7332, P101-U114

Frequency Abbreviated journal title Journal Subject Category Journal Subject Category1 ARCH OTO-RHINO-LARYN OTORHINOLARYNGOLOGY1 AUDIOLOGY OTORHINOLARYNGOLOGY1 BEHAV NEUROSCI BEHAVIORAL SCIENCES   NEUROSCIENCES

8 HEARING RES NEUROSCIENCES OTORHINOLARYNGOLOGY1 J ACOUST SOC AM ACOUSTICS1 J CLIN NEUROPHYSIOL CLINICAL NEUROLOGY NEUROSCIENCES 2 J NEUROPHYSIOL NEUROSCIENCES PHYSIOLOGY1 J NEUROSCI NEUROSCIENCES2 J NEUROSCI RES NEUROSCIENCES PHYSIOLOGY1 J PHARMACOL EXP THER PHARMACOLOGY & PHARMACY 1 JARO OTORHINOLARYNGOLOGY1 LANCET MEDICINE, GENERAL & INTERNAL 1 LARYNGOSCOPE MEDICINE, RESEARCH & EXPERIMENTAL OTORHINOLARYNGOLOGY1 NAT NEUROSCI NEUROSCIENCES2 NATURE MULTIDISCIPLINARY1 NEUROSCIENCE NEUROSCIENCES2 P NATL ACAD SCI USA MULTIDISCIPLINARY3 PROG BRAIN RES NEUROSCIENCES1 SCIENCE MULTIDISCIPLINARY1 TRENDS NEUROSCI NEUROSCIENCES

www.gu.se

Counting method: fractional counts or whole counts

• Whole counts– Every author, institution, department and country is

assigned one full paper• Fractional counts

– A paper is divided up over its authors– If a paper has 3 authors, every author is assigned 1/3

of the paper. – If 2 of these authors are associated with institution A

and 1 with institution B, A recieves 2/3 of the paper and B 1/3.

www.gu.se

Counting methods: fractional counts or whole counts

When more than one institutional affilation of an author:• Whole counts

– One author associated with two institutions – each insitution recieves one full paper

• Fractional counts– One author associated with two institutions – each

insitution recieves one half of a paper

www.gu.se

Counting methods: fractional counting of papersExample 1: 1 paper, 1 category, 2 authors, 3 institutions (A-C)

1c

1/2

1/2

1/4

1/4

14+14+12

A gets

B g C g

1/2

www.gu.se

Counting methods: fractional counting of papersExemple 2: 1 paper, 2 categories (1, 2), 2 authors, 3 institutions (A-C)

1c

1/2

1/2

1/4

1/4

1/4

1/4

1/8

1/8

1/8

1/8

14+18+18+14+18+18

Category 1

A gets

B gets

C gets

Category2

A gets

B gets

C gets

1/4

1/4

www.gu.se

Counting methods: fractional counting of citations.Example 3: 1 paper, 1 category, 2 authors, 3 institutions (A-C).The paper has recieved 10 citations (c).

1c

1/2

½ x 10

¼ x 10

¼ x 10

14∙10+

14∙10+

12∙10

A recieves 2.5 cB recieves 2.5 cC recieves 5 c

1/2

www.gu.se

Counting methods: fractional counting of citationsExample 4: 1 paper, 2 categories (1, 2), 2 authors, 3 institutions (A-C). The paper has recieved 10 citations (c)

1c

½

½

¼ x 10

¼

¼ x 10

¼

1/8 x 10

1/8 x 10

1/8 x 10

1/8 x 10

Category 1

A recieves

B recieves

C recieves

Category 2

A recieves

B recieves

C recieves

¼

¼

www.gu.se

Counting methods: whole counts of citationsExample 5: 1 paper, 2 categories (1, 2), 2 authors, 3 institutions (A-C). The paper has recieved 10 citations

1c

1 x 10

1 x 10

1 x 10

1 x 10

1 x 10

1 x 10

1 x 10

1 x 10

1 x 10

Kategori 1

A r

B rC r

Kategori 2

A r

B rC r

www.gu.se

Normalization of citation counts

• Observed value: institutional (departmental) citation counts• Expected value: approximated world averages

– based on Web of Science citation data

www.gu.se

Normalization of citation counts

The expected value (reference value)• Define groups on basis of the institutional publication pattern. Group

by:– Publication date (year)– Journal Subject Category– Document type

• For each such group, compute the Field Citation Score (FCS)– Based on citations between papers indexed in WoS

» Approximations of world-citation-averages for each group

www.gu.se

Normalization of citation counts: defining groups• Example 1: The department (group) publishes in journals assigned to

one journal subject category only (neurosciences).– Brain Research 4 articles

2004– Brain Research 1 letter

2004– Journal of Neuroscience 3 articles 2002– Journal of Neuroscience 1 review 2003

• All authors are from one department

• Gather all articles published in journals assigned to neurosciences and published 2004

• Gather all letters published in journals assigned to neurosciences and published 2004

• Gather all articles published in journals assigned to neurosciences and published 2002

• Gather all reviews published in journals assigned to neurosciences and published 2003

www.gu.se

Normalization of citation counts: reference values

For each group (1- 4) compute the world-average number of citations (Field Citation Score [FCS]):

• Neurosciences-articles 2004 FCS = 4.2 • Neurosciences-letters 2004 FCS = 1.0 • Neurosciences-articles 2002 FCS = 3.8 • Neurosciences-reviews 2003 FCS = 8.5

www.gu.se

Normalization of citation counts: the department’s counts

Paper DT PY # citations1 article 2004 72 article 2004 43 article 2004 54 article 2004 55 letter 2004 16 review 2003 77 article 2002 78 article 2002 69 article 2002 1

www.gu.se

Normalization of citation counts: comparing

Paper DT PY # citations FCS1 article 2004 7 4.22 article 2004 4 4.23 article 2004 5 4.24 article 2004 5 4.25 letter 2004 1 1.06 review 2003 7 8.87 article 2002 7 3.88 article 2002 6 3.89 article 2002 1 3.8

Pic 20

www.gu.se

Normalization of citation counts: averages

• CPP = Citation Per Publication (department)• FCSm = Mean Field Citation Score• CPP/FCSm = the Crown Indicator

• Compute CPP: (3 x 7 + 1 x 6 + 2 x 5 + 1 x 4 + 2 x 1) / (3+1+2+1+2) = 4.78

• Compute FCSm: (4 x 4.2 + 1 x 1.0 + 1 x 8.8 + 3 x 3.8) / (4 + 1 + 1 + 3) = 4.22

• Compute CPP/FCSm: 4.78 / 4.22 = 1.13

www.gu.se

Normalization of citation counts: defining groups

• Example 2: The department (group) publishes in journals assigned to more than one journal subject category.

– Brain Research 4 articles2004

– Brain Research 1 letter2004

– Journal of Neuroscience 3 articles 2002– Neuroimage 1

article 2003

Four classifications:• Gather all articles published in journals assigned to neurosciences and

published 2003• Gather all articles published in journals assigned to neuroimaging and

published 2003• Gather all articles published in journals assigned to Radiology and

published 2003• Gather all articles published in journals assigned to Nuclear Medicine

& Medical Imaging and published 2003

www.gu.se

Normalization of citation counts: reference values

• For each group (1- 7) compute the world-average number of citations (Field Citation Score [FCS]):

– Neurosciences-articles 2004 FCS = 4.2

– Neurosciences-letters 2004 FCS = 1.0

– Neurosciences-articles 2002 FCS = 3.8

– Neurosciences-articles 2003 FCS = 5.5– Neuroimaging-articles 2003 FCS

= 3.2– Radiology-articles 2003 FCS

= 2.8– Nuclear Medicine &

Medical Imaging articles 2003 FCS = 4.6

www.gu.se

Normalization of citation counts: comparing

Paper DT PY # citations Fields FCS1 article 2004 7 Neurosciences 4.2

2 article 2004 4 Neurosciences 4.2

3 article 2004 5 Neurosciences 4.2

4 article 2004 5 Neurosciences 4.2

5 letter 2004 1 Neurosciences 1.0

6 article 2002 7 Neurosciences 3.8

7 article 2002 6 Neurosciences 3.8

8 article 2002 1 Neurosciences 3.8

9 article 2003 7 / 4 Neurosciences 5.5 / 4

9 article 2003 7 / 4 Neuroimaging 3.2 / 4

9 article 2003 7 / 4 Radiology 2.8 / 4

9 article 2003 7 / 4 Nuclear Medicine & Medical Imaging

4.6 / 4

www.gu.se

Normalization of citation counts: averages

• CPP = Citation Per Publication (department)• FCSm = Mean Field Citation Score• CPP/FCSm = the Crown Indicator

• Compute CPP: (2 x 7 + 1 x 6 + 2 x 5 + 1 x 4 + 2 x 1 + ¼ x 7 + ¼ x 7 + ¼ x 7+ ¼ x 7) / (2+1+2+1+2+¼ + ¼ + ¼ + ¼) = 4.78

• Compute FCSm: (4 x 4.2 + 1 x 1.0 + 3 x 3.8 + ¼ x 5.5 + ¼ x 3.2 + ¼ x 2.8 + ¼ x 4.6) / (4 + 1 + 3 + ¼ + ¼ + ¼ + ¼) = 3.69

• Compute CPP/FCSm: 4.78 / 3.69 = 1.30

www.gu.se

Normalization of citation counts: paper by paper • Normalization on paper level: 10.48 / 9 = 1.16• Ratio of averages vs. average of ratios

Paper DT PY # citations FCS # citations /FCS

1 article 2004 7 4.2 1.6666672 article 2004 4 4.2 0.9523813 article 2004 5 4.2 1.1904764 article 2004 5 4.2 1.1904765 letter 2004 1 1.0 16 review 2003 7 8.8 0.7954557 article 2002 7 3.8 1.8421058 article 2002 6 3.8 1.5789479 article 2002 1 3.8 0.263158

Sum 10.47967

www.gu.se

Total performance indicator

• The average impact times the total number of papers:

• Where ci is the citation score for paper i, ei the expected number of citations for paper i and N the total number of papers for a department

N

i

N

i i

i

e

cN

N

ec

1 1

11

www.gu.se

A real-life example

• TI- Engineering of a novel Saccharomyces cerevisiae wine strain with a respiratory phenotype at high external glucose concentrations

• JN - Applied and Environmental Microbiology• SC- Biotechnology & Applied Microbiology; Microbiology• DT- article• PY- 2005• AU- Cecilia Henricsson (Institutionen för kemi- och bioteknik, Chalmers)

Maria de Jesus-Ferreira (-)

Kristina Hedfalk (Institutionen för kemi)

Karin Elbing (-)

Christer Larsson (Institutionen för kemi- och bioteknik, Molekylär bioteknik, Chalmers)

Joakim Norbeck (Institutionen för kemi- och bioteknik, Molekylär bioteknik, Chalmers)

Roslyn Bill (-)

Stefan Hohmann (Institutionen för cell- och molekylärbiologi)

Lena Gustafsson (Institutionen för kemi- och bioteknik, Molekylär bioteknik, Chalmers)

www.gu.se

A real-life example

Bibliometric data:• FCS or = field citation score (reference value), self-citations excluded• C = observed citations, self-citations excluded• Observation period 2004-2009

– A citation window of 5 years• Fractionalized author counts• Fractionalized subject category counts

– Journal subject categories: » Biotechnology & Applied Microbiology » Microbiology

www.gu.se

A real-life example

Computing the reference value:– Find all papers published in journals assigned to Biotechnology & Applied

Microbiology and/or Microbiology during 2005-2009.– Count all citation to these papers

– Fractionalize in case of multiple journal subject category assignments

– Compute FCS or :– for Biotechnology & Applied Microbiology , and – for Microbiology.– N = 247 (journals)

www.gu.se

A real life example

C FCS C/FCS PYCitation Interval # authors # departments Count

8 10.709 0.747 2005 5 9 1 0.083

www.gu.se

How to affect the average field-normalized citation score?

• Publish in journals covered by Thomsen ISI (SCOPUS)• Publish in high esteem journals• Use a standard departmental name• Use a standard institutional name