1

b

b

www.know-center.at

Altmetrics

Open Science – Technologie und Wissenschaft

Wien, 13. Jänner

Dr. Peter Kraker

2

AN EXPONENTIALLY

EXPANDING UNIVERSE

Part 1

3

The Hitchhiker‘s Guide to the Galaxy

3 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics

4

How to Stay on Top of the Universe

4 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics

5

How to Stay on Top of Scientific Literature

5 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics

6 6 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics

“One of the diseases of this age is the

multiplicity of books;

they doth so overcharge the world that it is not

able to digest the abundance of idle matter that

is every day hatched and brought forth into the

world.“

Attributed to Barnaby Rich in 1613

7 7 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics

Price (1963)

8

SCIENTOMETRICS:

THE QUANTITATIVE

ANALYSIS OF SCIENCE

Part 2

9

Eugene Garfield: Pathways through Science

9 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics

Science Citation Index (Garfield 1955)

Web of Science/Web of Knowledge

An index of incoming citations

Outgoing citations

Incoming citations

10 10

Garfield et al.

(1964)

Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics

11 11 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics

12 12 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics

Van Eck and Waltman

(2010)

13

Evaluative Scientometrics: Impact Factor

13 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics

A measure to quantify the relative importance of a scientific

journal

The average number of citations in a given year y to papers of a

journal in the years y-1 and y-2

𝐼𝐹𝐽𝑜𝑢𝑟𝑛𝑎𝑙 𝑋, 2015 =

𝐶𝑖𝑡𝑎𝑡𝑖𝑜𝑛𝑠 𝑖𝑛 2015 𝑡𝑜 𝑎𝑟𝑡𝑖𝑐𝑙𝑒𝑠 𝑝𝑢𝑏𝑙𝑖𝑠ℎ𝑒𝑑 𝑏𝑦

𝐽𝑜𝑢𝑟𝑛𝑎𝑙 𝑋 𝑖𝑛 2013 𝑎𝑛𝑑 2014

𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑎𝑟𝑡𝑖𝑐𝑙𝑒𝑠 𝑝𝑢𝑏𝑙𝑖𝑠ℎ𝑒𝑑 𝑏𝑦 𝐽𝑜𝑢𝑟𝑛𝑎𝑙 𝑋

𝑖𝑛 2013 𝑎𝑛𝑑 2014

14 14 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics

Source: Thomson Reuters

15 15 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics

Journal X

Paper 1 Citations 15

Paper 2 Citations 17

Paper 3 Citations 14

Paper 4 Citations 18

Paper 5 Citations 15

Paper 6 Citations 15

Impact Factor 15.5

Median 15 2

Journal Y

100

2

1

2

1

2

18.0

Rank 2 1

16

Criticisms of the Impact Factor

16 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics

Source: Amin & Mabe (2000)

17

Antidotes

17 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics

„Blockbuster“ papers can skew the IF

Use robust statistics (e.g. the median)

Report variation in the data (CWTS Journal Indicators)

Publication and citation behavior varies wildly between fields

Source-normalised impact factor (SNIP)

The IF window is too short

3- and 5-year Impact Factors

18

Criticisms of citation-based metrics

18 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics

Citations take very long to appear in meaningful quantities

Citation-based metrics are dependent on the corpus that is used

for calculation

Citations are mostly assigned to articles, leaving other products

of research out of the picture (data, code etc.)

Citation-based metrics are often intransparent and irreproducible

19

ALTMETRICS:

A NEW HOPE

Part 3

20

Setting the Stage for Alternative Metrics

20 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics

Open Science and the publication of more research products

Increased use of online services in the scientific community

Source: figshare

21

Altmetrics

21 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics

Altmetrics: metrics based on data generated in online systems

Seeing academic literature through the eyes of the readers (Rowlands

& Nicholas 2007)

Usage data (downloads, views, readership)

Links, likes and shares

Blogs and comments

Promises of altmetrics

Assess publications quicker and on a broader scale

Consider all outputs of research, not just papers

The altmetrics manifesto: http://altmetrics.org

22 22 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics

http://zenodo.org/record/34079

23 23 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics

http://www.altmetric.com/details/4822452

24 24 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics

Source:

https://impactstory.org/Carl

Boettiger

25 25 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics

26 26 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics

http://openknowledgemaps.org

Kraker et al.

(2014)

27 27 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics

http://stellar.know-center.tugraz.at/umap

Evolution of a Knowledge Domain

Kraker et al.

(2014)

28

The Emerging Open Digital Science Ecosystem

Bibliographic data – bibliometric data – full texts – open source software

29

CAVEATS

Part 4

30 30 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics

31 31 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics

32 32 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics

Kraker & Lex

(2014)

33

The Leiden Manifesto: Ten Principles for Research

Metrics (Hicks et al. 2015)

1) Quantitative evaluation should support qualitative, expert

assessment

…

4) Keep data collection and analytical processes open,

transparent and simple

5) Allow those evaluated to verify data and analysis

…

9) Recognize the systemic effects of assessment and indicators

10) Scrutinize indicators regularly and update them

34

Further Reading

Blog: http://blog.wikimedia.de/2015/09/15/freies-wissen-und-

wissenschaft-teil-4-ueber-die-notwendigkeit-fuer-mehr-offenheit-

und-transparenz-in-der-qualitaetssicherung-und-evaluierung-

open-peer-review-und-open-metrics/

Publication: Kraker, P., & Lex, E. (2015). A Critical Look at the

ResearchGate Score as a Measure of Scientific Reputation.

In ASCW’15 Workshop at Web Science 2015.

doi:10.5281/zenodo.35401

Source Code: https://github.com/pkraker/Headstart

35

b

Thank you for your attention!

Peter Kraker

pkraker@know-center.at

Twitter: @PeterKraker