Are Altmetrics a panacea for citation

delays? A critical view of the journey

of the citing article through the

research and impact cycles

Omwoyo Bosire Onyancha

University of South Africa

Department of Information Science

Department of Information Studies (Univ of Zululand)

Conference, 3-5 September 2014

Agenda

Introduction

Interpretation of citations in scholarly communication

Problems with citations

Citation delays

The concept

Statistical data on time delays

Causes of citation delays

Altmetrics – an introduction

Methods and materials

Findings and discussions

Conclusions and recommendations

Introduction

the scientific tradition of citing or bibliographic referencing

(the act of scientists, while documenting their own research,

refer to earlier works that relate to the subject matter of

their reported work) is as old as science itself (Nicolaisen

2008: 609)

The use of citations in research evaluation – “supposed to

identify those earlier researchers whose concepts, theories,

methods equipment and so on, inspired or were used by

the author in the process of conducting and presenting his

or her own research” (Nicolaisen 2008: 610)

How should citations be interpreted?

Citations are signposts left behind after information has

been utilized (Smith, 1981)

Citations are frozen footprints in the landscape of

scholarly achievement ... which bear witness to the

passage of ideas (Cronin, 1981)

Citations are one important form of use of scientific

information within the framework of documented

science communication (Glänzel and Schoepflin 1999)

Citations may be used as a formalised account of the

information use and can be taken as a strong indicator

of reception at this level (Glänzel and Schoepflin 1999)

Citing Smith (1981) and Merton (1973), Bornmann & Daniel (2008: 48)

express the use of citation counts (or citation analysis) as follows:

The use of citation counts as an indicator for research impact is

appropriate only when the citation of a document means that the citing

author used that document; the citation of a document reflects the merit

(quality, significance, impact) of that document; and citations are made to

the best possible works (Smith, 1981)… the normative theory, following

Robert K. Merton's – the founder of the modern sociology of science –

sociological theory of science (Merton, 1973), basically states that scientists

give credit to colleagues whose work they use by citing that work

Some problems with citations and citation

analysis

Assumptions (Wallace 1989)

The citing author has actually used the cited work

Citing a given work is an indicator of its quality

The citing author has provided references to the best works

the content of the citing work is significantly related to the content of the cited

works

that all citations are of equal value

Peters (2006: 786) argues that

authors receive citations for ‘bad’ as well as ‘good’ work;

there are] problems with accuracy, consistency and reliability in citation data

collection;

[there are] problems of self-citation

Kostoff (1998: 28) argues thus:

I could write a paper including Lotka's law without providing a reference to

Lotka's law, or without even mentioning the name 'Lotka's law'

Wang, Wang & Xu (2013: 718) observe that

“It is parochial to regard impact just as citations, since some influential theories,

such as the Merton Miller theorem and Mendelian genetics, are widely

accepted but seldom cited”

Citations to an article happen or are normally counted once the citing article

has been published and, more often, subsequently indexed in the same

indexing service

Citation delay – the concept

The citation delay is sometimes referred to as response time (to the cited document) or citation time lag or citation speed or delayed recognition

Diodato (1994: 157) defines it as “the number of months or years between the publication of a document and the publication of the first item that cites it”.

Although many journals publish articles online before going print, the online-to-print time lag has continued to increase over time Tort, Targino & Amaral (2012) observed that a journal’s impact factor is still

reliant on the publication of the print version of its articles.

The implication of the above is that citation delays will still occur in scholarly communication, thereby leading to “important distortions, with concrete effects on policy decisions concerning the evaluation of journals and/or individual scientists” (Tort, Targino & Amaral 2012: 5)

Causes of citation delay

Wang, Wang & Xu (2013: 718) have observed that “the publication of a scientific paper

requires months to execute the review process, and as a result, significant publication

delay will cause citation delay, and thus cause delay in the current research trend

analysis.

van Els, Janz, and Pair (1989) opine that the number of citations that an author or

article misses may be due to the format of disseminating the research findings. The

authors argue, for instance, that one may miss citations because “the results of

his/her research are made public in the form of an instrument instead of via an

article in a journal”.

Publication delay has also been identified as one of the possible causes of citation

delays by Aman (2013; Amat (nd); Bornmann and Daniel (2010); and Wang, Wang &

Xu (2013)

Indexing time lag, defined as the “number of months or years between the publication

of an article and the publication of an index that refers to the article” Diodato (1994:

157), can delay citations

Studies on citation delays

21.5% of all paper published in 1980 have not received any citation between the

publication year and 2000. Most of these papers will probably never be cited (Glazel,

Schlemmer, &Thijs, 2003: 574).

The number of articles published without a previous preprint in this time period

(1996-2009) is 424, of which 53 remained uncited in their first three years (12.5%).

The number of articles with a foregoing preprint is 287, of which 21 remained uncited

in their first three years (7.3%). To include the uncited articles into the examination,

their citation delay was set to 1,095 days (Aman, 2013: 39).

The median value of the citation delay for articles having a preprint is 304 days. Articles

without a preprint receive their first citation on average three months later; the median

is 395 days (Aman, 2013:39).

Citation delays have been studied under different contexts (see Bornmann and Daniel,

2010; van Els, Janz, and Pair, 1989; RINIA, et al, 2001)

Altmetrics – an introduction

Altmetrics is sometimes called social web metrics (Costas, Zahedi & Wouters, nd) and/or influmetrics (Cronin & Weaver, 1995; Rousseau & Ye, 2013)

Refers to the use of social media, particularly Web 2.0 media, in assessing the influence of researchers on all type of users (Rousseau & Ye, 2013:1)

Refers to mentions of scientific outputs in social web tools such as Facebook, Twitter, blogs, news media or online reference management tools (Costas, Zahedi & Wouters, nd)

Some of the metrics that have been identified as constituting Alt-metrics include downloads, tweets, likes, posts on Facebook, blogs, bookmarks, views, visits,

However, Altmetrics go beyond raw usage factors (such as downloads or click-through rates) in that they focus on readership, diffusion and reuse indicators that can be tracked via blogs, social media, peer production systems, collaborative annotation tools (including social bookmarking and reference management services).

How many times an output article, website, blog, dataset, grey literature, software, etc

has been: Viewed (Publisher websites, Dryad)

Downloaded (Slideshare, publisher websites, Dryad)

Cited (PubMed, CrossRef, Scopus, Wikipedia, DOI, Web of Science)

Reused/Adapted (Github)

Shared (Facebook, Twitter)

Bookmarked (Mendeley, CiteULike, Delicious)

Commented upon (Twitter, Mendeley, blogs, publisher websites, Wikipedia, Faculty of 1000)

Konkiel (nd) believes that Altmetrics is concerned with assessing:

Research question

Are Altmetrics a panacea for citation delays?

Methods and materials

For us to answer this question, we first sought to revisit as well as establish the citation delays using the citing ‘work’ as opposed to the cited ‘work’ as the unit of analysis

A closed-ended question containing 9 questions (8 of which were closed-ended) was sent to18 academics in LIS schools in South Africa

The 18 researchers were purposively selected on the basis of having obtained a PhD degree

So far, we have received only 14 completed questionnaires

We also obtained data on the file visits and views of the top most visited or viewed documents in academic institutional repositories and compared it with the Google Scholar citations to determine whether or not Altmetrics could have a role to play in citation delays

Findings and discussions

At what stage of the research cycle do researchers often use literature to

inform their research?

Never Rarely Sometime

s Most of

the Time Always

Before identifying of the research idea 2 5 8

Identification of the research idea 1 2 9

Identification of theory 2 12

Identification of empirical entities 1 3 9

Data acquisition 1 3 4 4 2

Data conversion 2 2 4 5 1

Data analysis 1 4 5 4

Publication 1 4 5 3

At what stage of the research cycle do researchers often cite literature to

inform their research?

Never Rarely Sometimes Most of

the Time Always

Before identifying the research idea 1 3 4 2 2

Identification of the research idea 4 3 6

Identification of theory 5 9

Identification of empirical entities 2 3 7

Data acquisition 4 4 2 3

Data conversion 2 4 2 4

Data analysis 2 3 2 6

Publication 2 3 3 5

Which of the following best describes how researchers cite the sources that

they have consulted when writing a research publication (i.e. article, chapter in

a book, conference paper, etc)?

a. I list the sources even before I have started writing the publication [ 1 ]

b. I cite the sources as I write the publication [ 13 ]

c. I cite the sources after I have completed writing the publication [ 0 ]

d. Any other, please specify………………………………………… [ 0 ]

No of respondents Average time (in months) Research article 8 1,8 Chapter in Book 5 1,3 Masters Dissertation* 10 4,5 Doctoral thesis 10 7,5 Technical reports 5 1,6 Conference papers 9 1,4 Application for funding 11 2,6

How long, on average, did it take the researchers to complete the

research proposal, in terms of the number of months, for the following

categories/purposes of research?

Average completion period 3.3 months

Now, let us compare citation opportunities

missed along the research process vis-à-vis

the altmetrics opportunities

Identification of

research problem

Identification of

theory

Identification of

empirical methods

Identification of

empirical entities

Data acquisition

Concept and

operationalization

Data conversion

Data analysis

Publication

Mind maps

Reference management

Literature search

Compilation of question

Interview procedure

Transcription

Qualitative data analysis

Quantitative data analysis

Scientific writing

Figures

Identification of

research problem Mind maps

You can share your mind map and citations can be picked by social media

But if not published

4.7 months ‘citation’

period is lost

If shared, an

Altmetric data is

created instantly

Identification of

theory

Reference management

Literature search

You can share the literature

so far identified by sharing

what you have read

Mendeley for example informs

your followers in Tweeter what

you are reading

If the literature you have read and identified as relevant to your study is not shared (or

published) a citation period of about 2.6 months is lost

Identification of

empirical methods

Identification of

empirical entities

Concept and

operationalization

Reference management

Literature search

These processes take an average of 3.5

months which can be missed opportunity

for citation if the outcomes are not

published in mainstream publication outlets

The illustration can be shared

through social media – with

citations supporting its

formulation

The sharing may be

immediately upon its

completion

Data acquisition

Compilation of question

Interview procedure

On average, data acquisition takes about 2.1 months to complete

Some respondents indicated

that they do cite sources while

collecting data

If the product so far developed

as at the data collection stage

is not published, another

citation opportunity is lost for

‘cited sources’

One can share information about the methods of data collection, population determination,

sampling procedures, etc on social media – instant citation impact for cited source realized

Data conversion Transcription

Researchers spend about 2.7 months to complete

data conversion

NB: out of 14 respondents, only 2 who indicated

that they do not cite consulted sources at the

time of data conversion (e.g. transcription).

If the product of data conversion is not shared

through publication, another 2.7 months of

citation period are missed.

At this stage, one can share data (with cited sources) through social media

Data analysis

Qualitative data analysis

Quantitative data analysis

Average time it takes a LIS

researcher to complete data

analysis = 2.1 months

A few researchers indicated that they

cite sources during this stage of the

research process

Data analysis techniques,

analyzed data, presentation

skills, can be shared among

researchers as research

progresses

If any product resulting from this stage is

not published, the cited author will not

be recognized before a total of 17.7

months elapse

Publication

Scientific writing

Figures

According to the 14 respondents

investigated for this paper,

publication of various types of

research products takes an

average of 7.8 months

Publication can be online or print

or both.

Online takes a shorter period but

citation delay is still long (Aman,

2013)

We all know that publication of a

manuscript takes much longer

when done in print

Identification of

research problem

Identification of

theory

Identification of

empirical methods

Identification of

empirical entities

Data acquisition

Concept and

operationalization

Data conversion

Data analysis

Publication

Mind maps

Reference management

Literature search

Compilation of question

Interview procedure

Transcription

Qualitative data analysis

Quantitative data analysis

Scientific writing

Publication of manuscript

4.7

2.6

3.5

2.1

2.7

2.1

7.8

2.1

3.2

2.9

4.0

1.6

2.8

2.7

2.5

1.7

6.1

9.7

What does the above illustration imply?

Although the citing author (Author A or product A) has mentioned (cited Author B or

Article B) in his/her work at or before point A, only after his/her work reaches point B

does the citation get recognized in traditional citation analysis.

Identification of a

research idea for

Article A

Publication of

Article A

t0 tt

25.5 months

Article B was

first ‘used’ by

author A

Article B receives

its first citation in

Article A

Are Altmetrics the answer to citation

delays?

Citations vs altmetrics (downloads, visits

and views)

Median average Total Least

intenger Greatest intenger

Citations 0 14,32 859 0 750 File visits 809 2177,02 130,621 25 28,452

Views 2571,5 3327,8 199,668 313 24,979

Uncited

documents (60%)

0 1 2 3 4 5 7 17 27 750

Number of items 36 4 3 9 1 2 2 1 1 1

0

5

10

15

20

25

30

35

40

Nu

mb

er

of

cit

ati

on

s p

er

item

N = 60 docs

Empirical results – IR downloads and views

vs Google citations

0

5000

10000

15000

20000

25000

30000

35000

a b c d e f g h i j a b c d e f g h i j a b c d e f g h i j a b c d e f g h i j a b c d e f g h i j a b c d e f g h i j

DUT UP NWU Unisa UJ Wits

Citations File Visits Views

Conclusion – are altmetrics a panacea for

citation delays?

Altmetrics can appear more rapidly than citations (Thelwall, Haustein, Lariviere, & Sugimoto,

2013; Eysenbach, 2011: abstract)

Piwowar & Priem (2013: 11), argue that citations take years to accrue… “This delay is a big

problem for graduate students who are applying for jobs soon after publishing their first

papers and for those promotion candidates whose most profound work is published only

shortly before review”

Altmetrics should be aimed at complementing and improving the limitations of both

traditional (i.e. bibliometrics) and web based (e.g. download and usage data) impact metrics

and giving new insights to the analysis of impact (Galligan & Dyas-Correia, 2013 as cited in

Costas, Zahedi & Wouters, nd)

BUT

YES

Altmetrics considers all the stages and products of scholarly research from ‘‘[…] social

literature search via Facebook to discussion of published results via Twitter, including any

impact a publication or author may have on other people […]’’ (Bar-Ilan et al. 2012: 2 in

Hammarfelt, 2014: nd)

Strongly

agree Agree

Neither agree nor disagree

Disagree Strongly disagree

Altmetrics should be used to supplement traditional metrics (e.g. citations and citation impact)

6 4 3 1

Altmetrics should be used as replacement of traditional metrics (e.g. citations and citation impact)

3 4 7

Altmetrics should be cautiously used to measure research impact

3 7 2 2

Altmetrics should NEVER be used to measure research impact

1 1 7 5

I prefer Altmetrics to traditional citation metrics

1 6 6 1

I prefer traditional citation metrics to Altmetrics

7 5 1 1

In your opinion, what would your preference be in terms of the usage of

these metrics in supporting the measurement of research impact?

END

Thanks