J U L I A L A N EA M E R I C A N I N S T I T U T E S F O R R E S E A R C H

THE USES, BENEFITS AND PITFALLS OF BIBLIOMETRICS

TO DESCRIBE SCIENTIFIC COMMUNICATION

OUTLINE

• Context• Scientific Communication• Bibliometrics

• Uses, Benefits and Pitfalls• Some other ideas and opportunities• What can be done

SCIENTIFIC COMMUNICATION

Knowledge• Creation• Transmission• Adoption

OPEN ACCESS CONTEXT

• Authors: of such articles, who will see their papers more read, more cited, and better integrated into the structure of science

• Academic readers: in general at institutions that cannot afford the journal, or where the journal is out of scope

• Researchers: at smaller institutions, where their library cannot afford the journal

• Readers: in general, who may be interested in the subject matter• The general public: who will have the opportunity to see what

scientific research is about• Taxpayers: who will see the results of the research they pay for• Patients: and those caring for them, who will be able to keep

abreast of medical researchSource: http://en.wikipedia.org/wiki/Open_access_journal accessed May 26, 2012

Bo-Christer Björk, 2007

NODE: TITLE: NUMBER:Do research, communicate and apply the resultsA0

Disseminatedscientificknowledge

Publication

Funding for R&D

New scientific knowledgeExisting ScientificKnowledge

Scientificproblems

Better qualityof life

Fundingforresearch

Funding forresearchcommunication

Funding forindustrialdevelopment

Public sectorfunding

Private sectorfunding

Priorities for scientificproblems to be solved

Research funders

ScientistsCompanies Government

Publishersandinfomediaries

Readers

1

Fund R&D

2

Perform the research

3

Communicate the results

4

Apply the Knowledge

SCIENTIFIC COMMUNICATION

• Is about people

BIBLIOMETRICS: USES

• …a set of methods to quantitatively analyze scientific and technological literature..used.

• …in library and information sciences• …to explore the impact of

• their field, • a set of researchers, or • a particular paper.

• …. in quantitative research assessment exercises of academic output which is starting to threaten practice based research

• Paraphrased from http://en.wikipedia.org/wiki/Bibliometrics

BIBLIOMETRICS

• Is about documents

BENEFITS

Benefits• Have focused attention on quantitative measures

of impact• Attracted some smart people to think about hard

problems• Identified some interesting patterns

PITFALLS(ILLUSTRATIVE NOT EXHAUSTIVE)

Scientific validity• Limited in its behavioral micro-foundations• Based on suspect scientific frame (unit of

analysis, currency, coverage etc.)• Not generalizable or replicableInferential validity• Generates spurious results• Subject to misuse and gamingValue for evaluation• Creates perverse incentive structure

ILLUSTRATIVE CRITICISM

Evaluators often rely on numerically–based shortcuts drawn from the closely related fields (Hood and Wilson, 2001) of bibliometrics and scientometrics — in particular, Thompson Scientific’s Journal Impact Factor (JIF). However, despite the popularity of this measure, it is slow (Brody and Harnad, 2005); narrow (Anderson, 2009); secretive and irreproducible (Rosner, et al., 2007); open to gaming (Falagas and Alexiou, 2008); and based on journals, not the articles they contain.

Priem and Hemminger, 2010

ILLUSTRATIVE EXAMPLE

WHY GETTING IT RIGHT MATTERS

MEASURING COMMUNICATION:SOME IDEAS AND OPPORTUNITIES

Focus on people (scientists)• How to start a movementMake use of scientific advances• New theories (graph theory; RCT)• New applications (social networks)• New ways of communicating knowledge• New application (graph oriented databases)• New data (natural language processing; computational

linguistics)New opportunities=> Potential for new science, new scientific field and theoretically grounded, metrics

CONCEPTUAL FRAMEWORK

Source: Ian Foster University of Chicago

SCIENTIFIC ADVANCES

• Graph Theory applied to social networks• E.g. Jason Owen Smith

• dyadic measures of the strength of individual ties; • structural measures of cohesion in the overall network; and • node demographic measures that highlight the degree of

heterogeneity of academic researchers

• Randomized Controlled Trials• E.g John Willinsky

• Physicians• Community Health Organizations

NEW WAYS OF COMMUNICATING KNOWLEDGE

Table 1: A partial list of popular Web 2.0 tools, and similar tools aimed at scholars.Description General–use application Scholarship–specific application

Social bookmarking Delicious(http://delicious.com/)

CiteULike(http://www.citeulike.org/,Connotea(http://www.connotea.org/)

Social collection management iTunes(http://www.apple.com/itunes/)Mendeley(http://www.mendeley.com/,Zotero(http://www.zotero.org/)[reference managers]

Social news/recommendationsDigg(http://digg.com/),Reddit(http://www.reddit.com/),FriendFeed(http://friendfeed.com/)

Faculty of 1000(http://facultyof1000.com/),[similar, but curated]

Publisher–hosted comment spaces (e.g., blog comments) Most Web 2.0 applications

British Medical Journal http://www.bmj.com/),PloS(http://www.plos.org/),BioMed Central(http://www.biomedcentral.com/),Bioinformatics (Oxford University Press journal)(http://bioinformatics.oxfordjournals.org/)

Microblogging Twitter(http://twitter.com/)

User–edited reference Wikipedia(http://www.wikipedia.org/)Encyclopedia of Life(http://www.eol.org/),Scholarpedia(http://www.scholarpedia.org/),Citizendium(http://en.citizendium.org/)

Blogs Wordpress.com(http://wordpress.com/),Blogger(https://www.blogger.com)

Research Blogging(http://researchblogging.org/),Blogger(https://www.blogger.com)

Social networks

Facebook(http://www.facebook.com/),MySpace(http://www.myspace.com/),Orkut(http://www.orkut.com/)

Nature Networks(http://network.nature.com/),VIVOweb(http://vivoweb.com/);

Data repositories DBPedia(http://dbpedia.org/About) GenBank(http://www.ncbi.nlm.nih.gov/genbank/)

Social videoYouTube(http://www.youtube.com/),Vimeo(http://www.vimeo.com/)

SciVee(http://www.scivee.tv/)

Table 2: Calls for Web 2.0 metrics of scholarship.

SourceSuggested Web 2.0 sources for

metricsMain use

M. Jensen (2007)Tags, “discussions in blogspace, comments in posts, reclarification, and continued discussion.”

Establishing scholars’ authority

Taraborelli (2008)Social bookmarking: CiteULike, Connotea

Augmenting or replacing peer review

Anderson (2009)Twitter, blogs, video and “Wikipedia, or any of the special ‘–pedias’ out there”

Broadening the scope of the JIF

Neylon and Wu (2009)Zotero, Mendeley, CiteULike, Connotea, Faculty of 1000, article comments

Filtering articles

Norman in Cheverie, et al. (2009)“scholastic bookmarking, and tagging (e.g., the ‘Slashdot index’) … academic networks like LinkedIn”

Tenure and promotion

Patterson (2009)

“… social bookmarks; blog coverage; and the Comments, Notes and ‘Star’ ratings that have been made on the article.”

“[A]ssessing research articles on their own merits …”

Scientometrics 2.0: Toward new metrics of scholarly impact on the social Web by Jason Priem and Bradley M. Hemminger.First Monday, Volume 15, Number 7 - 5 July 2010http://firstmonday.org/htbin/cgiwrap/bin/ojs/index.php/fm/article/viewArticle/2874/

NEW DATA

• Natural language processing (=> mine actual text and identify topics)

• Information extraction to capture data on people and institutions

WHAT CAN BE DONE

• Build on current advances to develop and automatically extract de-duplicated cross-referenced database of • papers (and references)• Topics• People• Grants• publication venues• Institutions

• Build a common data infrastructure• STAR METRICS

WHAT CAN BE DONE TO CREATE NEW SCIENCE

• Agree on handful of people based metrics• Engage social scientists to develop open and

transparent data and standardized measures• Use them

REFERENCES (PLUS THE NEW YORKER)

• McCabe, Mark J. and Snyder, Christopher M., The Economics of Open-Access Journals (July 9, 2010). Available at SSRN: http://ssrn.com/abstract=914525 or http://dx.doi.org/10.2139/ssrn.914525

• McCabe, Mark J. and Snyder, Christopher M., Did Online Access to Journals Change the Economics Literature? (January 23, 2011). Available at SSRN: http://ssrn.com/abstract=1746243 or http://dx.doi.org/10.2139/ssrn.174624

• Philip M. Davis Open access, readership, citations: a randomized controlled trial of scientific journal publishing FASEB J July 2011 25:2129-2134; published ahead of print March 30, 2011, doi:10.1096/fj.11-183988

• Evans, James and Jacob Reimer (2009) “Open Access and Global Participation in Science,” Science 323: 1025.

• Leydesdorff, L. (2008), Caveats for the use of citation indicators in research and journal evaluations. J. Am. Soc. Inf. Sci., 59: 278–287. doi: 10.1002/asi.20743

• Scientometrics 2.0: Toward new metrics of scholarly impact on the social Webby Jason Priem and Bradley M. Hemminger.First Monday, Volume 15, Number 7 - 5 July 2010http://firstmonday.org/htbin/cgiwrap/bin/ojs/index.php/fm/article/viewArticle/2874/

• Kaye Husbands-Fealing, Julia Lane, Jack Marburger, Stephanie Shipp, and Bill Valdez The Handbook of Science of Science Policy,, Stanford University Press, 2011.

• Julia Lane and Stefano Bertuzzi “Measuring the Results of Science Investments” Science, Volume 331, pages 678-680, February 11, 2011.

• Julia Lane “Let’s Make Science Metrics More Scientific” Nature, Volume 464, pages 488–489, March 25, 2010.

• Julia Lane“Assessing the Impact of Science Funding” Science, Volume 324. no. 5932, pp. 1273 – 1275, 5 June 2009.