bibliometrics: an insider’s view
TRANSCRIPT
BibliometricsBibliometrics: An : An insiderinsider’’s Views View
Sabina Robertson, University of MelbourneSabina Robertson, University of MelbourneLynne Horwood, University of MelbourneLynne Horwood, University of Melbourne
Jenny Cameron, Victoria UniversityJenny Cameron, Victoria University
CRIG: Information Literacy in Interesting Times CRIG: Information Literacy in Interesting Times November 25, 2009November 25, 2009
Presentation OverviewPresentation Overview
Part One: Context (Sabina Robertson)Part One: Context (Sabina Robertson)
Part Two: Measures, products and Part Two: Measures, products and applications (Lynne Horwood)applications (Lynne Horwood)
Part Three: Models of research support: Part Three: Models of research support: (Jenny Cameron)(Jenny Cameron)
Part One: ContextPart One: Context Government policies setting Government policies setting
the research scenethe research scene
Rankings of universitiesRankings of universities•• Promotion, awarding of Promotion, awarding of
research fundsresearch funds
Our academics responding Our academics responding to new research environmentto new research environment
ERAERA•• issues issues –– different disciplines different disciplines
need different measuresneed different measures
Government imperativesGovernment imperatives Positioning Australia in Positioning Australia in
the International arenathe International arena Competition from India & Competition from India &
China China Identifying areas for Identifying areas for
research investmentresearch investment Attracting Researchers to Attracting Researchers to
AustraliaAustralia Universities accountable Universities accountable
for R & D fundingfor R & D funding
Perspective from a University Perspective from a University Vice Chancellor Vice Chancellor
Rigorous monitoring and reviews:Rigorous monitoring and reviews:
InternationalInternational ranking systems: Shanghai Jiao Tong World Universities & ranking systems: Shanghai Jiao Tong World Universities &
Times Higher Education RankingsTimes Higher Education Rankings
National National •• Performance reports, National protocols & auditsPerformance reports, National protocols & audits
LocalLocal•• Internal faculty, departmental and divisional reviews with exterInternal faculty, departmental and divisional reviews with external nal
reviewersreviewers
ReputationReputation•• Individual reputations of institutions built up over long periodIndividual reputations of institutions built up over long periods of times of time•• Includes community engagementIncludes community engagement
High Quality University SystemsHigh Quality University SystemsCurrent review and monitoring systems have achieved:Current review and monitoring systems have achieved:
Universities aspiring to quality and excellence in:Universities aspiring to quality and excellence in: ResearchResearch InnovationInnovation Student learningStudent learning TeachingTeaching Community and industry engagementCommunity and industry engagement
Universities actively pursuing their different missionsUniversities actively pursuing their different missions
Demand for university placesDemand for university places
Demand from government and industry for university researchDemand from government and industry for university research
Australian Universities: Australian Universities: International RankingsInternational Rankings
Australia has an international reputation of having a strong uniAustralia has an international reputation of having a strong universityversitySystem in research; teaching and learning; international EducatiSystem in research; teaching and learning; international Educationon
Reflected in Universities in International Rankings system:Reflected in Universities in International Rankings system:
Shanghai Jiao TongShanghai Jiao Tong 17 of 38 Australian Universities in Top 500;17 of 38 Australian Universities in Top 500; 7 in Top 2007 in Top 200
Times Higher Education (THE)Times Higher Education (THE) 12 Australian Universities in Top 20012 Australian Universities in Top 200
Australia performs highly on international rankings.Australia performs highly on international rankings.
World University Rankings: World University Rankings: indicators used indicators used
The rankings derived from calculations based on The rankings derived from calculations based on data gathered from the following indicators:data gathered from the following indicators: Academic peer review Academic peer review Employer review Employer review International academic staff ratio International academic staff ratio International student ratio International student ratio Student : academic staff ratio Student : academic staff ratio
Citations per academic staff member Citations per academic staff member (citation data supplied by Scopus) (citation data supplied by Scopus)
Role of The Australian Research Role of The Australian Research Council (ARC)Council (ARC)
ARC roles:ARC roles: A statutory authority within the Australian GovernmentA statutory authority within the Australian Government’’s s
Innovation, Industry, Science and Research portfolio.Innovation, Industry, Science and Research portfolio.
Advises the Government on research matters and Advises the Government on research matters and manages the National Competitive Grants Program manages the National Competitive Grants Program (NCGP) . (NCGP) .
Responsible for the Australian research quality and Responsible for the Australian research quality and evaluation system evaluation system –– the the Excellence in Research for Excellence in Research for AustraliaAustralia (ERA) initiative.(ERA) initiative.
Excellence in Research for Excellence in Research for Australia (ERA): the experienceAustralia (ERA): the experience
Cluster Trials 2009: Cluster Trials 2009: Cluster 1: Cluster 1:
•• Physical, Chemical & Physical, Chemical & Earth Sciences PCE)Earth Sciences PCE)
Cluster 2:Cluster 2:•• Humanities and Creative Humanities and Creative
Arts (HCA)Arts (HCA)
Data collection rangesData collection ranges
Submission data for ERA was collected for the Submission data for ERA was collected for the following reference periods:following reference periods:
Data Type Reference Period YearsData Type Reference Period YearsResearch Outputs 1 January 2002Research Outputs 1 January 2002––31 December 31 December
2007 (6 years)2007 (6 years)Research Income 1 January 2005Research Income 1 January 2005––31 December 31 December
2007 (3 years)2007 (3 years)Applied Measures 1 January 2005Applied Measures 1 January 2005––31 December 31 December
2007 (3 years)2007 (3 years)
ERA: Eligible Research ERA: Eligible Research OutputOutput
For the PCE and HCA clusters, the common For the PCE and HCA clusters, the common eligible research output types were:eligible research output types were:
Books: Authored ResearchBooks: Authored Research Book: Chapters in Research BookBook: Chapters in Research Book Journal Articles: Refereed, Scholarly Journal Articles: Refereed, Scholarly
JournalJournal Conference Publication: Full Paper Conference Publication: Full Paper
Refereed.Refereed.
Humanities and Creative Arts (HCA) Humanities and Creative Arts (HCA) cluster criteria. cluster criteria.
Criteria used to select 20% of research output as Criteria used to select 20% of research output as follows:follows:
articles that are published in articles that are published in A*A* or or A A category category journals journals ERA journal listsERA journal lists
books that discipline heads consider to have had a books that discipline heads consider to have had a major impact in the field major impact in the field -- not limited by publishernot limited by publisher
book chapters book chapters –– assessed on criteria set by ERA assessed on criteria set by ERA Committee of expertsCommittee of experts
Use of Information gathered from Use of Information gathered from ERA cluster trialsERA cluster trials
The Australian Research Council (ARC) will The Australian Research Council (ARC) will publicly release outcomes of the evaluations publicly release outcomes of the evaluations for Clusters One and Two aggregated to the for Clusters One and Two aggregated to the national level but not broken down to the national level but not broken down to the level of individual institutions.level of individual institutions.
ARC will also make available to individual ARC will also make available to individual institutions analyses of their own disciplines. institutions analyses of their own disciplines. These individual analyses will not be publicly These individual analyses will not be publicly released by ARC. released by ARC.
University of Melbourne Research & University of Melbourne Research & Innovation OfficeInnovation Office
University of Melbourne Research University of Melbourne Research PerformancePerformance
An AcademicAn Academic’’s response: Dr s response: Dr Christopher MarshallChristopher Marshall
ERA ranking: A journal
“A” ranked ERA journal“A” ranked ERA journal
“A” ranked ERA journal
Measures, products and usesMeasures, products and uses
Evaluating researchers Evaluating researchers
Measuring citation impact (research Measuring citation impact (research impact)impact)
Evaluating journalsEvaluating journals
hh--indexindex
The The hh--index was developed in 2005 by index was developed in 2005 by Jorge Hirsch, a condensedJorge Hirsch, a condensed--matter matter physicist at the University of California in physicist at the University of California in San Diego.San Diego. HirschHirsch’’s aim was to qualify the s aim was to qualify the impact and quantity of an individual impact and quantity of an individual scientistscientist’’s research output.s research output.
HirschHirsch’’s original paper appears at: s original paper appears at: http://arxiv.org/abs/physics/0508025http://arxiv.org/abs/physics/0508025
hh--index and citation patternsindex and citation patterns
Calculating the Calculating the hh--indexindex
hh--index and index and Web of ScienceWeb of Science
Google Scholar Universal Google Scholar Universal GadgetGadget
A Google Scholar Universal Gadget A Google Scholar Universal Gadget enables users to search for the total enables users to search for the total number of citations of number of citations of author(sauthor(s). It ). It provides a total citation count, total provides a total citation count, total number of cited publications and Jorge E. number of cited publications and Jorge E. HirschHirsch’’s s hh--Index.Index.
Sources for Sources for hh--index calculation index calculation ––comparison of the three productscomparison of the three products
Web of ScienceWeb of Science117 papers 117 papers hh--index = 31index = 31
Google ScholarGoogle Scholar158 publications158 publicationshh--index =34index =34
ScopusScopus84 papers84 papershh--index = 25index = 25
Evaluating researchersEvaluating researchers
Citation analysisCitation analysis
Examines the elements of bibliographic Examines the elements of bibliographic recordsrecords
Studies relationships between citationsStudies relationships between citationsMeasures the amount of activity in a Measures the amount of activity in a
subject category, journal, country or topic subject category, journal, country or topic
Citation counts and 3 productsCitation counts and 3 products
McGorryMcGorry, P. D., Edwards, J., , P. D., Edwards, J., MihalopoulosMihalopoulos, C., , C., HarriganHarrigan, S. M., & Jackson, H. J. (1996). EPPIC: , S. M., & Jackson, H. J. (1996). EPPIC: An evolving system of early detection and An evolving system of early detection and optimal management. optimal management. Schizophrenia Bulletin, Schizophrenia Bulletin, 2222(2), 305(2), 305--326.326.
Cited Cited 460460 times (times (Google ScholarGoogle Scholar))Cited Cited 339339 times (times (Web of ScienceWeb of Science))Cited Cited 349349 times (times (ScopusScopus))
ScopusScopus : Arts and Humanities : Arts and Humanities coverage increased in 2009coverage increased in 2009
Subjects with the most journals include Subjects with the most journals include literature and literary theory (30% of new titles) literature and literary theory (30% of new titles)
general arts and humanities (22%) general arts and humanities (22%) history (17%) history (17%) visual/performing arts (16%) visual/performing arts (16%)
More than half (60%) are published from the EMEA More than half (60%) are published from the EMEA (Europe, Middle East and Africa) region, 38% from the (Europe, Middle East and Africa) region, 38% from the Americas and 2% are from AsiaAmericas and 2% are from Asia--Pacific.Pacific.
ScopusScopus –– title, abstract, keyword title, abstract, keyword searchsearch
ScopusScopus Citation trackerCitation tracker
Alternatives to Alternatives to Web of ScienceWeb of Science and and ScopusScopus
CiteSeerXCiteSeerX (Information Technology) (Information Technology) Google Scholar (Multidisciplinary) Google Scholar (Multidisciplinary) ScienceDirect(MultidisciplinaryScienceDirect(Multidisciplinary)) Business Source Premier (Business Business Source Premier (Business
research database) research database)
Most cited computer science Most cited computer science articlesarticles
ScienceDirectScienceDirect
Business Source PremierBusiness Source Premier
Researcher IDResearcher ID
http://http://researchanalytics.thomsonreuters.comresearchanalytics.thomsonreuters.com//
Tracking hot papersTracking hot papers
Science WatchScience Watch Essential Science Essential Science
IndicatorsIndicators Highly Highly Cited.comCited.com
ISIHighlycited.comISIHighlycited.com
Essential Science IndicatorsEssential Science Indicators
Journal evaluationJournal evaluation
To which journal should you submit your To which journal should you submit your high profile research for publication?high profile research for publication?
RejectaRejecta MathematicaMathematica
Journal for papers Journal for papers rejected elsewhererejected elsewhere
Measuring journal quality Measuring journal quality –– journal journal ranking systemsranking systems
Impact factor is one measureImpact factor is one measure Journal rankings by subject groupingJournal rankings by subject grouping ARC list of journals ARC list of journals CORE CORE –– computing journalscomputing journals ERIH ERIH ––humanitieshumanities BARDsNETBARDsNET -- economicseconomics
ScopusScopus –– Journal Journal AnalyzerAnalyzer
Journal Citation ReportsJournal Citation Reports
Journal Citation ReportsJournal Citation Reports
5 year impact factor as a new measure5 year impact factor as a new measure
Impact factorsImpact factors
Impact factor criticismImpact factor criticism““Beware the tyranny of impact factorsBeware the tyranny of impact factors””..““Challenging the tyranny of impact factorsChallenging the tyranny of impact factors””..““Corruption of journal impact factorsCorruption of journal impact factors””..““Impact factors Impact factors ‘‘flawed, misleading and unfairflawed, misleading and unfair’”’”..““Journal impact factors for the individual scientist: an unnecessJournal impact factors for the individual scientist: an unnecessary evilary evil””..““LetLet’’s dump the factorss dump the factors””..““Nightmare impact factorNightmare impact factor””..““The dreaded impact factor is back to haunt us!The dreaded impact factor is back to haunt us!””..““The impact factor The impact factor –– ‘‘Misleading, unscientific and unjustMisleading, unscientific and unjust’”’”..““The impact factor The impact factor –– what it is and where it is uselesswhat it is and where it is useless””..““The journal The journal ‘‘impact factorimpact factor’’: a misnamed, misleading, misused measure: a misnamed, misleading, misused measure””..““The malign influence of impact factorsThe malign influence of impact factors””..““The perfidy of impact factorsThe perfidy of impact factors””..““The tyranny of the impact factorThe tyranny of the impact factor””..““Worshiping false idols: the impact factor dilemmaWorshiping false idols: the impact factor dilemma””..
Food for thoughtFood for thought
Elite journals wield huge influence and Elite journals wield huge influence and control over scientistscontrol over scientists’’ lives (lives (CorbynCorbyn 2009)2009)
Sir John Sulston US Nobel Laureate :Sir John Sulston US Nobel Laureate : Pressure to publish prematurelyPressure to publish prematurely His work not published in a top journalHis work not published in a top journal Treadmill of re submissionsTreadmill of re submissions Provides citations for many, but does little for Provides citations for many, but does little for
scientific enterprise, eg long author lists, but scientific enterprise, eg long author lists, but some may have marginal contributionsome may have marginal contribution
Part Three: Part Three: Models of research Models of research supportsupport
Understanding the contextUnderstanding the context
Understanding the usesUnderstanding the uses
Levels of supportLevels of support
Libraries responding to the Libraries responding to the research environmentresearch environment
Librarians need to understand the context:Librarians need to understand the context: Research funding highly competitiveResearch funding highly competitive Culture of evaluationCulture of evaluation Imperative to be Imperative to be ‘‘research activeresearch active’’ BibliometricsBibliometrics can be confusing!can be confusing!
Seizing the momentSeizing the moment
BibliometricsBibliometrics –– a perfect opportunity to a perfect opportunity to reconnect with academic staffreconnect with academic staff
They are looking to us for guidanceThey are looking to us for guidance Show our expertise (and can spill over to other Show our expertise (and can spill over to other
areas, eg trust to embed IL)areas, eg trust to embed IL) But we need to understand the uses (in detail!) But we need to understand the uses (in detail!)
that they are making of that they are making of bibliometricsbibliometrics Professional development for library staffProfessional development for library staff
Use of metricsUse of metrics
For grant applicationsFor grant applications Academic promotionAcademic promotion RecruitmentRecruitment Research assessment (school, research centre, Research assessment (school, research centre,
whole university)whole university) Where to publishWhere to publish
One use: ARC grant applicationOne use: ARC grant application
Researchers need to nominate their ten careerResearchers need to nominate their ten career--best publications: best publications:
Can be up to 3 pages, Can be up to 3 pages, ieie paragraph per itemparagraph per item Say why each one is best (they must FIGJAM to Say why each one is best (they must FIGJAM to
be competitive!)be competitive!)
Levels of supportLevels of support
Intensive: UNSWIntensive: UNSW SelfSelf--help: Vic Unihelp: Vic Uni Something in between?Something in between?
Intensive supportIntensive support
RIMS at UNSWRIMS at UNSW Restructure to create Research Impact Restructure to create Research Impact
Measurement ServiceMeasurement Service Produce Produce ‘‘Research Impact StatementResearch Impact Statement’’ and more and more
specialised specialised ‘‘Grant Application StatementGrant Application Statement’’ for for academicsacademics
7 EFT staff / 50% of their time spent on creating 7 EFT staff / 50% of their time spent on creating citation reports for researcherscitation reports for researchers
SelfSelf--helphelp
Train the researcher to do it themselves (VU)Train the researcher to do it themselves (VU) Develop, advertise and deliver Develop, advertise and deliver ‘‘Research Research
QualityQuality’’ or or ‘‘Citation TrackingCitation Tracking’’ workshopsworkshops Timed to critical dates Timed to critical dates ieie grant applications or grant applications or
academic promotions applications due soonacademic promotions applications due soon Collaborate with University Research Office, coCollaborate with University Research Office, co--
presenterspresenters Sessions at different campusesSessions at different campuses Wording used to market sessions very importantWording used to market sessions very important
Something in between?Something in between?
Ad hoc intensive support to individuals (even Ad hoc intensive support to individuals (even with a self help model this happens at VU) with a self help model this happens at VU)
Intensive support to target groupsIntensive support to target groups ……combined with workshopscombined with workshops Good web pagesGood web pages…… some examplessome examples
UQUQ
UNSWUNSW
UWSUWS
Issues re supportIssues re support
Intensive Intensive -- full full ‘‘research impactresearch impact’’ report:report: Sustainability?Sustainability? Responsibility with the librarian to corroborate Responsibility with the librarian to corroborate
from multiple sources and provide accurate data from multiple sources and provide accurate data (this is what is SO time consuming)(this is what is SO time consuming)
It is not always clear what measures should be It is not always clear what measures should be used in which context used in which context –– shouldnshouldn’’t the researcher t the researcher decide?decide?
But UNSW service has elevated the libraryBut UNSW service has elevated the library’’s s profileprofile
Issues re supportIssues re support
Ad hoc:Ad hoc: Inequitable? CanInequitable? Can’’t do for everyonet do for everyone Does everyone know about this service?Does everyone know about this service? People (managers/researchers) donPeople (managers/researchers) don’’t know just t know just
how time intensive it is (unrecognized)how time intensive it is (unrecognized)
Self help:Self help: Academics are time poorAcademics are time poor And may leave a workshop still confusedAnd may leave a workshop still confused
W h a t a r e y o u d o in g ?W h a t a r e y o u d o in g ?
How much effort should you go to?How much effort should you go to?
Workshop discussionWorkshop discussion
University of Melbourne:University of Melbourne: BibliometricsBibliometrics workshops for library staff; RHD workshops for library staff; RHD
students; training for supervisorsstudents; training for supervisors Staff assist with grant applications; ERAStaff assist with grant applications; ERA DonDon’’t call it a class, call it a forum/seminart call it a class, call it a forum/seminarDeakin University:Deakin University: Developing a research support strategy Developing a research support strategy
framework: framework: bibliometricsbibliometrics just a part of thisjust a part of this
Workshop discussionWorkshop discussionRMIT:RMIT: Individual support from liaison librariansIndividual support from liaison librarians Citation tracking covered briefly in Citation tracking covered briefly in postgradpostgrad
sessions (for Eng/sessions (for Eng/SciSci)) Developing a research support planDeveloping a research support planOther issues:Other issues: Combining staff & students in single session Combining staff & students in single session
trickytricky Some admitted at end of workshop that they Some admitted at end of workshop that they
hadnhadn’’t focussed in this area yett focussed in this area yet
Initiatives Initiatives
Positioning librarians to support researchPositioning librarians to support research Role of CRIG in supporting developments Role of CRIG in supporting developments
in research supportin research support RecommendationRecommendation
Establishment of regular Establishment of regular forafora librarians librarians working in the field of research supportworking in the field of research support