Bibliometrics in Computer ScienceMyRI project team

Outline•Issues relating to Computer Science

•Additional tools for Computer Science

•Metrics of journal paper quality

•Metrics of conference paper quality

•Additional metrics for Computer Science

•Who’s top in my field?

•Conclusions

Issues relating to Computer Science (CS)•CS is a fast-changing, multi-disciplinary research area

– every CS research area has its own journals and conferences•CS researchers tend to publish more conference than journal papers due to their quicker publication timeframe•CS journals tend to have lower impact factors than in other disciplines•Some top conference venues are considered to be as prestigious as high quality journals•Most conference publications in CS are rigorously peer reviewed and are published as full-length papers rather than as abstracts

Issues relating to Computer Science (CS)• In CS, unlike most other disciplines, many conference papers are

considered archival publications

• Impact factors are not yet available for conferences

• Many bibliometric resources do not index conference publications – multiple resources must be used to measure the quality of

conferences

• There is no one definitive bibliometric resource for CS– this will depend on your field of expertise– discuss this with your supervisor and research group

Additional tools for Computer Science• Of the generic tools, Google Scholar/Publish or Perish is the most

commonly used resource for finding citation counts. Scopus is not commonly used. While ISI Web of Science provides the commonly used ISI journal impact factors, some CS researchers complain about coverage (important journals and conferences are missing)

– Link to our related resource: A short video in which Prof. Gregory O’Hare from the School of Computer Science, University College Dublin discusses the limitations of the ISI data source and product set for bibliometrics purposes in the Computer Science field

• ACM Digital Library - provides citation counts and downloads per journal/conference paper as well as some acceptance rates for conferences – Link to our related resource: A product profile for ACM Digital Library

• Microsoft Academic Search - provides citation counts and g-index, h-index, total citations, number of publications for authors. Microsoft Academic Search is not commonly used, but is one to watch– Link to our related resource: A product profile for Microsoft Academic Search

Additional tools for Computer Science• Australian Research Council's ERA rankings (previously called CORE

rankings) - ranked journal/conference lists; a reliable quality indicator for journals/conferences in some CS sub-fields; may be missing journals in some multidisciplinary fields– 2010 listing: www.arc.gov.au/era/era_journal_list.htm– Link to our related resource: A product profile for the ERA rankings

• Conference acceptance rates - some individuals maintain websites where they list conference acceptance rates and rankings for specific sub-fields – e.g. www.adaptivebox.net/CILib/CICON_stat.html

• CiteSeerx - provides h-index, total and individual paper citation counts for authors; automatically maintained so the index is incomplete and information can be poorly formatted; used to be popular, but not as much now.– Link to our related resource: A product profile for CiteSeerx

Additional tools for Computer Science

• These resources provide no bibliometric information (even citations), but if a paper is indexed in these repositories it provides good evidence that the paper (journal or conference) has been appropriately peer reviewed:

– IEEE Xplore – DBLP Computer Science bibliography

Link to our related resource: A decision chart to pick the best product to use for your Computer Science bibliometrics query

Metrics of journal paper quality

•In order of importance:– Number of citations (excluding self-citations)

– Impact factor (problematic if sub-field is emerging, e.g. Digital Forensics)

– Australian Research Council's ERA rankings (suitable for some sub-fields)

– Download statistics from, for example, the ACM Digital Library (less important)

Metrics of conference paper quality• In order of importance:

– Number of citations (excluding self-citations)

– Conference acceptance rate for year of publication (<20% is indicative of a top conference)

– Australian Research Council's ERA Conference rankings (suitable quality indicator for some CS sub-fields)

– Inclusion in resources such as ACM Digital Library, DBLP Computer Science bibliography, IEEE Xplore and ISI’s Web of Science gives some assurance that a paper has been peer-reviewed (but this does depend on the CS sub-field)

Additional metrics for Computer Science

• There are many metrics other than citation counts that CS researchers can use to indicate the quality of a publication; these metrics are particularly relevant for early career researchers who haven’t accumulated significant citation counts:

– Datasets/Open Source software• Download statistics from your webpage or other repository

– Web services• No. of hits, list of high profile clients using the service (e.g. pharmaceutical

industry)

– Awards• Best paper awards; scholarship awards etc.

Additional metrics for Computer Science– Technology transfer

• No. of invention disclosures, patents, licensing agreements, spinout companies etc.

– Research internships• Invitations to work in industry or a host university

– Professional services• Reviewing invitations (journals, conferences)• Organisation of conferences• Journal editorial board membership (later career researcher)

Link to our related resource: a short video in which Professor Pádraig Cunningham of the School of Computer Science, University College Dublin outlines alternative metrics to the h-index that early career Computer Science researchers should consider using to capture their research impact, in a CV for example

Who’s top in my field?

• In CS the following sources can be used to find the top researchers in your field:– Editorial board members of top journals in your area– Programme committee members of top conferences– Microsoft Academic Search has introduced an academic ranking

functionality for different research fields– Authors of seminal text books in your field– Winners of Life achievement awards, e.g. the Gerard Salton Award in

Information Retrieval Research– Invited speakers at top conferences– CiteSeerx maintains an automatically generated list of the most cited

CS papers, most cited authors, most cited citations, and venue impact ratings (this last one has not been updated since 2008)

Conclusions• For early-stage researchers, citation counts and h-index scores are

not as relevant as the quality of the journal or conference, e.g. impact factor, conference acceptance rate

• No metric is perfect– Citation counts and h-index scores can be enhanced by self-citations

• Recognising the difficulty of assessing conference paper quality, CS researchers are moving towards balancing their publications between conference and journal venues

• CS researchers need to use multiple measures from multiple sources to provide as full a picture as possible on the quality of their research achievements