The Use & Abuse of Usage MeasuresIan BannermanManaging Director, Journals

The Use & Abuse of Usage MeasuresCOUNTER and the Usage FactorConsistency, credibility, compatibilityUsage as an indicator of qualityThe Observer EffectRecommendations

COUNTER and the Usage FactorLaunched in March 2002, COUNTER is an international initiative serving librarians, publishers and intermediaries by setting standards that facilitate the recording and reporting of online usage statistics in a consistent, credible and compatible way

Usage Factor:J. Bollen & H. Van de Sompel (2006) Usage Impact Factor: the effects of sample characteristics on usage-based impact metrics. arXiv:cs.DL/0610154 v2Oct 06 - COUNTER director Peter Shepherd interviewed 7 authors, 9 librarians, 13 publishersMarch 07 web survey of 155 librarians, 1,400 authorsJune 07 UKSG report published www.uksg.org April 08 Invitation to tender to investigate and test the feasibility of developing a new metric, the Journal Usage Factor, based on COUNTER usage data

Usage Factor:Thomson Scientific Impact Factor:Total cites in 2007 for items published during 2005/6Total items published during 2005/6

Usage Factor:Total usage over period x of items published during period yTotal items published online during period y

Lies, damned lies and [usage] statisticsImplicit Assumptions:That usage data is consistent, credible and compatibleThat the Usage Factor would be a meaningful indicator of something (Utility? Readership? Quality? Value?)

Consistent, credible, compatibleCOUNTER: guidelines on filtering for robots and pre-fetching are in draft (release 3)By the end of 2007, bepress predicts that, without filtering, one out of every two logged downloads from academic sites will be made by machine or mistake bepress press releaseIve successfully downloaded my own article thousands of times without setting off any alarms Phil Davis, Cornell (Lib-license)Most known robots wont get past access control on subscribed content, but its the unknown ones that distort the numbers

Consistent, credible, compatibleDownloaded 6,372 times on 25/4/07 by a Russian institution

Every article from vol 5-7 downloaded ~58 times by a Korean institutionConsistent, credible, compatible

Downloaded 1,183 timesConsistent, credible, compatible

Davis & Price (2006) eJournal interface can influence usage statistics: implications for libraries, publishers, and Project COUNTER. JASIST v57 n9,1243-1248A meaningful indicator?

A meaningful indicator?Reading or just Viewing: `power browsing form of information seeking: skimming and bouncing along the information surface. CIBERIn particular, we found a general negative correlation between the CSU IF [California State Usage Impact Factor] and the ISI IF, which indicates usage over the entire CSU community is inversely related to general citation impactUsage Impact Factor: the effects of sample characteristics on usage-based impact metrics, Johan Bollen and Herbert Van de Sompel, October 2006 arXiv:cs.DL/0610154v2

Lies, damned lies and [usage] statisticsAny economy that is based on non-transparency, blind-trust, and few (if any) consequences for unethical behaviour is wide open to abuse. When cancellation decisions ride solely on measures for article downloads, librarians should approach these figures with a healthy degree of caution. Phil Davis, Cornell (Lib-license)

Lies, damned lies and [usage] statisticsGoogle's share price fell ~4% on 26th Feb, 2008, following the release of comScore data that showed a 7% decline in clicksthe evidence suggests that the softness in Googles paid click metrics is primarily a result of Googles own quality initiatives that result in a reduction in the number of paid listings.

Observer EffectRefers to changes that the act of observing will make on the phenomenon being observed

In 1955 it did not occur to me that impact would become so controversial Eugene Garfield, ISI

Potential Observer Effect: Impact FactorSelf-citing the journal in other articles and editorialsAlerting authors to content they should citeSeeking out prolific, high quality authors who will self-citeBuilding editorial boards that will attract citationsPublishing themed issues with prestigious Guest EditorsPublishing the most citable papers early in the yearKeeping review times short so citations dont miss the 2-year impact factor windowTargeting topical areas rather than long-term studiesPublishing review articlesPublishing news, letters, obituaries, book reviews, editorials (that get cited without counting as citable articles)

Potential Observer Effect: Impact FactorThe Number Thats Devouring Science. Richard Monastersky, Chronicle of Higher Education, October 2005.

The impact factor maybe a pox on the land because of the abuse of that number- Robert H Austin, Physics, PrincetonIt is easy to catch attention when one describes a previously unknown gene or protein follow-up studies, to uncover the true functions of the molecules or sometimes to challenge the initial analysis, are typically more difficult to publish - Yu-Li Wang, Physioliogy, Univ. MassachusettsAcademia is being held to ransom by these citations- Alan Nevill, Biostatistics, Univ. Wolverhampton

Potential Observer Effect:Usage FactorGetting your friends, your dog and your mother to download articles or writing a bot to do it for themLeaving usage data unfiltered or worsePublishing for students, not for researchersSexing-up title and key-wordsPutting the HTML in the way of the PDFUsing the abstract to tease, not to informStopping printed journalsEncouraging online coursepacks, discouraging printed onesBlogging it, tagging it, posting itBroadcasting metadata but keeping articles where they are counted not in OA repositories!

Potential Observer EffectImpact Factor:Not all attempts to improve impact factor are badThey leave an audit trail in the literatureThe act of citing is usually meaningfulCitation requires significant investment of time, effort and reputation

Usage Factor:Would attempts to improve usage factor be bad?They are largely untraceableThe act of downloading is often meaninglessDownloading requires little investment and is practically anonymous

The Use and Abuse of Usage MeasuresRecommendations:Extreme caution in (over) interpreting usage dataConsider whether the available data can address the questionCompare like with likeFurther research into the factors that influence article downloadsImproved guidelines on detection, blocking and filtering of robots and other aberrant useCareful attention to the Observer Effect when developing usage-based metrics

Ian BannermanManaging Director, Journalsian.bannerman@informa.com