METRICS

· Whither Metrics, Cont.: Assessing

Publication Output of AcademicLibrary Practitioners Across DifferentTypes of Institutionsby Christopher Stewart

There is wide debate in the profession about the value placed onpublication activity as a measure of our success in informingpractice through traditional channels of scholarly communica-

tion and as a reward system for promotion. In examining publicationactivity of academic library practitioners, it is useful to begin byinvestigating institutional context, both quantifiable and qualitative,in which our activity as practitioner scholars can be measured. Whattypes of colleges and universities produce the significant amounts ofpractitioner scholarship? Previous research has suggested that themajority of practitioner research is generated at large researchuniversities.1

In this second column on academic library practitioner scholarlypublication activity, I will use three major institutional variables toassess patterns in practitioner publication activity for sample year 2009.The next step is to take a closer look at these institutions to identifypatterns. In doing so, we can make reasonable observations aboutwhere practitioner research in our field originated in the sample yearand to what degree. These data and the methods used to derive themcan also help develop a framework for a larger study, confirm somefindings from previous research, and add to the portfolio of toolsavailable for assessing the research output of our profession.

In the previous column2 on using metrics to assess scholarlypublishing activity among library practitioners, the top ten libraryscience journals were ranked by 5-year impact factor as derived fromThompson Reuters ISI Web of Knowledge Journal Citation Reports.The 5-year impact factor for the most recent JCR year, 2009, was used.A total of 781 articles were identified and subsequently narrowed byarticle type, as well as whether the piece was authored by LIS facultyor academic library practitioners. Library practitioners authoredapproximately 70% of all articles, but this percentage narrowedsomewhat for research articles specifically, of which approximately60% were authored by library practitioners. Based on data fromprevious studies,3 it was determined that, while there was generalagreement in quality, there was considerable difference between LISDeans and ARL directors in the ranked prestige of the ten journals usedin this analysis—most of which were ranked highly by ARL directors,however. Finally, among library practitioner authors, there was a gooddeal of distribution among numerous institutions.4

Christopher Stewart is Dean of Libraries, Illinois Institute of Technology35 West 33rd Street, Chicago, IL 60616-3793, USA<stewart@iit.edu>.

546 The Journal of Academic Librarianship, Volume 36, Number 6,

,

pages

There are three major institutional variables that will be used in thisanalysis: institutional public/private governance; Association ofResearch Library (ARL) membership; and basic (revised as of 2005)institutional classifications assigned by the Carnegie Foundation forthe Advancement of Teaching: Carnegie Classifications. ARL member-ship includes many of the largest college and university researchlibraries in NorthAmerica. Of the 125 ARLmember libraries, 99 areU.S.colleges and universities.5

The 2005 revisions of the Carnegie classifications “providedifferent lenses through which to view U.S. colleges and universities,offering researchers greater flexibility in meeting their analyticneeds.” In their most general sense, the new Basic Carnegieclassifications describe institutional governance, highest level ofdegree, and level of research activity for doctoral granting institutions.The current analysis will use eight Basic classifications that describethe institution's instructional program and research activity. Most ofthe descriptive information that follows is derived from the CarnegieFoundation for the Advancement of Teaching website.6 The firstgeneral classification includes include doctorate-granting universities,which includes “institutions that award at least 20 doctoral degreesper year (excluding doctoral-level degrees that qualify recipients forentry into professional practice, such as the JD, MD, PharmD, DPT,etc.). Within the category are three classifications that describe“explicit measure of research activity”:7 RU/VH: Research Universities(very high research activity); RU/H: Research Universities (highresearch activity); and DRU: Doctoral/Research Universities. Institu-tions in the RU/VH category comprise the nation's highest research-level institutions, many of which are ARL members. The secondgeneral category, Master's Colleges and Universities, includes institu-tions that award at least 50 master's degrees and fewer than 20doctoral degrees per year. There are three groups in this generalcategory, Master's/L: Master's Colleges and Universities (largerprograms); Master's/M: Master's Colleges and Universities (mediumprograms); Master's/S: Master's Colleges and Universities (smallerprograms). Remaining categories, each with numerous sub-catego-ries, include Baccalaureate Colleges, Associate's Colleges, TribalColleges, and Special Focus Institutions.

Carnegie Classifications include U.S. institutions only. There are ofcourse a number of Canadian institutions, many of which wereidentified in the previous column, from which a good amount ofpractitioner scholarship emanates. For the current analysis, however,only U.S. institutions are considered. The sample was further refined byincluding research articles only. As shown in Table 1, with thesecontrols, there were 223 U.S. authors of research articles in 2009, and

546–548

Table 12009 Research Article Output by U.S. Academic Library

Practitioners in Leading Library Science Journals

Number of Authors 223

Number of Articles 138

Number of Institutions 101

Table 2Practitioner Publication Across ARLMember/Non-Member Institutions

InstitutionalControl

ARLMember

ARLNon-Member Total

Public 103 77 180

Private 19 24 43

Total 122 101 223

TLibrary Practitioner Publication Activity

Carnegie Classification Number of Autho

Doctoral research 17

Research university/very high research 115

Research university/high research 53

Masters large 18

Masters medium 10

Masters small 2

Baccalaureate 4

Specialty 1

Community colleges 3

Total 223

101 institutions represented. If an article was co-authored by U.S. andnon-U.S. practitioners, it was not eliminated from the sample, althoughthe non-U.S. author affiliations was removed.

The next step in developing an institution-focused model foranalyzing practitioner-based publication activity is exploring the firstand second variables, institutional governance and ARL membership.A large majority of practitioner authors of research articles in the top-ten impact factor journals in 2009 were from public institutions.Looking at this from the perspective of expected values, while publicinstitutions comprise roughly 40% of post secondary institutions inthe United States, they are represented over in over 80% of the caseshere. If one were to infer from the sample, a reasonable hypothesiscould be that, across the population of U.S. higher education, librariansat public institutions are more likely to publish work in leadingjournals than their counterparts at private institutions. While theexistence of tenure systems for librarians at public institutions likelycontributes to this bias, at least one recent study found no strong linksbetween the presence of tenure systems and practitioner publicationrates of the top 10% of institutions represented in five major LISjournals.8

ARL membership was more evenly divided, although a slightmajority (55%) of institutions in the sample were ARL members. Thelarge majority (n=101) of ARL institutions represented in thissample are pubic institutions. As ARL membership includes many ofthe nation's research universities, the high percentage of authorsfrom these institutions would suggest that there is a link betweenhigh ARL membership and public universities for library practitionerpublication activity. However, as there were 101 non-ARL institu-tions also represented in the sample, only a very low correlationcoefficient (r=0.103) was derived for these two variables.However, when including private research universities with veryhigh levels of research activity, there is a not surprising highcorrelation (r=0.739; pb0.01) between ARL membership and level

able 3Across Institutional Carnegie Classifications

rs PercentExpected Percentage

ValueCumulativePercent

7.6% 1.9% 7.6%

51.6% 2.2% 59.2%

23.8% 2.3% 83.0%

8.1% 7.9% 91.0%

4.5% 3.4% 95.5%

0.9% 2.9% 96.4%

1.8% 17.4% 98.2%

0.4% 19.7% 98.7%

1.3% 41.4% 100.0%

November 2010 547

of institutional research output. The cross-tabulation in Table 2breaks down the number of ARL and public/private institutionsrepresented in the sample.

The distribution of institutions becomes quite different, however,when arranged according to Basic Carnegie Classification. There is onevariable and one value. The variable, of course, is the CarnegieClassification. For this model, eight classifications were used. As willbe shown, individual categories within Specialty Institutions, TribalColleges, and Baccalaureate Colleges are not separated out becausethere were so few results in these categories (indeed, out of the 223author/institution affiliations, there were no authors from TribalColleges and only one from a Specialty Institution). The value is theexpected value of the number of author/institution affiliations basedon the number of institutions in the population of U.S. post secondaryinstitutions as listed by the Carnegie Foundation (N=4391). Theresults are striking. Fully 83% of the articles published in the top tenLibrary Science journals in 2009 were authored by practitioners atresearch universities. Of these, three quarters were from “very high”and “high” research institutions. Each institutional group wasrepresented at far greater percentages than in the population. Forexample, there are 96 research universities in the “very high” categoryin the United States, representing only 2.2% of the population of postsecondary institutions in the population. In this sample, these types ofresearch universities represented 52% of the author/institutionaffiliations. The difference between results data and expected valuesis somewhat less dramatic for the high research category ofinstitutions, but is wide nonetheless. Therewere 53 author/institutionaffiliations in this category of institutions, or approximately 24% of theoverall results, compared to an expected value of only 2.3% of highresearch institutions in the overall population of U.S. post secondaryinstitutions. Table 3 lists percentage of institutions where practitioner

scholarship is produced versus expected values for those institutionsbased on the overall population.

There was a far more representative sample of author affiliationsfrom Masters Colleges and Universities, with 8.1% of authors fromlargemasters institutions versus 7.9% of this category of institutions inthe U.S. higher education population. For medium-sized mastersinstitutions, the percentage of institutions was also similarly matchedbetween the sample and the population. When one looks to see howmuch practitioner scholarship is produced at baccalaureate institu-tions, community colleges, and specialty institutions, it is clear thatauthors at these types of institutions are represented in leading libraryscience journals at far lower levels than their counterparts at researchuniversities. Out of 223 author/institution affiliations in the sample,only four (1.8%) were from baccalaureate institutions, versus anexpected value of 17.4%. For community colleges, the results are evenmore striking: 1.3% of the author/institutions affiliations were fromcommunity colleges, versus 41.4% of community colleges in theoverall population of U.S. higher education.

If one were to infer from this single year sample, it can be fairlyassumed that there is, at least in the field's top journals, very littlepublished work being authored by librarians at undergraduateinstitutions and community colleges. One way of looking at this isfrom the “most of” perspective: most of the practitioners publishingresearch today are from large research universities. As previouslymentioned, there is research that suggests this fact. Another way oflooking at this is from the “how little” perspective: the lack ofresearch being published in leading journals by our colleagues atundergraduate institutions and community colleges. The issuesfacing academic librarianship–particularly issues pertaining toinformation literacy, library space, and pedagogy–are certainly noless important at these institutions. Indeed, these institutions areoften on the front lines of some of the most important challengesfacing academic librarianship today. Most certainly, academiclibrarians at all types of institutions could benefit from theirinsights.

Hopefully, these past two columns have provided illustrations inthe use of some basic tools and descriptors for assessing journalvalue and measuring scholarly output of library practitioners. Thevariables, expected values, and basic descriptive methods used herecan be enhanced with the addition of new variables, especiallycontinuous variables such as enrollment, organizational factors suchstaff size, and financial data (Seaman's work incorporates thesetypes of variables to a certain degree). Including these variables

548 The Journal of Academic Librarianship

would allow for deeper statistical analysis, perhaps even thecreation of a model based on correlations and multivariateregressions that would identify the key organizational factorspresent in libraries that are most likely to (and not to) producehigh levels of practitioner-generated scholarship. Of course, thesample size should be expanded to include additional years, and itis my goal to devise a 10-year study built on the frameworkpresented here and in the previous column. It is clear that there isimportant and interesting data to be found in exploring ourscholarly output as practitioners. As other researchers haveshown, there are different research strategies for going about thistask, and different ways of presenting the information depending onthe questions asked. Hopefully, the strategies presented in thesecolumns will be useful additions to the work begun by others inassessing research output of academic library practitioners on abroad scale.

NOTES AND REFERENCES

1.Wiberly, Stephen E., Hurd, Julie M., and Weller, Ann C. “PublicationPatterns of U.S. Academic Librarians from 1993 to 1997.” College &Research Libraries 60, no. 4: 352-362.

2. Stewart, Christopher, “Whither Metrics? Tools for AssessingPublication Impact of Academic Library Practitioners,” The Journalof Academic Librarianship 36 (2010): 449–453.

3. Nisonger, Thomas E., and Davis, Charles H. “The Perception ofLibrary and Information Science Journals by LIS Education Deansand ARL Library Directors: A Replication of the Kohl-Davis Study.”College & Research Libraries 66, no. 4: 341-377, p. 341.

4. After the top five institutions were listed (there was a two-way tie),the remaining 149 (approximately 80% of the total number ofinstitutions represented) become so dispersed as to prevent aranked list from being provided without resulting in a tied scoreconsisting of numerous institutions.

5. “Association of Research Libraries Member Libraries.” http://www.arl.org/arl/membership/members.shtml (Accessed July 22, 2010).

6. “The Carnegie Foundation for the Advancement of Teaching -Classification Descriptions” http://classifications.carnegiefounda-tion.org/descriptions/basic.php (Accessed July 22, 2010).

7. Ibid.8. Seaman, Scott, “North American Institutions Most FrequentlyRepresented in High Impact Journals,” LIBRES 18, no. 2, http://libres.curtin.edu.au/libres18n2/. (Accessed July 19, 2010).