Here Comes the Sunburst:Measuring and Visualizing Scholarly Impact

John BarnettScholarly Communications Librarian Jennifer ChanAssistant Scholarly Communications Librarian Office of Scholarly Communication and PublishingUniversity Library SystemUniversity of Pittsburgh

Here Comes the Sunburst:Measuring and Visualizing Scholarly Impact

University of Pittsburgh

Pittsburgh campus + regional campuses in Bradford, Greensburg, Johnstown, and Titusville

16 undergraduate, graduate, and professional schools

456+ degree programs 2012: conferred 8,949 degrees

University of Pittsburgh

Top 10 American higher ed. in federal funding (NSF)

Top 5 in annual research support (NIH)

5,369 faculty; 4,470 full-time faculty Research conducted: more than 300

centers, institutes, laboratories, clinics

University Library System

ARL 22nd largest academic library system in North

America 25 libraries; 6.6 million volumes 279,000 current serials

Sum of the Parts

Liaison Librarians

ULS Department of

Information Technology

Office of Scholarly

Communication and Publishing

Why Pitt?

Strategic goal: Innovation in scholarly communication

Providing services that scholars understand, need, and value

Putting ourselves in faculty “spaces” Re-envisioning our librarian liaison program Deepening our understanding of scholarly

communications issues

Why PlumX? Making research “more assessable

and accessible”– Gathering information in one place– Making it intelligible and useful

Measuring and visualizing research impact

Correlating metrics from traditional and new forms of scholarly communication

Allowing researchers, labs, departments, institutions to track real-time scholarly impact

Promoting research, comparing with peers, connecting with new research

Altmetrics Project Timeline

Spring 2012: • First meeting with

Plum Analytics

Fall 2012• Gathered data

from pilot participants

Spring 2013• Faculty surveyed;

enhancements made

Pilot project aims

Develop a tool for measuring and visualizing research impact

Gathering information in one place Intelligible and useful Impact in social media and other scholarly

communication methods Traditional measures counted as well See where to disseminate works to increase impact

Traditional vs. new

•Traditional measures are also counted

•Findings are complementary to conventional methods of measuring research impact (e.g., H-Index)

•Not intended to replace them

New measures More comprehensive: Altmetrics = ALL METRICS

– Citations– Usage– Captures– Mentions– Social Media

Covers impact of online behavior– Because scholars increasingly work online

Measures impact immediately– Because citation counts take years to appear in literature

Pilot Process

Created Altmetrics Task Force

Engaged Liaison Librarians

CV receipt and records creation

Pilot Project Participants

• 32 researchers, various disciplines

• 9 schools• 18 departments• 1 complete research group• Others joined as they

learned about the project

Pilot Project Participantsdiscipline

school/department

online behavior

level of career advancement

Selected faculty

participants, diversified by:

Technologies

Internal– IR built on Eprints Platform– Sharepoint– Microsoft Office Suite– PMID/DOI data import tool

External– PlumX– DOIs– PMID

Data collection for pilot project• Created records in D-Scholarship@Pitt, our

institutional repository• Focused on articles, books, book chapters,

proceedings• Scholarly output with standard identifiers

• DOI, ISBN, PubMed ID, official URL, etc.

• Scholarship produced since 2000

Other Library work

• Developed guidelines to standardize record creation• Data entry from faculty CVs into IR (2 to 3 student

workers with QA by librarians)• Librarian liaisons and other staff trained in record

creation• SharePoint site used to track work completed• Coordination with pilot faculty• Gathered feedback and administered online survey

Sharepoint Altmetrics Meetings Minutes Faculty CVs Excel spreadsheets Word docs

External Data Sources

Metadata sources

Faculty CVs . . . But verify metadata! Books: PittCat, WorldCat, Books in Print, publisher

sites, online retailers Journals: Serials Solutions list, journal websites,

JournalSeek, UlrichsWeb, DOAJ, PubMed Conference presentations: Websites, PittCat,

indexes, WorldCat

PMID Import Tool

Custom build by SysAdmin for Eprints Platform Utilizing PMIDs from PubMed, able to import

records that prepopulate metadata fields– Item Type, Title, Abstract, Creators, Publication Title, ISSN,

Volume/Issue, Page ranges, Date and Date type, DOI, MeSH Headings, Grant Information, Keywords, etc.

Data Ingestion

IR

External Data

Ingestion

CV input

Self-archiving

Full-text sources

DOAJ ERIC PLOS SSRN* Other repositories* Federal government websites* Conference websites*

* Use with caution

Plum Analytics processing activities

Harvest records from Pitt IR for each participant

Build profile for each researcher in PlumX

Harvest additional online artifacts NOT in Pitt IR

Key features Faculty profiles Online ‘artifacts’

– Article– Book– Book chapter– Video– Etc.

Impact graph Sunburst

Faculty profile

Online ‘artifact’ display

Impact graph

Sunburst

Feedback• Solicited via email and online survey• Generally positive in most cases• Data corrections

• Errors in profiles• Links to wrong data• Quickly corrected by Plum staff

• Requests for results from additional online sources (Google Scholar, SlideShare, Reddit, etc.)

• PlumX collects data from these but did not gather information in advance for profiles

The survey says

Surveyed pilot project faculty in spring 2013 @ 1/3rd responded to the survey Meaning 13 out of 32 participants responded

Accurate and useful data

Summary profile data page0

10

20

30

40

50

60

70

80

90

15.387.69

76.92

0

Strongly disagreeDisagreeAgreeStrongly agree

The bar graph

Usefulness of interactive bar graph0

10

20

30

40

50

60

70

25

58.33

16.67

Strongly disagreeDisagreeAgreeStrongly agree

0

The sunburst

Usefulness of sunburst0

10

20

30

40

50

60

41.67

50

8.33

Strongly disagreeDisagreeAgreeStrongly agree

0

Traditional & new measures

Conveying traditional and new measures0

10

20

30

40

50

60

36.36

54.55

9.09

Strongly disagreeDisagreeAgreeStrongly agree

0

Usefulness of altmetrics

The value of altmetrics0

10

20

30

40

50

60

18.18 18.18

54.55

9.09

Strongly disagreeDisagreeAgreeStrongly agree

Learning something new

Learning something new about my own research0

5

10

15

20

25

30

35

40

9.09

36.36

27.27 27.27

Series 1Series 2Series 3Series 4

Comments

Affiliations/bio inaccurate or has missing information

“Mentions” by whom & when? Publications misclassified

– Books vs. conference proceedings

Data not collected– Google Scholar– Slideshare

Comments

Filter out unwanted information Data are wrong—and not useful Overabundance of information in sunburst “I only care what a select group of scholars thinks

of my work” “I did not find this useful for my discipline”

Observations

Lacked information about faculty practices Are the results useful to all faculty, all disciplines? May appeal more to faculty who are early in their

careers or whose work is more contemporary Will the data be used against faculty or programs? Labor-intensive strategy When it comes down to it . . . Does anyone care?

Embeddable widgets(in development)For researchers, to add to:

• their own Web pages• department directories• IR researcher profile page

For individual artifacts,to build article level metrics for imbedding in:

• IR document abstract page• Article abstract page for

journals we publish

Roll-out challenges

Who creates profiles? Who edits? What information should be included in profiles?

Who can view them? Separate data gathering from D-Scholarship

deposits? Who promotes the service? Who trains? Timing . . .

Future plans

Data checking Additional data gathering Record merging/deduping Ability to edit user profiles and artifact records locally Open API

To allow integration with other online systems

More exhaustive scholarly practices survey for all faculty Rollout to all Pitt Researchers

Will use automatic feed from Pitt IR to PlumX

Discussion

How would you “sell” PlumX to additional faculty?