american evaluation association 27 th annual conference october 16, 2013 washington, d.c

Data provided by the Division of Statistical Analysis & Reporting (DSAR)/OPAC/OER Contact: [email protected] 1

Evaluating the Diffusion of Information from a Federal Research Agency Blog: A Case

Study from the National Institutes of Health

American Evaluation Association 27th Annual Conference October 16, 2013Washington, D.C.

Katherine Catevenis, MSPH, Sam Ryan, MPP, Lindsey Scott, PhD, Nicole Garbarini, PhD, David Rosen, Michael Dorsey, Katrina Pearson,

Megan Columbus, Robin Wagner, PhD

Office of Planning, Analysis and CommunicationsOffice of Extramural Research (OER), Office of the Director (OD)

National Institutes of Health (NIH)


Overview

• Background on Rock Talk blog– Blog goals– Distribution– Audience

• Evaluation Goals• Methods• Results• Conclusions• Limitations• Lessons learned• Contact information


The Rock Talk Bloghttp://nexus.od.nih.gov/all/category/blog/


Rock Talk Blog Goals

• Provide visibility into NIH decision-making

• Engage, educate, and provide transparency into NIH policy, research investments and workforce

• Establish a new avenue for dialogue

with the extramural community


Transparency and Open Government

“My Administration is committed to creating an unprecedented level of openness in Government. We will work together to ensure the public trust and establish a system of transparency, public participation, and collaboration. Openness will strengthen our democracy and promote efficiency and effectiveness in Government.”

Presidential Memorandum on Transparency and Open Government


Qualitative Evidence ofAchieving Transparency

“Transparency: Two years of blogging the NIH”Nature 493, 298–299 (17 January 2013)

http://www.nature.com/nature/journal/v493/n7432/full/493298a.html





Rock Talk Distribution

• Average of 1 blog post/week on NIH website• NIH Extramural Nexus monthly email update sent to

listserv of ~90,000 email addresses– Nexus website includes blog posts as well as

other news • NIH also disseminates blog through:

– RSS feed– Tweets by Dr. Sally Rockey (@RockTalking) and

NIH OER communications team (@NIHgrants)

Data provided by the Division of Statistical Analysis & Reporting (DSAR)/OPAC/OER Contact: [email protected]

Blog Audience: Biomedical Research Community

8

Rock Talk

Researchers

Grant Administrators

Trainees

Media

Other influencers

e.g., professional

society leadership

NIH Staff


Reaching the Media: Initial Evidence of Success


Evaluation Goals

To complement qualitative evidence of blog success to better understand:

• How blog topics align with audience interests

• What topics inspire reader comments

• What drives visits to the blog


Methods

• Evaluated all blog posts since inception on January 19, 2011 through July 24, 2013 for– Pageviews

• Defined as an instance of a page being loaded by a browser

• Used Google Analytics to measure pageviews over time starting in late May 2011

– see http://www.google.com/analytics/

– Reader comments • Comment counts collected manually

• Developed topic classification scheme and categorized each blog into main topic area

• Analyzed pageviews over time for top 10 viewed blogs

http://www.google.com/analytics/

http://www.google.com/analytics/


Summary Statistics

Metric ResultNumber of blog posts 138

Total pageviews to Nexus 1,119,158

Total pageviews to blog posts 631,196 (56% of Nexus pageviews)

Total blog comments 2,253

Top viewed blog post Paylines, Percentiles and Success Rates

28,705 pageviews (5% of total blog pageviews)

Top commented blog post “How Do You Think We Should Manage Science in Fiscally Challenging Times”

254 comments (11% of total comments)


Blog Content at a Glance


Blog Topic Areas• Budget

– Overall NIH appropriated budget-related issues• Resources

– Announcements of new tools, FAQs, etc.• Award programs

– Specific, high profile and/or trans-NIH programs• Grants policy

– Changes to grants policies and processes• Funding data

– Who, what, and how NIH funds• Peer review

– The review process for NIH grant applications• Biomedical workforce

– People involved in biomedical research• Other


What Are We Blogging About?

funding data17%

grants policy17%

biomedical workforce17%

budget4%

peer review7%

award programs15%

other16%

resources7%

Percentage of Blog Posts in Each Topic Area*

*Each blog is shown under only one topic area. Blogs tagged to more than one topic area were assigned a main topic area.


What Are People Reading?

funding data32%

grants policy16%

biomedical workforce15%

budget11%

peer review9%

award programs8%

other5%

resources5%

Percent of Total Pageviews for Each Topic Area*



On Which Topics Do People Comment?

funding data31%

grants policy7%

biomedical work-force19%

budget17%

peer review13%

award programs5%

other6%

resources3%

Percent of Total Comments for Each Topic Area*



Number of Blog Posts vs. Total Pageviewsby Topic Area

funding data17%

grants policy17%

biomedi-cal work-

force17%

budget4%

peer review

7%award

programs15%

other16%

resources7%


funding data32%

grants policy16%


budget11%

peer review

9%

award pro-

grams8%

other5%

resources5%




Number of Blog Posts vs. Total Commentsby Topic Area

funding data17%

grants policy17%

biomedi-cal work-

force17%

budget4%

peer review

7%

award programs

15%

other16%

resources7%



funding data31%

grants policy

7%

biomedical workforce

19%

budget17%

peer review

13%award

programs5%

other6%

resources3%



Total Pageviews vs. Total Comments by Topic Area


funding data31%

grants policy

7%

biomedical workforce

19%

budget17%

peer review

13%award

programs5%

other6%

resources3%


funding data32%

grants policy16%


budget11%

peer review

9%

award pro-

grams8%

other5%

resources5%



Rock Talk Blog Traffic Drivers


What Drives Visits to the Blog?

1/1/2012

1/16/2012

1/31/2012

2/15/2012

3/1/2012

3/16/2012

3/31/2012

4/15/2012

4/30/2012

5/15/2012

5/30/2012

6/14/2012

6/29/2012

7/14/2012

7/29/2012

8/13/2012

8/28/2012

9/12/2012

9/27/2012

10/12/2012

10/27/2012

11/11/2012

11/26/2012

12/11/2012

12/26/2012

1/10/2013

1/25/2013

2/9/2013

2/24/2013

3/11/2013

3/26/2013

4/10/2013

4/25/2013

5/10/2013

5/25/2013

6/9/2013

6/24/2013

7/9/2013

7/24/2013

0

2,000

4,000

6,000

8,000

10,000

12,000

14,000

16,000

18,000

20,000

Pageviews Over Time for the Entire Nexus Website

Page

view

s

Nexus Email


Typical Blog* Reader Behavior for a Post:Single Spike Following Nexus Email

1/1/

2011

1/27

/201

1

2/22

/201

1

3/20

/201

1

4/15

/201

1

5/11

/201

1

6/6/

2011

7/2/

2011

7/28

/201

1

8/23

/201

1

9/18

/201

1

10/1

4/20

11

11/9

/201

1

12/5

/201

1

12/3

1/20

11

1/26

/201

2

2/21

/201

2

3/18

/201

2

4/13

/201

2

5/9/

2012

6/4/

2012

6/30

/201

2

7/26

/201

2

8/21

/201

2

9/16

/201

2

10/1

2/20

12

11/7

/201

2

12/3

/201

2

12/2

9/20

12

1/24

/201

3

2/19

/201

3

3/17

/201

3

4/12

/201

3

5/8/

2013

6/3/

2013

6/29

/201

30

1000

2000

3000

4000

5000

6000

Date

Pag

evie

ws

*Blog post: "Age Distribution of NIH Principal Investigators and Medical School Faculty"

Published on Web 2/13/2013

Nexus Email 3/1/2013


Atypical Blog* Reader Behavior: Same Post Gets Recurring Hits

1/1/2011

1/22/2011

2/12/2011

3/5/2011

3/26/2011

4/16/2011

5/7/2011

5/28/2011

6/18/2011

7/9/2011

7/30/2011

8/20/2011

9/10/2011

10/1/2011

10/22/2011

11/12/2011

12/3/2011

12/24/2011

1/14/2012

2/4/2012

2/25/2012

3/17/2012

4/7/2012

4/28/2012

5/19/2012

6/9/2012

6/30/2012

7/21/2012

8/11/2012

9/1/2012

9/22/2012

10/13/2012

11/3/2012

11/24/2012

12/15/2012

1/5/2013

1/26/2013

2/16/2013

3/9/2013

3/30/2013

4/20/2013

5/11/2013

6/1/2013

6/22/2013

7/13/2013

0

100

200

300

400

500

600

Date

Page

view

s

Nexus Email2/5/2013

Published on Web

2/15/2011

*Blog post: “Paylines, Percentiles, and Success Rates”

6/17/11 10/17/11 2/8/12

6/11/12

10/22/12 2/14/13

6/7/13


Conclusions

• Qualitative evidence suggests we are reaching our target audience

• Blog topic areas are relatively evenly distributed• Audience shows greatest interest in funding data, the

NIH budget and peer review, as these topics were more highly represented in pageviews and reader comments than in blog posts

• Nexus email is a much more effective way to disseminate the blog to our entire audience than other distribution methods – However, highly interested influencers monitor blog

posts in real time • Some content/data may be more appropriate to post

on other NIH web sites, using the blog to point to them


Limitations

• Pageview data only available since started using Google Analytics (May 2011)

• Comment counts on Google Analytics were not the same as the comment counts on the blog

• Could not use average visit duration of individual blog posts due to the way Google Analytics calculates it

• Some blog posts were classified to more than one topic, but were reported here under one main topic area due to sparseness of blogs in combined categories

• Could not easily characterize readers into different audiences due to the unique source URLs


Lessons Learned

• Establish evaluation goals and criteria when setting up blog– Create custom variables if needed from the beginning– Additional settings can be added to track reading

behavior further• For example: scrolling on the page or reaching the end of the

page

• Learn about and understand your audience’s interests

• Identify what blog content can be reused to strengthen other web resources

• Consider additional tools for more comprehensive evaluation


She [Dr. Rockey] has also won kudos for engaging with the research community through her blog ‘Rock Talk.’ “What a buzz about her blog,” Blum said. “It is the kind of engagement with the community that has done more for NIH in terms of reputation in research administration than anything else.”

Carol Blum, Director of Research Compliance and Administration for the Council on Governmental Relations


Contact Information

Katherine Catevenis, MSPH

Mathematical Statistician, Division of Statistical Analysis and Reporting

Office: 301-435-2691

Email: [email protected]

Robin M. Wagner, PhD, MS

Director, Division of Statistical Analysis and Reporting

Office: 301-443-5234


Megan Columbus

Director, Division of Communication and Outreach

Office: 301-435-2496


Office of Policy, Analysis and Communication

Office of Extramural Research, Office of the Director

National Institutes of Health

mailto:[email protected]

mailto:[email protected]

american evaluation association 27 th annual conference october 16, 2013 washington, d.c

Documents