american evaluation association 27 th annual conference october 16, 2013 washington, d.c
Post on 13-Jan-2016
52 Views
Preview:
DESCRIPTION
TRANSCRIPT
Data provided by the Division of Statistical Analysis & Reporting (DSAR)/OPAC/OER Contact: OERStats@mail.nih.gov 1
Evaluating the Diffusion of Information from a Federal Research Agency Blog: A Case
Study from the National Institutes of Health
American Evaluation Association 27th Annual Conference October 16, 2013Washington, D.C.
Katherine Catevenis, MSPH, Sam Ryan, MPP, Lindsey Scott, PhD, Nicole Garbarini, PhD, David Rosen, Michael Dorsey, Katrina Pearson,
Megan Columbus, Robin Wagner, PhD
Office of Planning, Analysis and CommunicationsOffice of Extramural Research (OER), Office of the Director (OD)
National Institutes of Health (NIH)
Data provided by the Division of Statistical Analysis & Reporting (DSAR)/OPAC/OER Contact: OERStats@mail.nih.gov 2
Overview
• Background on Rock Talk blog– Blog goals– Distribution– Audience
• Evaluation Goals• Methods• Results• Conclusions• Limitations• Lessons learned• Contact information
Data provided by the Division of Statistical Analysis & Reporting (DSAR)/OPAC/OER Contact: OERStats@mail.nih.gov 3
The Rock Talk Bloghttp://nexus.od.nih.gov/all/category/blog/
Data provided by the Division of Statistical Analysis & Reporting (DSAR)/OPAC/OER Contact: OERStats@mail.nih.gov 4
Rock Talk Blog Goals
• Provide visibility into NIH decision-making
• Engage, educate, and provide transparency into NIH policy, research investments and workforce
• Establish a new avenue for dialogue
with the extramural community
Data provided by the Division of Statistical Analysis & Reporting (DSAR)/OPAC/OER Contact: OERStats@mail.nih.gov 5
Transparency and Open Government
“My Administration is committed to creating an unprecedented level of openness in Government. We will work together to ensure the public trust and establish a system of transparency, public participation, and collaboration. Openness will strengthen our democracy and promote efficiency and effectiveness in Government.”
Presidential Memorandum on Transparency and Open Government
Data provided by the Division of Statistical Analysis & Reporting (DSAR)/OPAC/OER Contact: OERStats@mail.nih.gov 6
Qualitative Evidence ofAchieving Transparency
“Transparency: Two years of blogging the NIH”Nature 493, 298–299 (17 January 2013)
http://www.nature.com/nature/journal/v493/n7432/full/493298a.html
Data provided by the Division of Statistical Analysis & Reporting (DSAR)/OPAC/OER Contact: OERStats@mail.nih.gov 7
Rock Talk Distribution
• Average of 1 blog post/week on NIH website• NIH Extramural Nexus monthly email update sent to
listserv of ~90,000 email addresses– Nexus website includes blog posts as well as
other news • NIH also disseminates blog through:
– RSS feed– Tweets by Dr. Sally Rockey (@RockTalking) and
NIH OER communications team (@NIHgrants)
Data provided by the Division of Statistical Analysis & Reporting (DSAR)/OPAC/OER Contact: OERStats@mail.nih.gov
Blog Audience: Biomedical Research Community
8
Rock Talk
Researchers
Grant Administrators
Trainees
Media
Other influencers
e.g., professional
society leadership
NIH Staff
Data provided by the Division of Statistical Analysis & Reporting (DSAR)/OPAC/OER Contact: OERStats@mail.nih.gov 9
Reaching the Media: Initial Evidence of Success
Data provided by the Division of Statistical Analysis & Reporting (DSAR)/OPAC/OER Contact: OERStats@mail.nih.gov 10
Evaluation Goals
To complement qualitative evidence of blog success to better understand:
• How blog topics align with audience interests
• What topics inspire reader comments
• What drives visits to the blog
Data provided by the Division of Statistical Analysis & Reporting (DSAR)/OPAC/OER Contact: OERStats@mail.nih.gov 11
Methods
• Evaluated all blog posts since inception on January 19, 2011 through July 24, 2013 for– Pageviews
• Defined as an instance of a page being loaded by a browser
• Used Google Analytics to measure pageviews over time starting in late May 2011
– see http://www.google.com/analytics/
– Reader comments • Comment counts collected manually
• Developed topic classification scheme and categorized each blog into main topic area
• Analyzed pageviews over time for top 10 viewed blogs
Data provided by the Division of Statistical Analysis & Reporting (DSAR)/OPAC/OER Contact: OERStats@mail.nih.gov 12
Summary Statistics
Metric ResultNumber of blog posts 138
Total pageviews to Nexus 1,119,158
Total pageviews to blog posts 631,196 (56% of Nexus pageviews)
Total blog comments 2,253
Top viewed blog post Paylines, Percentiles and Success Rates
28,705 pageviews (5% of total blog pageviews)
Top commented blog post “How Do You Think We Should Manage Science in Fiscally Challenging Times”
254 comments (11% of total comments)
Data provided by the Division of Statistical Analysis & Reporting (DSAR)/OPAC/OER Contact: OERStats@mail.nih.gov 13
Blog Content at a Glance
Data provided by the Division of Statistical Analysis & Reporting (DSAR)/OPAC/OER Contact: OERStats@mail.nih.gov 14
Blog Topic Areas• Budget
– Overall NIH appropriated budget-related issues• Resources
– Announcements of new tools, FAQs, etc.• Award programs
– Specific, high profile and/or trans-NIH programs• Grants policy
– Changes to grants policies and processes• Funding data
– Who, what, and how NIH funds• Peer review
– The review process for NIH grant applications• Biomedical workforce
– People involved in biomedical research• Other
Data provided by the Division of Statistical Analysis & Reporting (DSAR)/OPAC/OER Contact: OERStats@mail.nih.gov 15
What Are We Blogging About?
funding data17%
grants policy17%
biomedical workforce17%
budget4%
peer review7%
award programs15%
other16%
resources7%
Percentage of Blog Posts in Each Topic Area*
*Each blog is shown under only one topic area. Blogs tagged to more than one topic area were assigned a main topic area.
Data provided by the Division of Statistical Analysis & Reporting (DSAR)/OPAC/OER Contact: OERStats@mail.nih.gov 16
What Are People Reading?
funding data32%
grants policy16%
biomedical workforce15%
budget11%
peer review9%
award programs8%
other5%
resources5%
Percent of Total Pageviews for Each Topic Area*
*Each blog is shown under only one topic area. Blogs tagged to more than one topic area were assigned a main topic area.
Data provided by the Division of Statistical Analysis & Reporting (DSAR)/OPAC/OER Contact: OERStats@mail.nih.gov 17
On Which Topics Do People Comment?
funding data31%
grants policy7%
biomedical work-force19%
budget17%
peer review13%
award programs5%
other6%
resources3%
Percent of Total Comments for Each Topic Area*
*Each blog is shown under only one topic area. Blogs tagged to more than one topic area were assigned a main topic area.
Data provided by the Division of Statistical Analysis & Reporting (DSAR)/OPAC/OER Contact: OERStats@mail.nih.gov 18
Number of Blog Posts vs. Total Pageviewsby Topic Area
funding data17%
grants policy17%
biomedi-cal work-
force17%
budget4%
peer review
7%award
programs15%
other16%
resources7%
Percentage of Blog Posts in Each Topic Area*
funding data32%
grants policy16%
biomedical work-force15%
budget11%
peer review
9%
award pro-
grams8%
other5%
resources5%
Percent of Total Pageviews for Each Topic Area*
*Each blog is shown under only one topic area. Blogs tagged to more than one topic area were assigned a main topic area.
Data provided by the Division of Statistical Analysis & Reporting (DSAR)/OPAC/OER Contact: OERStats@mail.nih.gov 19
Number of Blog Posts vs. Total Commentsby Topic Area
funding data17%
grants policy17%
biomedi-cal work-
force17%
budget4%
peer review
7%
award programs
15%
other16%
resources7%
Percentage of Blog Posts in Each Topic Area*
*Each blog is shown under only one topic area. Blogs tagged to more than one topic area were assigned a main topic area.
funding data31%
grants policy
7%
biomedical workforce
19%
budget17%
peer review
13%award
programs5%
other6%
resources3%
Percent of Total Comments for Each Topic Area*
Data provided by the Division of Statistical Analysis & Reporting (DSAR)/OPAC/OER Contact: OERStats@mail.nih.gov 20
Total Pageviews vs. Total Comments by Topic Area
*Each blog is shown under only one topic area. Blogs tagged to more than one topic area were assigned a main topic area.
funding data31%
grants policy
7%
biomedical workforce
19%
budget17%
peer review
13%award
programs5%
other6%
resources3%
Percent of Total Comments for Each Topic Area*
funding data32%
grants policy16%
biomedical work-force15%
budget11%
peer review
9%
award pro-
grams8%
other5%
resources5%
Percent of Total Pageviews for Each Topic Area*
Data provided by the Division of Statistical Analysis & Reporting (DSAR)/OPAC/OER Contact: OERStats@mail.nih.gov 21
Rock Talk Blog Traffic Drivers
Data provided by the Division of Statistical Analysis & Reporting (DSAR)/OPAC/OER Contact: OERStats@mail.nih.gov 22
What Drives Visits to the Blog?
1/1/2012
1/16/2012
1/31/2012
2/15/2012
3/1/2012
3/16/2012
3/31/2012
4/15/2012
4/30/2012
5/15/2012
5/30/2012
6/14/2012
6/29/2012
7/14/2012
7/29/2012
8/13/2012
8/28/2012
9/12/2012
9/27/2012
10/12/2012
10/27/2012
11/11/2012
11/26/2012
12/11/2012
12/26/2012
1/10/2013
1/25/2013
2/9/2013
2/24/2013
3/11/2013
3/26/2013
4/10/2013
4/25/2013
5/10/2013
5/25/2013
6/9/2013
6/24/2013
7/9/2013
7/24/2013
0
2,000
4,000
6,000
8,000
10,000
12,000
14,000
16,000
18,000
20,000
Pageviews Over Time for the Entire Nexus Website
Page
view
s
Nexus Email
Data provided by the Division of Statistical Analysis & Reporting (DSAR)/OPAC/OER Contact: OERStats@mail.nih.gov 23
Typical Blog* Reader Behavior for a Post:Single Spike Following Nexus Email
1/1/
2011
1/27
/201
1
2/22
/201
1
3/20
/201
1
4/15
/201
1
5/11
/201
1
6/6/
2011
7/2/
2011
7/28
/201
1
8/23
/201
1
9/18
/201
1
10/1
4/20
11
11/9
/201
1
12/5
/201
1
12/3
1/20
11
1/26
/201
2
2/21
/201
2
3/18
/201
2
4/13
/201
2
5/9/
2012
6/4/
2012
6/30
/201
2
7/26
/201
2
8/21
/201
2
9/16
/201
2
10/1
2/20
12
11/7
/201
2
12/3
/201
2
12/2
9/20
12
1/24
/201
3
2/19
/201
3
3/17
/201
3
4/12
/201
3
5/8/
2013
6/3/
2013
6/29
/201
30
1000
2000
3000
4000
5000
6000
Date
Pag
evie
ws
*Blog post: "Age Distribution of NIH Principal Investigators and Medical School Faculty"
Published on Web 2/13/2013
Nexus Email 3/1/2013
Data provided by the Division of Statistical Analysis & Reporting (DSAR)/OPAC/OER Contact: OERStats@mail.nih.gov 24
Atypical Blog* Reader Behavior: Same Post Gets Recurring Hits
1/1/2011
1/22/2011
2/12/2011
3/5/2011
3/26/2011
4/16/2011
5/7/2011
5/28/2011
6/18/2011
7/9/2011
7/30/2011
8/20/2011
9/10/2011
10/1/2011
10/22/2011
11/12/2011
12/3/2011
12/24/2011
1/14/2012
2/4/2012
2/25/2012
3/17/2012
4/7/2012
4/28/2012
5/19/2012
6/9/2012
6/30/2012
7/21/2012
8/11/2012
9/1/2012
9/22/2012
10/13/2012
11/3/2012
11/24/2012
12/15/2012
1/5/2013
1/26/2013
2/16/2013
3/9/2013
3/30/2013
4/20/2013
5/11/2013
6/1/2013
6/22/2013
7/13/2013
0
100
200
300
400
500
600
Date
Page
view
s
Nexus Email2/5/2013
Published on Web
2/15/2011
*Blog post: “Paylines, Percentiles, and Success Rates”
6/17/11 10/17/11 2/8/12
6/11/12
10/22/12 2/14/13
6/7/13
Data provided by the Division of Statistical Analysis & Reporting (DSAR)/OPAC/OER Contact: OERStats@mail.nih.gov 25
Conclusions
• Qualitative evidence suggests we are reaching our target audience
• Blog topic areas are relatively evenly distributed• Audience shows greatest interest in funding data, the
NIH budget and peer review, as these topics were more highly represented in pageviews and reader comments than in blog posts
• Nexus email is a much more effective way to disseminate the blog to our entire audience than other distribution methods – However, highly interested influencers monitor blog
posts in real time • Some content/data may be more appropriate to post
on other NIH web sites, using the blog to point to them
Data provided by the Division of Statistical Analysis & Reporting (DSAR)/OPAC/OER Contact: OERStats@mail.nih.gov 26
Limitations
• Pageview data only available since started using Google Analytics (May 2011)
• Comment counts on Google Analytics were not the same as the comment counts on the blog
• Could not use average visit duration of individual blog posts due to the way Google Analytics calculates it
• Some blog posts were classified to more than one topic, but were reported here under one main topic area due to sparseness of blogs in combined categories
• Could not easily characterize readers into different audiences due to the unique source URLs
Data provided by the Division of Statistical Analysis & Reporting (DSAR)/OPAC/OER Contact: OERStats@mail.nih.gov 27
Lessons Learned
• Establish evaluation goals and criteria when setting up blog– Create custom variables if needed from the beginning– Additional settings can be added to track reading
behavior further• For example: scrolling on the page or reaching the end of the
page
• Learn about and understand your audience’s interests
• Identify what blog content can be reused to strengthen other web resources
• Consider additional tools for more comprehensive evaluation
Data provided by the Division of Statistical Analysis & Reporting (DSAR)/OPAC/OER Contact: OERStats@mail.nih.gov 28
She [Dr. Rockey] has also won kudos for engaging with the research community through her blog ‘Rock Talk.’ “What a buzz about her blog,” Blum said. “It is the kind of engagement with the community that has done more for NIH in terms of reputation in research administration than anything else.”
Carol Blum, Director of Research Compliance and Administration for the Council on Governmental Relations
Data provided by the Division of Statistical Analysis & Reporting (DSAR)/OPAC/OER Contact: OERStats@mail.nih.gov 29
Contact Information
Katherine Catevenis, MSPH
Mathematical Statistician, Division of Statistical Analysis and Reporting
Office: 301-435-2691
Email: katherine.catevenis@nih.gov
Robin M. Wagner, PhD, MS
Director, Division of Statistical Analysis and Reporting
Office: 301-443-5234
Email: wagnerr@mail.nih.gov
Megan Columbus
Director, Division of Communication and Outreach
Office: 301-435-2496
Email: columbuM@od.nih.gov
Office of Policy, Analysis and Communication
Office of Extramural Research, Office of the Director
National Institutes of Health
top related