Evaluating the Alaska EPSCoR Phase III: Resilience and Vulnerability in a Rapidly Changing North: The Integration of Physical, Biological and Social Processes

Dr. Julia Melkers, Associate Professor of Public PolicyGeorgia Institute of Technology

Dr. Eric Welch, Associate Professor of Public AdministrationUniversity of Illinois at Chicago

Year 4 Evaluation PresentationAlaska EPSCoR All Hands Meeting

May 2011Fairbanks, Alaska

Year 4 Evaluation

Extends prior years formative evaluations to not only collaborative development, but also academic production.

Summative Knowledge-Based Impacts Research Capacity

New research ideas and related activities Research Outcomes

Publications and knowledge-based impacts Looking forward:

Lessons learned and possible future directions

Today

Data sources and focus Co-authorship connections Shared knowledge Disciplinary placement Research impacts Conclusions

Evaluation Design & Data

Faculty survey Brief online survey re: newest developments and

emerging plans Articles/papers Grant proposals Research ideas

Bibliometric analysis Analysis of publication-related outputs

Alaska EPSCoR Phase III Annual Reports ISI Web of Science:

Bibliometric Data Bibliometric Measures:

Publication counts Co-authorship patterns

Citation counts and disciplines (Moed, 2005) Integration Score: Degree of interdisciplinarity based on

cited works (Porter and Rafols, 2009) Bibliographic coupling: “Coalescence” based on knowledge

sharing (shared citations)

SOURCES:

SCI-EXPANDED - Science Citation Index

SSCI - Social Sciences Citation Index

A&HCI - Arts & Humanities Citation Index

CPCI-S - Conference Proceedings Citation Index – ScienceCPCI-SSH - Conference Proceedings Citation Index - Social Science & Humanities

Survey Focus

Respondents: 99 faculty (67% response rate) Higher response among those involved for multiple years

and supported by EPSCoR

Content: New activities & outputs Qualitative:

Details on new research Most important outcomes Student impacts Outreach and stakeholder interaction/impacts

Results of Alaska EPSCoR Involvement & Support

77% reported new ideas or plans for research

44% reported submission of grant proposals in 2010/2011

(range = $4500 to $3M) (average = $636,000)

32% reported submission of a journal article in 2010/2011 Mean of ~ 1 articles per person

Detailed qualitative results provided in final report.

BIBLIOMETRIC ANALYSIS

Findings: Publication Counts: 2007-2010

54 AK EPSCoR faculty produced 122 peer reviewed journal publications

Additional publications include 45 books/one time pubs Other (non-EPSCoR) journal publications (pre/non) = 680

2003 2004 2005 2006 2007 2008 2009 20100

20406080

100120140160180200

Publication Counts by YearEPSCoR & Other Publications

AllEPSCoR

Finding: Co-Authorship on Alaska EPSCoR Publications -- Alaska EPSCoR faculty have broadco-authorship ties, within and outside of Alaska

443 authors54 EPSCoR affiliated authors122 EPSCoR publications

Finding:Co-authorship ties within AK EPSCoR reflect fewer broad relationships, but clear cross- component ties

54 EPSCoR authorsE-I index (component) = -0.073E-Index (institution) = -0.707

Other Co-

authorships

Finding: Knowledge sharing/coalescenceMost central faculty showed some common knowledge base of AK EPSCoR

(N=2

20 a

nd 6

6)

EP

SC

oR P

ublic

atio

ns (N

=86)

Non

-EP

SC

oR P

ublic

atio

ns

(N=1

86 a

nd 6

3)

2003-6 2007-10

0%

10%

20%

30%

40%

22%35%

20%

36 core network authors

• Percentage of references shared by at least one EPSCoR author• EPSCoR publications show higher % of shared reference than pre or

non-EPSCoR publications

Rafols, Porter and Leydesdorff (2009)

Cognitive Sci.

Agri Sci

Biomed Sci

Chemistry

Physics

Engineering

Env Sci & Tech

Matls Sci

Infectious Diseases

Psychology

Social Studies

Clinical Med

Computer Sci.Business & MGT

Geosciences

Ecol Sci

Econ. Polit. & Geography

Health & Social Issues

Global Map of Science, 2007221 SCI-SSCI Subject Categories

EPSCoR Publications in the Map of Science

Cognitive Sci.

Agri Sci

Biomed Sci

Chemistry

Physics

Engineering

Env Sci & Tech

Matls Sci

Infectious Diseases

Psychology

Social Studies

Clinical Med

Computer Sci.Business & MGT

Geosciences

Ecol Sci

Econ. Polit. & Geography

Health & Social Issues

Integration Score = 0.523

Disciplinary placement of AK EPSCoR Pubs

Finding: EPSCoR publications overall demonstrate higher interdisciplinarity than non-EPSCoR pubs

Publications Integration Score

Number of articles

pre-2007 pre-EPSCoR 0.469 516

2007-2010 EPSCoR 0.523 122

Non-EPSCoR 0.479 532

2007 2008 2009 2010 20113

25

87

325

156

AK EPSCoR Citation Record Count

FINDING: AK EPSCoR show early signs of having a multiplicative effects:

Results:122 EPSCoR publications are cited 596 times in the WoS (686 times with self citation (9% ) 2267 Total Authors, 836 Institutions 282 Journals, 96 Subject Categories Articles citing AK EPSCoR publication are in turn cited by another

1687 articles Caveats

Lag times Partial 2011 data Faculty reporting accuracy

AK EPSCoR Early Impacts:Work cited by authors across the U.S. (all 50 states)

Created using addresses of authors listed on the publication citing AK EPSCoR

Finding:Global impact of Alaska EPSCoR Publications: cited by authors in 71 countries

11111111112443286

105

171735

0 10 20 30 40 50 60 70 80 90 100 110

10251

4635

3421

1918

1412

1110

98

76

54

32

10

Times Cited

Finding: Citation Impacts based on broad set of articles.

11111111112443286

105

171735

0 10 20 30 40 50 60 70 80 90 100 110

10251

4635

3421

1918

1412

1110

98

76

54

32

10

Times Cited

Finding: Citation Impacts based on broad set of articles.

Some “Chapin Effect” exists. Yet, citations of EPSCoR work are more

broadly based

EPSCoR Publications in the Map of Science

Cognitive Sci.

Agri Sci

Biomed Sci

Chemistry

Physics

Engineering

Env Sci & Tech

Matls Sci

Infectious Diseases

Psychology

Social Studies

Clinical Med

Computer Sci.Business & MGT

Geosciences

Ecol Sci

Econ. Polit. & Geography

Health & Social Issues

N= 122Integration Score = 0.523

Disciplinary placementof AK EPSCoR Pubs

Cognitive Sci.

Agri Sci

Biomed Sci

Chemistry

Physics

Engineering

Env Sci & Tech

Matls Sci

Infectious Diseases

Psychology

Social Studies

Clinical Med

Computer Sci.Business & MGT

Geosciences

Ecol Sci

Econ. Polit. & Geography

Health & Social Issues

Knowledge Impacts:Articles citing EPSCoR Pubs

N=596 articles

Citing Pieces:EPSCoR Cited- Publications in the Map of Science

Overall Findings

Alaska EPSCoR Phase III research has: generated new research ideas and related

collaborative relationships; generated work that has been recognized

nationally and internationally; generated or leveraged global research ties,

expanding the knowledge resources of Alaskan researchers and institutions; and

produced work that has been based on interdisciplinary knowledge resources

been visible in a cross disciplinary international research community.