conceptual and empirical framework for measuring science ... · • approach: doing an evaluation...
TRANSCRIPT
![Page 1: Conceptual and Empirical Framework for Measuring Science ... · • Approach: Doing an evaluation • Conceptual Framework • Empirical Framework • Next steps . 3 . Outline •](https://reader030.vdocument.in/reader030/viewer/2022040419/5e030219d9e2ea2f204153f7/html5/thumbnails/1.jpg)
Conceptual and Empirical
Framework for Measuring
Science Impact
1
![Page 2: Conceptual and Empirical Framework for Measuring Science ... · • Approach: Doing an evaluation • Conceptual Framework • Empirical Framework • Next steps . 3 . Outline •](https://reader030.vdocument.in/reader030/viewer/2022040419/5e030219d9e2ea2f204153f7/html5/thumbnails/2.jpg)
Key ideas Conceptual Framework
• Basic unit of analysis is individual scientists/tems
• Core outcome is creation, transmission and adoption of
ideas
• Potential analytical framework is theory of the firm
Empirical Framework
• Leverage existing data and tools, particularly STAR
METRICS and SciENCV
• Build analytical community (e.g. SciSIP)
2
![Page 3: Conceptual and Empirical Framework for Measuring Science ... · • Approach: Doing an evaluation • Conceptual Framework • Empirical Framework • Next steps . 3 . Outline •](https://reader030.vdocument.in/reader030/viewer/2022040419/5e030219d9e2ea2f204153f7/html5/thumbnails/3.jpg)
Outline • Motivation
• Approach: Doing an evaluation
• Conceptual Framework
• Empirical Framework
• Next steps
3
![Page 4: Conceptual and Empirical Framework for Measuring Science ... · • Approach: Doing an evaluation • Conceptual Framework • Empirical Framework • Next steps . 3 . Outline •](https://reader030.vdocument.in/reader030/viewer/2022040419/5e030219d9e2ea2f204153f7/html5/thumbnails/4.jpg)
Outline • Motivation
• Approach: Doing an evaluation
• Conceptual Framework
• Empirical Framework
• Next steps
4
![Page 5: Conceptual and Empirical Framework for Measuring Science ... · • Approach: Doing an evaluation • Conceptual Framework • Empirical Framework • Next steps . 3 . Outline •](https://reader030.vdocument.in/reader030/viewer/2022040419/5e030219d9e2ea2f204153f7/html5/thumbnails/5.jpg)
Motivation • Congressional and Administration interest
in building a scientific basis for science
policy
Science of Science Policy Interagency Group
NSF SciSIP program
5
![Page 6: Conceptual and Empirical Framework for Measuring Science ... · • Approach: Doing an evaluation • Conceptual Framework • Empirical Framework • Next steps . 3 . Outline •](https://reader030.vdocument.in/reader030/viewer/2022040419/5e030219d9e2ea2f204153f7/html5/thumbnails/6.jpg)
Motivation
Agencies should demonstrate the use of
evidence throughout their Fiscal Year (FY)
2014 budget submissions. Budget submissions
also should include a separate section on
agencies' most innovative uses of evidence and
evaluation, addressing some or all of the issues
below….
6
![Page 7: Conceptual and Empirical Framework for Measuring Science ... · • Approach: Doing an evaluation • Conceptual Framework • Empirical Framework • Next steps . 3 . Outline •](https://reader030.vdocument.in/reader030/viewer/2022040419/5e030219d9e2ea2f204153f7/html5/thumbnails/7.jpg)
Motivation
The Office of Science and Technology Policy
has created a "community of practice" for
agency personnel involved in designing and
managing incentive prizes and has organized
a Science of Science Policy working group
that is developing tools aimed at establishing
a more scientific, empirical evidence basis for
science and technology policymaking.
7
![Page 8: Conceptual and Empirical Framework for Measuring Science ... · • Approach: Doing an evaluation • Conceptual Framework • Empirical Framework • Next steps . 3 . Outline •](https://reader030.vdocument.in/reader030/viewer/2022040419/5e030219d9e2ea2f204153f7/html5/thumbnails/8.jpg)
8
Motivation
8
![Page 9: Conceptual and Empirical Framework for Measuring Science ... · • Approach: Doing an evaluation • Conceptual Framework • Empirical Framework • Next steps . 3 . Outline •](https://reader030.vdocument.in/reader030/viewer/2022040419/5e030219d9e2ea2f204153f7/html5/thumbnails/9.jpg)
9
Motivation:
Buttressing Anecdotes
![Page 10: Conceptual and Empirical Framework for Measuring Science ... · • Approach: Doing an evaluation • Conceptual Framework • Empirical Framework • Next steps . 3 . Outline •](https://reader030.vdocument.in/reader030/viewer/2022040419/5e030219d9e2ea2f204153f7/html5/thumbnails/10.jpg)
10
Outline • Motivation
• Approach: Doing an Evaluation
• Conceptual Framework
• Empirical Framework
• Next steps
![Page 11: Conceptual and Empirical Framework for Measuring Science ... · • Approach: Doing an evaluation • Conceptual Framework • Empirical Framework • Next steps . 3 . Outline •](https://reader030.vdocument.in/reader030/viewer/2022040419/5e030219d9e2ea2f204153f7/html5/thumbnails/11.jpg)
I
Classic Questions for
Measuring Impact
What is the impact or causal effect of a program on outcome of interest?
s a given program effective
compared to the absence of the
program?
When a program can be
implemented in several ways, which
one is the most effective?
11
![Page 12: Conceptual and Empirical Framework for Measuring Science ... · • Approach: Doing an evaluation • Conceptual Framework • Empirical Framework • Next steps . 3 . Outline •](https://reader030.vdocument.in/reader030/viewer/2022040419/5e030219d9e2ea2f204153f7/html5/thumbnails/12.jpg)
12
Classic Example:
Measuring Impact
Illustration of swan-necked flask experiment used by Louis Pasteur to test the
h ypothesis of spontan eous generatio n
![Page 13: Conceptual and Empirical Framework for Measuring Science ... · • Approach: Doing an evaluation • Conceptual Framework • Empirical Framework • Next steps . 3 . Outline •](https://reader030.vdocument.in/reader030/viewer/2022040419/5e030219d9e2ea2f204153f7/html5/thumbnails/13.jpg)
13
Classic Challenge: Identifying
Counterfactual
“A key goal of an impact evaluation is to identify
a group of program participants (the treatment
group) and a group of nonparticipants (the
comparison group) that are statistically identical
in the absence of the program.” World Bank
![Page 14: Conceptual and Empirical Framework for Measuring Science ... · • Approach: Doing an evaluation • Conceptual Framework • Empirical Framework • Next steps . 3 . Outline •](https://reader030.vdocument.in/reader030/viewer/2022040419/5e030219d9e2ea2f204153f7/html5/thumbnails/14.jpg)
14
Classic Challenge: Theory of
Change
![Page 15: Conceptual and Empirical Framework for Measuring Science ... · • Approach: Doing an evaluation • Conceptual Framework • Empirical Framework • Next steps . 3 . Outline •](https://reader030.vdocument.in/reader030/viewer/2022040419/5e030219d9e2ea2f204153f7/html5/thumbnails/15.jpg)
15
Outline • Motivation
• Approach: Doing an evaluation
• Conceptual Framework
• Empirical Framework
• Next steps
![Page 16: Conceptual and Empirical Framework for Measuring Science ... · • Approach: Doing an evaluation • Conceptual Framework • Empirical Framework • Next steps . 3 . Outline •](https://reader030.vdocument.in/reader030/viewer/2022040419/5e030219d9e2ea2f204153f7/html5/thumbnails/16.jpg)
16
Key Concept: Individual is
basic unit of analysis
Source: Ian Foster, University of Chicago
![Page 17: Conceptual and Empirical Framework for Measuring Science ... · • Approach: Doing an evaluation • Conceptual Framework • Empirical Framework • Next steps . 3 . Outline •](https://reader030.vdocument.in/reader030/viewer/2022040419/5e030219d9e2ea2f204153f7/html5/thumbnails/17.jpg)
17
Core outcome is ideas • Goal of project/firm: to create and transmit scientific ideas and
push for their adoption (by other scientists, policy-makers or
businesses)
• Behavioral Framework; Ideas are transmitted by workers in a
variety of potentially measurable ways, including publications,
presentations, blogs, internal project workspaces, and emails
• Behavioral Framework: Social networks/collaboration are a major
vehicle whereby ideas are transmitted
![Page 18: Conceptual and Empirical Framework for Measuring Science ... · • Approach: Doing an evaluation • Conceptual Framework • Empirical Framework • Next steps . 3 . Outline •](https://reader030.vdocument.in/reader030/viewer/2022040419/5e030219d9e2ea2f204153f7/html5/thumbnails/18.jpg)
Possible theory of Change:
Theory of the firm
Repurposing the theory of the firm 𝑌 = 𝑋 𝛽 + 𝜀 𝑋 = 𝑍 𝜑 + 𝜔
Y – transmission of ideas
X - networks/collaborations
Z – level of funding
𝜑, 𝛽 –
𝜀, 𝜔 - error terms
18
![Page 19: Conceptual and Empirical Framework for Measuring Science ... · • Approach: Doing an evaluation • Conceptual Framework • Empirical Framework • Next steps . 3 . Outline •](https://reader030.vdocument.in/reader030/viewer/2022040419/5e030219d9e2ea2f204153f7/html5/thumbnails/19.jpg)
Outline • Motivation
• Approach
• Conceptual Framework
• Empirical Framework
• Analytical Framework
• Next steps
19
![Page 20: Conceptual and Empirical Framework for Measuring Science ... · • Approach: Doing an evaluation • Conceptual Framework • Empirical Framework • Next steps . 3 . Outline •](https://reader030.vdocument.in/reader030/viewer/2022040419/5e030219d9e2ea2f204153f7/html5/thumbnails/20.jpg)
20
Challenge: The data didn ’t exist
![Page 21: Conceptual and Empirical Framework for Measuring Science ... · • Approach: Doing an evaluation • Conceptual Framework • Empirical Framework • Next steps . 3 . Outline •](https://reader030.vdocument.in/reader030/viewer/2022040419/5e030219d9e2ea2f204153f7/html5/thumbnails/21.jpg)
Leverage Existing Data Portfolio Explorer
21
tent Datab
Source: Ian Foster University of
Chicago
Scientific Workforce e.g. Pa ase
![Page 22: Conceptual and Empirical Framework for Measuring Science ... · • Approach: Doing an evaluation • Conceptual Framework • Empirical Framework • Next steps . 3 . Outline •](https://reader030.vdocument.in/reader030/viewer/2022040419/5e030219d9e2ea2f204153f7/html5/thumbnails/22.jpg)
22
Leverage existing tools to
capture ideas
Source: David Newman, UCI
![Page 23: Conceptual and Empirical Framework for Measuring Science ... · • Approach: Doing an evaluation • Conceptual Framework • Empirical Framework • Next steps . 3 . Outline •](https://reader030.vdocument.in/reader030/viewer/2022040419/5e030219d9e2ea2f204153f7/html5/thumbnails/23.jpg)
•
•
•
23 23
Tools to capture level of
funding
• 1000 topics learned from
200k submitted NSF
proposals
Topics ideal for research
portfolio analysis
Measure relationship
between research
investment and research
outcomes (publications,
patents)
Topic model unifies
research categories across
funding agencies and
document genre
![Page 24: Conceptual and Empirical Framework for Measuring Science ... · • Approach: Doing an evaluation • Conceptual Framework • Empirical Framework • Next steps . 3 . Outline •](https://reader030.vdocument.in/reader030/viewer/2022040419/5e030219d9e2ea2f204153f7/html5/thumbnails/24.jpg)
24
Tools to capture level of
funding
24
NIH-
specific
topics
NSF-
specific
topics
Topics
relevant
to both
NIH and
NSF
NIH grant
NSF grant
Topic modeling can
learn categories that
cut across federal
funding agency
boundaries
Source: David Newman, UCI
![Page 25: Conceptual and Empirical Framework for Measuring Science ... · • Approach: Doing an evaluation • Conceptual Framework • Empirical Framework • Next steps . 3 . Outline •](https://reader030.vdocument.in/reader030/viewer/2022040419/5e030219d9e2ea2f204153f7/html5/thumbnails/25.jpg)
25
Combine tools and data to capture
networks
Source: Jason Owen-Smith
![Page 26: Conceptual and Empirical Framework for Measuring Science ... · • Approach: Doing an evaluation • Conceptual Framework • Empirical Framework • Next steps . 3 . Outline •](https://reader030.vdocument.in/reader030/viewer/2022040419/5e030219d9e2ea2f204153f7/html5/thumbnails/26.jpg)
Outline • Motivation
• Approach: Doing an evaluation
• Conceptual Framework
• Empirical Framework
• Next steps
26
![Page 27: Conceptual and Empirical Framework for Measuring Science ... · • Approach: Doing an evaluation • Conceptual Framework • Empirical Framework • Next steps . 3 . Outline •](https://reader030.vdocument.in/reader030/viewer/2022040419/5e030219d9e2ea2f204153f7/html5/thumbnails/27.jpg)
Engage analytical community
• Patent Network Dataverse Lee Fleming Harvard
• COMETS (Connecting Outcome Measures in Entrepreneurship Technology and Science): Lynne Zucker and Michael Darby UCLA
• Randomized Controlled Trials John Willinsky, Stanford
27
27
![Page 28: Conceptual and Empirical Framework for Measuring Science ... · • Approach: Doing an evaluation • Conceptual Framework • Empirical Framework • Next steps . 3 . Outline •](https://reader030.vdocument.in/reader030/viewer/2022040419/5e030219d9e2ea2f204153f7/html5/thumbnails/28.jpg)
Build on key ideas Extend Conceptual Framework
• Basic unit of analysis is individual scientists/teams
• Core outcome is creation, transmission and adoption of
ideas
• Potential analytical framework is theory of the firm
Extend Empirical Framework
• Leverage existing data and tools, particularly STAR
METRICS and SciENCV
• Build analytical community (e.g. SciSIP)
28
![Page 29: Conceptual and Empirical Framework for Measuring Science ... · • Approach: Doing an evaluation • Conceptual Framework • Empirical Framework • Next steps . 3 . Outline •](https://reader030.vdocument.in/reader030/viewer/2022040419/5e030219d9e2ea2f204153f7/html5/thumbnails/29.jpg)
29
29
And a reminder of why!