towards indicators for ‘opening up’ science and technology policy ismael rafols, tommaso ciarli,...
Post on 15-Jan-2016
218 Views
Preview:
TRANSCRIPT
Towards indicators for ‘opening up’ science and technology policy
Ismael Rafols, Tommaso Ciarli,
Patrick van Zwanenberg and Andy Stirling
Ingenio (CSIC-UPV), Universitat Politècnica de València &
SPRU —Science and Technology Policy Research, University of Sussex
Stockholm, October 2013
Building on work with Loet Leydesdorff and Alan Porter
Paper born out of the reflection on the contrast between interdisciplinary research and journal rankings
Interdisciplinary maps
versus rankings
On the role of scientific advice in policy(scientometric is the science of science –hence scientific advice)
The linearity-autonomy model of scientific advice (Jasanoff, 2011) Scientific knowledge is the best possible foundation for public
decisions Scientists should establish the facts that matter independently.
– S&T indicators produce evidence of these facts.
However, this (enlightenment) model has been challenged The mechanisms to establish facts and make decisions is a social
process – “knowledge enables power, but power structures knowledge” (Stirling, 2012)
Modes of advice: The pure scientist vs. honest broker (Pielke, 2007)
What is (should be) the role of STI indicators in policy advice? Closing down vs. Opening up
The challengeProblems with current use of S&T indicators
Use of conventional S&T indicators is en *problematic*
(as many technologies, in particular those closely associated with power, e.g. nuclear)
Narrow inputs (only pubs!) Scalar outputs (rankings!) Aggregated solutions --missing variation Opaque selections and classifications (privately owned databases) Large, leading scientometric groups embedded in government /
consultancy, with limited possibility of public scrutiny Sometimes even mathematically debatable
Impact Factor of journals (only 2 years, ambiguity in document types) Average number of citations (pubs) in skewed distributions
From S&T indicators for justification and disciplining…
Justification in decision-making• Weak justification, “Give me a number, any number!”• Strong justification, “Show in numberrs that X is the best choice!”
S&T Indicators have a performative role: They don’t just measure. Not ‘just happen to be used’ in science
policy (neutral) Constitutive part incentive structure for “disciplining” (loaded) They signal to stakeholders what is important.
Institutions use these techniques to discipline subjects Articulate framings, goals and narratives on performance,
collaboration, interdisciplinarity…
… towards S&T indicators as tools for deliberation
Yet is possible to design indicators that foster plural reflection rather than justifying or reinforcing dominant perspectives
This shift is facilitated by trends pushed by ICT and visualisation tools
More inputs (pubs, pats, but also news, webs, etc.) Multidimensional outputs (interactive maps) Multiple solutions -- highlighting variation, confidence intervals More inclusive and contrasting classifications (by-passing
private data ownership? Pubmed, Arxiv) More possibilities for open scrutiny (new research groups)
1. Conceptual framework:
“broadening out” vs. “opening up” policy appraisal
Policy use of S&T indicators: Appraisal
Appraisal:
‘the ensemble of processes through which knowledges are gathered and produced in order to inform decision-making and wider institutional commitments’ Leach et al. (2008)
Breadth: extent to which appraisal covers diverse dimensions of knowledge
Openness: degree to which outputs provide an array of options for policies.
Policy use of S&T indicators: Appraisal
Appraisal:
‘the ensemble of processes through which knowledges are gathered and produced in order to inform decision-making and wider institutional commitments’ Leach et al. (2010)
Example: Allocation of resources based on research “excellence”
Breadth: extent to which appraisal covers diverse dimensions of knowledgeNarrow: citations/paper
Broad: citations, peer interview, stakeholder view, media coverage, altmetrics
Openness: degree to which outputs provide an array of options for policies. Closed: fixed composite measure of variables unitary and prescriptive
Open: consideration of various dimensions plural and conditional
narrow
broad
closing-down opening-up
range of appraisals inputs(issues, perspectives, scenarios, methods)
effect of appraisal ‘outputs’ on decision-making
Leach et al. 2010
Appraisal methods: broad vs. narrow & closing vs. opening
narrow
broad
closing-down opening-up
range of appraisals inputs(issues, perspectives, scenarios, methods)
effect of appraisal ‘outputs’ on decision-making
Appraisal methods: broad vs. narrow & close vs. open
cost-benefit analysis
open hearings
consensusconference
scenarioworkshops
citizens’ juries
multi-criteria mapping
q-method
sensitivityanalysis
narrative-based participant observation
decision analysis
risk assessment structured interviews
Stirling et al. (2007)
narrow
broad
closing-down opening-up
range of appraisals inputs(issues, perspectives, scenarios, methods)
effect of appraisal ‘outputs’ on decision-making
Appraisal methods: broad vs. narrow & closing vs. opening
Most conventionalS&T indicators??
narrow
broad
closing-down opening-up
range of appraisals inputs(issues, perspectives, scenarios, methods)
effect of appraisal ‘outputs’ on decision-making
Broadening out S&T Indicators
ConventionalS&T indicators??
Broadening out
Incorporation plural analytical dimensions:
global & local networkshybrid lexical-actor netsetc.
New analytical inputs: media, blogsphere.
narrow
broad
closing-down opening-up
range of appraisals inputs(issues, perspectives, scenarios, methods)
effect of appraisal ‘outputs’ on decision-making
Appraisal methods: broad vs. narrow & closing vs. opening
Journal rankings
University rankings Unitary measuresthat are opaque, tendency to favour the established perspectives
… and easily translated into prescription
European InnovationScoreboard
narrow
broad
closing-down opening-up
range of appraisals inputs(issues, perspectives, scenarios, methods)
effect of appraisal ‘outputs’ on decision-making
Opening up S&T Indicators
ConventionalS&T Indicators??
opening-up
Making explicit underlying conceptualisations and creating heuristic tools to facilitate exploration
NOT about the uniquely best methodOr about the unitary best explanationOr the single best prediction
2. Examples of Opening Up
a. Broadening out AND Opening up
b. Opening up WITH NARROW inputs
narrow
broad
closing-down opening-up
range of appraisals inputs(issues, perspectives, scenarios, methods)
effect of appraisal ‘outputs’ on decision-making
1. Preserving multiple dimensions in broad appraisals
ConventionalS&T indicators??
Leach et al. 2010
Broadening out opening-up
Composite Innovation Indicators (25-30 indicators)
European (Union) Innovation Scoreboard
Grupp and Schubert (2010) show that order is highly dependent on indicators weightings. Sensitivity analysis
Solution: representing multiple dimensions(critique by Grupp and Schubert, 2010)
Use of spider diagramsallows comparing like with like
U-rank, University performance Comparison tools(Univ. Twente)
5.4 Community trademarks indicator
U-Map: Comparison of Universities in Multiple Dimensions
2. Examples of Opening Up
b. Opening up WITH NARROW inputs
narrow
broad
closing-down opening-up
range of appraisals inputs(issues, perspectives, scenarios, methods)
effect of appraisal ‘outputs’ on decision-making
Opening up S&T Indicators
ConventionalS&T Indicators??
Leach et al. 2010
opening-up
Making explicit underlying conceptualisations and creating heuristic tools to facilitate exploration
NOT about the uniquely best methodOr about the unitary best explanationOr the single best prediction
Manchester Inn Inst
WarwickBusinessSchool
Disciplinary Diversity of Publications Variety 19 20Shannon Entropy 2.966 3.078
Disciplinary Diversity of References Variety 17 20Shannon Entropy 3.378 3.153
Disciplinary diversity of Citations Variety 22 24Shannon Entropy 3.415 3.503
Interdisciplinarity as diversity
Rafols, Porter and Leydesdorff (2010)
Cognitive Sci.
Agri Sci
Biomed Sci
Chemistry
Physics
Engineering
Env Sci & Tech
Matls Sci
Infectious Diseases
Psychology
Social Studies
Clinical Med
Computer SciBusiness & MGT
Geosciences
Ecol Sci
Econ Polit. & Geography
Health & Social Issues
A Global Map of Science222 SCI-SSCI Subject Categories
Warwick Business SchoolSubject Categories of publications
Nodes labelled if >0.5% publications
Manchester MIoIRSubject Categories of publications
Nodes labelled if >0.5% publications
Heuristics of diversity
(Stirling, 1998; 2007)
Diversity:‘attribute of a system whose elements may be apportioned into categories’
Characteristics: Variety: Number of distinctive categoriesBalance: Evenness of the distribution Disparity: Degree to which the categories
are different.
Variety
Balance Disparity
Herfindahl (concentration): i pi2
Shannon (Entropy): i pi ln pi
Dissimilarity: i di
Generalised Diversity (Stirling) ij(ij) (pipj)a (dij)
b
Manchester Innov Inst
WarwickBusiness S
Diversity of Publications Variety 19 20Balance 0.543 0.460Disparity 0.817 0.770Shannon Entropy 2.966 3.078Rao-Stirling Diversity 0.726 0.680Diversity of References Variety 17 20Balance 0.415 0.325Disparity 0.846 0.780Shannon Entropy 3.378 3.153Rao-Stirling Diversity 0.729 0.689Diversity of Citations Variety 22 24Balance 0.505 0.454Disparity 0.836 0.801Shannon Entropy 3.415 3.503Rao-Stirling Diversity 0.771 0.736
Comparing degree of interdisciplinarity of two university units: Manchester is more??
Multiple concepts of interdisciplinarity:
Conspicuous lack of consensus but most indicators aim to capture the following concepts
Integration (diversity & coherence)• Research that draws on
diverse bodies of knowledge • Research that links different
disciplines
Intermediation• Research that lies between or
outside the dominant disciplines
Coherence
Low High
Diversity
Low
High
InterdisciplinaryMultidisciplinary
Monodisciplinary
Intermediation
Low High
Monodisciplinary Interdisciplinary
Diversity
ISSTI Edinburgh WoS Cats of references
Assessing interdisciplinarity
ISSTI EdinburghObserved/ExpectedCross-citations
CoherenceAssessing interdisciplinarity
RiskAnal
PsycholBull
PhilosTRSocA
Organization
JPersSocPsychol
JLawEconOrgan
JIntEcon
Interfaces
EnvironSciPolicy
CanJEcon
ApplEcon
AnnuRevPsychol
RandJEcon
JPublicEcon
JManage
JLawEcon
HumRelat
BiomassBioenerg
AtmosEnviron
PolicySci
JIntBusStud
JApplPsychol
Econometrica
PublicUnderstSci
PsycholRev
JFinancEcon
JApplEcolJAgrarChangeClimaticChange
AcadManageJ
JRiskRes
JDevStud
Scientometrics
HarvardBusRev
IntJMedInform
GlobalEnvironChang
EconJ
JFinanc
StudHistPhilosSci
DrugInfJ
Futures
WorldDev
StrategicManageJ
SciTechnolHumVal
EconSoc
PublicAdmin
Lancet
IndCorpChange
AccountOrgSoc
EnergPolicy
Nature
AmJSociol
ResPolicy
TechnolAnalStrateg SocStudSciBritMedJ
ISSTI EdinburghReferences
IntermediationAssessing interdisciplinarity
Summary: IS (blue) units are more interdisciplinary than BMS (orange)
More DiverseRao-Stirling Diversity
More CoherentObserved/Expected
Cross-Citation Distance
More InterstitialAverage Similarity
0.02
0.0300000000000001
0.0400000000000001
0.0500000000000001
0.0600000000000001
0.0700000000000002
2. Excellence: Opening Up Perspectives
Provide different perspectives of performance(alternative measures of the same type of indicator)
Are measures of “excellence” consistent and robust?
Good
Average
Bad
Van Eck, Waltman et al. (2013)
More basic
More applied
Clinical neurologyIs basic always better than applied?
Citations: not stable to changes in classification and granularity (Zitt et al., 2005; Adams et al., 2008).
Measures of “excellence”
ISSTI SPRU MIoIR Imperial WBS LBS0
0.5
1
1.5
2
2.5
3
3.5
4A
BS R
ank
ISSTI SPRU MIoIR Imperial WBS LBS0
1
2
3
4
Cita
tions
/pub
Jo
urna
l-fiel
d N
orm
alis
ed
ISSTI SPRU MIoIR Imperial WBS LBS0
0.05
0.1
0.15
0.2
Cita
tions
/pub
Citi
ng-p
aper
Nor
mal
ised
Which one is more meaningful??
ISSTI SPRU MIoIR Imperial WBS LBS0
1
2
3
4
Jour
nal I
mpa
ct F
acto
r
The new Leiden ranking (2011-12) • Different measures of performance
• MNC, MNCS, MNCJ, Top 10%, • Under different conditions (fractional, language)• Include confidence interval (bootstrapping)
3. Summary and conclusions
S&T indicator as a tools to open up the debate
• ‘Conventional’ use of indicators (‘Pure scientist ‘--Pielke) Purely analytical character (i.e. free of normative assumptions) Instruments of objectification of dominant perspectives Aimed at legitimising /justifying decisions (e.g. excellence) Unitary and prescriptive advice
• Opening up scientometrics (‘Honest broker’ --Pielke) Aimed at locating the actors in their context and dynamics
Not predictive, or explanatory, but exploratory Construction of indicators is based on choice of perspectives
Make explicit the possible choices on what matters Supporting debate
Making science policy more ‘socially robust’ Plural and conditional advice
Barré (2001, 2004, 2010), Stirling (2008)
Strategies for opening up or how to “keep it complex” yet “manageable”
• Presenting contrasting perspectives At least TWO, in order to give a taste of choice
• Simultaneous visualisation of multiple properties / dimensions Allowing the user take its own perspective
• Interactivity Allowing the user give its own weigh to criteria / factors Allowing the user manipulate visuals
.
Is ‘opening up’ worth the effort? (1)
Sustaining diversity in S&T system
Decrease in diversity.
Potential unintended consequence of the evaluation machine:
Why diversity matters
Systemic (‘ecological’) understanding of the S&T S&T outcomes depend on synergistic interactions between
disparate elements.
Dynamic understanding of excellence and relevance New social needs, challenges, expectations from S&T
Manage diverse portfolios to hedge against uncertainty in research Office of Portfolio Analysis (National Institutes of Health)
http://dpcpsi.nih.gov/opa/
Open possibility for S&T to work for the disenfranchised Topics outside dominant science (e.g. neglected diseases)
Is ‘opening up’ worth the effort? (2)
Building robustness against bias
Do conventional indicators tend to favour incumbents?
Hypothesis:
Elites and incumbents (directly or not) influence choice of indicators, which tend to benefit them…
“knowledge enables power, but power structures knowledge” (Stirling, 2012)
Crown indicator –Standard measure of performance (~1990-2010)– ‘systematic underrating of low-ranked scientists’ (Opthof and
Leydesdorff, 2010) (Not spotted for 15 years!) Journal rankings in Business and Management.
– systematic underrating of interdisciplinary (heterodox) depts. (Rafols et al., 2012).
Others?? H-index??– favours established academics over younger.
‘lock-in’ to policy favoured by incumbent
power structures
multiple practices, and processes, for informing social agency (emergent and unstructured as well as deliberately designed )
complex, dynamic, inter-coupled and mutually-
reinforcing socio-technical configurations
in science
narrow scope of attention
Conventional Policy Dynamics
SOCIAL APPRAISAL
GOVERNANCE COMMITMENTS
simple ‘unitary’ prescriptions
POSSIBLE FUTURES
expert judgements /
‘evidence base’
“best / optimal /legitimate”
S&T indicators
risk assessment
cost-benefit analysis
also: restricted options, knowledges, uncertainties in participation
incomplete knowledges
Res. Excellence
$IIIIII
GUIDANCE / NARRATIVE
Stirling (2010)
POSSIBLE PATHWAYSMULTIPLE
TRAJECTORIES
SOCIAL APPRAISAL
GOVERNANCE COMMITMENTS
broad-based processes of ‘precautionary appraisal’
‘opening up’ with ‘plural conditional’
outputs to policymaking
dynamic portfolios pursuing diverse
trajectories
viable options under: conditions, dissonant views,
sensitivities, scenarios, maps, equilibria, pathways, discourses
multiple: methods, criteria, options, frames, uncertainties, contexts, properties, perspectives
Breadth, Plurality and Diversity
Sustainability
$
Stirling (2010)
S&T indicator as a tools to open up the debate
• ‘conventional’ use of indicators Instruments of objectification Analytical character (i.e. free of normative assumptions) Aimed at making decisions (e.g. excellence) Unitary and prescritive advice
• Opening up scientometrics Construction of indicators is based on choice of perspectives
implicit normative choice on what matters Aimed at locating the actors in their context and dynamics
Not predictive, or explanatory, but exploratory Supporting debate
making science policy more ‘socially robust’ Plural and conditional advice
Barré (2001, 2004, 2010), Stirling (2008)
Heuristics of diversity
(Stirling, 1998; 2007)
Diversity:‘attribute of a system whose elements may be apportioned into categories’
Characteristics: Variety: Number of distinctive categoriesBalance: Evenness of the distribution Disparity: Degree to which the categories
are different.
Variety
Balance Disparity
Herfindahl (concentration): i pi2
Shannon (Entropy): i pi ln pi
Dissimilarity: i di
Generalised Diversity (Stirling) ij(ij) (pipj)a (dij)
b
Rafols, Porter and Leydesdorff (2010)
Cognitive Sci.
Agri Sci
Biomed Sci
Chemistry
Physics
Engineering
Env Sci & Tech
Matls Sci
Infectious Diseases
Psychology
Social Studies
Clinical Med
Computer SciBusiness & MGT
Geosciences
Ecol Sci
Econ Polit. & Geography
Health & Social Issues
A Global Map of Science222 SCI-SSCI Subject Categories
• CD-ROM version of the JCR of SCI and SSCI of 2009.• Matrix of cross-citations between journals (9,000 x
9,000)• Collapse to ISI Subject Category matrix (222 x 222)• Create similarity matrix using Salton’s cosine
Diversity indexes
Stirling Generalised Diversity
Diversity indexes
Stirling Generalised Diversity
a=0, b=0
Number of disciplines
Diversity indexes
:Stirling Generalised Diversity
a=0, b=1Simpson(Herfindahl) Index
Diversity indexes
Generalised Stirling Diversity
a=1, b=1quadratic entropy
Different aspects of diversity are uncorrelated
rvar,bal = 0.18, p < .001rvar,dis = 0.32, p < .001
rbal,dis = -0.20, p < .001Yegros et al. (2010)
Which diversity measureshould we choose?
Leiden Ranking: high correlation but important individual differences
mcs_rank mncs_rank mncs_frac_rank mncs_doc_rank mncs_SM_rank mncs_LW_rank
mcs_rank 1.000 0.911 0.815 0.915 0.904 0.937
mncs_rank 0.911 1.000 0.959 0.995 0.998 0.987
mncs_frac_rank 0.815 0.959 1.000 0.953 0.958 0.927
mncs_doc_rank 0.915 0.995 0.953 1.000 0.993 0.983
mncs_SM_rank 0.904 0.998 0.958 0.993 1.000 0.982
mncs_LW_rank 0.937 0.987 0.927 0.983 0.982 1.000
Spearman‘s rank correlation coefficient matrix.
(Thanks to Daniel Sirtes and Ludo Waltman for sharing this data)
top related