Pre-WorkshopAssessment Results

Science of Science Policy WorkshopGeorge Washington University

December 2008

Were the Correct 10 Questions Identified?

Answer Total Percentage

No 49 24.62%

Yes 150 75.38%

YES: But Only a Start• “A good coherent set of questions to start. As the number of

practitioners increase the questions are likely to change, or new ones will emerge. It is important to have a open and flexible leadership to incorporate different perspectives.”

• “Further understanding of the policy instruments used by agencies is important. We need more modeling to understand how policies (and regulations) affect the practice of science.”

• “They are broad enough labels to encompass the core set of problems.”

• “I feel the answers to these 10 questions should provide the insight to develop the roadmap. I commend the authors on a job well done.”

YES: But You Missed Some Things• “How can we appropriately assess science management? • “How can we advance trans-disciplinary integration of knowledge to

promote innovation?”• “There is no mention of the role of ‘mental competition’ in behavioral

foundation of innovation.”• “The humanitarian dimension of why we do science seemed to be

missing.”• “I could think of 5 more, but I don’t know that any of them would

displace 5 from your list.”• “I would add one more question under theme two: What is the value

of public investment in Internationally collaborated research? “• “Where is the question on data?”

YES: But not Detailed Enough• “There is no mention of the critical role of professional organizations

and societies, in the evolution of scientific communities.”• “In question 2.2 in addition to ‘predict’, ‘induce’ could be relevant to

explore.”• “I might have pushed further on the methodological challenges

involved in addressing some of those questions (e.g., rate of return, link to competitiveness.”

• “Some will require further depth, but the questions are excellent.”• I understand there will be a framework for creating the application,

but I think there should be more investigation into a practical application (tools, methods, data, etc).”

• “The context for the questions relies too much on the economics literature. An important part of the SoSP effort MUST be breaking the stranglehold that economics has on Federal agencies.”

NO: Way too Simple• “There's a bit of a reductionist bias throughout, which belies talk

about the ‘ecology of innovation.’”• “The scope is too narrowly defined. Instead of focusing solely on

the government's role, there should be a focus on how to better develop public/private/academic (and, to a degree, international) collaborations in establishing national science policies and investment strategies.”

• “What data problems; e.g., access, quality, uncollected data, currently impede the study of innovation -- in science and business?”

NO: You Missed Important Questions• “Study ‘knowledge spillovers’ as a crucial determinant for the

creation of knowledge; these spillovers in turn occur mainly in urban agglomerations. Another important topic is the role of entrepreneurship in bringing intellectual novelty to market, thereby creating wealth.

• “Pay systematic attention to the unanticipated and negative outcomes of innovation and science policy (for example, nuclear technology).”

• “Should there be a national industrial and science policy? If so, what role should the federal government play in developing, supporting and promoting this policy?"

NO: You Fundamentally Misunderstand• “Two concerns: First, some of the questions are quite similar, so I

don't see them as addressing 10 distinct issues. Second, they seem to be missing something very basic about what emerges from research investments -- i.e. in addition to the knowledge, you get students that turn into professionals, you get new instruments, etc.”

• “The psychology of innovators and scientists may be important, but more important is the role of education. The excessive emphasis on psychology seems misguided.”

• “There's nothing about whether there is a way to figure out the right size of the enterprise (in terms of dollars or people) or the right mix of gov't, indus., acad, or R vs. D, or military vs. civilian. These seem to be the main debating points in science policy.”

NO: Just too Narrow• “Question 8 dealt only with the impact of science on innovation and

competitiveness. There are numerous other national priorities where the impact of science can be important. I see understanding these connections as part of science policy.”

• “Lacks global perspective. For example, would the same Top 10 be relevant in Asian society?”

• “It seems that the questions are completely skewed towards stimulating discovery and technological advance. Little attention is paid to the "assessment" side of SoSP, i.e. understanding and valuing the externalities associated with technologies and their use.”

• “Questions 8 & 9 are unnecessarily narrow in their focus on ‘science.’ Both should be broadened to include scientific knowledge, technological knowledge, and business/management/finance knowledge.”

Who Should Take the Lead?

Answer Total Percentage

Academia 29 14.57%

International Community 18 9.05%

Other 30 15.08%

Private Sector 3 1.51%

U.S. Federal Government 119 59.80%

Those who Said: FEDERAL GOVERNMENT• “The Federal role would ideally be neutral, but it is the best place for

the ‘convening’ function, at least to begin with. That is not to rule out private sector role, but to insure broad participation and diverse viewpoints.”

• “The Federal government possesses resources to stimulate and encourage the development of SoSP needed to develop cutting edge science and technology.”

• “The government should nurture the SoSP COMMUNITY, but academia likely should nurture SoSP itself. Two different things...”

• “The government should pose questions as challenges to the SoSP community. Example: I want to know if the 10+ year investment in nanotech has had an impact. How can I find out?”

• “An unanswered, and indeed, unasked, question is: the role of Congress.”

Those who Said: ACADEMIA• “While incentives and coordination can come from governments, the

intellectual nurturing must come from scholars - professional societies can play an intermediate role.”

• “Academic science provides the international communities that have the skills, tools, knowledge, and to some extent the communication abilities to develop a science of SP.”

• “I think that better decisions are made by those who are experts in that field, thus I think that academia should have a strong voice in the SoSP community.”

• “While all the parties are needed as active participants, academia should take the lead. Academia is charged with nurturing the next generations of research scientists and of science policy decision-makers.”

• “Well, I really wanted to check both “Federal" and "academia" as I think this should be a collaborative undertaking.”

Those who Said: INTERNATIONAL• “The science of science policy has become an international issue.

International collaboration in science has increased rapidly in the past ten years and the fruit of science is a public good to be shared, and it follows that the cost of supporting science should be shared by the international community as well.”

• “Establishing a SoSP is a common issue for every country.”• “International cooperation policies in S&T is a policy area requiring

a redesign under the light of SOSP (and Science of Science and Innovation Policy).”

• “The major problem and the reason why we have the valley of death is because each of these parts of the complex innovation process works on its own.”

Those who Said: OTHER• “The best approach would be a joint committee comprised of NGO, state

and federal members of the science, policy and management community. It requires the unique perspective of each group combined to see the whole picture.”

• “No one entity should take the lead alone. The U.S. Federal Government cannot take the lead in nurturing SoSP in a vacuum. Universities do a great deal of the research and must be included in the leadership along with the U.S. Federal Government. Only in shared leadership will there be buy-in on any SoSP decisions.”

• “There should be no leader, but an OPEN marketplace of ideas and tools.”• “Effective SoSP will involve significant commitments by all 4 groups you

identify: long-term Federal funding for research and links to implementation, the development of a coherent academic community (both US and international) and perhaps the rise of a new type of private sector engagement with these issues.”

Data on Creativity

0%

20%

40%

60%

80%

100%

Exist High Quality Documented Access

Prop

ortio

n Ag

reei

ng

Data on Workforce

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Exist High Quality Documented Access

Pro

po

rtio

n A

gre

ein

g

Data on Funding

0%20%40%60%80%

100%

Exist High Quality Documented Access

Prop

ortio

n Ag

reei

ng

Data on Scientific Papers

0%20%40%60%80%

100%

Exist High Quality Documented Access

Prop

ortio

n Ag

reei

ng

Data on High Impact Papers

0%20%40%60%80%

100%

Exist High Quality Documented Access

Prop

ortio

n Ag

reei

ng

Data on Doctoral Students

0%20%40%60%80%

100%

Exist High Quality Documented Access

Prop

ortio

n Ag

reei

ng

DATA: Some Representative Comments• “I cannot answer these questions at this high level of generality.

Different questions/insights require very different datasets.”• “The answer to many of the questions, is, ‘It depends.’ For doctoral

students (to take one example), the Survey of Earned Doctorates has excellent data at a specific point in every graduate student's career, but there is not comparable individual-level data up until that point. NSF, of course, has excellent summary data, but what they collect is highly generic. Hence lots of 2s and 3s in my answers.

• “Some specific comments on this section. Many of the concepts are opaque. What does, ‘Discovery to innovation infrastructure (institutions)’ mean? Is the question, ‘Is there good data on the research being conducted at institutions?’ Why is the STEM workforce both an input and an output?”

Econometric Models

0%

20%

40%

60%

80%

100%

Value Applicability Cost Risk

Prop

ortio

n Ag

reei

ng

Risk Modelling

0%20%40%60%80%

100%

Value Applicability Cost Risk

Prop

ortio

n Ag

reei

ng

Options Modelling

0%20%40%60%80%

100%

Value Applicability Cost Risk

Prop

ortio

n Ag

reei

ng

Agent Based Modelling

0%20%40%60%80%

100%

Value Applicability Cost Risk

Prop

ortio

n Ag

reei

ng

MODELING: Some Representative Comments• “I will be interested in seeing how agent based modeling applies to

this activity. • “Why aren’t the upper models stochastic as well?”• “Value, applicability, cost and risks very much depend on the specific

type and purpose of the models.”• “You need a multi-model approach and to use these models for

decision support (not decision making).”• “I believe my definition of options modeling differs from the intent of

this survey.”• “Quantitative analysis, while a very useful tool, must be used with

caution. The value of each tool must be assessed on a case-by-case basis. There is always danger in using a ‘one-size-fits-all' model.”

Case Studies

0%20%40%60%80%

100%

Value Applicability Cost Risk

Prop

ortio

n Ag

reei

ng

Peer Review

0%20%40%60%80%

100%

Value Applicability Cost Risk

Prop

ortio

n Ag

reei

ng

Delphi Approach

0%20%40%60%80%

100%

Value Applicability Cost Risk

Prop

ortio

n Ag

reei

ng

Data: 30-35 respondents

QUALITATIVE: Some Representative Comments• “A combination of qualitative and quantitative is inevitably best - a

basket of such factors should be used to establish the level or value of any desired factor. Critical Instance analysis is a more forensic kind of case study approach and more insightful in terms of understanding system dynamics and values.”

• “I take umbrage at this section.”• “There is always a very important place for qualitative analysis, but

it must be balanced with a deeper methodological understanding of the drivers and consequences of scientific discovery and innovation, as well as more rigorous quantitative analysis that can be used to undertake counterfactual analysis.”

• “These techniques are very well developed and widely used.”

Models 28-30 respondents

METRICS: Some Representative Comments• “I cannot answer this at such a general level.”• “These are not the correct measures for systems level performance.

One needs measures of gross domestic innovation benefit.”• “This section begins with the implicit assumption that economic value

is the goal of all issues with which SoSP is concerned - an assumption with which I disagree. Moving on, the program/portfolio 'outcome metrics' are meaningless….”

• “At systems level, it may be very difficult to disaggregate outcome metrics from other success factors for such poorly defined or multifactorial terms as ‘quality of life’ and GDP.”

• “Such metrics used in performance management can be dangerous as they over simplify the connections.”

• “Failure is predicted with these measures.”

Models 28-30 respondents

TOOLS: Some Representative Comments• “In general, much too much time, effort and money has been spent on

visual analytics and scientometrics (bibliometrics), tools that might look pretty but offer little insight. Network analysis and science mapping have much more promise, but are much more difficult.

• “SoSP analysis results have to be communicated in an easy to understand form to diverse stakeholders. Visualizations have a major role to play here. Scientometrics is more of an analysis technique then a visualization tool.”

• “With regard to network analysis, Federal agencies MUST be more open to the use of network analysis if it is going to be of any use.”

• “The increasingly interesting areas of "cloud computing" and data mining are pointing to something (trends, frequency, quasi-sentience, crowd psychology), but it seems premature to think that such an approach can be scientific and/or predictive rather than anecdotal or suggestive in nature.”

Pre-Workshop Conclusions• A great deal of variation exists in the communities’ views on the maturity

and quality of data, tools, models and other aspects of the SoSP.• This passionate interchange gives us hope that there is a structured way

that this community of practice could communicate in meaningful ways.• The quality and insightfulness of the discussion from this form of

communication could be of great value when building the community of practice, but we definitely need to improve the experiment and the tools we used.

• The academic community, working with the Federal government, will be able to identify gaps that the academic community could remedy, such as data on the workforce, models, use of high impact scientific papers, etc.

• We are amazed at how many people responded and the thoughtfulness of the comments.