Assessing the Value of Transdisciplinary Research: The Science of Team Science

Daniel Stokols, Ph.D.Chancellor’s Professor of Social Ecology

University of California, Irvine

Fall 2008 Lecture Series

Robert Wood Johnson Foundation Center for Health Policy at the University of New Mexico, Albuquerque

October 23, 2008

In collaboration with:

Kara Hall, Ph.D.Health Scientist, DCCPS-NCI

Richard Moser, Ph.D.Research Psychologist, DCCPS-NCIResearch Psychologist, DCCPS-NCI

Annie Feng, Ed.D.Cancer Research Training Award Fellow, DCCPS-NCI

Brandie Taylor, M.A.Office of Portfolio Analysis and Strategic Initiatives, NIH

The Emergence of Big Science

(See Wuchty, S., B. F. Jones, et al. (2007, Science). "The increasing dominance of teams in production of knowledge.”)

Substantial Investments in Team Science Initiatives Have Been Made Over the Past Three Decades

• MacArthur Foundation Networks in Mental Health and Human Development (1980-present)

Private Foundations:

• Robert Wood Johnson Foundation Tobacco Policy and Active Living Research Programs (2000-present)

• NAS/Keck Foundation Initiative to Transform Interdisciplinary Research (2003-present)

Interest From Public and Private Agencies

• NIH Roadmap Initiative (2003-present)

• MacArthur Foundation Networks in Mental Health and Human Development (1980-present)

Substantial Investments in Team Science Initiatives Have Been Made Over the Past Three Decades

• Robert Wood Johnson Foundation- Active Living Research Program (2002-present)

• NAS/Keck Foundation Initiative to Transform Interdisciplinary Research (2003-present)

Earlier Conferences on Facilitating Cross-Disciplinary Research and Training

• The National Research Council Conference on Interdisciplinary Research (1990)

• The National Academy of Sciences Conference on Bridging Disciplines in the Brain, Behavioral, and Clinical Sciences (2000)Clinical Sciences (2000)

• The National Institutes of Health Bioengineering Consortium Symposium on Catalyzing Team Science (2003)

• The National Academy of Sciences Convocation on Facilitating Interdisciplinary Research (2004)

Burgeoning Interest in Studying and Facilitating Cross-Disciplinary Collaboration

• Working definitions of key terms: team science initiatives (TS), the science of team science (STS); multi-, inter-, and transdisciplinary (TD) research and training

• The role of logic models in guiding evaluations of team science; importance of tailoring program theories and evaluation strategies to the size, complexity, and stated

Overview

evaluation strategies to the size, complexity, and stated goals of TD initiatives

• Examples of quantitative and qualitative methods that have been used to assess TD processes and outcomes in earlier and ongoing evaluations of TD initiatives

• Translating and disseminating team science knowledge into practical applications – lessons learned and priorities for future studies of team science

1. Defining Key Terms and Units of AnalysisUnits of Analysis

• STS is a rapidly emerging field concerned with understanding and managing circumstances that

The Science of Team Science

managing circumstances that facilitate or hinder the effectiveness of large-scale research, training, and translational initiatives

Conference Aims:

• Generate scholarly papers, posters, panels, and informal discussions that address cutting edge issues in the science of team science

• Review state of the art knowledgeabout the antecedents, processes, and outcomes of transdisciplinary

2006 NCI Conference on the Science of Team Science:Assessing the Value of Transdisciplinary Research

and outcomes of transdisciplinary team science and training programs

• Identify methods and metricspresently available for evaluating transdisciplinary collaboration in large scientific initiatives

• Identify priorities for future researchand opportunities for implementing research findings to enhance TD collaboration in scientific, training, clinical, and health policy contexts

http://videocast.nih.gov/launch.asp?13474http://videocast.nih.gov/launch.asp?13471

l

http://cancercontrol.cancer.gov/brp/scienceteam/presentations_day1.htmlhttp://cancercontrol.cancer.gov/brp/scienceteam/presentations_day2.html

Presentations:

Videocast:

AJPM Supplement on the Science of Team Science, August 2008

Continuum Ranging from Disciplinary to Cross-Disciplinary Forms of Collaboration

• Unidisciplinary - researchers from a single discipline work together to address a common problem

• Multidisciplinary - researchers from different disciplines work independently or sequentially, each from his or her own disciplinary-specific perspective, to address a common problem

• Interdisciplinary - researchers from different disciplines work jointly to address a common problem and although some integration of their diverse perspectives occurs, participants remain anchored in their own fields

• Transdisciplinary - researchers from different disciplines work jointly to create a shared conceptual framework that integrates and moves beyond discipline-specific theories, concepts, and approaches, to address a common problem

(Rosenfield, 1992)

Definition of Interdisciplinary Research in the NIH Roadmap Initiative

“Interdisciplinary research integrates the analytical strengths of two or more often disparate scientific disciplines to create a new hybrid discipline.”

Examples of interdisciplinary hybrid fields:• Pharmacogenetics• Pharmacogenetics

• Bioinformatics

• Proteomics

• Populomics

• Psychoneuroimmunology

• Translational Genomics

(http://www.nihroadmap.nih.gov/interdisciplinary)

Integration of Disciplines

Horizontal Integration of DisciplinesCrosses disciplines within one level or category of

analysis and discourse (e.g., the molecular or biogenetic level)

Vertical Integration of DisciplinesLinks disciplines across analytic levels:

1. molecular/genetic/biological 2. psychological/developmental 3. social/organizational/institutional4. societal/community policy levels

Environmental Racism

Social Fact

An Example of Vertical Integration of Disciplines in Cancer Epidemiology Research

.

High Levels of Air and Water Pollution in One’s Neighborhood

Elevated Cancer RatesAmong Residents

Physical Environmental Fact Biomedical Fact

Key Question:

Do large cross-disciplinary initiatives lead to scientific breakthroughs and lead to scientific breakthroughs and public health improvements that would not have occurred without those initiatives?

Types of Transdisciplinary Collaborations

Scientific Teams - researchers from different disciplines who are working to develop collaborative intellectual and research products (e.g., novel theories, methods for studying particular problems)

Community Coalitions - researchers from various fields who are working with community members to translate scientific are working with community members to translate scientific evidence into new interventions aimed at reducing societal problems

Inter-Sectoral Partnerships - representatives of community organizations (at local, state, national, and international levels) who coordinate their efforts to implement new policies and programs for improving environmental, social, and public health outcomes

• Team science initiatives are not uniformly generative; their effectiveness is context-dependent and is jointly influenced by several key environmental, organizational, technological, intra- and interpersonal factors

Gauging the Effectiveness of Team Science

Some Caveats…

technological, intra- and interpersonal factors

• The generative processes and outcomes achieved through team science are stage-dependent and vary substantially over the course of a collaborative initiative (e.g., from its initial 5-year funding phase through its subsequent stages)

Intrapersonal

�Members' attitudes toward collaboration and their willingness to devote substantial time and effort to TD activities

�Members' preparation for the complexities and tensions inherent in TD collaboration

�Participatory, inclusive, and empowering leadership styles

Organizational

�Presence of strong organizational incentives to support collaborative teamwork�Non-hierarchical organizational structures to facilitate team autonomy and participatory goal setting�Breadth of disciplinary perspectives represented within the collaborative team or organization�Organizational climate of sharing �Frequent opportunities for face-to-face communication and informal information exchange

Interpersonal

�Members' familiarity, informality, and social cohesiveness�Diversity of members' perspectives and abilities�Ability of members to adapt flexibly to changing task requirements and environmental demands�Regular and effective communication among members to develop common ground and consensus about shared goals�Establishment of an hospitable conversational space through mutual respect among team members

Collaborative

Typology of Contextual Factors Influencing TD Scientific Collaboration at Each Level of Analysis

Physical Environmental

�Spatial proximity of team members' workspaces to encourage frequent contact and informal communication�Access to comfortable meeting areas for group discussion and brainstorming�Availability of distraction-free work spaces for individualized tasks requiring concentration or confidentiality�Environmental resources to facilitate members' regulation of visual and auditory privacy

Societal/Political

�Cooperative international policies that facilitate exchanges of scientific information and TD collaboration�Environmental and public health crises that prompt inter-sectoral and international TD collaboration in scientific research and training�Enactment of policies and protocols to support successful TD collaborations (e.g., those ensuring ethical scientific conduct, management of intellectual property ownership and licensing)

Technological

�Technological infrastructure readiness

�Members' technological readiness

�Provisions for high level data security, privacy, rapid access and retrieval

Collaborative Effectiveness of Transdisciplinary Science Initiatives

(Stokols, Misra, Hall, Taylor, & Moser, 2006)

Org

aniz

atio

nal

Sco

pe

Intra-Organizational

Inter-Organizational

Inter-Sectoral

Organizational, Geographic, and Analytic Scope of Team Science Initiatives

Geographical Scale

Local Group Regional

Org

aniz

atio

nal

Sco

pe

Organizational

CommunityBiological

Psychological

Social/Environmental

Community/Policy

National/Global

(Stokols, 2006)

2. The Role of Logic Models in Guiding Evaluations of Team ScienceEvaluations of Team Science

• Traditional evaluative criteria of scientific qualityemphasize conceptual validity, originality, methodological rigor, and the quantity of research outputs such as peer-reviewed publications

• Criteria for evaluating TD team science add the following considerations:

Defining and Measuring Program Effectiveness

following considerations:

(1) the quality and scope of cross-disciplinary integration reflected in new conceptual models and methodological strategies

(2) the impact of integrative intellectual products in forging new avenues of scientific research, training, clinical applications, health policy, and improved health outcomes

NCI Transdisciplinary Research Initiatives

Collaboration

Professional Validation

Communication

Training Policy Implications

Communication

Recognition

TD ResearchInstitutionalization

Publications

Logic Model for TTURC-I

Evaluation(Trochim et al., 2005)

Scientific Integration

Health ImpactsCollaboration

TransdisciplinaryIntegration

ImprovedInterventions

Health Outcomes

Methods

Science &Models

TranslationTo Practice

Implications

Intermediate MarkersImmediate Markers Long-Term Outcomes

• Behavioral

• Affective

• Intrapersonal

• Social

Antecedents Processes Outcomes

• Novel ideas

• Integrative models

Conceptual Model of Transdisciplinary Scientific Collaboration

• Affective

• Interpersonal

• Intellectual

• Physical environmental

• Organizational

• Institutional

• New training programs

• Institutional changes

• Innovative policies

(Fuqua et al., 2002; Stokols et al., 2003)

Collaboration Readiness Factors

• Team members’ history of collaboration on prior projects

• Institutional support for cross-disciplinary collaboration

• Leaders with collaborative orientation and experience•

• Members share a strong commitment CD collaboration

• Spatial proximity of investigators’ offices and laboratories

• Similarity of researchers’ scientific worldviews

• Overlapping departmental identities of team members

Near Term

Markers / Outcomes

Intermediate

Markers / Outcomes

Long Term

Markers / Outcomes

AN

TE

CE

DE

NT

S

HE

ALT

H

IMP

AC

TS

Conceptual Model for Evaluating Collaborative Initiatives

YEAR 1 YEAR 3YEAR 2 YEAR 4 YEAR 5 BEYONDPRIOR TOINITIATIVE

AN

TE

CE

DE

NT

S

HE

ALT

H

IMP

AC

TS

Centers for Population Health and Health

Disparities Conceptual Framework

Investigator Development

Community

Translation of KnowledgeDissemination

Policy

CPHHD Initiative Flowchart(Revised 12-14-04)

Community Stakeholder – Investigator Incubator

Methods

PracticeStakeholder

Participation/ Integration

Collaboration

Transdisciplinary Activity/Capacity

Building

Increased Awareness

Publication

Interventions

Training

Scientific Innovation

Health Outcomes

(INPUTS)

Immediate Markers

What we invest

(ACTIVITIES) (OUTPUTS)

Intermediate Markers

What we do, who we reach

(OUTCOMES)

Short-term Outcomes

Learning & Action

(OUTCOMES)

Long-term Outcomes

Ultimate Transformations

CommunityEmpowerment

Models Findings

Transdisciplinary Integration

Transdisciplinary Processes

3. Multi-Method Tools for Evaluating Processes and Outcomes of Processes and Outcomes of

TD Collaboration

• Strategic Evaluations of TS Initiatives - those that apply evaluation resources efficiently to yield information about major contributions and limitations of particular programs, in a manner that is responsive to the needs of multiple stake-holder

Methodological and Measurement Issues

is responsive to the needs of multiple stake-holder groups:

� scientists and trainees

� funding organizations

� policy makers

� translational partners

• Guidelines for Strategic Evaluations of TS Initiatives

� Specify program theory underlying the evaluation

� Use weighted measures of program success

Methodological and Measurement Issues (cont.)

� Incorporate multiple methods of evaluation

� Temporally sequence evaluative measures

� Work toward convergent validation of evaluation data

� Account for research design and sampling limitations

Peer Review of Progress Reports

Please rate progress in each of the following outcome areas:

No progress

Limited progress

Moderate progress

Significant progress

Not applicable

Transdisciplinary Integration � � � � �

Collaboration � � � � �

Training � � � � �

Methods � � � � �Methods � � � � �

Science and Models � � � � �

Interventions � � � � �

Publications � � � � �

Internal Recognition & Support � � � � �

External Recognition & Support � � � � �

Communication � � � � �

Policy Implications � � � � �

Translation to Practice � � � � �

Health Outcomes � � � � �

Financial Analysis: Cumulative Percent of Federal Funds Spent by Grantee

(Data from Financial Status Reports of TTURC grantees)

80%

99%

78%77%

91%

74%

93%

82%

94%

87%

80%

90%

100%

Center 1

Center 2

Center 3

Center 4

62%

72%

46%

74%

68%

57%

66%

40%

50%

60%

70%

Year 1 Year 2

Center 4

Center 5

Center 6

Center 7

Total

Criteria for Evaluating Successful Transdisciplinary Collaboration

Multiple “Windows” on processes of TD Collaboration:

• Self-Reported Changes in TD Activities and Values

• Semantic Differential Measures of Affective Experiences

• Social Networks Analyses of Teamwork

• Archive of “Intellectual Capital” and Related Products

The TREC Baseline Survey March-June 2006

•New survey measures derived from theoretical and empirical analyses of “collaboration readiness” measures

•Development of an Online System for Survey Administration

•Coordination of IRB Approvals at Multiple Sites

Baseline Survey Dimensions

• History of Collaboration

• Researcher Orientation

• Semantic Differential

• Fields of Training • Fields of Training

• Training

• Institutional Resources

• Collaborative Attitudes

• Collaborative Processes

• Collaborative Activities

Selected Survey Associations: Institutional Resources

� The better the researcher felt their center’s institutional resources were:

� The more:

� Positive their impressions of the center and as a TREC member (r = .311)

� Satisfied they were with previous collaborators (r = .340)

� Confident they were that their TREC center would achieve TD research � Confident they were that their TREC center would achieve TD research

(r = .397) and training (r = .298) goals

� The more they felt:

� The members of their center are a socially cohesive group (r = .275)

� Their CD is effective at promoting collaboration and trust (r = .306)

� The better they felt:

� The collaborative productivity (r = .495) and interpersonal collaboration (r = .497) was within their center or with respect to center-related research

Research Orientation Scale (ROS)

• Component of collaborative readiness

• Designed to assess “continuum” of disciplinary • Designed to assess “continuum” of disciplinary integration as defined by Rosenfield (1992)

• Provides an empirical index of investigators’ and trainees’ research orientations

Sample Research Orientation Items from the TREC Year-1 Evaluation Survey

TYPE ITEMS

UNI

There is so much work to be done within my field that I feel it is important to focus my research efforts with others in my own discipline.

While working on a research project within my discipline, I

MULTI

sometimes feel it is important to seek the perspective of other disciplines when trying to answer particular parts of my research question.

INTER/TRANS

In my own work, I typically incorporate perspectives from disciplinary orientations that are different from my own.

TRANSIn my collaborations with others I integrate theories and models from different disciplines.

Items rated on a 5-Point Likert Scale: Strongly Disagree to Strongly Agree

Path Diagram for the Research Orientation Scale Including Factor Loadings and Factor Correlations

(Hall et al., 2007)

TREC Collaborative Activities Scale

Participation in Working Groups

Increased

TREC Written Products Protocol

Protocol Criteria

• Listing of centers involved in proposal, researchers and their disciplines, departments and institutions

• Coding of disciplines, levels of analysis, and methods of analysis

• Ratings of type of cross-disciplinary integration and scope of TD integration

Review and Consensus Process

• Two independent reviewers of each proposal

• Moderator and expert panel member participated along with the two reviewers in the review and consensus process

Sample Size (Proposals)

• 2006: N= 21

• 2007: N= 27

Cross-Disciplinary Representation in TREC Developmental Proposals During Year-1

•• More than 35 different disciplines represented across the proposals

9

Number of Disciplines Represented

within Proposal

0123456789

# of Proposals

1 2 3 4 5 6

# of Disciplines

Multiple Levels of Analysis Reflected in TREC Developmental Proposals at Year-1

• 4 of 7 levels of analysis were represented across the proposals:

� Molecular & Cellular 10

Number of Levels of Analysis within

Proposals

� Molecular & Cellular� Individual� Group & Interpersonal� Organizational &

Institutional� Community & Regional� Societal & National� Global

0

2

4

6

8

# of Proposals

1 2 3 4

# of Levels of Analysis

TREC Developmental/Pilot Projects

� No pilot proposals included other center researchers or resources during 2006

� Multi-rater assessments of pilot proposals during 2007 revealed a substantial increase in cross-center collaborations

Bibliometric Analysis

• Quantitative assessment of scientific publications, the works they cite, and the citations of them

• Source of objective information about the quality and productivity of scientific workproductivity of scientific work

• Can be used to estimate the influence and impact of a single publication, or the quality and recognition of the entire published opus of a researcher, a research journal, or even a field of research

• Possible to track changes in quality & productivity of science over time; these comparative analyses could help to identify the role of the TTURC initiative

Examples of Bibliometric Measures

• Citation Impact: The number of citations received per paper published (total citations received/total papers published); usually in specific citing and cited periods.

• Number of Citing Journals: Refers to the number of unique journals that cite a publication.

• Journal Impact Factor: The average number of times recent articles in a specific journal were cited.

Social Networks Map of TTURC-II Collaborative Linkages Within and Between Centers

(Gray & Ren, 2007)

Cross-Center Network Ties Prior to Funding of the CPHHD Initiative

KeyRed=OSUGray=TuftsGreen=UTMBBlue=UPenn Pink=UC Turquoise=Wayne StateYellow=Rand Orange=UICDots in upper left represent researchers with no ties to others in the network

(Gray & Ren, 2007)

Cross-Center Network Ties Subsequent to the Funding of the CPHHD Initiative

(Gray & Ren, 2007)

Charting The Intellectual History of a Research Organization

Evolution of Intellectual Themes

• Which ideas were present at the outset of the project?

• Which ideas were dropped in subsequent years?

• Which new ideas emerged later in the project?

• Which initial ideas were modified over time?

• Which ideas were integrated with previously separate ideas?

0.7

2.1

3.5

1 3

4 6

2

8

Correspondence Analysis of the Degree to Which UCI TTURC-I Investigators Work Closely With Each Other to Integrate Ideas

-3.5 -2.1 -0.7 0.7 2.1 3.5-3.5

-2.1

-0.7

32

910

7

5

(Stokols et al, 2005)

Semantic Differential Scales

SDS items assessed TTURC members’ affective impressions about the Center. Sample anchor words included:

a. Satisfying/ Frustrating b. Optimistic/ Pessimisticc. Socially Integrated/ Alienatedd. Enjoyable/ Unenjoyable e. Appreciated/ Unappreciated

Neuroscientists and Behavioral Scientists’ Semantic Differential Ratings of the UCI TTURC Over Successive Years

(Stokols et al., 2005)

Worlds of Difference Among Behavioral and Neuroscientists

� Alternative “World Views” of Science

� Dissimilar Sociospatial Environments and Cultural Norms

Scientific Retreats as a Setting for Cross-Disciplinary Integration of Ideas

What did you enjoy the most?

•“Developing new research questions based on preliminary findings as well as the creative aspect of brainstorming with bright people.”

•“The retreats were an enjoyable forum for seeing what’s going on in other projects, reconnecting and checking in.”going on in other projects, reconnecting and checking in.”

•“The promise. The TTURC had such great potential to do great things in new ways.”

•“Meeting other investigators, hearing new ideas was the best part. Sharing an occasional meal with colleagues was also enjoyable.”

What were the biggest challenges to collaborating? What frustrated you?

•“The biggest challenge was time spent at meetings.”

•“There were several choices made about allocation of funds that were frustrating.”

•“Some people were afraid of losing control over intellectual property, making collaboration difficult. You have to trust collaborators and share intellectual property such as preliminary data.”

•“The hardest part was not having a primer or overview of other people’s research before going to a presentation. I did not follow all of the material.”

4. Practical Applications of Team Science Knowledge and Priorities Science Knowledge and Priorities

for Future Research

Presumed Benefits of Transdisciplinary Scientific Collaboration

• Greater Explanatory Power

• Methodological Pluralism• Methodological Pluralism

• Broad-Gauged Public Policies

• Advantages of Generalist Training Programs

Qualities of Cross-Disciplinary Collaborations

• Labor intensive

• Conflict prone

• Require preparation, practice, and trust

• Impact of contextual factors on collaboration• Impact of contextual factors on collaborationreadiness(e.g., institutional constraints, proximity)

• Outcomes are often uncertain, as are the metrics and timeframes for measuring them

• Not all researchers are well-suited to CD collaboration - one size does not fit all - match researchers and research arrangements more effectively in order to do “smarter science”

What makes a “Successful” TD Researcher?

•Abstract thinking

•Selflessness

•Assertiveness

•Cognitive flexibility

•Tolerance for ambiguity

•Plays well with others

•Intellectual curiosity

•Good communication skills

•Openness to new ideas

•Feels supported by institution

•Socially focused

•Committed to team

•Social intelligence

•Respect for other views

•Inductive thinking

Challenges/Lessons Learned

• Establish partnership with and provide feedback to grantees (from the start)

• Balancing the use of objective and subjective data

• Importance of cross-initiative comparisons

• maintaining support and resources• Importance of maintaining support and resources for sustained evaluation activities

• Minimize burden on grantees

• Need for repeated measures (including baseline data) to analyze trends and do predictive modeling

• Results should be well disseminated & utilized

• Utilizing social science methods with basic scientists

• Translating results into practical applications

• Translating research findings from TS evaluation studies to enhance future collaborative initiatives

• Strategies for building greater capacity for scientific collaboration in TS initiatives

Translational Strategies

collaboration in TS initiatives

� Collaboration readiness audits

� Workshops and training modules

� Formative evaluation for continuous quality improvement

Additional Strategies for Improving Future Collaborations

• Continue to Expand Cumulative Database on TD Collaborations

• Develop and Evaluate Educational • Develop and Evaluate Educational Strategies for Training Future TD Scientists and Practitioners

• Tailor Collaborative Goals and Arrangements to the Circumstances Surrounding Particular Projects

Resources

• Science of Team Science Information: http://cancercontrol.cancer.gov/brp/scienceteam/index.html

• Evaluation of Large Initiatives (ELI): • Evaluation of Large Initiatives (ELI): http://cancercontrol.cancer.gov/brp/eli.html

• Presentations from NCI Science of Team Science Conference October, 2006: http://cancercontrol.cancer.gov/brp/presentations_day1.html

• Special Issue of Nicotine Tobacco and Control: Volume 5, December, 2003