Real and Perceived Barriers to Research Assessment

Parallel Session 1.2

ORCID-CASRAI JOINT CONFERENCEBARCELONA, SPAIN

18 MAY 2015

FROM LOGIC MODEL TO DATA MODEL

22015 ORCID – CASRAI Joint Conference Barcelona, Spain

32015 ORCID – CASRAI Joint Conference Barcelona, Spain

Barriers to Research Assessment

1. Program managers are not familiar with evaluation concepts or do not have the capacity to carry out evaluation

2. Evaluation is not top-of-mind at the program design stage

3. Insufficient operating budget is available to carry out evaluation

4. Data to support evaluation are not ready-made

5. Burden on the R&D community to support evaluation

42015 ORCID – CASRAI Joint Conference Barcelona, Spain

NIH/NCI/CSSI/OPSO

Established in 2009

$150M, $10M U54 Research

Grants

Physical Sciences – Oncology Centers Program (PS-OC)

• To unite the fields of physical science with cancer biology and oncology

• To develop trans-disciplinary teams and infrastructure

• To generate new knowledge and catalyze new fields of study

Program Goals

• Twelve centers were funded 2009-2014, U54 research grants

• 150 main investigators from the fields of physics, mathematics, chemistry, engineering, cancer biology and clinical oncology

• 110 Institutions involved across the US

PS-OC Network

2015 ORCID – CASRAI Joint Conference Barcelona, Spain 5

6Confidential

2015 ORCID – CASRAI Joint Conference Barcelona, Spain 7

Evolution of data available for to support PSO program management

Manual data extraction, organization, deduplication and visualization in Excel

Difficult to track individuals’ contributions over time

No ability to collectively search progress reports

Data are entered through an intuitive web-based interface

Identify data relationships

Data deduplication

Flexible, unified search

On-demand Bar, pie, and line charts and network graphs

One-click tabular exportfor data behind graphs or “report card” tables

Data can be visualized by network, center, person or time, for a total of over 100 possible charts and graphs

At-a-glance view of research output• Personnel +2,900• Publications

+2,300• Collaborations +1,900• Conferences

+4,200• Workshops +400

Manual Data Analysis Before

iTRAQR

Automated Data Analysis With iTRAQR

2015 ORCID – CASRAI Joint Conference Barcelona, Spain 8

Deduplication

1. Collaboration2. Course3. Funding4. Meeting5. Patent6. Publication7. Training

Transition8. Workshop9. Exchange10. Project11. Person

The value of structured data: clarity, communication and change

2015 ORCID – CASRAI Joint Conference Barcelona, Spain 9

Lessons learned for overcoming barriers

1. Start with your evaluation logic model and translate it to your data model

2. Think about the level of analysisa. Analyzing subprojects activities at outputs would have been impossible

without iTRAQR

b. Understanding people involved beyond key personnel

3. Data structure is key

4. Have a flexible approach to evaluation (adjust based on findings using initial data)

102015 ORCID – CASRAI Joint Conference Barcelona, Spain

Individual-Level• Publications• Patents• Grants (NIH, other)• Science Awards (innovative,

translational, training)• Clinical Trials• Conference presentations• Courses and workshops taught• Trainee disciplines

Center-Level• Cost, content and people

involved in research projects, pilot projects and cores

• Stage, content and people involved in collaborations

• Datasets, techniques, technologies and bio-specimens generated and utilized

• Enumeration and content of transdisciplinary team science activities

Network-Level• Cost, content and people

involved in trans-network projects and outside network pilot projects

• Stage, content and people involved in collaborations

• Datasets, techniques, technologies and bio-specimens generated and utilized

• People and centers involved in trainee exchanges

• Location and content of outreach activities

Inputs and Activities Outputs Outcomes Relative to Comparison Groups

Generated Robust Collaborations that Resulted in Significant Transdisciplinary Research • Accelerated the formation of a greater quantity of transdisciplinary

collaborations• Accelerated the creation of a greater quantity of field convergent research• Communicated effectively across disciplines to form optimal team sizes• Effectively contributed to team based activities and outreach

Connected Physical Sciences Perspectives with Clinical Research• Accelerated the formation of a greater quantity of collaborations among

physical and physician scientists• Reduced the time between the appearance of a physical sciences perspective

or technology to its application in translational research• Acted as key investigators leading a convergence of physical sciences

perspectives within translational research and motivating transdisciplinary translational research

Bridged Oncology Research Gaps• Accelerated the generation of innovative and impactful transdisciplinary

solutions to outstanding questions in oncology (e.g. integrated transdisciplinary datasets, technologies and bio-specimens, prominently positioned in citation networks and commercialized cancer-relevant patented technology)

Trained a New Generation of Transdisciplinary Scientists• Conducted a greater quantity of transdisciplinary training activities• Attracted a greater volume of training grant applications to the PS-OC

program• Graduated a greater quantity of transdisciplinary scientists

• Accelerated the trainee development path toward a career in physical sciences-oncology

Generated a Sustainable Transdisciplinary Infrastructure• PS-OC alumni sustained a transdisciplinary perspective by integrating team

science into their infrastructure and attracting new investigators to the field • Motivated the formation of other inter-/intra- national programs promoting

physical sciences perspectives in cancer research

PS-OC Program Logic Model: Dec 2013

Network-Level• Coordinate Expertise

Trans-network Projects Physical or virtual

infrastructure Integrative training Data Coordinating Center Research Contracts to further

support clinical translation, cross-validation and integration of datasets, techniques, technologies, bio-specimens

• Communicate with PS-OC and Broader Research Community

Center-Level• Primary leading physical

scientist and cancer researcher

• Research framework: 3-5 projects

• Shared Resources: 1-3 non-redundant core facilities

• Pilot Projects • Transdisciplinary lectures,

workshops, working groups, courses

Individual-Level• Research findings: pre-award

publications, grants, patents, clinical trials and business development

• Research discipline• Organization associations

(location, Title/Rank, department)

• Degrees received• Other demographics

Evaluation is only as good as the data available

• Program design and management is enhanced by early evaluation design and evaluation efforts

• Data are not infallible and should be part of a holistic evaluation approach as well as close engagement with program participants

• Data do not exist today to measure all of your program goals

• Careful consideration for what actions can be taken following the evaluation should help to prioritize data collection

2015 ORCID – CASRAI Joint Conference Barcelona, Spain 12

Acknowledgments

Nicole Moore, ScD Program DirectorNCI Division of Cancer BiologyPhysical Sciences-Oncology

Unni Jensen, PhD Sr Scientific AnalystThomson Reuters

Jodi Basner, PhD Scientific AnalystThomson Reuters

132015 ORCID – CASRAI Joint Conference Barcelona, Spain

THANK YOU

Systems to link research to outputs and outcomes

152015 ORCID – CASRAI Joint Conference Barcelona, Spain