ctsa program common metrics initiative

Module One:

Maximizing CTSA Program Impact

Philip L. Lee

Clear Impact

www.clearimpact.com

CTSA Program

Common Metrics Initiative

Based on the principles of Results-Based Accountability from Trying Hard is Not Good Enough: How to

Produce Measurable Improvements for Customers and Communities by Mark Friedman (Trafford 2005)

1. The Common Metrics

2. Expectations

3. Turn-the-Curve Thinking

a. Analogy

b. Application

4. Scorecard

Module One

1. Median IRB Review Duration (Time from IRB

submission to IRB approval)

2. Pilot Funding Publications and Subsequent Funding

3. Careers in clinical and translational research (T1-T4).

The Common Metrics

Expectations

1. Define the desired impact of the CTSA Program’s support and

innovations:

A skilled CT research workforce

Improved translation: more & better treatments & cures get to

more patients faster & at less cost

Q.: How will the Common Metrics will be used?

2. Gauge the extent to which that impact is occurring:

% of K & T scholars who become CT researchers

Improved translation speed, cost, quantity & quality

Rates of innovation – develop, demonstrate, disseminate, and

uptake – in CT research workforce development and translation

The Common Metrics will not be used to:

1. Answer questions of contribution/attribution:

Why scholars did (or did not) become successful

researchers

Why translation did (or did not) improve

2. Define, assess or compare performance:

Performance can only be defined, assessed or compared

when all of the factors that influence a measure are known

and considered.

The Common Metrics will be inherently insufficient to define,

assess, or compare performance.

Expectations

Expectations

1. Align decision making across the CTSA Program:

Hub management

Network management

Q.: How will Results-Based Accountability (RBA) and

the Scorecard be used with the Common Metrics?

2. Focus decision making on the Common Metrics:

the rate of scholars going into CT research

the speed, cost, quantity and quality of translation

the rate of innovation in CT workforce development and

translation

3. Standardize the systematic and rigorous use of data and analysis

to determine what will work to “turn the curve” for the Common

Metrics

4. Make decision making collaborative and transparent

5. Share with and learn from other stakeholders at the hubs and

across the Network

Expectations

6. Keep decision making dynamic:

improve strategies on an ongoing basis, incorporate

learning/insights as they are garnered

anticipate and adapt nimbly to changing circumstances – new

challenges and opportunities

7. Trade accountability for flexibility:

Accountability focused on impact, rigorous use of data and

analysis, collaborative and transparent

Flexibility customize strategies to the unique

circumstances of a hub and its stakeholders

8. Develop, implement, and continue to manage a plan for at least

one Common Metric now … eventually for all.

Expectations

We are implementing the Common Metrics

collaboratively. Together we will learn how to

ensure the usefulness of the Common Metrics for

maximizing the impact of the CTSA Program.

Expectations

PM% of studies that achieved accrual goal

within time specified in study design

Apply to your Common Metric

1. Median IRB Review Duration

(Time from IRB submission to IRB approval)

1. Pilot Funding Publications and Subsequent Funding

2. Careers in clinical and translational research (T1-T4).

How are we doing?

Why?What are we going to do?

Who can help?Options?

Change the trend?

“Turn the Curve”?

A Disciplined Approch to Decision MakingTurn-the-Curve Thinking

Start with the End

Where are we headed?STRATEGY

Work backwards to the Means

Baseline History and Forecast

History Forecast

Turning the Curve

OK?

Return

on

Investment*

* The “ROI” is not financial, it is the change in the trend line.

13

How Are We Doing?

PM

50 %

Current Value:Q2 2016

% of studies that achieved accrual goal within time

specified in study designPM

Force Field Analysis

Factors Contributing?

Factors Restricting?

The Story Behind the Curve

Root Causes (ask “Why?” five times)

Positive and negative

current and anticipated

internal and external

Prioritize – Which are the most important to

address to turn the curve?

Do we need additional data/analysis?

The Story Behind the Curve

PM

What is the story

behind the curve?

• Non-investigator clinicians not screening patients for potential study

participants during clinic consultations. (See: CTSI report)

• Lack of systems for data-driven cohort discovery in planning clinical studies

• Inadequate knowledge/skills in recruitment planning/implementation

• Study designs that impede participation

• Sites inconveniently located.



50 %Current Value:

Q2 2016

Who are partners who might have a role to

play in turning the curve?

Does the story behind the curve suggest any

new partners?

Partners

Do we know what would work to turn the curve?

Low-cost/no-cost ideas?

Off-the-wall ideas?

Research/evidence-based?

Do we need additional information?

What Would Work?

PM

Partners?

What would work?

• University/Medical Center Leadership

• University Communications School

• IT staff, Other Hubs

• Education and training department

• Develop strategies to motivate non-investigator clinicians to screen patients.

(Research: German Heart Center Munich)

• Social media (See: eRecruitmentWhitePaper.pdf)

• IT Solutions:

• Enterprise cohort discovery database.

• Cohort discovery tools.

• Provide education to inexperienced investigators in how to develop and

implement recruitment strategies.



50 %Current Value:

Q2 2016

What are our actions (in order of priority)?

Criteria

• Leverage

• Feasibility

• Specificity

• Consistent with organizational values

Strategy

PM

Strategy to

turn the curve?



Research, develop, and institute targeted strategies to

motivate non-investigator clinicians to screen patients.

Engage institutional leadership.

Develop and implement social media strategies.

Partner with the communications school.

50 %Current Value:

Q2 2016 STRATEGY

PM

55 %Current Value:

Q2 2017

% of studies that achieved accrual goal within

time specified in study designPM

PM

What is the story

behind the curve?

Targeted clinician motivation programSee EvaluationofTargetedClinicianMotivationProgram.docx

Social media campaignsSee Evaluationof2016RecruitmentSocialMediaCampaigns.docx



55 %Current Value:

Q2 2017

PM

55 %Current Value:

Q2 2017



Expand clinician motivation program to all clinicians.

Increase number of trials using social media campaigns.

Strategy to

turn the curve?

STRATEGY

1. New data

2. New story behind the curve

3. New partners

4. New information on what works

5. Changes to strategy

6. Adjourn

Network/Hub/Division/Core/Team

Agenda

1. How are

we doing?

2. What is the story behind the curve?

2. What is our strategy to turn the curve?

(c) Results Leadership Group, LLC 28

Turn-the-Curve Plan

the use of measures/data

and transparency in the

management of the CTSA Program.

but

encourages

does not

discourage

Together build a culture that

% Job Training Trainees Placed in Jobs

100%

0%

Yr1 Yr2 Yr3 Yr4 Yr5 Yr6 Yr7

What’s the story

behind the curve?

Accrual Rates

Story behind the curve?

What works?

Partners?

(c) Results Leadership Group, LLC 31

Getting to Our Best Thinking

"All the significant

breakthroughs

were

'break-withs’

old ways of thinking.”

Thomas Kuhn

Goals,Values& Strategies

From Ed Batista: Double-Loop Learning www.edbatista.com/2006/12/doubleloop_lear.htmlMore on Double-Loop Learning: www.edbatista.com/2006/10/chris_argyris_d.html

Results

Most Learning (Single-Loop)Improvement within an existing system thatrests on underlying assumptions thatare implicit and unchallenged.

UnderlyingAssumptions

http://www.edbatista.com/2006/12/doubleloop_lear.html

http://www.edbatista.com/2006/10/chris_argyris_d.html

Results

Most Learning (Single-Loop)Improvement within an existing system thatrests on unchallenged assumptions thatare implicit and unchallenged.

UnderlyingAssumptions

Goals,Values& Strategies

More on Double-Loop Learning: www.edbatista.com/2006/10/chris_argyris_d.html

http://www.edbatista.com/2006/10/chris_argyris_d.html

Implicit assumptions

How to

(1) Identify

and

(2) “break with”?

The discipline of working with

mental models

starts with turning the mirror inward,

learning to unearth

our internal pictures of the world,

to bring them to the surface,

and hold them to rigorous scrutiny.

Peter Senge

The Fifth Discipline

To tell, sell, persuade,

decide

To inquire to learn

Discussion vs. Dialogue

To evaluate and select

the best

To integrate multiple

perspectives

To uncover and examine

assumptions

To justify/defend

assumptions

“I wonder which of these is

the right one?”“I wonder how these pieces

combine to create a whole?”

The purpose of a dialogue is to go beyond any one individual's understanding.

In dialogue, individuals gain insight that simply could not be achieved individually.

Suspend Judgment

Listen

Inquire*

Explore Assumptions

* Start with questions

Dialogue

The governing Values of Model I are:

• Achieve the purpose as the actor defines it

• Win, do not lose

• Suppress negative feelings

• Emphasize rationality

Primary Strategies are:

• Control environment and task unilaterally

• Protect self and others unilaterally

Model I

Usually operationalized by:

• Un-illustrated attributions and evaluations e.g.. “You seem unmotivated”

• Advocating courses of action which discourage inquiry e.g.. “Lets not talk about the past, that’s over.”

• Treating ones’ own views as obviously correct

• Making covert attributions and evaluations

• Face-saving moves such as leaving potentially embarrassing facts unstated

Consequences include:

• Defensive relationships

• Low freedom of choice

• Reduced production of valid information

• Little public testing of ideas

Model I

The governing values

of Model II include:

• Valid information

• Free and informed

choice

• Internal commitment

Strategies include:

• Sharing control

• Participation in design

and implementation of

action

Operationalized by:

• Attribution and evaluation illustrated with relatively directly observable data

• Surfacing conflicting view

• Encouraging public testing of evaluations

Consequences should include:

• Minimally defensive relationships

• High freedom of choice

• Increased likelihood of double-loop learning

Model II

Scorecard Software and the CTSA Program

Common

Metric

With Turn-the-Curve Page

Common Metrics and Performance Measures

Common Metric

Performance Measures

Performance Measures

With Turn-the-Curve Page

50

1. Watch lecture recordings and scorecard videos

2. Develop and enter your Common Metric turn-the-curve plan into

Results Scorecard.

3. Develop your own analogy to explain the turn-the-curve steps (like

Mark Friedman’s leaking roof analogy). Explain the turn-the-curve

process to a colleague using your analogy.

4. Sorting Exercise

Interim Learning Activities

Mark Friedman, Trying Hard Is Not Good Enough (Trafford,

2005)

Peter Senge, The Fifth Discipline: The Art and Practice of

the Learning Organization (Doubleday 1990); Fifth

Discipline Fieldbook (Doubleday)

Mark K. Smith, Chris Argyris: theories of action, double-

loop learning and organizational learning

(2001)http://infed.org/mobi/chris-argyris-theories-of-action-

double-loop-learning-and-organizational-learning/

Portions of these materials draw upon the work of:

Acknowledgement

ctsa program common metrics initiative

Documents