case study: determining the right metrics to monitor and ...case study: determining the right...

Case Study: Determining the Right Metrics to Monitor and Develop a Sponsor-CRO Collaboration

Steve Crow, Associate Director, Performance & Training, Clin Ops, GW Pharmaceuticals

Keith Dorricott, Director, Dorricott Metrics & Process Improvement

© 2020 Dorricott Metrics & Process Improvement, All Rights Reserved

Disclaimer2

The opinions expressed are solely those of the presenters and do

not necessarily reflect the views or opinions of their employers.

Steve Crow

[email protected]

Keith Dorricott

[email protected]


Agenda3

Intro and Background

KPIs for Outsourcing

Standard KRIs

Training & Education

Summary and Next Steps



Standard KRIs




Outsourcing

Clinical trial growth outpacing

internal resources

Initial focus: Clinical, Regulatory,

Data, Stats, Medical Monitoring

Strategic CRO collaboration

4


GW: MCC and an MCC Ambassador

GW had little experience on using standard metrics – no

desire to reinvent

Joined the MCC in 2019

GW now participates across nine work groups

Decided to engage an MCC Ambassador to gain

expertise quickly – and as an independent ‘chair’ for

meetings with CRO

5


GW Metric Needs

Metric definitions / standards:

‒ KPIs for outsourcing

‒ Standard KRIs

‒ Functional-level metrics (e.g. Data Management)

Training & Education on ‘Critical Thinking’:

‒ RBQM – KRIs, Critical KRIs, thresholds & QTLs for risks

‒ Interpretation and use of data

‒ Root cause analysis

6


Agenda7



Standard KRIs




KPIs for Outsourcing8

Define Critical

Success Factor

(CSF) Define Key

Performance

Question (KPQ)

Define Key

Performance

Question (KPQ)

Generate Metric

Display/Report

that Answers KPQ

Generate Metric

Display/Report

that Answers KPQ

Define & Measure

Metric that

Answers KPQ

Define & Measure

Metric that

Answers KPQ

Format display/report to highlight answer(s)

We used the MCC Metric Development Framework – starting with the questions

We determined 34 Key Performance Questions – most pre-defined in MCC toolkits



We asked the CROs for their standard KPIs and lined them up with our 34 KPQs.

We were intrigued by the results:

Full metric definitions not initially provided by CROs

Questions the metrics are designed to answer were not initially provided by

CROs

Some metrics were simple counts e.g. number of open queries

9

CRO #1 CRO #2 CRO #3 CRO #4

# KPIs 23 14 19 13

# KPIs aligned with GW KPQs 11 3 9 8



We had another challenge: the CROs were not members of the MCC

CROs did not have access to full metric definitions

In the spirit of collaboration, GW offered to pay for a one-year membership

10



Next steps:

‒ Meetings have started with each CRO

‒ We started by agreeing the Key Performance Questions – not the KPIs

‒ Next will be the discussion on what metrics best help answer the KPQs – MCC

metrics definitions are available to all parties

‒ Focus will be on study level and smaller number of governance level KPIs

‒ Implement and use – via reports from the CROs

‒ Review / revise the metric set as needed

‒ Plan to move to data feed from CROs to GW so GW can ‘own’ and

interrogate the data

11


Functional-Level Metrics

Separate functional teams have been

set up e.g. Finance, Data Management

They will also develop functional metrics

Guidance and assistance is being

provided to follow the same approach

Has led to a discussion on KPIs vs

Metrics

12

Nic

k Y

ou

ng

son

CC

BY

-SA

3.0

Alp

ha

Sto

ck Im

ag

es


Agenda13



Standard KRIs




Standard Key Risk Indicators (KRIs)

Review of one CRO KRIs showed use of lagging rather than leading metrics

We pulled together a list of MCC KRIs and recommendations from Centralized Monitoring Community of Practice meetings including for COVID-19

Next steps:

‒ Review existing processes and KRIs at CROs

‒ Work together to agree minimum set of KRIs

‒ Ensure KRIs are leading metrics

‒ Review / revise the KRIs as needed

14


Agenda15



Standard KRIs





Need to drive culture change to more data-driven decisions

Decided to tie this to ICH E6 (R2) and ICH E8 (R1): Critical Thinking and Quality

by Design

‒ RBQM – KRIs, Critical KRIs, thresholds & QTLs for risks

‒ Interpretation and use of data

‒ Root cause analysis

► 5 live training sessions carried out to date

► Training recorded and in GW LMS using Camtasia

16

An

dy B

righ

t CC

BY

-SA

2.0


Training Example: Critical Thinking17

Analyzing

Applying

Standards

Logical

Reasoning

Discriminating

Predicting

Transforming Knowledge

Information

Seeking Use your critical thinking skills

Encourage others to use theirs

Be pragmatic – you are not

trying to capture every risk. Just

the most important ones.

ICH E8 (R1) [Draft]:

“Perfection in every aspect of an

activity is rarely achievable or can

only be achieved by use of

resources that are out of proportion

to the benefit obtained.”

Ref: B. K. Scheffer and M.G. Rubenfeld,"Critical Thinking: What Is It and How Do We Teach It?"


Training Example: COVID-19 Testing

On 2-Apr-2020, the UK

Government announced a

target of 100,000 COVID-19

tests a day by May 1st

Did they meet their target?

18

Source: UK Department of Health & Social Care

Maybe!


Training Example: Leading Metrics19

Ref: Metrics Champion Consortium – “Measure the Right Things at the Right Time: Design Key Risk Indicators and Key Performance Indicators that Provide Timely Insights During Study Conduct”

The risk – If there is under-reporting of AEs

then an incorrect safety profile might result

So you decide to use the KRI of AEs per

patient and compare the blue site with all

the others

You want your metric within the thresholds

– but this metric means it will only be within

the thresholds near the end of the study!

top threshold

bottom threshold


Training Example: Leading Metrics20

Ref: Metrics Champion Consortium – “Measure the Right Things at the Right Time: Design Key Risk Indicators and Key Performance Indicators that Provide Timely Insights During Study Conduct”

The risk – If there is under-reporting of AEs

then an incorrect safety profile might result

So you decide to use the KRI of AEs per

patient and compare the blue site with all

the others

You want your metric within the thresholds

– but this metric means it will only be within

the thresholds near the end of the study!

Using AEs per patient-week transforms the

approach. This is a leading metric that

takes account of the time the patients

have been on the trial


Agenda21



Standard KRIs





GW Pharmaceuticals has moved quickly into an outsourcing model

They recognized the need for KPIs with their CROs

And also the need for culture change within the organization on the use of

process data

MCC membership has provided access to standard metrics, and a forum for

discussion across the industry

Engaging an MCC Ambassador has accelerated the process of KPI

development and training aimed at assistant with the desired culture change

This is the start of a journey!

22

Case Study: Determining the Right Metrics to Monitor and Develop a Sponsor-CRO Collaboration

Steve Crow, Associate Director, Performance & Training, Clin Ops, GW Pharmaceuticals ([email protected])

Keith Dorricott, Director, Dorricott Metrics & Process Improvement ([email protected])

case study: determining the right metrics to monitor and ...case study: determining the right...

Documents