draft title gates mri experience utilizing a site audit ... · of your study sites relative to your...

Gates MRI Experience Utilizing a Site Audit Selection Tool Derived from the MCC Site Selection/Conducting Scoring Tool

Maryann Livolsi, GCP Compliance Leader

Bill & Melinda Gates Medical Research Institute

Linda B Sullivan, Executive Director

WCG-MCC

Meet the Presenters

Maryann Livolsi – Bill & Melinda Gates Medical Research Institute, GCP Compliance Leader responsible for fostering and promoting GCP compliant environment, supporting a culture of quality, regulatory compliance and continuous inspection readiness. Over 29 years of experience in clinical research operations and quality assurance at small biopharmaceutical and large pharmaceutical companies.

Linda Sullivan is Co-Founder & Executive Director of WGC’s Metrics Champion Consortium (MCC), an industry association dedicated to leading the drug-development enterprise in adoption and utilization of standardized metrics and benchmarks to drive performance improvement. She has more than 30 years of experience helping healthcare and clinical research organizations improve processes that strengthen financial and quality outcomes.

Agenda

• Background Gates Medical Research Institute Mission

• Business Need Identified for Investigator Site Audit Selection

• Project Approach

• Risk-Based Auditing Tool Development

• Risk – Based Auditing Tool Implementation Results

• Future Considerations

• Summary

Introducing Gates MRI

September 2020

© 2019, Bill & Melinda Gates Medical Research Institute. All rights reserved.

Maryann Livolsi, Sr. Director GCP Compliance

s

Location Cambridge, MA (HQ), Seattle

StructureFully funded by a grant from

the Gates Foundation

Size ~67 as of January, 2020

Portfolio

Quality

Management

System

Quality and compliance systems implemented; first audit completed

Initial focus on TB drugs, BCG booster, malaria vx with novel adjuvants, shigella vaccines, MNCH portfolio

5

/ GATES MRI AT A GLANCE

DISCOVERY/

RESEARCH TRANSLATIONAL DEVELOPMENT

LATE-PHASE DEVELOPMENT

Lead candidate

optimization, pre-

clinical development,

working with early

research partners

GxP studies through

clinical POC in

the target population,

working with translational

development partners

Effective handoff

to late-phase

development partners


/ the Foundation: We seek to execute

6

Alliances such as

TB Drug

EnablingTool

Development

InfluencingRegulatory

policy

InfluencingDelivery

and Pricing

Project Leadership

Global Partnerships

Program Strategy

Gates

MRI

• We share a common belief: all lives have equal value.

• Wholly-owned subsidiary of the foundation with secure

funding (15-year business plan and operating budget).

• The foundation’s extensive skills, expertise and long-

standing relationships across academia and industry

provide synergy.

• We are committed to bringing the latest industry

innovations to bear in our collective work.And many other capabilities…..

EnablingDrug Discovery


Target

Validation

Pre-Clinical

Phase 1

Phase 3

Phase 2 Innovative

Clinical Trials

Translating from

Preclinical to

Development

CMC

Nonprofit

Biotech Model

Bridging

Innovation from

Private Sector

Translational

Medicine

Assay

Development

7

Our focus is translational medicine: advancing novel drug and vaccine candidates from the laboratory to human studies. Funneling many possible ideas into a few viable solutions is the most challenging stage of the research and development process, but we embrace this challenge.

We learn continuously from everything we do. If one, two, or 100 ideas don't turn into solutions today, they yield insights that help us succeed tomorrow.

/ Our Approach

8

"The best way to predict the future is to

invent it."

- Alan Kay

Business Need, Project Approach, Risk-Based Auditing Tool Development

Evidence –Based Decision

Making

Target Auditees Based on

Performance & Risks

Focus: HSP, Data Reliability,

The quick brown fox

jumps over the lazy dog.

Business Need: Risk – Based AuditingNew non-profit biotechnology company faced with many challenges

• Starting TB vaccine and drug clinical trials• Sites in South Africa

Gates MRI is small and still developing systems to support our clinical trial oversight processes

5Fundamental project management skills Of 2019

1 2 3 4 5

FDA presentation at MCC SummitBenchmark

Research

Initial Assessment

Information aligned with Gates MRI experiences and

approaches

Assessment Analysis

Leverage the MCC Site Selection Tool

and Metrics

Contacted Linda Sullivan

Substituted Quality Indicators as the

Risk Score

Risk Based Auditing Tool

Initial site audits based on their Risk

Score

ID of Sites

Project Approach: Investigational Site Risk – Based Auditing

Weighted Criteria Individual Site Scores Country & Site List Composite Scores

Criteria weighted as to relevance per protocol before

beginning scoring process

Each site is rated (Y/N) on a standard set of site

quality criteriaTool calculates score based on responses

Individual sites can be excluded from country & site list composite scoresComposite scores reported for all sites and selected sites only

MCC Site Selection & Site Study Conduct Quality Scoring Tool Key Features

Site Selection Quality (SSQ) which helps you evaluate the quality of the sites you plan to initiate. This tool should be used prior to selecting any sites for your protocol.

Site Study Conduct Evaluation (SSCE) tool which helps you evaluate the ongoing performance of your selected sites. This tool should be used during study conduct and/or study close.

MCC Site Selection & Site Study Conduct Quality Scoring Tool –Key Features

These Tools are

Linked

Scoring CriterionImportance to success of this protocol

Weight

(click on criterion to see a list of useful scoring questions and

related MCC metrics)

None

(0)

Low

(1)

Moderate

(2) Critical (3)

1 Site is meeting enrollment goals with evaluable subjects

2 Site has stable research staff and re-training is seldom required

3 Site is compliant with sponsor/CRO, ethical, legal, privacy requirements

4Site is able to produce quality, timely data (query rate is not an outlier

with study and/or regional norms)

5 Site resolves data queries per required timelines

6 Site is compliant with dosing/device use requirements

7 Site is compliant with management of drug/device supply

8 Site has demonstrated state of regulatory agency inspection readiness

9 Site is compliant with reporting and management of safety events

10 Site performs procedures in compliance with protocol requirements

11 Site handles and maintains equipment per protocol requirements

12Site's vendor data and vendor/sponsor escalation requirements are

satisfied

13 Site is compliant with/does not deviate from the study protocol

14 Site is compliant with applicable GCP requirements

Total of all weights 0

Number of criteria scored 0

©2011 Metrics Champion Consortium, Inc. All rights reserved

MCC Site Audit Selection Draft Tool – Key Features

14

att

rib

ute

s

Weight the importance of attributes

Click on criterion to see a list of

useful scoring questions and

related metrics

Select Y/N if expectations met

©2020 Metrics Champion Consortium, All rights reserved

Criterion Weighting


Criteria from the weighting worksheet

Each one links to listing with useful scoring

questions and related metrics

Tool generates score using criteria weighting

established in criteria weights tab



Criterion from the weighting worksheet

Each one links to listing with useful scoring

questions and related metrics

Risk Based Auditing Tool Development

The initial categories were based on the metrics the CRO was collecting at each site and included:

# Enrolled

Participants

% Screen failed

Number of early discontinuations participants

( a QTL)

Number of participants missing QFT testing at required visit (a QTL)

% Protocol Deviations

Site reporting SAEs late

Query Status

Risk-Based Site Auditing Selection Tool

Provide evidence-based insights to understand the strengths and weaknesses of your study sites relative to YOURprotocol. • There are 14 criteria that a site must meet to

be successful• There is an option of selecting the 14 that

are most and least critical to the success ofYOUR protocol

• You should amend, as indicated, based on your critical data, processes, and other protocol-related risks, etc.

• Use this tool to select sites for an audit

Risk – Based Auditing Tool Implementation Results

Risk-Based Auditing Tool: Implementation Results

• Positive Outcomes• Successfully repurposed the Site Selection Tool

to a Site Audit Selection Tool• Utilized study specific assessments including

QTL criteria• Identified sites for audit selection using the

risk score• It is a work in progress as we grow our clinical

study portfolio• Issues:

• Data was assessed based on third month of the quarter to plan audits for next quarter

• Data is not normalized across sites• Due to lack of experience with the sites we did

not have any historical data to contribute to assessment

• Study specific criteria time consuming and prevents ability to assess study site risk scores across a program

budgeting

scheduling

communication

ThoughtLeadership

Implement

exe

cutesu

pp

ort

plan

Plan

Utilize KQIs from investigator sites as criteria to assign a risk score to each site

Execute

Adapt the MCC Site Selection Tool to create a Site Audit Selection Tool for active clinical studies.

Implement

Replaced the Site Selection Criteria with study specific KQIs for TB vaccine study

EvaluatePilot was success for initial site risk scores but use of normalized data is required for better on going assessment of sites during active studies

Summary

Future ConsiderationsAdding Leading Indicators

Next Update:Leading vs Lagging Metrics

Normalization has a variety of meaning in

statistics. In the context of metrics, it refers to

adjusting the scale used to make it easier to

compare values. Typically, this is accomplished

by dividing a count value by a relevant factor to

enable easier comparison. That relevant factor

is termed a normalization factor.


Next Update:Leading vs Lagging Metrics

Lagging metrics provide information you can use for future trials or for baselining process-improvement efforts. They are typically generated after a finite step or a trial is completed.

Leading metrics provide information you can use during study conduct and should be defined to give as early a signal as possible so you can take action quickly to avoid the indicated issue and/or minimize its consequences. The normalization factor used can significantly impact how early a signal is detected. In some cases, the normalization factor that works well for a lagging metric is not the best option for a leading metric.


Do Site Performance Metrics Need to be Modified When Used During Study Conduct?

Does the opportunity for the measurement increase over time?

When assessing an activity where the opportunity for an event to occur changes over time, you should consider including a time component as part of normalization.


This is an important factor to consider when comparing site performance during study conduct if sites have patients enrolling at differing time points

Do Site Performance Metrics Need to be Modified When Used During Study Conduct?

Item being

Measured

Lagging Metric

Normalization

Factor

Leading Metric

Normalization

Factor

Rationale

Adverse EventsRandomized

Patient

Patient-Week

(data collection

with ePRO)

Patient Visit

(data collection at

patient visits)

Each week a patient is on the

study provides an opportunity

for AEs.

The number of Patient Visits

could also be used –

depending on the data flow.

Protocol

Deviations

Randomized

Patient

Patient-Week

Patient Visit

Each week a patient is on the

study provides an opportunity

for protocol deviations.

Patients with

premature drug

discontinuation

Randomized

patients

Patient-Weeks

Patient Visits

Each patient visit provides an

opportunity for the decision

to discontinue drug.

The number of Patient-Weeks

could also be used but may

not be as effective.


Thank You! Please submit questions to the presenters using

the submit question link

Maryann LivolsiGCP Compliance Leader Bill & Melinda Gates Medical Research Institute

Liz Wool, CCRA, CMT PresidentWool Consulting Group, Inc.

Join us at the Summit for a “live” Risk-Based Auditing Community Discussion Group where participants will

discuss how organizations decide which vendors and/or investigative sites to audit?

draft title gates mri experience utilizing a site audit ... · of your study sites relative to your...

Documents