quality metrics journey to excellence

Quality Metrics

Journey to Excellence

Steve Greer

Corp QA External Engagement Leader

Procter & Gamble

Russ Wesdyke – FDA, CDER/OPQ/OS

Tara Gooen Bizjak – FDA, CDER/OPQ/OPPQ

Nelson Webb – P&G

ISPE Quality Metrics Core Team

PCPC Quality Assurance Committee

Tracy Kreuser – ASQ

Acknowledgements

Journey to …

FDA Quality Metrics

ISPE Quality Metrics Pilots

Metrics that Matter

PCPC Industry Support

Journey Mileposts

FDA Quality Metrics

Overview

Quality Metrics help FDA:

Prioritize inspectional resources

Recognize high performers

Reduce product shortages

Encourage operational excellence

Why?

Lot Acceptance Rate Operational Excellence

Invalidated OOS Rate Laboratory Excellence

Product Quality Complaint Rate Voice of Consumer

APR/PQR On Time Rate Quality System

Senior Management Engagement Quality Culture

CAPA Effectiveness Quality of Investigations

Process/Performance Capability Voice of the Process

FDA Quality Metrics

Start small

Don’t start with reporting of APIs or Optional Metrics

Voluntary during learning period

Report metrics by site

Allow grouping of formulae

Clarify definitions

CMOs report directly

Context

OTC Industry Comments

Draft Guidance comment period closed Nov ‘15

Revised Draft will be issued by end of 2016

Start of Reporting?

Q Metrics Guidance Timing

Start: Phased Approach, Voluntary

Simplified Approach

Revised Guidance Insights

ISPE Quality Metrics

ispe.orgConnecting Pharmaceutical Knowledge

2012 2013

Quality Metrics Timeline

ISPE Wave

2 Pilot

Industry Whitepapers

Launch Quality Metrics Wave 1 Pilot

July 2012 FDASIA signed into law

2014 2015

Launch Wave 2 Pilot

2016 Beyond

…

Brookings Institute FDA FRN/Draft Guidance

Wave 2 Pilot enrollment Data collection and submission

ISPE Quality Metrics Wave 2 Pilot Timeline

Individual reportsIndustry analysis

Jun 2015

Wave 2 pilot Report

Jun 2016

14ispe.orgConnecting Pharmaceutical Knowledge

Wave 2 Metricsblue = FDA guidance

metrics

SOURCE: ISPE Quality Metrics Initiative

Quantitative metrics▪ Total complaints rate

– Per million packs, incl. LOE

– Per million packs, excl. LOE

– Per ’000 attempted lots released,

incl. LOE

– Per ’000 attempted lots released,

excl. LOE

▪ Critical complaints rate

– Per million packs

– Per ’000 attempted lots released

▪ Total recall events per year

▪ Lot acceptance (%)

– Per finally dispositioned lots

– Per attempted lots

▪ Invalidated OOS rate

– Per ’000 lots tested

– Per ’000 tests performed

– Per total OOS per tests performed

▪ Right first time (%) per released lots

attempted

▪ Deviations rate

– Per ’000 finally dispositioned lots

– Per ’000 attempted lots

▪ Recurring deviations rate (%)

▪ Lots pending disposition more than 30

days (%) per lots attempted

▪ Culture survey scores (% top boxes)

– Total score

– Leadership score

– Integrity score

– Mindset score

– Governance score

– Capabilities score

▪ CAPAs with preventive actions (%)

▪ Planned maintenance rate (%)

▪ Employee turnover rate (%)

▪ Human error deviations (%)

▪ Deviations with no assigned root cause

(%)

▪ CAPA requiring retraining (%)

External Quality Outcomes Internal Quality Outcomes Culture Indicators


Wave 2 Correlations Overview


FRN metrics

Total Complaints

(per pack)

CAPA with

Preventative

Actions

Planned

Maintenance

Rate

External

Quality

Outcomes

Internal

Quality

Outcomes

Culture

Indicators

Total Complaints

(per Released

Lots)

Correlations with p-value <0.05


Planned maintenance rate to total complaints (n=15

solids sites)

Planned Maintenance Rate & Product Q Complaints Rate

Note: Major outliers excluded – more than 2 interquartile ranges away from sample median – 2 sites excl. on total complaints both with very high LOE rates

R2 measures to what extent metric Y (dependent variable) is explained by the variability of metric X (independent variable)

P- value is probability that correlation between X and Y is zero, value below 0.05 indicates statistically significant results.


▪ Planned maintenance rate explains

~45% of the variability in total

complaints per million packs

▪ This relationship may be related to the

common factors that influence both

planned maintenance rate and total

complaints – such as focus on

prevention, operational excellence,

quality improvement mindset

R² = 46%

P=0.00

-10

0

10

20

30

40

50

60

70

80

1000 4020 806010 30 50 70 90

Total complaints rate (including LOE) per million pack

Planned maintenance rate, %


FDA Guidance Metrics Definitions

Product Quality Complaint Rate – proposed normalization by packs

released

– When normalized by released lots attempted has no connection to other

quality metrics

– When normalized by packs there is correlation to cultural metrics



Available from ISPE

Metrics that Matter

Pursue the Red

Wide variability in the Quality of information

Only 50% of consumers have product at time of reporting

Financially motivated fraud artificially inflates the numbers

New product launch and product changes result in an increase in complaints

OTC Complaint Reality

Consumer Comment Journey

P&G Approach to FDA Q Metrics

Leadership role in Industry initiatives

ISPE Pilots: 2 sites

Developing IT solutions – $$$$

Internal FDA Quality metrics reporting program

Leadership reviews -> Quality Council integration

Internal FDA Q Metrics Reporting

Top-Level Learning

FDA Q Metrics have provided new insights

Product grouping approach

Initial challenges to align on definitions and adapt to supply chain differences

Manual reporting burden high at start – improves over time

Complaint rate normalized by released lots is not meaningful

ASQ Quality Culture Workshop

28

Cultureofquality.org/assessment

http://cultureofquality.org/assessment

Quality Culture Assessment

P&G

Consumer Products

All Respondents Understanding CustomerNeeds

Leadership Commitment toExcellence

Adapting AcrossGeographies/Cultures

Investment in PerformanceImprovement

Employee Engagement

Performance Metrics andMeasurement

Pursuit of PerformanceExcellence

Quality Vision and Values

Risk Management Culture

Customer Engagement

Average Scores

Overall By Theme

30

ASQ Culture of Quality Roadmap

Define

Align

Manage

1. Evaluate Current

Performance 2. Identify How Culture

Plays a Role

3. Clarify Improvement

Vision

4. Define Values andExpected Behaviors

5. Clarify Strategies and Goals6. Engage

Team to define

SMART Goals

7. Clarify & Track Key Measures

8. Maintain a Management System for Priorities/

Goals

9. Manage Communicati

on Habits

10. Build Motivations Throughout

PCPC Industry Support

32

60%

40%

2016 Awareness of FDA Q Metrics Program

Yes

No

0

2

4

6

8

10

12

14

16

18

1 2 3 4 5

2016 Level of Readiness

44%56%

2016 Level of Interest Participating in

Benchmarking

Yes

No

Group Discussion

What can PCPC do to help your company prepare?

How can you leverage the FDA Quality Metrics program to accelerate your journey?

Where is Your Journey Headed

… Compliance or Excellence?

quality metrics journey to excellence

Documents