Clinical trials and the impactClinical trials and the impactof regulationsof regulationsof regulationsof regulations

Cytel ConferenceCytel ConferenceOctober 12, 2012October 12, 2012Octobe , 0Octobe , 0

David L DeMets, PhD

Department of Biostatistics &Department of Biostatistics &Medical Informatics

U i it f Wi i M diUniversity of Wisconsin-Madison

TopicsTopics• Proliferation of multinational trials• Regulatory Guidelines & their• Regulatory Guidelines & their

interpretationAd t ti f f t• Adverse event reporting for safety

• One site monitoring• SOPs & SAPs• DMCsDMCs• Adaptive Designs

N G id li f Di b t D• New Guidelines for Diabetes Drug Approval

Regulatory GuidelinesRegulatory GuidelinesRegulatory GuidelinesRegulatory Guidelines• Over the past 20 years, there are more

large phase III trials done by industry• As a result, there are more regulatory , g y

guidelines on the design, conduct and analysis of clinical trials (Eg ICH, FDA, y ( g , ,EMA)

• Are the recent regulatory guidelines onAre the recent regulatory guidelines on the design, conduct and analysis of industry sponsored trials making trialsindustry sponsored trials making trials better?

OpinionOpinion• Clinical trials are not necessarily better

today than 20 years agotoday than 20 years ago• The guidelines are generally consistent

ith li i l t i l f d t lwith clinical trial fundamentals• The problem is in the interpretation of

those guidelines and practice• Industries have been built up on the p

interpretation and implementation of regulatory guidelines g y g

• Not getting a better bang for the cost!

Globalization of InvestigatorsGlobalization of Investigators

Percent of Total1572s Filed

Average Number of CountriesAverage Number of CountriesWhere Clinical Trials Conducted

(2010)

Phase I Studies 2

Phase II Studies 13

20011997 2005 2009

Phase III Studies 34

5Source: Tufts CSDD

20011997 2005 2009

Adult NonAdult Non--US TrialsUS Trials• More than doubled in previous 10 years• One-third of clinical trials by 20 largestOne third of clinical trials by 20 largest

US-based companies are conducted solely outside the USsolely outside the US

• US Inspector General Report80% of drugs approved had non US sites– 80% of drugs approved had non-US sites

– 80% of subjects from non-US sitesT i l f d lit ll– Trials of good quality generally

R f Gli k t l NEJM 2009• Ref: Glickman et al, NEJM 2009

US SituationUS Situation

• Sponsors often view US sites asp– Too slow to get studies approved– Too expensivep– Too slow to enroll

• US investigators often view trials as• US investigators often view trials as– Competing with pressures to see patients

Regulatory overhead more costly than– Regulatory overhead more costly than actually taking care of the patientNot enough academic currency– Not enough academic currency

Estimated Costs of DrugEstimated Costs of DrugEstimated Costs of Drug Estimated Costs of Drug Development ($Millions)Development ($Millions)

Fee R: The cost of clinical trials. Drug Discovery and Development Webcast. March 1, 2007DiMasi JA et al: The price of innovation: new estimates of drug development costs. J Health Economics 2003; 22:151-185p g p ;

DiMasi JA: New drug development in the United States from 1963 to 1999. Clin Pharmacol Ther 2001; 69:286-96

Multiple Countries / RegulatoryMultiple Countries / RegulatoryMultiple Countries / Regulatory Multiple Countries / Regulatory Agencies / regulationsAgencies / regulations

• Sponsors want their produce approved in multiple countries – especially large p p y gmarkets

• Want to abide by each country’sWant to abide by each country s regulations for approval

• Not all countries have same regulations• Not all countries have same regulations• Sponsors feel the need to meet lowest

d i tcommon denominator

International Conference on International Conference on Harmonization (ICH)Harmonization (ICH)

Att t t t d i l 1990’ t k• Attempt started in early 1990’s to make regulatory guidance between US and E i t t l ithEurope more consistent, also with Japan

• Many guidelines developed on design, conduct, analysis, reporting for submissions

• Guidelines generally support good g y pp gprinciples (E.g. ICH-E9 Statistical )

Regulatory Side BusinessRegulatory Side Business• Many companies (CROs) have been

developed to help sponsors &developed to help sponsors & investigators interpret the various regulations avoid problems & passregulations, avoid problems & pass auditsSimilarly companies developed to• Similarly, companies developed to conduct audits for regulatory compliancecompliance

• Sometimes the same company• All of this activity adds substantial cost

Clinical Trials Transformation Clinical Trials Transformation I iti ti (CTTI)I iti ti (CTTI)Initiative (CTTI)Initiative (CTTI)

• Collaborative effort between US FDA,Collaborative effort between US FDA, EMEA, academia and drug/device industryindustry

• Centered at Duke UniversityGoverned by an Executive Committee• Governed by an Executive Committee, & operated by a Steering CommitteeId ifi d i h• Identified some practices that were costly, perhaps not effective, and areas f li ifor streamlining

CTTI ProjectsCTTI ProjectsTwo examplesTwo examplesTwo examplesTwo examples

• SAE ReportsCopies from a trial shipped to all participating sites– Copies from a trial shipped to all participating sites

– Not unblinded & so not useful to sites– Not required by US federal regulations

A l di i i f i– A costly tradition, not very informative– Use DMC process, either internal or external– FDA has new IND safety reporting guideliney p g g

• CRF Complete Auditing– Line by line on site auditing by CRA’s

Very costly to sponsor and investigators– Very costly to sponsor and investigators– Two natural experiments suggest errors in NIH

(academic) type approach are smallGUSTO• GUSTO

• Breast Cancer Trial / Montreal Site Fraud

Clinical SafetyClinical Safetyyy

• Many possible AEs and SAEs• Some can be prespecified, e.g.

– LFTs– QT Interval

• Unexpected events challengingUnexpected events challenging• Rare events challenging

S d t d l b k• Some adverse events are deal breakers• Multiple comparison problem• Short term vs long term surveillance

Coronary Drug ProjectCoronary Drug ProjectMortality ResultsMortality ResultsMortality ResultsMortality Results

Life table cumulative mortality ratesLife-table cumulative mortality rates,Coronary Drug Research Project Group

Coronary Drug Project Coronary Drug Project “ f“ f“Unreliability of small numbers”“Unreliability of small numbers”

Placebo Superior

Clofibrate Superior

z values for clofibrate-placebo differences in proportion of deathsby calendar month since beginning of studyby calendar month since beginning of study

(Month 0 = March 1966, Month 100 = July 1974)

AE Standard Coding SystemsAE Standard Coding SystemsAE Standard Coding SystemsAE Standard Coding Systems• AE’s often collected from patient

complaints, text recorded & coded• Several adverse event coding systemsg y• Often organized by body systems• Subcategorized into events as reported• Subcategorized into events as reported

by investigator/patientS f t itt ft i th i• Safety committees often review these in tabular form by treatment arms

• (AE listings not helpful after awhile)

AE Standard Coding Systems AE Standard Coding Systems g yg y• Generally these tables not helpful

At l l t l d t bl fill d• At one level, too granular and tables filled with small number of events – difficult to interpretinterpret

• At higher level, too many critical and non critical events pooled together – also difficult to interpret

• DMCs often need to select critical events and create new safety variable(s) best not to becreate new safety variable(s), best not to be done post hoc

• Need to sharpen our safety focus on AdverseNeed to sharpen our safety focus on Adverse Events of Special Interest (AESIs)

CTTI/FDA AE Reporting CTTI/FDA AE Reporting I iti tiI iti tiInitiativeInitiative

• As a result, FDA has issued updatedAs a result, FDA has issued updated guidelines on AE reporting: IND Rule

• Focus on AE’s that matter• Focus on AE s that matter• However,

Gl b l l t i t– Global regulatory agencies may not agree– Tradition is hard to break– Many jobs depend on the current practice– What trial wants to be the “first” ?

• Industry struggling to deal with new rule

Site MonitoringSite MonitoringSite MonitoringSite Monitoring• Regulations & good practice suggest

that CTs need to establish that trial had– Real patients– Relevant disease– No allocation bias / randomization – Intervention applied without bias– Unbiased Outcome ascertainment– All relevant outcomes reported– Consent forms signedConsent forms signed

Site Monitoring:Site Monitoring:C t P tiC t P tiCurrent PracticeCurrent Practice

• Need for site performance validation phas led to industry practice of – On site monitoring, multiple timesg, p– 100% Case Report Form (CRF) line by line

validation– Costly to both sponsor and sites– Perhaps as much as 30-35% of trial costp

• CTTI On Site Monitoring Survey– Industry/CROs: > 80% of trials– Industry/CROs: > 80% of trials– Academic networks: < 30% of trials

Is 100% CRF validation cost Is 100% CRF validation cost ff ti ?ff ti ?effective?effective?

• Two natural experimentsTwo natural experiments– GUSTO-I trial of tpa in cardiac patients

• NEJM, 1993NEJM, 1993 – Breast cancer trial

• NEJM, 1995,

• Some errors detected but no influence or even noticeable affect on analysisor even noticeable affect on analysis

• But public often expects perfection but we can’t afford itwe can t afford it – Need to minimize bias

CTTI & Site MonitoringCTTI & Site Monitoringgg

• CTTI Recommendations– Focus on what matters– Develop on-line, not post-hoc, quality p , p , q y

management– Assess quality in key parametersq y y p

• Improve training of investigators– 1000 sites with 1 patient vs 10 sites each1000 sites with 1 patient vs 10 sites each

with 100 patients?• Share experiences work together toShare experiences, work together to

improve QC without adding burden

Multiple IRB ReviewMultiple IRB Reviewpp• Multi-center trials will typically have

lti l IRB imultiple IRB reviews• Multi-country trials will have multiple

IRBs operating under different “regulatory” rules or guidelines

• Each protocol may get different review & requests for different changesq g

• Every amendment must go through the process across all IRBsprocess across all IRBs

• Need to promote more central IRBs

Standard Operating Standard Operating Procedures (SOPs) ManiaProcedures (SOPs) ManiaProcedures (SOPs) ManiaProcedures (SOPs) Mania

• Good practice to document certain keyGood practice to document certain key aspects for a trial– DefinitionsDefinitions– Procedures & equipment– Training / certification– Training / certification– Data collection

• Current practice is to document• Current practice is to document everything imaginable, whether critical or not it’s now a businessor not – it s now a business

• Audit teams demand your SOPs

Statistical Analysis Plans (SAPs)Statistical Analysis Plans (SAPs)ManiaManiaManiaMania

• Many statistical methods exist for any single question or datasingle question or data

• Multiple comparison, repeated testing, endless subgroup issuesendless subgroup issues

• Good to write down before trial starts an analysis plan for major question(s)

• BUT not all contingencies can be anticipated and planned for

• Need to use statistics as a tool to helpNeed to use statistics as a tool to help us think, not an excuse to stop thinking

Greenberg Report Greenberg Report Recommendations for CT ConductRecommendations for CT ConductRecommendations for CT Conduct Recommendations for CT Conduct

• NIH Report 1967, CCT 1988NIH Report 1967, CCT 1988 • Develop a mechanism to terminate early if

Question has been answered– Question has been answered– Trial can’t achieve its goals

Unusual circumstances– Unusual circumstances– Hypothesis no longer relevant

S h ld i i l• Sponsor should not terminate a trial without outside consultants

• Led NIH to use of external DMCs

DMCs in IndustryDMCs in Industry• Increased use of DMCs since 1990• FDA 1989 guidelines very brief mention of dataFDA 1989 guidelines very brief mention of data

monitoring and DMCs• International Conference on Harmonization

(ICH)– ICH/E9

• Section 4.5 Interim Analyses• Section 4.6 Independent DMCs

ICH/E6– ICH/E6

• Gene therapy patient death (2000)

29• FDA DMC Guidelines (2001, 2005)

FDA: When External DMCs AreFDA: When External DMCs AreFDA: When External DMCs Are FDA: When External DMCs Are Probably NeededProbably Needed

• Trials with mortality or major morbidity endpointsp

• Trials for which assessment of serious toxicity requires comparison of ratestoxicity requires comparison of rates

• Trials of novel, potentially high-risk treatmentstreatments

• Trials with high risk or vulnerable subjects

DMC Guideline ImpactDMC Guideline ImpactDMC Guideline ImpactDMC Guideline Impact

• DMCs formed for Phase III and some Phase II trials, more & more

• More DMCs than available experiencedMore DMCs than available experienced members

• No sustained training programs for• No sustained training programs for DMCs

Despite books and numerous papers– Despite books and numerous papers– No regulatory science training

FDA till ibl f d t• FDA still responsible for adequate DMCs implementation

Industry-Modified NIH Model(PROMISE 1991 NEJM)

Pharmaceutical Industry Sponsor

SteeringCommittee

Regulatory Agencies

(PROMISE, 1991, NEJM)

Sponsor

Independent Data Monitoring

Committee Agencies

Central Units

Data Monitoring Committee (IDMC)

Statistical Data Central Units (Labs, …)Data Coordinating

Center (Sponsor or CRO)

Statistical Data Analysis Center

Clinical Centers Institutional Review Board

Patients

Statistical Data Analysis Statistical Data Analysis Center (SDAC)Center (SDAC)

• FDA DMC guidelines suggest a preference for an SDAC independent of the sponsor (industry or NIH)

• Not enough such centers• Most CROs not experienced at DMC

supportsupport– DMC Reports often not focused or

presented effectivelyp y– Often lack flexibility via contracting

mechanism

DMC ChartersDMC ChartersDMC ChartersDMC Charters• Early charters were written to provide

guidance to DMC– Membership– Responsibilities– Guidelines for early terminationy

• FDA guidance supports this approach• Recent charters becoming more• Recent charters becoming more

contractual, suggest legal liabilityDMCs expected to live by the charter– DMCs expected to live by the charter

– Rules rather than guidelines

DMC SAPsDMC SAPs• Difficult to develop a total SAP for the

final analysis of a trial• Impossible to develop an SAP for the p p

DMC of a trial– Too many unexpected scenariosy p– “Expect the unexpected”– Attempts to have detailed DMC SAP haveAttempts to have detailed DMC SAP have

failed• Plan for primary outcome a leadingPlan for primary outcome, a leading

secondary outcome, let DMC do it’s job

Indemnification of DMCs Indemnification of DMCs • DMCs or members have been

subpoenaed and become defendants in plitigation.– Experience indicates investors most likely

• DMCs must be indemnified by the sponsor or through some other defined process

• Indemnification language should be part of the DMC Charter as well as contracts

• Should not be compromised by the insertion of “negligence” or similar terms as an exclusion for protection.

• Ref: DeMets, Fleming et al (2004, CCT)

Erosion of DMC IndependenceErosion of DMC IndependenceErosion of DMC Independence Erosion of DMC Independence

• Charters read like a legal document, not aCharters read like a legal document, not a set of principles to protect patients

• Charters may limit the number of interim• Charters may limit the number of interim analyses, or endpoint data to be reviewed (don’t spend any alpha!)(don t spend any alpha!)

• SDACs not always experienced in DMCsT i i f DMC b l ki• Training of DMC members lacking

• Threat of litigation against DMCs may influence their judgment negatively

DMCs and Adaptive DesignsDMCs and Adaptive Designs• Adaptive designs popular but not new

Adaptation of sample si e based on• Adaptation of sample size based on interim results still problematic

St ti ti l th d i t t t l T I– Statistical methods exist to control Type I– Logistics still not fully resolved

• DMCs being asked to make sample size adjustment recommendations

• May not be a good idea– DMCs know totality of datay– Sample size adjustment may be contrary

Some Common MythsSome Common MythsSome Common MythsSome Common Myths

• DMCs should be blindedDMCs should be blinded• DMCs must follow the SAP exactly• DMC meetings must be scheduled• DMC meetings must be scheduled

precisely & limited in number• DMC reports must be on cleaned• DMC reports must be on cleaned

adjudicated data• DMC reports can be totally• DMC reports can be totally

preprogrammed – following SAP• DMCs review each AE or SAE

39

• DMCs review each AE or SAE

New Guidelines for Diabetes New Guidelines for Diabetes D A lD A lDrug ApprovalDrug Approval

• Recent guidelines require a ruling out ofRecent guidelines require a ruling out of cardiovascular risk for drug approval

• For initial approval rule out risk of > 1 8• For initial approval, rule out risk of > 1.8• Must follow with further data to rule out

i k f 1 3a risk of > 1.3• OK if there are two consecutive trials• Problematic if done within a single trial

– Filing “1.8 results” NDA may compromiseFiling 1.8 results NDA may compromise trial continuation to get “1.3 results”

SummarySummary• RCTs our best methodology if correctly

designed, conducted & analyzed• Unnecessary burdens growing with narrow

regulatory interpretation• Burdens increasing costs for sponsors &

investigators dramatically• US role diminishing• US role diminishing• Must streamline and focus more on real

critical issuescritical issues• If we fail, other external forces will play a

major role & not be as well informed• In the end, our patients lose