FDA Regulatory Perspective: Data Integrity

Steve Wilson, Dr.P.H., CAPT USPHSDeputy Director, Division of Biometrics II, CDER/FDA

NIH Roadmap ProgramFeasibility of Integrating & Expanding Clinical Research

Networks 4th Steering Committee Meeting

Friday, May 12, 2006NIH Natcher Conference Center, Bethesda, MD

FDA’s Mission• The FDA is responsible for protecting the public

health by assuring the safety, efficacy, and security of human and veterinary drugs, biological products, medical devices, our nation’s food supply, cosmetics, and products that emit radiation.

• The FDA is also responsible for advancing the public health by helping to speed innovations that make medicines and foods more effective, safer, and more affordable; and helping the public get the accurate, science-based information they need to use medicines and foods to improve their health.

Data Integrity• The quality of correctness, completeness,

wholeness, soundness and compliance with the intention of the creators of the data.

• It is achieved by preventing accidental or deliberate but unauthorized insertion, modification or destruction of data in a database.

• Data integrity is one of the six fundamental components of information security.

http://www.pcmag.com/encyclopedia_term/0,2542,t=data+integrity&i=40792,00.asp

Data Integrity• 21 CFR 11 ( In final rule “integrity”

is used 50 times!)• Electronic Submission• Gateway• EDR SOPs

21 CFR 11 -- IntegrityFor electronic records and submissions to have the

same integrity as paper records, they must be developed, maintained, and used under circumstances that make it difficult for them to be inappropriately modified. Without these assurances, FDA’s objective of enabling electronic records and signatures to have standing equal to paper records and handwritten signatures, and to satisfy the requirements of existing statutes and regulations, cannot be met.

Data QualityData are of high quality "if they are fit

for their intended uses in operations, decision making and planning” (J.M. Juran)

http://en.wikipedia.org/wiki/Data_quality

FDA Regulatory Perspective: Data Quality

Steve Wilson, Dr.P.H., CAPT USPHSDeputy Director, Division of Biometrics II, CDER/FDA

NIH Roadmap ProgramFeasibility of Integrating & Expanding Clinical Research

Networks 4th Steering Committee Meeting

Friday, May 12, 2006NIH Natcher Conference Center, Bethesda, MD

DisclaimerViews expressed in this

presentation are those of the speaker and not, necessarily, of the Food and Drug Administration

Truth-in-Advertising• CDER-Centric View• Customer -- Not in Office of

Compliance or ORA

• Center for Biologics Evaluation and Research (CBER) • Center for Devices and Radiological Health (CDRH) • Center for Drug Evaluation and Research (CDER) • Center for Food Safety and Applied Nutrition (CFSAN) • Center for Veterinary Medicine (CVM) • National Center for Toxicological Research (NCTR)• Office of the Commissioner (OC)• Office of Regulatory Affairs (ORA)

Organization: FDA

Outline• Background• Data Quality

– A Bimo perspective– A Review perspective

• Changing Landscape– The Critical Path – Data standards & electronic submissions– PRO and ePRO– Organizational Changes

• Bimo Initiative• CDER’s new Office of Translational Science

(OTS)• Connections/References

Science, Statistics and Science, Statistics and Experimental DesignExperimental Design

• Science is concerned with understanding variability in nature

• Statistics is concerned with making decisions about nature in the presence of variability

• Experimental design is concerned with reducing and controlling variability in ways which make statistical theory applicable to decisions about nature.

Winer, et.al., Statistical Principles in Experimental Design, New York, 1962, 1971, 1991

Data Quality• Guidance for Industry: Computerized

Systems Used in Clinical Trials, April 1999

• Data should be (ALCOA)– attributable – legible– contemporaneous– original – accurate

Responsibility• The sponsor is responsible for

implementing and maintaining quality assurance and quality control systems with written SOP’s to ensure that trials are conducted and data are generated...

• ... Quality control should be applied to each stage of data handling to ensure that all data are reliable and have been processed correctly…

Section 5.1, “Quality Assurance and Quality Control,” Good Clinical Practice Consolidated Guideline -- ICH E6

The LawSection 505(k)(2) of the Food, Drug, and

Cosmetic Act mandates FDA shall have access to and copy and verify the required clinical

study records.

The ProgramThe Bioresearch Monitoring (BIMO) Program

was established in 1977 to verify the data submitted in support of marketing applications

and to provide oversight of the conduct of studies with regulated products.

Ketek – The “Fraud” Word Infected Data

– Fraud, Errors Taint Key Study of Widely Used Sanofi Drug

– Despite Some Faked Results, FDA Approves Antibiotic; One Doctor's Cocaine Use

– Company Defends Safety

By ANNA WILDE MATHEWS Wall Street Journal, May 1, 2006; Page A1

online.wsj.com/article_print/SB114644463095840108.htm

Not the First Time (Never Just a Scientific Question)

•Halcion•Tamoxife

n•Fiddes

Our Responsibility for the Our Responsibility for the Quality of Clinical Trials Data Quality of Clinical Trials Data

Trust, but Verify

Worrying About Data Quality: NDA Review

• Erroneous values• Missing values• Imputation• Adjudication• Deviations from protocol• “Unconscious Bias” • Fraud

Enforcer Colleague

The Terminator Mr. Rogers

Programs That Worry About Data Integrity/Quality

•BIMO

•REVIEW

Data Integrity/Quality A BIMO Perspective

• FDA calls its program of on-site inspections for GCP and GLP its “Bioresearch Monitoring Program” or “BIMO”

• The program includes inspections of:– Clinical Investigators– Sponsors, monitors, CROs– Institutional Review Boards– Bioequivalence Laboratories and Facilities– GLP Facilities (nonclinical studies)

• Each Center has its own BIMO groupWoollen, 2002

CDER’sDivision of Scientific

Investigations

Our BIMO People

DSI’s Program Responsibilities

• Good Laboratory Practices • In vivo Bioequivalence • Good Clinical Practices

– Institutional Review Boards (IRBs)– Clinical Investigators– Sponsor-Monitors, CROs

Program Objectives• To verify the quality and integrity

of bioresearch data• To protect the rights and welfare

of human research subjects

GCP Inspections:Routine vs. Directed

• Routine– Inspections assigned for NDA/PMA’s

• Directed– Problems identified at IND/IDE stage– Complaints to FDA

• FDA, other Agencies• Sponsors/monitors• Institutions/IRB’s• Subjects/Public

CDER-GCP Bioresearch Monitoring (BIMO) Program

• Clinical Investigator Inspection Program• Sponsor/Monitor/CRO Inspection Program• IRB/RDRCThese “on-site” inspection programs collectively allow

the agency to determine:– Adherence to applicable FDA regulations– Validity of data from studies in support of pending

marketing applications (Data Integrity)– Whether the rights and safety of subjects have been

protected (Safety)

J. Rhoads, 2005

CDER BIMO Inspections (FY 2004)

CI = 351BEQ=

136IRB = 120GLP = 68S-M = 17

Total = 692

J. Rhoads, 2005

When a Submission is Received:

• Review Division Invites DSI staff for filing meeting• Team effort in selecting sites for inspection based

on “risk based approach”– Impact to review of the study

• Site(s) influencing significance on outcome • Outliers, e.g., site with large proportion of treatment

responders• Drop-outs; Adverse events; Protocol violations; large

N#– Impact to the clinical trial process

• Volume of work performed by the Clinical Investigator• Past inspectional history

• Inspection request assignment to the field offices (ORA)

• PDUFA Time Clock! J. Rhoads, 2005

Criteria for Assigning International Inspections

International sites may be audited• if there are insufficient domestic data;• only foreign data are submitted to support

an application;• domestic and foreign data show conflicting

results pertinent to decision-making; or• there is a serious issue to resolve, e.g.,

suspicion of fraud, scientific misconduct, significant human subject protection violations.

J. Rhoads, 2005

After an Inspection• Form FDA 483: Inspectional Observations

– Left with CI at close of inspection– Immediately available via FOI

• Establishment Inspection Report (EIR)– Prepared by field investigator after

inspection– Includes exhibits supporting observed

deficiencies

J. Rhoads, 2005

After an Inspection• Clinical Inspection Summary to the

Review Division– Summary of all inspections assigned for an

application– Recommend accept or reject data– Provided to review division in advance of

PDUFA action goal date• Letter to the inspected party

– emphasize deviation from regulations, if any.– Copied to the review division– include “DSI note to Review Div. Medical

Officer”J. Rhoads, 2005

Regulatory/Administrative Follow-up

• Rejection of study

• Disqualification• Prosecution

J. Rhoads, 2005

Clinical Investigator Deficiencies

CDER Inspections - FY 2004

0%

10%

20%

30%

40%

50%

60%

NAI Protocol Record Consent DrugAcct

AEs

ForeignDomestic

34%28%

55%

29%

48%

7%

21%

5%

Foreign n = 71*Domestic n = 243*

2%

17%

4%10%

*% does not includeBiologic inspectionsJ. Rhoads, 2005

Some GCP Challenges and Initiatives

• Increased international inspections• Risk-based approach to inspection

site selection • Accountability of study staff/site

management organizations in clinical research

• Linked real-time inspections (CI/Sponsor/IRB)

J. Rhoads, 2005

Data Integrity/Quality: A Review Perspective

•Medical Reviewers•Statistical

Reviewers

NDA/BLA Review: MedicalNDA/BLA Review: Medical• Exclusion of patients from primary analyses• Reliance on unplanned subset analyses• Lack of consistency of results within and across

studies• Patient entry errors (cancelled patients, ineligible

patients)• Non-evaluable patients• Missing data• Inconsistent or clearly inaccurate data• Extensive or non-random corrections to case report

forms• Missing source documents• Possibility for bias in patient treatment or patient

assessments• ……

Robert DeLap,Robert DeLap, PhRMA Meetings, PhRMA Meetings, Washington DC, October 26, 1994Washington DC, October 26, 1994

NDA/BLA Review: NDA/BLA Review: StatisticalStatistical• Assess compliance with protocol / blinded analysis

plans• Assist DSI in planning investigator audits• Check appropriateness of statistical models and

conclusions. • Verify results reported in the NDA.• Modify models and assess robustness/sensitivity of

the results. • Modify data sets and reanalyze.• Examine the trial and data for bias:

– Results by center – Baseline predictors– Important subgroups (sex, age, race, etc,)

• Assess impact of audits

The reduction in sudden deaths, said the FDA, had been exaggerated. Some patients who died according to Dr. Robert Temple Director of FDA’s Cardio-Renal Drug Products Division, had been excluded for “minor protocol violations”; and most of the excluded patients turned out to have been taking anturane … Nearly all the miss-classifications turned out to favor the hypothesis that anturane prevents sudden deaths.

“Unconscious Bias” and Readjudication

Lancet, August 9, 1980, p. 306

Case Study #1

“Unconscious Bias” and Readjudication

The reason for these doubts is that Ciba Geigy, which had a stake in the outcome, not only paid the bills (around four million dollars) but also was deeply involved in the daily processing and collection of data before turning over the information to an independent policy committee.”

Lancet, August 9, 1980, p. 306

Case Study #1

Analyzing Potential FraudTreatment-by-Center Interaction: p = 0.056

CENTER

Mea

n C

hang

e fro

m B

asel

ine

1 2 3 4 5 6 7 8 9

-3

-2

-1

0

1

2

Drug X

Placebo12 10 19 1 7 19 5 17 1

7 5 9 0 4 10 3 9 0

?

Case Study #2

Analyzing Potential Fraud (con.)

Sym

pB

SympC

0 1 2 3

0 68 4 0 01 17 4 0 02 16 21 2 03 1 0 0 0

Total: 133

Center 1

SympC0 1 2 3

0 33 2 0 01 3 33 0 02 5 4 63 23 0 0 1 31

Total:177

Sym

pB

Center 8

Case Study #2

Analyzing Potential FraudSymptom Score

0

20

40

60

80

100

1 2 3 4 5 6 7 8 9

Center

Perc

ent o

f Val

ues

NoneMildModerateSevere

Case Study #2

Halcion: For-Cause AuditCase Study #3

From the Summary Basis for ApprovalFrom the Summary Basis for Approval

Protocol 6045 (investigators: Franklin, Howard, Reeves, Protocol 6045 (investigators: Franklin, Howard, Reeves, Simson): 129 insomniacs (aged 22-62 years) participated in Simson): 129 insomniacs (aged 22-62 years) participated in four double-blind 28-day studies comparing triazolam four double-blind 28-day studies comparing triazolam 0.5mg to placebo; Franklin and Reeves enrolled 78% of the 0.5mg to placebo; Franklin and Reeves enrolled 78% of the patients. Pooled data showed triazolam was significantly patients. Pooled data showed triazolam was significantly better than placebo at end of week 1 on efficacy better than placebo at end of week 1 on efficacy measures…measures…

Drowziness, dizziness, and impaired coordination were Drowziness, dizziness, and impaired coordination were significant side effects; 13/61 patients in drug group who significant side effects; 13/61 patients in drug group who dropped out due to side effects, 10/13 dropped out in the dropped out due to side effects, 10/13 dropped out in the first week; no abnormal changes in clinical significance first week; no abnormal changes in clinical significance were observed in lab analyses or vital signs.were observed in lab analyses or vital signs.

Halcion: For-Cause AuditEfficacy Analysis (Protocol 6045)Efficacy Analysis (Protocol 6045)

Investigator 8 Days 15 Days 28 Days

Franklin 0.0001 0.0001 0.0001Reeves 0.001 0.99 0.48Howard 0.6 0.45 0.45Simson 0.04 0.62 0.99

TOTAL 0.0001 0.0001 0.0001(-Franklin) 0.0001 0.34 0.11

Case Study #3

Fraud: Robert Fiddes “If ever there was a wonder boy in the lucrative

business of drug testing, it was Dr. Robert Fiddes. In just a few years, Fiddes transformed his sleepy

medical practice ... into a research juggernaut, recruiting his patients for drug experiments at a breakneck pace. His success made him a magnet for an industry desperately scouring the nation for test subjects. Companies large and small showered him not only with almost 200 studies to conduct, but with millions of dollars in compensation for his work.”

Case Study #3

Kurt Eichenwald and Gina Kolata“A Doctor's Drug Studies Turn Into Fraud”New York Times, May 17, 1999

Fraud: Robert Fiddes“The abuses of this one doctor point to

weaknesses in the new system that has developed in recent years for testing experimental drugs. No longer does the pharmaceutical industry rely on career researchers at academic medical centers, whose professional reputations are forged on the quality of their data. Rather, the industry has turned to thousands of private-practice doctors, for whom testing drugs has become a sideline for making money.”

Kurt Eichenwald and Gina Kolata“A Doctor's Drug Studies Turn Into Fraud”New York Times, May 17, 1999

Case Study #3

When Things Go Wrong• Sensitivity Analysis

(Reanalyze)• Expand Audit (FDA, Sponsor,

Third Party)• Resubmission (delayed

approval)

Changing Landscape• The Critical Path • Data standards & electronic submissions• PRO and ePRO• Organizational Changes

– Bimo Initiative– CDER’s new Office of Translational Science

(OTS)• Connections

Critical Pathhttp://www.fda.gov/oc/initiatives/criticalpath/

Opportunities List

The Critical Path—Data Standards

FDA Vision: e-Review

eSubSponsor

Document Mgmt System

Sponsor Data

Warehouse

FDAElectronicDocument

RoomServers

Desktop Tools (Acrobat, PPV,

JMP, Excel etc.)

Sponsor FDAGateway Repository Review Environment

DocumentsCRTsListingsPatient ProfilesAnalysis Data

CRTs Oracle

Database

JanusData

Warehouse

Document

Share

WebSDMData

Load &Validation

WebSDMData

Viewer

“The Standards Business”Kush/CDISC (2005)

CDA

RCRIM Technical Committee

Protocol Representation

ADaM

U.S. Dept. of Health and Human Services(HHS)

Health Level 7 (HL7)

U.S. FDA

CDISC

NIH/NCI NLM

EFPIA

EMEA MHLW

PhRMAJPMA

CDC

Reference Information Model

RIM

LAB

eCTD

LOINC

SNOMEDMedDRA

ODMSDS

= Organization= Dictionary, Codelist = Standard = Model

= Document Standard, or Architecture

BRIDG Model

International Conference on Harmonization (ICH)

World Health Organization (WHO)

FDA/CDISCwww.cdisc.org

• Clinical Data Interchange Standards Consortium

• Workgroups– Operational Data Model (ODM)– Submission Data Standards (SDS) – Analysis Data Models (ADaM)– Protocol– Electronic Source Data Interchange (eSDI)– Lab Data

ODM: Clinical Data Structure

*StudyEventData~*SubjectData~

*Annotation

*StudyEventData~

*Annotation

*SubjectData~

*ItemGroupData~*ItemData~

?Signature

?AuditRecord

*Annotation

*ItemGroupData~

*FormData~

?ArchiveLayoutRef~

?Signature

?AuditRecord

*Annotation

?Signature

?AuditRecord

?InvestigatorRef~

?SiteRef~

?Signature

?AuditRecord

Kubick

ADaM: Analysis-level Metadata• ANALYSIS NAME – A unique identifier for this

analysis. May include a table number or other sponsor-specific reference.

• DOCUMENTATION – A text description documenting the analysis performed.

• REASON – The reason for performing this analysis. Examples may include Pre-specified, Data-driven, Exploratory, and Regulatory Request.

• DATASET – the name of the analysis dataset used should be linked to the analysis dataset used for this analysis. In most cases, this will be a single dataset. If multiple datasets are used, they should all be listed here.

• PROGRAM – Analysis programs using the DATASET above as input can be described or included here.

New Draft Guidancehttp://www.fda.gov/cder/guidance/5460dft.pdf

ePRO: Specific Concerns When Using Electronic PRO Instruments

• …sponsors should plan carefully to ensure that FDA regulatory requirements are met for sponsor and investigator record keeping, maintenance, and access.

• These responsibilities are independent of the method … apply to electronic PRO data.

• Sponsors are responsible for providing investigators with the information they need to conduct the investigation properly, for monitoring the investigation, for ensuring that the investigation is conducted in accordance with the investigational plan, and for permitting the FDA to access, copy, and verify records and reports relating to the investigation…

BIMO Initiative• Ref. “Modernizing Human

Subject Protection/Bioresearch Monitoring,” Rachel E. Behrman, Deputy Director, Office of Medical Policy & Lead, Cross-Center Initiatives Task Force presented at PhRMA Bimo Meeting, March 31, 2006

FDA’s HSP/BiMO Initiative• Part of FDA’s Critical Path

Initiative• Begun December 2004• Steering committee chartered –

includes representatives from all centers and relevant offices

• Scoped out dimensions of issues and formed working groups

R. Behrman, 2006

FDA’s Oversight Must Evolve • Must provide regulatory guidance and perhaps

new regulatory scheme that encompasses modern trial arrangements – Responsibilities of investigators– Data integrity

• Must facilitate effective IRB oversight of evolving clinical trials arena to facilitate– IRB oversight of human subject protection – FDA oversight of IRB function

• Need common standards and regulatory requirements for electronic data handling

• Must be able to accommodate globalization of clinical trials

• Must ensure comprehensive approach to protection of vulnerable populations

R. Behrman, 2006

Guiding Principles of FDA Initiative

• Collaborative efforts among government, academia, industry and patient groups

• Infrastructure and “toolkit” development, not product development

• Build support for academic science bases in relevant disciplines

• Build opportunities to share existing knowledge & database

• Develop enabling standardsR. Behrman, 2006

CDER/OTS• CDER’s new Office of Translational

Science• “Coming together”

– Office of Biostatistics (OB)– Office of Clinical Pharmacology (OCP)

• Focus for Critical Path activities at CDER

Comments/Connections• Parsimony: Guidance for Industry --

Cancer Drug and Biological Products —Clinical Data in Marketing Applications

• Critical Path: Case report form Standards• DQRI – Data Quality Research Institute• SCDM – Society for Clinical Data

Management

Regulation Guidance

The Terminator Mr. Rogers

Thank Youstephen.wilson@fda.hhs.gov