1 fda evaluation of prescription genetic tests reena philip, ph.d. oivd/cdrh/fda march 9, 2011...

1

FDA Evaluation of Prescription Genetic Tests

Reena Philip, Ph.D.

OIVD/CDRH/FDA

March 9, 2011

Molecular and Clinical Genetics Panel for Direct-to-Consumer (DTC) Genetic Tests

2

SafetyAre there reasonable assurances, based on valid scientific

evidence that the probable benefits to health from use of the device outweigh any probable risks? [21 CFR 860.7(d)(1)]

Effectiveness Is there reasonable assurance based on valid scientific

evidence that the use of the device in the target population will provide clinically significant results? [21 CFR 860.7(e)(1)]

IVD Device Regulation

3

Genetic Tests: Categories

Single analyte tests Genotyping

Multiple analyte genetic tests Multiplex

4

FDA Cleared Prescription Genetic Tests: Examples

Single analyte genetic tests Factor II / Factor V / MTHFR (aid in

diagnosis claim) Multiple analyte genetic tests

CFTR (carrier testing, newborn screening, and confirmatory diagnosis claim)

CYP2D6 genotyping (drug metabolism claim)

5

What Does FDA Review for Prescription Genetic Tests?

Intended use/indications for use Device description (platform, software) Pre-analytical Analytical validation Clinical validation Instrumentation, software validation (if applicable) Labeling (package insert)

6

What Does FDA Review for Prescription Genetic Tests?

Intended use/indications for use Device description (platform, software) Pre-analytical Analytical validation Clinical validation Instrumentation, software validation (if applicable) Labeling (package insert)

7

Intended Use/Indications for Use

Intended use specifies What the test measures Why (the clinical indication for use) In what population it is intended to be used Setting in which the device is meant to be used

8

Example: Intended Use The –---- Cystic Fibrosis Kit is a device used to simultaneously

detect and identify a panel of mutations and variants in the cystic fibrosis transmembrane conductance regulator (CFTR) gene in human blood specimens. The panel includes mutations and variants currently recommended by the American College of Medical Genetics and American College of Obstetricians and Gynecologists (ACMG/ACOG), plus some of the worlds most common and North American-prevalent mutations. The ----- Cystic Fibrosis Kit is a qualitative genotyping test which provides information intended to be used for carrier testing in adults of reproductive age, as an aid in newborn screening, and in confirmatory diagnostic testing in newborns and children. 

The kit is not indicated for use in fetal diagnostic or pre-implantation testing. This kit is also not indicated for stand-alone diagnostic purposes.

For Prescription use only.

9

Device needs to have a clinical indication The types of validation studies that are

needed depend on the claims that are made in the intended use

Intended Use/Indications for Use

10

What Does FDA Review for Prescription Genetic Tests?

Intended use/indications for use Device description (platform, software) Pre-analytical Analytical validation Clinical validation Instrumentation, software validation (if applicable) Labeling (package insert)

11

Pre-analytical

Sample collection/transport/storage Sample preparation/conditions Nucleic acid isolation Stability of the analyte in the patient

specimen

12

Analytical Performance

Does my test measure the analyte I think it does?

Correctly? Reliably?

13

Accuracy Precision (repeatability, reproducibility) Performance around the cut-off Limit of Detection Interference, cross-reactivity Sample type/matrix Potential for carryover, cross-hybridization Effect of excess/limiting sample

Analytical Performance

14

Analytical Performance: Accuracy

Closeness of the agreement between the result of a test and result of reference method.

15

Analytical Performance: Accuracy

Compare to ………. Comparison to a reference method

e.g., bi-directional DNA sequencing

Comparison to a clinical truth

Real clinical samples Multiple clinical samples per allele

Cover every claimed allele/result, genotypes, subtypes/classes

16

Analytical Performance: Accuracy(Continued)

Alleles % Agreement

Total 98.40

Alleles 1 - 20 100.00

Allele 21 87.50

Allele 22 66.67

Allele 23 96.55

An example of why we ask accuracy data of every individual allele the test claims to detect:

17

Analytical Performance: Precision Studies should demonstrate that the intended users can

get reliable results All sources of variability should be identified and assessed

for its impact on assay precision Should use clinical samples where possible

Adequate coverage of all genotypes/tumor types In limited cases (i.e., very rare alleles) may use contrived

samples Samples should mimic the molecular composition and

concentration of real clinical samples

All analytical steps of the assay should be included

18

Test Performance: Evaluation Analytical performance - does my test

measure the analyte I think it does? Correctly? How reliably?

Clinical performance - does my test result correlate with target condition of interest in a clinically significant way?

19

Clinical Performance: Genetic Tests

When there is sufficient information that establishes well-known association between genetic variants and medical condition –

For each claimed allele:

Peer-reviewed articles

Genotype – Phenotype

20

Clinical Performance: Genetic Tests (Continued)

When there is not enough information that establishes well-known association between genetic variants and medical condition –

May require clinical studies

21

CFTR mutation panel ACOG/ACMG recommendation Published literature

Mutations in a novel gene to predict risk of developing cancer Most likely needs clinical studies

Clinical Performance: Examples

22

Clinical Effectiveness

New Markers

(Should meet FDA standard for effectiveness)

(based on valid scientific evidence that the use of the device in the target population will provide clinically significant results [21 CFR 860.7(e)(1)])

Established Markers

(Medical literature)

23

What Does FDA Review for Prescription Genetic Tests?

Intended use/indications for use Device description (platform, software) Pre-analytical Analytical validation Clinical validation Instrumentation, software validation (if applicable) Labeling (package insert)

24

Labeling and Reporting Results

21 CFR part 809 (subpart B) Tests provide results, limited interpretation

required

25

Most Frequent Issues…FDA Evaluation of Genetic Tests

Lack of clinical samples covering all genotypes

Lack of literature to support validity One or two references may not be sufficient Genotype and Phenotype not indicated

Pre-analytical issues Lack of specimens from start to end (e.g., whole

blood assay result) Sample matrix issues

26

Summary: What Does FDA Review for Prescription Genetic Tests?

Safety and effectiveness generally determined based on:

Satisfactory analytical performance Clinical performance in the context of use Labeling that is compliant with the labeling

regulations for IVDs (21 CFR 809 Subpart B) And other factors such as ability to repeatedly

manufacture the device to specifications

27

Relevant Guidance Documents

Guidance on Pharmacogenetic Tests and Genetic Tests for Heritable Markers http://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/ucm071075.pdf

Class II Special Controls Guidance Document: CFTR Gene Mutation Detection Systems http://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/ucm071104.pdf

Class II Special Controls Guidance Document: Drug Metabolizing Enzyme Genotyping System - Guidance for Industry and FDA Staff http://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/ucm071085.pdf

Class II Special Controls Guidance Document: Instrumentation for Clinical Multiplex Test Systems - Guidance for Industry and FDA Staff http://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/ucm071061.pdf

28

Performance of FDA Approved/Cleared Genetic Tests are

Publicly Available Decision summaries of 510(k)s

http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfPMN/pmn.cfm

Summary of Safety & Effectiveness of PMAshttp://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfPMA/pma.cfm

29

Thank You!

Reena.Philip@fda.hhs.gov

30

Principles of FDA Regulation for In Vitro Diagnostic Tests for

Home Use

Carol C. BensonOIVD/CDRH/FDA

March 9, 2011

Molecular and Clinical Genetics Panel for Direct-to-Consumer (DTC) Genetic Tests

31

Examples of home use testBenefitsRisksInterpretation of results Device performanceLabeling Human factors

Overview

32

Examples of Home Use Tests FDA Regulates Test Collect

samplePerform test

Interpret results

Glucose Yes Yes Yes

Pregnancy Yes Yes Yes

Drugs of abuse Yes Yes Yes

Breath alcohol Yes Yes Yes

Ovulation and Menopause

Yes Yes Yes

33

Examples of Home Use Tests FDA Regulates (Continued)

Test Collect sample

Perform test

Interpret results

Collection Hep. C kit

Yes No No

Collection HbA1c kit

Yes No Yes

Collection HIV-1 kit

Yes No No

And others …but “No” genetic tests

34

What are the Benefits for Home Use Tests?

Condition/disease that needs to be monitored at home?

Diabetes – home glucose meters monitor the management of diabetes home user under care of a physician

Not for diagnosis – no performance

35

What are the Benefits for Home Use Tests?

Condition/disease that can be identified to allow for early detection at home?

Pregnancy – urine hCG testsusers retest, go to HCP

36

What are the Benefits for Home Use Tests?

Condition/disease that can be screened for at home?

Drug detection – Home DOA urine testNot a definitive test Mitigation - send sample for confirmation testing

37

What are the Risks of Home Use Tests?

Is the device robust? Simple to use Works correctly every time Not affected by environmental conditions

or different operators Can a home user read instructions and

Collect the sample correctly Perform the test Get accurate results and interpret results

38

Interpretation of the Results to Ensure Safe and Effective UseDoes the home user know what to do with the results? Test again on another dayCollect another sample and retestContact HCP – seek treatmentNot seek treatment Not suspect the test may be wrong

39

What are the Risks of False or Inaccurate Results at Home?

Failure to seek treatment Delay in seeking treatment Improper self management/treatment No follow up with health care provider Unnecessary worry False sense of security

40

Do the benefits outweigh the risks? If yes, then…

41

Evaluate Performance of the Test in the Hands of the Intended User:

Home User Study Compare home user results to

laboratory method How well the test should work at

home depends upon benefit and mitigation of risks

Likelihood of incorrect results

42

Does the labeling provide adequate information so home user can perform

the test and interpret the results for safe and effective use?

Labeling

43

How Does FDA Review Labeling for Home Use Tests?

Written at 8th grade reading level Simple instructions Pictures and diagrams on how to get sample and

perform test Clear instructions on how to interpret the results

(what to do with the results – call HCP – retest) Users know when device did not work User know what to do if device does not work Telephone number to call for assistance

44

How Do Human Factors Play a Role in Home Tests?

People have different abilities to follow directions Home users are not trained users so no “good laboratory practice” standard for them Fail to get adequate or appropriate sample Can perform test incorrectly Can interpret results incorrectly

45

Summary – FDA Principles for Regulation of Home Use Tests FDA regulates home use tests Benefits vs. risks Mitigation of risks Interpretation of results by home user Performance of the device by home user Labeling Human factors

46

http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfIVD/Search.cfm

Website for Database Search for Home Use Tests:

47

Thank You!

Carol.Benson@fda.hhs.gov

48

Risk Assessment Tests

Marina Kondratovich, Ph.D.

OIVD/CDRH/FDA

March 9, 2011

Molecular and Clinical Genetics Panel for Direct-to-Consumer (DTC) Genetic Tests

49

1. Introduction2. Basic concepts: risks, relative risks, likelihood ratios, odds ratios3. Description of a typical DTC risk assessment test4. Clinical validation (discrimination and calibration)

Overview

50

1. Introduction Susceptibility/Pre-dispositional tests (Risk Assessment tests): tests that estimate the lifetime risk (relative or absolute) that an individual will develop a condition. Examples: test for Alzheimer’s disease, test for prostate cancer, test for type 2 diabetes

Possible Intended Use Claim:“…to estimate the likelihood that an individual will develop <target condition> during the lifetime…”

51

Typical DTC Risk Assessment Test

Individual

Covariates Markers

Race Gender SNP1 SNP2 SNP3 SNP4

Pre-test risk (baseline risk)

Race and gender specific

Relative risk Absolute risk Risk category(as “Low”, “Average”,

“High”)

20%Race=European

Gender=Male

1.5 30%

(1.5 x 20%)

“High”

52

2. Basic Concepts Absolute risks, relative risks

Likelihood ratios, odds ratios

Test with more than two outcomes

53

Consider Test with Two Outcomes (Pos., Neg.)

D+ D- Total

T + 70 160 230

- 30 240 270

Total 100 400 500

Clinical Performance of the Test

Sensitivity 70.0% (70/100)

Specificity 60.0% (240/400)

Let us have 500 subjects who are representative subjects from intended use population (target population). Each subject has results of the Test (Pos., Neg.) and a Clinical Reference Standard (D+, D-). Prevalence of 20% (100/500) reflects prevalence in the IU population.

54

Risks (Absolute Risks)

Clinical Performance of the Test

R1=Risk of D+ for T+ (PPV)* 30.4% (70/230)

R0=Risk of D+ for T- (1-NPV)* 11.1% (30/270)

π= Pre-test risk (baseline risk, prevalence, average risk (averaged over other risk factors))

20.0% (100/500)

D+ D- Total

T + 70 160 230

- 30 240 270

Total 100 400 500

*Post-test risk for T Pos, post-test risk for T Neg.

55

Relative Risks

R1/π = 1.52 : For a subject with T+, the risk increases by

1.52 times with regard to pre-test risk (=30.4/20.0); R0/π = 0.56 : For a subject with T-, the risk increases by 0.56

times (decreased by 1.80 (1/0.56) times) with regard to pre-test risk (=11.1/20.0); R1/R0 = 2.74 : For a subject with T+, the risk increases by 2.74

times with regard to the subjects with T- (=30.4/11.1)

Clinical Performance of the Test

R1=Risk of D+ for T+ (PPV) 30.4% (70/230)

R0=Risk of D+ for T- (1-NPV) 11.1% (30/270)

π= Pre-test risk 20.0% (100/500)

56

11

1

1)|Pr(1

SeSp

PPVTDR

1

11

11)|Pr(0

SeSp

NPVTDR

Absolute risks and relative risk depend on the sensitivity, specificity and also on the pre-test risk.

D+ D- Total

T + 70 160 230

- 30 240 270

Total 100 400 500

Se Sp

R1

R0

57

“Odds” are the ratio of the probability of one outcome to the probability of its opposite outcome.

Example: Single fair coin with outcomes {Head, Tail}: odds =1 because Pr (Head)=0.5 and Pr (Tail)=1-0.5=0.5 => odds=1 (0.5/0.5=1).

Likelihood Ratios (LR) are another way to describe the performance of a test.

Likelihood Ratios, Odds Ratios

58

Subject from the IU population with pre-test risk π, two outcomes (D+, D-); Pr(D+) = π and Pr(D-)=1-π.

-1

oddstest -Pre

Likelihood Ratios, Odds Ratios (Continued)

After the test is performed (with knowledge of the test results):

1

1

1)T|Pr(D-1

)|Pr( )odds(Ttest -Post

R

RTD

0

0

1T-)|Pr(D-1

)|Pr( odds(T-)test -Post

R

RTD

Is there a relationship between post-test odds and pre-test odds?

59

1

)(1 1

1 TLRR

R

1

)(1 0

0 TLRR

R

Post-test odds = Likelihood Ratio x Pre-test odds

Likelihood Ratios, Odds Ratios (Continued)

Sp

SeTLR

1)(

Sp

SeTLR

1)(

60

Post-test odds = Likelihood Ratio x Pre-test odds

Likelihood Ratios, Odds Ratios (Continued)

Sp

SeTLR

1)(

Sp

SeTLR

1)(

)(

)((OR) Ratio Odds

TLR

TLR

Likelihood Ratios do not depend on the pre-test risk.

Odds Ratio does not depend on the pre-test risk.

61

Consider Test with More than Two Outcomes.

Let us have 500 subjects who are representative subjects from the intended use population (target population). Each subject has results of the Test and a Clinical Reference Standard (D+, D-). Prevalence of 20% (100/500) reflects prevalence in the IU population.

In the hypothetical example, the test examines four markers: each marker has three possible results (aa, Aa, AA) Then the test has 81 possible results (=3 x 3 x 3 x 3).

For the sake of simplicity, consider test with three outcomes: as (Result1, Result2 and Result3).

62

Test with Three Outcomes: as (Result1), (Result2) and (Result3).

D+ D- Total Risk

TResult3 24 72 96 25.0%Result2 56 216 272 20.6%Result1 20 112 132 15.2%

100 400 500 20.0%

Pre-test odds: 0.200/(1-0.200) = 0.250 Post-test odds(Result3): 0.250/(1-0.250) = 0.333 Post-test odds(Result2): 0.206/(1-0.206) = 0.259 Post-test odds(Result1): 0.152/(1-0.152) = 0.179

Is there a relationship between post-test odds and pre-test odds?

63

Post-test odds (Resulti) = LR(Resulti) x Pre-test odds

Likelihood Ratios, Odds Ratios

D-) |Pr(Result

)D|Pr(Result)LR(Result

i

ii

LR is a way of quantifying how much given test result changes the pre-test (baseline) risk of the target condition.

D+ D- Risk D+ D- LR

TResult3 24 72 96 25.0% 24.0% 18.0% 1.33

Result2 56 216 272 20.6% 56.0% 54.0% 1.04

Result1 20 112 132 15.2% 20.0% 33.0% 0.61

100 400 500 20.0% 100% 100%

64

Likelihood Ratios, Odds Ratios (Continued)

ORs are usually considered with regard to the Result with the lowest risk (normalized to the lowest risk): ORi=LRi/LR1. LRs are related to the pre-test risk (average risk).

D+ D- D+ D- LR OR

TResult3 24 72 96 24.0% 18.0% 1.33 2.18

Result2 56 216 272 56.0% 54.0% 1.04 1.70

Result1 20 112 132 20.0% 33.0% 0.61 1.00

100 400 500 100% 100%

65

Summary

Risks and relative risks Risks and relative risks depend on corresponding likelihood ratios and pre-test (baseline) risk.

Because risks (and relative risks) depend on pre-test risk, in some study designs, they cannot be estimated (as in case-control studies).

Risks and relative risks measure probabilities of events in a way that is interpretable and consistent with how the people think.

66

Likelihood Ratios (LR) and Odds Ratio (OR) LRs and ORs do not depend on the pre-test risk.

Because they do not depend on the pre-test risk, LRs and ORs can be calculated even in the case-control studies.

It is easy to adjust an ORs for other variables (logistic regression)

LRs and ORs are more difficult for interpretation because they are related to pre-test and post-test odds, which are not intuitive.

Summary (Continued)

67

3. Description of Typical DTC Risk Assessment Test

For the sake of simplicity, consider a test which measures four markers: each marker has three possible results: (aa, Aa, AA). Then the test has 81 possible results (=3 x 3 x 3 x 3).

Post-test Odds = Likelihood Ratio x Pre-test Odds

1

),,,(1 ,,,

,,,lkji

lkji

lkji DCBALRR

R

Consider that an individual has test result (Ai, Bj, Ck, Dl).Basic idea of calculation of the risk for this individual is

68

),,,( lkji DCBALRNote 1

For a given race/ethnicity, information from case-control studies in published literature is used (independent confirmations of GWAS)

1) Even for the same set of published papers related to the target condition (disease), different markers (SNPs) can be included in the test (different approaches for selection of SNPs are used).

2) Even for the same set of published papers and for the same SNP included in the test, different OR estimates can be used in the calculation of the LR for the test result (different approaches are used). For example, estimates of OR are: 1.2 in paper 1; 1.4 in paper 2; 1.1 in paper 3

(study with largest sample size? meta-analysis? …)

3) Information about OR in the case-control studies is used for calculation of LR (OR with regard to the average risk). Different assumptions are considered. For example, an assumption that “Controls” are not

subjects without disease but a random sample from population.

69

),,,( lkji DCBALRNote 2

Consider the test for the target condition with 4 markers (SNPs) for a given race/ethnicity; ORs for individual markers are obtained from literature.

SNP1 SNP2 SNP3 SNP4

LRResult3 1.27 1.55

Result2 1.05

Result1 0.77

Multiplicative Model: an assumption that all four SNPs are independent (no interactions). This assumption may be not correct.

59.155.177.027.105.1),,,( lkji DCBALR

70

Note 3 Pre-Test Risk, π

Absolute risk Ri,j,k,l is calculated based on corresponding LR and the pre-test risk (average risk).

Pre-test risk is provided based on the publicly available information about race- and gender- specific lifetime risks (for example, Surveillance Epidemiology and End Results (SEER) Cancer Statistics Review).

Pre-test risk (average risk) is gender- and race- specific (very limited number of factors). The average risks present risks averaged over other important risk factors (such as family history, smoking, environmental factors and so on). => An individual pre-test risk taking into account other important factors can be very different from the average risk.

71

Note 3 Pre-Test Risk, π (Continued)

Race and gender specific risk averaged over other

important factors

5% 20% 35%

Pre-Test Risk 5% 20% 35%

LR 1.5

Post-Test Risk* 7.3% 27.3% 44.7%

Increase in Risk 2.3% 7.3% 9.7%

Relative risk 1.46 1.36 1.28

Subjects of the same gender and race and with low

risk factors

Subjects of the same gender and race and with high

risk factors

1

5.11 R

R*

Hypothetical Example

R

risk Relative

72

Absolute values of the post-test risks are considerably affected by the pre-test risk. In hypothetical example of LR=1.5 and pre-test average risk =20% (range 5%-35%), post-test risks were from 7.3% to 44.7%.

Relative risks are also affected by the pre-test risks but to much lesser degree. In hypothetical example of LR=1.5 and pre-test average risk =20% (range 5%-35%), relative risks were from 1.28 to 1.46.

If the pre-test risk is very low, then relative risk ≈ LR In hypothetical example of LR=1.5 and pre-test average risk =3% (range 1%-5%), relative risks were from 1.46 to 1.49.

If there is an assumption that “Controls” in the case-control study are not subjects without disease but a random sample from the intended use population (this assumption may be not correct), then relative risk = LR.

Note 3 Pre-Test Risk, π (Continued)

73

Note 4 Risk Categories “Low”, “Average”, “High”

Various approaches (based on either relative risks, or likelihood ratios, or absolute risks) and different cutoffs can be used for defining these three categories.

Pre-test risk 20%

LR Low Average High 0.80 1.00 1.20

RR Low Average High

Risk Low Average High

0.833 1.00 1.154

16.67% 20.0% 23.08%

Hypothetical example:

The same person can be classified into different risk categories.

74

Risk assessment tests report an absolute risk for an individual.

Note that, with regard to the study designs, the absolute risks cannot be evaluated in the case-control studies.

For the absolute risks, a clinical validation includes two aspects: discrimination and calibration.

4. Clinical Validation

75

4. Clinical Validation (Continued)DiscriminationUnder discrimination, we understand the ability of the test to discriminate between subjects who have target condition and subjects who do not have one. We would like that the subjects with target condition have higher values of the absolute risk compare to the absolute risks of the subjects without target condition. For assessing discrimination, a receiver operating characteristic (ROC) analysis is used (ROC curve and area under ROC curve along with 95% confidence interval).

As a general rule, the larger is the area under ROC curve, the better is the discriminatory capability of the test.

76

ROC AnalysisTest Variable: absolute risk values for Diseased group (Y);

absolute risk values for Non-Diseased group (X)

AUC = P{Y>X}, probability that an absolute risk of a randomly selected Diseased subject is larger than an absolute risk of a randomly selected Non-Diseased subject.

X Y

Consider, for example, a wrong pre-test risk and correct pre-test risk (all other calculations are the same): ROC curves are the same.

77

CalibrationAbsolute risks should be well-calibrated. If one has 100 subjects and the test is telling that that their risk is 12%, then one can anticipate that among these 100 subjects, approximately 12 subjects will have the target condition in reality.

Calibration evaluates the degree of correspondence between the risk of the target condition provided by the test (Expected according to the absolute risk by the test) and the actual risk of the target condition (Observed). Calibration of the test which provides absolute risks cannot be evaluated in the case-control studies.

4. Clinical Validation (Continued)

78

Summary

1. Absolute and relative risks provided by the DTC risk assessment tests are calculated based on different approaches what can lead to inconsistencies in the results.

2. Absolute risks depend considerably on the pre-test risks.

3. Absolute risks in the DTC risk assessment tests do not include important risk factors other than markers measured by the DTC risk assessment tests and some limited number of factors (as race, gender, sometimes age).

79

Thank You!

Marina.Kondratovich@fda.hhs.gov