The Nitty Gritty of Test Validation/Verification

Roche User Group 2019Emma Liu

In Vitro Diagnostic Medical Device (IVD)

• is defined in Directive (98/79/EC) as: any medical device which is a reagent, reagent product, calibrator, control material, kit, instrument, apparatus, equipment, or system, whether used alone or in combination, intended by the manufacturer to be used in vitro for the examination of specimens, including blood and tissue donations, derived from the human body, solely or principally

• for the purpose of providing information:– concerning a physiological or pathological state, or– concerning a congenital abnormality, or– to determine the safety and compatibility with potential recipients, or– to monitor therapeutic measures.

FDA cleared & approved test system

CE IVD

• "Conformité Européene" which literaturely means "European Conformity“

• The CE mark is seen as a declaration by the manufacturer that the product meets all of the appropriate provisions of the relevant legislation.

• EN ISO 17511: In vitro diagnostic medical devices – Measurement of quantities in biological samples – Metrological traceability of values assigned to calibrators and control materials.

• EN ISO 18153: In vitro diagnostic medical devices – Measurement of quantities in samples of biological origin – Metrological traceability of values for catalytic concentration of enzymes assigned to calibrators and control materials (ISO/DIS 18153:2000).

FDA cleared & approved test system

• What does the FDA do for you? – Ensure that the indications for use (IFU) are justified – Evaluate initial test performance (manufacturer’s claims)– Ensure continued performance at the stated manufacturer’s claims– Establish a floor for “state of the art”

• What does the FDA NOT do for you? – Ensure that your patient needs are met Although a common level of performance is typically ensured across different manufacturers’ products that have the same indications for use, this may not provide adequate performance in specific situations – Implementation of a method is a director’s decision

Regulatory New Zealand for medial devices

• MEDSAFE: New Zealand Medicines and Medical Devices Safety Authority

• Responsible for the regulation of medicines and medical devices in New Zealand

• Ensure that medicines and medical devices are acceptably safe

• Australian equivalent is TGA

Standards & Guidelines

• ISO 17025, General requirements for the competence of testing and calibration laboratories.

• ISO 15189:2003, Medical laboratories – Particular requirements on quality and competence

• Clinical Laboratory Improvement Amendments (CLIA) of 1988

• The Clinical and Laboratory Standards Institute (CLSI, formerly NCCLS)

• National Pathology Accreditation Advisory Council (NPAAC)

• CAP Guidelines

• IANZ

Manufacturer

Laboratory

FDA

CE IVD

TGA

Medsafe

CAP

NPAAC

IANZ

ISO 13485

ISO 17025

ISO 17511

ISO 18153

ISO 15189

CLIA

CLSI

Assay Validation

• A process that determines the fitness of an assay that has been properly developed, optimised and standardised for an intended purpose.

• Includes estimates of the analytical and diagnostic performance characteristics of a test.

Three types of test systems - CLIA regulations

• Test systems that are cleared or approved by FDA and run by the laboratory without modification.

• Test systems that are cleared or approved by FDA and run after modification by the laboratory.

• Test system that are not subject to FDA clearance or approval.

Intended use

• The “intended use” of the new test must be defined as part of the test design.

• Intended uses may be to diagnose disease, confirm a serologic diagnosis, evaluate the effect of therapy, etc.

• The use of a test, once validated, must be restricted to the stated purposes(s).

• The intended patient population and specimen types should also be fully described.

Intended use

• cobas® HBV is an in vitro nucleic acidamplification test for the quantitation of hepatitisB virus (HBV) DNA in human EDTA plasma orserum of HBV-infected individuals using theautomated cobas® 4800 System for specimenprocessing, amplification and detection. This testis intended for use as an aid in the managementof patients with chronic HBV infectionundergoing anti-viral therapy. The test can beused to measure HBV DNA levels at baseline andduring treatment to aid in assessing response totreatment. The results from cobas® HBV must beinterpreted within the context of all relevantclinical and laboratory findings

INTENDED USE

• The MagNA Pure 24 System is an automated nucleic acid purification system consisting of the MagNA Pure 24 instrument, software, consumables and reagents. The MagNA Pure 24 System is intended for use by professional users for the purification of nucleic acids from biological samples for in vitro diagnostic purposes. The MagNA Pure 24 Total NA Isolation Kit is for use with the MagNA Pure 24 System.

INTENDED USE

LDT vs In-house assay & Modification vs Off-label use

• A laboratory developed test (LDT): is a type of in vitro diagnostic test that is designed, manufactured and used within a single laboratory.

• In-house assay: the assay has been developed and validated in your own laboratory and does not have the optimisation an validation data published for independent review.

• Modification: Changes in test components (extraction, amplification, and/or detection), procedural parameters, assay cutoff values, specimen types or collection devices, etc.

• Off-label use: In the United States, the regulations of the Food and Drug Administration (FDA) permit physicians to prescribe approved medications for other than their intended indications. This practice is known as off-label use.

Terminology - Validation

• Accuracy

• Precision

• Reportable range

• Reference interval

• Analytical sensitivity

• Analytical specificity

Accuracy

• Definition: refers to the closeness of the agreement between the results of a single measurement and the true value of the analyte.

• Trueness: when referring to the closeness of the agreement between the average value obtained from a large series of measurements and the true value (if there is an international standard) or accepted reference value (if there is not an international standard) of a measurement.

• Bias: Trueness is expressed numerically as bias (lack of agreement). Bias is inversely related to trueness and refers to the average deviation from the true value due to non-random effects caused by a factor(s) unrelated by the independent variable.

Accuracy study

• Comparison-of-methods studyspecimens are tested in parallel with both the new test and a valid comparative method.

• Recovery StudyFor new tests with no comparison method. Samples may need to constructed for testing by adding known amounts of analyte to negative specimens. The amount of analyte recovered is then compared to the amount added to the specimen.

Accuracy study

• Sample sources- Residual patient specimens, standards, quality control, reference

panels, etc.- Distributed over the clinically meaningful range of the test, and should

include both positive and negative specimens- Population, clinical conditions, age and gender, etc.

• Sample handling- Collected and handled according to accepted laboratory practice- Test within the stability of the analyte if possible

• Number of samples- No fewer than 20- A minimum of 100 specimens is suggested for manufactures. - Many factors to be considered, complexity, prevalence, cost, etc.

Data analysis and criteria for acceptance

(i) Assays with a gold standard- Equivalence & commutability, incompatible- Scatter diagram with correlation and regression analysis- A correlation coefficient of greater than 0.975 (or equivalently, r2

≥0.95) is generally accepted for analytical chemistry. Acceptable correlation coefficients have not been defined specifically for molecular assays.

- Difference plot, e.g. Bland and Altman approach

(ii) Qualitative or quantitative assays without a suitable comparator.- two-by-two table

2x2 table FALSE POSITIVE, FALSE NEGATIVE, TRUE POSITIVE & TRUE NEGATIVE

Sensitivity = TP/(TP+FN)

Specificity = TN/(FP+TN)

PPV = TP/(TP +FP)

NPV = TN/(FN +TN)

Precision

Definition: The term “precision” refers to how well a given measurement can be reproduced when a test is applied repeatedly to multiple aliquots of a single homogeneous sample.

Precision is defined as the closeness of agreement between independent test/measurement results obtained under stipulated conditions.

Repeatability vs Reproducibility

Repeatability and Reproducibility are considered to be the extreme measures of precision.

• Repeatability

within-run imprecision, being the smallest measure of precision and involving measurement carried out under the same conditions (same operator, reagent lots, instrument, laboratory, time, etc.)

• Reproducibility

run-to-run imprecision, day-to-day imprecision, etc. being the largest measure of precision and involving results of measurement under changed conditions (different operators, reagent lots, time, laboratory, etc.)

Precision

Qualitative assay - testing the positive and negative control material for 20 runs

or in duplicate for 10 runs.

Quantitative assay - multiple separate test runs.

Qualitative tests

• Precision studies for qualitative tests should provide an estimate of the imprecision of the method at analyte concentrations near the limit of detection.

• It is not appropriate to measure the imprecision of a qualitative assay with high-positive samples.

• CLSI document EP12-A2, protocol for performing a precision experiment at analyte concentrations near the limit of detection.

• Three samples, LoD, LoD – 20%, LoD + 20%

• Replicated up to 40, fewer is acknowledged due to cost-effective

Quantitative tests

• CLSI documents EP5-A2 suggests a high-level sample, a low-level sample and a sample as close as possible to a medical decision level should be tested.

• Duplicate twice a day over 20 working days

Sample sources

• Test materials could include standards, quality control materials, proficiency testing samples, or patient specimens

• The samples should be selected or prepared in a matrix as close as possible to the appropriate clinical specimens.

• Sample must be adequately stored, generally at -20 °C -70 °C or lower, to ensure stability over time.

• Number of samples, a larger number of observation leads to more confidence in an estimate.

Data analysis

• Standard deviation (SD) & Coefficient of variation (CV)

Reportable range

• Definition: refer to the span of test result values over which the laboratory can establish or verify the accuracy of the instrument or test system measurement response.

• Measuring interval, analytical measurement range, and linear range.

• Study: - testing a series of samples of known concentrations or a series of

known dilutions. - The measured values are compared to the assigned values by plotting

on the y axis and x axis. - Check precision, 2-4 replicates.- CLSI recommends 7-11 concentrations - Carried out for each sample type

Reportable range

• Data analysis– Linear regression analysis– Polynomial regression analysis

• y = a +bx (linear) • y = a + b1x + b2x 2 (quaratic, parábola)• y = a + b1x +b2x 2 + b3x 3( Cubic, S- shape)

– Degrees of freedom and t test

Linear range – cobas® HBV

Linear range from 10.0 IU/mL to 1.0E+09 IU/mL

Reference Interval – “normal range”

• The range of values typically found in individuals who do not have the disease or condition that is being assayed by the test.

• If a nucleic acid target is always absent in a healthy individual and the test is a qualitative test, the reference range is = negative or not detected.

• For quantitative assays, the reference interval will be reported as below a particular measurement, usually either the limit of detection or the limit of quantification.

• For some analytes, the reference interval may be different from the limit of detection or quantification, and a clinical decision limit may be used instead. e.g. CMV < cutoff

Analytical Sensitivity

• Definition: the ability of the assay to detect very low concentration of a given substance in a biological specimen.

• Often referred as the “limit of detection”(LOD)

• For both quantitative and qualitative tests.

• LOD is expressed as a concentration, e.g. copies/ml

Study:– Blank specimens – limited of blank (LOB)– Low-level specimens – LOB = meanblank +1.645SDblank; LOD = LOB + 1.645SDlow-concentration sample

– Each type of specimen matrix – Genetic variants, such as HCV genotypes– Data analysis: probit analysis– Multiplex assays

Analytical Sensitivity

Limit of Detection

• Each genotype of HCV

Analytical Specificity

• Definition: The analytical specificity of an assay is different from the diagnostic specificity of an assay.

• Analytical specificity refers to the ability of an assay to detect only the intended target and that quantification of the target is not affected by cross-reactivity from related or potentially interfering nucleic acids or specimen-related conditions.

• Diagnostic Specificity: refers to the percentage of individuals who do not have a given condition and are identified by the assay as negative for the condition.

Interference studies

• Cross-reactivity: Organisms that should be tested to rule out potential cross-reactivity include organisms with similar genetic structure, normal flora organisms that could concurrently be present in the specimen, and organisms that cause similar disease states or clinically relevant coinfections.

• Interfering substances: refers to the effect that a compound other than the analyte in question has on the accuracy of measurement of an analyte.

Package insert - cobas® HBV Quantitative nucleic acid test for use on the cobas® 4800 System

References – not limited

• Evaluation of Precision Performance of Quantitative Measurement Methods; Approved Guideline—Second Edition CLSI EP5-A2

• Validation of Laboratory-Developed Molecular Assays for Infectious Diseases. Eillen M.Burd. Clin Microbiol Rev, 2010 Jul; 23(3):550-576

• ISO15189:2012

• FDA.gov

• CE- marking.gov

• CAP.org

• IFU of Cobas HBV, HCV, HSV 1/2

• IFU of MagNA Pure 24 Total NA Isolation Kit package insert

Acknowledgments

• Dr. Fabrice Merien, Head of Department Biomedicine Medical diagnostic, AUT.

• Jenny Bennett, Technical Head Molecular Biology, Waikato Hospital Laboratory.

• Bernd Unterweger, GCS Senior Specialist, Roche Diagnostics GmbH

Doing now what patients need next