white paper on designverification

Design Verification – The Case for Verification, Not Validation

of 12

MEDIcept, Inc. 200 Homer Avenue Ashland, MA 01721

May not be reprinted or copied without expressed permission from MEDIcept 11/2010

Overview:

The FDA requires medical device companies to verify that all the design outputs meet the design inputs.

The FDA also requires that the final medical device must be validated to the user needs. When there are

so many more design inputs and outputs than specific user needs, why do companies spend so little

time verifying the device and so much time and money on validation? And what is the role of risk

management in determining the amount of testing required? This presentation will demonstrate that if

developers conduct more complete verifications of design outputs and risk mitigations, validations can

be completed in a shorter time, more reliably, and more successfully.

Introduction:

In our experience working with medical device manufacturers to improve their Quality Management

Systems and to gain regulatory clearance of new devices, we have found that Design Verification is an

often underutilized tool for ensuring success during the latter stages of device development efforts. In

many cases, limited verification efforts represent a lost opportunity to significantly reduce the scope of

validation efforts – and thereby reduce time to market and development costs.

This paper explores the use and misuse of Design Verification and how device manufactures can get the

most out of their verification efforts. But first, some background . . .

Background:

Medical device manufacturers have been working to align their design and development systems with

the FDA’s design control regulations (21 CFR 820.30) since their release in 1996. The regulations are

structured to ensure that device manufacturers maintain control over their designs throughout the

development process and that the marketed device is safe and effective for its intended use. At their

most basic level, the regulations require that manufacturers:

Clearly state what they intend to produce (Planning and Design Inputs);

Develop a design that meets those needs (Design Outputs and Design Review);

Confirm that the design meets the original intent (Verification);

Confirm that manufactured product can be produced reliably and achieve the desired result

(Transfer and Validation); and


of 12



Maintain records of key development activities and decisions (Design Changes and Design

History File)

It would be hard to argue that any of these steps should be excluded from any development effort.

Whether you are building a bridge, a cell phone, or a surgical tool, this combination of steps has a logical

flow that most engineers would simply consider to be “good practice”.

The challenge arises when you move from the conceptual world, where product development can be

seen as single stream of water flowing down a hill: steadily moving from one point to the next without

interuption; to the real world, full of rapids, eddies, branching streams, and (occasionally) deep pools of

water that don’t seem to be moving at all. In this turbulent environment, it’s can be difficult to maintain

a disciplined design approach – particularly when business schedules demand rapid progress and any

delays in moving on to the next development phase risk affecting project milestones and development

staff deployment plans. These challenges are multiplied when the device has multiple components and

integration points that must be managed throughout the development process.

Much has been written about the importance of early stage planning and problem solving to speed

time to market and reduce development costs – and the value of clear Design and Development Plans

and well defined Design Inputs cannot be overstated. An equal amount of attention has been focused

on Design Transfer and Validation activities. At this late stage of a development effort, the project scope

will have expanded to include more active involvement of Clinical, Operations and Marketing staff. In

addition, the cost of validation studies and the “make or break” aspect of these studies require a great

deal of attention and company resources. Problems at the validation stage will have serious implications

for the success of the project and especially for small companies could determine the fate of the

whole company.

Compared to these early and late stage activities, the importance of Design Verification tends to be

overlooked. There are several reasons why:

Definitions: Many practitioners (and established Quality Systems) continue to have trouble

defining exactly what “verification” is, and how it fits into the design control process. The term is

often considered to be synonymous with “validation” and the development of combined

“V&V” plans that do not establish a clear distinction between the objectives of the two efforts

don’t help. [Note: when discussing verification with the FDA, it’s best not to talk about your

“V&V plan” – increasingly, inspectors’ preferences are to address the two activities separately.]

Timeframe: Verification is often an ongoing effort conducted throughout the development of

outputs. So, a great deal of time may pass between when the first and last output is verified.


of 12



With this extended timeframe, verification activities can get lost within all of the “churning”

associated with developing final outputs.

Approaches: There are a variety of ways to complete the verification for an output, so it can be

difficult to communicate that all of these activities, as a group, represent the design verification.

The problem with this lack of attention is that it weakens a critical link in the design control “chain”,

affecting the strength of the overall design control effort. Poor design verification can lead to problems

during design transfer and validation, and can reduce your ability to track down and correct problems if

(and when) they occur. Just as importantly, it may represent a lost opportunity to optimize validation

activities, reduce time to market, and increase your overall confidence in the safety and efficacy of your

products.

Key Concepts:

To make sure that we’re aligned on the proper use of the term “verification”, here are a few key points

from the FDA’s Design Control Guidance For Medical Device Manufacturers:

Definition: Verification means confirmation by examination and provision of objective evidence

that specified requirements have been fulfilled [§820.3(aa)].

Types of Verification Activities: Verification activities are conducted at all stages and levels of

device design. The basis of verification is a three pronged approach involving: tests, inspections,

and analyses.

Any approach which establishes conformance with a design input requirement is an acceptable

means of verifying the design with respect to that requirement. In many cases, a variety of

approaches are possible . . . the manufacturer should select and apply appropriate verification

techniques based on the generally accepted practices for the technologies employed in their

products.

Table 1 provides some examples of the types of verification activities that device manufacturers often

employ.


of 12



Table 1: Types of Verification Activities

Tests Inspections Analyses

o Material performance / fatigue tests

o Package integrity tests* o Biocompatibility testing of

materials* o Bioburden testing of products

to be sterilized*

o First article inspection o Comparison of a design to a

previous product having an established history of successful use*

o Worst case analysis of an assembly to verify that components are derated properly and not subject to overstress during handling and use*

o Thermal analysis of an assembly to assure that internal or surface temperatures do not exceed specified limits*

o Fault tree analysis of a process or design*

o Failure modes and effects analysis*

o Engineering analyses: o Finite Element Analysis

(FEA)o Computational Fluid

Dynamics (CFD) o Tolerance stack-up

* Source: Design Control Guidance For Medical Device Manufacturers (1997)

A key distinction between design verification and design validation activities is that verification only

requires that a single unit be assessed. What constitutes that single unit will vary depending on the

intent of the verification. It might be one batch of raw material (for material performance tests), one

machined part (for first article inspection), on one package sample (for integrity tests). The intent of

verification is to confirm that the design outputs (i.e., the materials or components specified in design

documents) meet the design input requirements.

Validations, on the other hand, require that studies be conducted to ensure that the device can be

manufactured to meet design specifications on a consistent basis (i.e., process validation), and to ensure

that the finished device is safe and effective for its intended purpose (i.e., design validation). For the

process validation, multiple devices must be manufactured and evaluated to confirm that the

production process is capable of producing devices within specifications on a consistent basis. For the

design validation, multiple devices must be used to treat multiple patients to confirm that the treatment

is safe and effective.


of 12



The number of devices that need to be produced and the number of patients that need to be treated is

a function of the risk associated with the particular aspect of the device and the need to establish

confidence in the study results. As we will see, the effective use of risk analysis tools and solid

verification results can help to reduce the size of the validation studies without negatively affecting the

confidence in the study results.

What Goes Wrong?

As described above, too often design verification does not receive the amount of attention needed to

ensure success in the latter stages of a device development program. While not a perfect representation

of what goes on in all device companies, it is interesting to look at the FDA’s design control audit

findings to see what problems they have found with companies’ design verification programs. Figure 1,

shows the number of warning letters in the FDA’s database that include findings related to specific

sections of the Design Control regulation (21 CFR 820.30).

Figure 1: Design Control Warning Letters


of 12



Interestingly, the most observations were found with regard to the Design Change, Design Validation,

and the General design control sub headings (the “General” findings suggest an overall failure of the

company’s design control process). We argue that failures in these three areas are largely the result of

poor performance in the earlier stages of the program (e.g., validation failures due to poorly structured

input requirements, outputs and incomplete verification; and design change failures due to an

ineffective system to update verifications and validations when changes are made).

Of the remaining categories, most warning letters cite 21CFR 820.30(f) – Design Verification. In these 64

warning letters, the FDA identified that the manufacturer did not complete all required verifications in

nearly 45% of the cases. A lack of (or significant gaps in) verification procedures were identified in about

25% of the warning letters. In addition, out of specification results, a lack of records in the DHF, and a

lack of acceptance criteria were found in 16%, 14%, and 14% of these companies, respectively. Figure 2

provides a breakdown of all of the types of violations identified in these warning letters.

Figure 2: Types of Design Verification Violations


of 12



Note: 92 separate violations were identified in the 64 warning letters.

Why are these failures occurring? One possibility is that by viewing verification as a “regulation”, device

developers are losing sight of the fact that conducting verifications is simply “good engineering”. For

example, it makes no sense to move forward with the development of a component before you are sure

that the material properties meet the performance and safety requirements established in the design

inputs.

This “meeting the regulations” mindset can lead engineers to view design verification simply as a task

that must be checked off by QA instead of valuable tool for building knowledge about their device.

Failures occur because sometimes the “checkoffs” get missed.

Even when all the boxes are checked, the “regulation” mindset can drive engineers to focus only the

minimum number of verification activities needed to satisfy the design control requirements. While this

approach may help satisfy short term budget or schedule constraints, the development tea m will lose

the opportunity to learn potentially important information about the performance and behavior of its

device and its components. If that learning is put off to the validation stage, the cost of failure can grow

dramatically.

What To Do?

There are a variety of tools that developers can use to improve the effectiveness of their design

verification process. In this paper we discuss three tools that can help ensure that verification activities

are appropriate and complete and, if well documented, can provide regulators with sufficient

confidence in the design that excessive validation can be avoided. The three tools are

Traceabilty Matrix

Failure Modes and Effects Analysis (FMEA)

“Fault Tree” of Design Inputs

Traceability Matrix: The traceability (trace) matrix is a basic design control tool that all device

developers should use throughout the design control process to establish clear linkages between Design

Inputs, Outputs, Verifications, Validations and Risk Analyses. Too often the trace matrix is left for QA to

complete just in time for the final design review. However, if it is begun as soon as Design Inputs are

approved and managed throughout the development effort, the trace matrix provided an excellent

roadmap – guiding developers through key steps of the design control process and ensuring that

required documentation is created and controlled.


of 12



To illustrate how the trace matrix can be used, Table 2 shows what one row in a trace matrix might look

like when the matrix is first developed and approved. At this stage of the development effort, the matrix

is solely a list of the design inputs (the reference number from the Product Requirements Document,

and a description). At this stage there is no information about how that design input will be satisfied, but

it provides a roadmap for what questions need to be asked and what documents need to be developed.

As the project proceeds through the develop stages, subsequent cells will be filled in, identifying how

the design input is being met.

Table 2: Traceability Matrix at Design Inputs Stage

Traceability Matrix

Design

Req.

No.

Design Input

(Requirement)

Design

Output

(Specification)

Risk

Analyses

Verification Process

Validation

Design

Validation

Comments

1.1.1 Can be ETO

sterilized

Table 3 illustrates how a row of a trace matrix might look when being reviewed at the Final Design

Review. At this point the design outputs have been approved and the requirement that the material in

question is appropriate for ETO sterilization is established in Drawing No. 1.2. The matrix also identifies

that identified risks associated with the device have been mitigated (in part) through the use of this

material. The references to design FMEA 1.3 and process FMEA 5.4 identify two places where this

material is addressed. Verification was achieved by confirming that the specified material is included in

the purchasing bill of material (BOM 2.3). The Process Validation column identifies that production of

the material was assessed in an operation qualification (OQ Study 3.5) and a performance qualification

(PQ Study 4.3). Finally, it shows that a design validation (Sterilization Study 2.1) was conducted to

confirm that the final manufactured device could in fact be effectively sterilized using ETO.

Table 3: Traceability Matrix at Final Design Review

Traceability Matrix

Design

Req.

No.

Design Input

(Requirement)

Design

Output

(Specification)

Risk

Analyses

Verification Process

Validation

Design

Validation

Comments

1.1.1 Can be ETO

sterilized

Material A

(Dwg 1.2)

dFMEA

1.3

pFMEA

5.4

BOM 2.3 OQ Study

3.5

PQ Study

4.3

Sterilization

Study 2.1

None


of 12



The trace matrix now serves as a clear record of the chain of analyses and studies used to ensure that

the specific design input is met. All of the documents referenced in the trace matrix are included in your

design history file (DHF), so if a question arises regarding any stage in the process, the trace matrix

points to the key documentation developed at that stage.

While the trace matrix is a necessary and valuable tool, it largely serves just a recordkeeping role. It does

nothing to inform the team about how to prioritize verification and validation efforts. For that input, risk

analyses are needed.

FMEA: Like the trace matrix, risk analyses need to be started early in the design and development

process and filled in and enhanced as project proceeds. While this paper will not discuss the details of

the application of risk analyses, and FMEAs in particular, we will address the link between FMEAs and

verification.

Too often, risk analyses are conducted in isolation from the other design control activities. As discussed

earlier, the “regulation” mindset can lead to the perspective that risk analyses are just a required task to

be completed and not as tools to support decision making – which they are.

FMEAs are conducted to provide developers with a shared understanding of the risks associated with

their device – usually from three perspectives: design, use/application, and processing. The result of this

tool, as illustrated in Figure 1, is the classification of risks to people, property, and the environment into

three categories: Broadly Acceptable, As Low as Reasonably Practicle (ALARP), and Intolerable. Typically,

application of this tool focuses on the Intolerable risks and the development of mitigation strategies to

reduce the probability of occurrence for those risks, bringing them into the ALARP region.


of 12



Figure 2: Risk Classification

While risk identification and mitigation is clearly the primary objective of the risk management activity,

these analyses can also be used to focus verification and validation efforts. For example, consider a

device that includes two components, one affects how the device is held by the user and the other

comes into contact with the patient. Both components have a similar number of physical dimensions to

verify, but risks associated with the user facing component are “Broadly Acceptable”, while the patient

facing component risks are “ALARP”. While a First Article inspection of the user facing component may

be appropriate, a more thorough analysis of the patient facing component may be warranted. This

analysis could include a tolerance analysis or the inspection of multiple pieces to provide a better

indication of the potential capability to produce the component within specification.

While the additional efforts described above may not be required at the verification stage (conducting a

First Article would be sufficient to claim that the verification is complete), the ALARP risk classification is

an indication that validation of this component may be challenging. The more you learn about this

component during verification, the better prepared you will be once you get to the validation. If fact, if

the verification is thorough, it may eliminate the need for some aspects of validation – saving time and

money later in the development effort.

The key point is that incomplete or minimal verifications may not provide a sufficient understanding of

the likely performance of a device during validation, and excessive verification can be a waste of

resources. To effective focus your verification efforts, risk analyses, and FMEAs in particular, can provide


of 12



a clear rationale for how to focus your time and resources – potentially reducing validation

requirements.

Fault Tree Analysis: While FMEAs are a well accepted risk analysis tool, one weakness of the approach is

that each risk is considered in isolation. There is no ability to assess the risk of two independent failures

(i.e., dimension A is too short and the user applies too much force). As illustrated in Figure 3, Fault tree

analysis (FTA) allows you to consider the risk of two independent failures occurring in parallel (the

“AND”), and the risk of any one of a series of failures (the “OR”). For example, in this “Bad Coffee”

example, adding “Old Cream” requires the cream to be past its expiration date AND for the coffee

drinker not to read the date. Unsatisfactory serving temperature could be due to the coffee being either

too hot OR too cold.

Figure 3: Fault Tree Analysis

While more challenging to structure and assess than an FMEA, the FTA provides a better understanding

of the risks associated with the “system”. Since the performance of the system is the objective of

validation activities, FTA is an ideal tool to prepare for validations. With the top of the fault tree

representing the intended use of the device, this tool allows device developers to fill out the tree –


of 12



identifying that range of design or usage faults that could lead to a problem. Just as the FMEA helped to

focus attention on independent risks, the FTA helps to focus on system risks.

Once the branches of a tree associated with a significant risk are identified, additional verification

resources can be focused on verifying the design of components linked to that risk. Again, the objective

of verification and validation activities is to build confidence within your organization and with

regulators that your device is going to be safe and effective. A well structure FTA combined with

thorough verifications of key components can be instrumental in the identification and resolution of

problems early in the development process, allowing your team to enter into the validation phase of the

process with confidence in the performance of the device and to focus its validation on those system

elements that most directly affect user needs.

What are the Benefits?

While it can sometimes get lost in the churning of engineering process, verification is a critical element

of the design control process. While meeting the threshold requirement of documenting a verification

for each design input may help to move the design through the development stages, such an approach

does not provide the value that a strong verification effort could provide. By developing a trace matrix

to ensure that all design inputs are properly addressed and leveraging risk analysis tools, medical device

developers would be better able to focus scarce resources on those verification activities that will

provide the greatest benefit. Well structured verification activities provide the foundation for

validations. If the verifications are sound, validations can be better focused, helping to reduce the scope

of these activities – reducing validation costs and accelerating time to market.

white paper on designverification

Documents

homeravenue ashland