agile and medical device software€¦ · “agile methods could be non-compliant because of the...

Agile and Medical Device Software

Incompatible? or Harmonious Union?

Brian Pate

Agenda

• Why Agile?

• Agile and Medical Device Software

• Lean and Compliant

Copyright SoftwareCPR®


We are uncovering better ways of developing software by doing it and helping others do it.

Through this work we have come to value:

Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

MANIFESTO FOR AGILE SOFTWARE DEVELOPMENT

© 2001, the above authors this declaration may be freely copied in any form, but only in its entirety through this notice

Kent Beck James Grenning Robert C. Martin

Mike Beedle Jim Highsmith Steve Mellor

Arie van Bennekum Andrew Hunt Ken Schwaber

Alistair Cockburn Ron Jeffries Jeff Sutherland

Ward Cunningham Jon Kern Dave Thomas

Martin Fowler Brian Marick

• Continuous planning and

feedback ensure value is

maximized.

• Continuously aligning the

delivered software with

business needs (adapting).

• Continuous testing provides

accurate visibility.

• Focusing on the rapid

delivery of business value

reducing the overall risk.

©2012, VersionOne, Inc. All rights reserved www.versionone.com

Agile & Lean Deliver on 9 Critical ROI Metrics

Copyright © Gear Stream, Inc.

• Improved Efficiency due to Optimized Process

• Higher Productivity due to Improved Morale

• Cost Control due to Superior On-time Performance

• Cost Reduction due to Improved Maintainability

• Higher Quality due to Improved Defect Removal

• Benefit Improvement due to Integral Customer Feedback

• Benefit Acceleration due to Incremental Delivery

• Benefit Recovery due to Superior On-time Performance

• Opportunity Recovery due to Superior On-time Performance

Questions

Agile methods for medical

device software…

• can it be compliant?

• can it be safe?


Questions

“Agile methods could be non-compliant because of the lack of

formal requirements.”

“Agile methods are not suited for medical device software

development because there is no formal verification and

validation.”

“Agile methods are not suited for medical device software

development because there is no formal process for ensuring all

hazards are properly mitigated.”


Resources

• Soon to be released Technical Information

Report from AAMI titled “Guidance on the use

of agile practices in the development of

medical device software”

• IEC 62304:2006 “Medical device software --

Software life cycle processes”


Copyright SoftwareCPR® Copyright SoftwareCPR®

Agenda

• Why Agile?




AAMI/TIR SW1:2012

TIR Content:

• Perspectives

• Concepts

• Practices


AAMI/TIR SW1:2012

Guidance on the use of Agile practices in the

development of medical device software


AAMI/TIR SW1:2012

Why Publish a TIR?

• Information is valuable

• Immediate need


AAMI/TIR SW1:2012

TIR Purpose

• Provide motivation for the use of Agile

• Clarify misconceptions about the suitability of

Agile

• Provide direction on the application of Agile

to meet quality system requirements


AAMI/TIR SW1:2012

What the TIR is not intended as:

• Educational tool for Agile development

practices

• Education tool for quality management

regulations


AAMI/TIR SW1:2012

How should the TIR be used?

• Resource for creation of SW Dev process

– Expert point of view

– Can present different points of view

• Assist with communication within

organization


AAMI/TIR SW1:2012

When will this TIR release?

• Soon …

• AAMI Software committee is reviewing …

hoping for June 1, 2012 meeting


AAMI/TIR SW1:2012

TIR Content:

• Perspectives

• Concepts

• Practices



TIR Perspectives


AAMI/TIR SW1:2012




TIR - Perspectives

The Regulatory Perspective:

• Values safety and performance

• Values process and methods to ensure safety

and efficacy

• Value demonstrable evidence that processes and

methods were used


TIR - Perspectives

Key regulatory principles:

• Established (controlled) processes

• No specific development methodology, but should be methodic

and contain essential activities

• Quality systems depend upon management control, professional

training, and professional work

• Documentation (evidence) necessary to demonstrate compliance

• Planning is important to ensure processes are executed effectively


Regulations/Standards • Governmental Laws (e.g. FDA, MDD, etc)

• Standards (e.g. ANSI/AAMI/ISO 14971:2000, EN

62304:2006) ANSI/AAMI/ISO 14971:2000, Medical

devices—Application of risk management to medical

devices

• ISO 13485:2003, Medical devices -- Quality management

systems -- Requirements for regulatory purposes

• EN 62304:2006


FDA

FDA regulates software in the following areas:

• Medical Devices

• Automation of Production Systems

• Automation of Quality Systems

• Computers used in Clinical Investigations

• Electronic Record & Electronic Signatures


TIR - Perspectives

IEC 62304:2006

• A framework – processes, activities and tasks

– Process is the top level, a process has activities and an activity has tasks. Specific

requirements in IEC 62304 are generally at the task level.

• Identifies requirements for what needs to be done and what

needs to be documented

• Specifies a software safety classification scheme

– Additional requirements apply as safety becomes more important

– Much more significant than a simple minor/moderate distinction


TIR - Perspectives

Do the goals of regulations and

standard align with the goals Agile?

• Agile and regulatory bodies both value

high quality software

• Keep balanced emphasis


TIR - Perspectives

Do the values of regulations and standard

align with the goals Agile?

• The Agile Manifesto is a clear, powerful, and

internationally recognized

• Regulatory value is inferred from regulations


AAMI/TIR SW1:2012

TIR Content:

• Perspectives

• Concepts

• Practices



TIR Aligning on Concepts


AAMI/TIR SW1:2012




TIR - Concepts

Is Agile an Incremental or an Evolutionary

Lifecycle?

• Yes


TIR - Concepts

Agile is Incremental …

• Software system broken into small pieces

• Uses backlog

• Pieces are realized in short increments


TIR - Concepts

But Agile is also Evolutionary …

• Allows product definition to evolve over time

• Detailed definition “emerges”

• Highly overlapping and dynamic


TIR - Concepts

Software Quality Assurance …

• Performed on small pieces

• Continuously applied

• Continuously integrated to demonstrate “sum of

parts” is consistent and a correct whole


TIR - Concepts

Regulations, standards, and guidance

applicable to medical device software neither

mandate nor prefer any particular lifecycle

model.

• Manufacturer to choose

• Should be appropriate to their situation/application


Process Flow, Timing

62304 shows as linear sequence: Requirements

analysis, Architectural design, Detailed design,

• Design Planning

• Design Review documentation

• “…contain historical versions of key DMR

documents to show how design evolved”


TIR - Concepts

Address these in your process:

• Process activities to be executed

• The timing and sequencing of those activities

• The inputs to and outputs of an activity

• The process artifacts to be produced

• How lifecycle integrates with risk management

• How change control and CM will be handled


Layers of Abstraction • Project

– What’s needed to deliver finished product

• Release

– What’s needed to deliver a usable product

• Increment

– What’s needed to deliver a set of useful functionality

• Story

– What’s needed to deliver a small piece of functionality


Configuration Management

Agile approaches to software development

require significantly more discipline in

managing the software system configuration

work products change constantly and frequently


Configuration Management

Benefit of

responding to

customer needs

[requirement]

changes

Pain of added CM

discipline


› greater than!

Defining Done

DONEness criteria should also address:

• Process requirements that must be satisfied

• Reviews to be performed

• Approvals to be obtained

• Product acceptance criteria used to evaluate work product

• Documentation requirements that must be satisfied


On-going Impact Analysis

Work that occurs in any iteration could affect

work previously done in an earlier iteration

• Use the concepts of impact analysis on each iteration

• Practices such as continuous integration, automated

regression testing, design review, and customer demos

provide necessary feedback (non-exhaustive list)


Inputs

Iterations only need to consider a limited set of

inputs

Use the concepts of impact analysis on each iteration

• Practices such as continuous integration, automated

regression testing, design review, and customer demos

provide necessary feedback (non-exhaustive list)


Inputs

Agile’s evolutionary nature allows for inputs to

evolve as the outputs are created

• Common at the start of an iteration to say “the input is

sufficient to begin”

• This is highly subjective and depends heavily on the context

in which it applies and on the team’s judgment to apply it.


AAMI/TIR SW1:2012

TIR Content:

• Perspectives

• Concepts

• Practices



TIR Aligning on Practices


AAMI/TIR SW1:2012




AAMI/TIR SW1:2012

TIR Content:

• Perspectives

• Concepts

• Practices: Topics related to planning


Planning

Is agile too undisciplined to meet planning

requirements?

• Agile approaches are sometimes perceived to

lack disciplined processes and process

execution

This perception is incorrect.


Planning

Agile has the concept of layers of planning

• Planning occurs at the story, the increment,

and the release


Development Planning

• Story – plan for elaboration and implementation

of a specific piece of functionality

• Increment – plan for the all of the functionality

that will be added in a fixed time window

• Release – plan for all the activities that are

needed to finish the software development

project


Planning

• Agile has the concept of the “DONE is DONE”

checklist that is executed and verified.

– This provides feedback on successful completion

of the planned activities.


Planning

• So, is agile suitable for medical device

software?

– Agile teams will find they do need more than just

a Backlog and a Release strategy to address all

regulatory requirements.


Planning

IEC 62304 5.1 Software Development Planning

requires:

• The processes to be used

• The deliverables (included documentation) of the activities and tasks

• Traceability between system reqm’ts, sw reqm’ts, test, and risk control

measures

• Software configuration management and change management (including

SOUP)

• Software problem resolution at each stage of the life cycle


Planning

• Is shippable software, after every increment,

a realistic expectation?

• This would seem to be rarely practical for medical

device software.

• Release hardening and formal verification &

validation could be cost prohibitive.


Planning

• Is the “shippable” practice valuable?

• Act as though it were shippable (minus a few

activities)

• Continuous Integration, Integration testing (for the

increment might be less formal than for the release)

• Regression testing schemes


Verification Planning

Agile’s “Done is Done” concept is core to creating a

verification plan

• Each story needs: elaboration (requirements analysis),

architectural and detailed design, sw unit coding and

verification, sw integration testing, and system testing

• Verification becomes part of the “Done is Done” checklist


Continuous Integration

Continuous Integration typically requires that

configuration management practices are

defined and perfected very early in the project,

and then relentlessly followed.

• Provides immediate feedback if things go

differently than planned.


AAMI/TIR SW1:2012

TIR Content:

• Perspectives

• Concepts

• Practices: Topics related to team structure

and collaboration


Pairing

Pairing is the practice of putting two developers on one

activity.

• Could be applied to requirements, design, test and coding

(commonly applied to coding, i.e. “pair programming”)

• Second person can assume reviewer role

• Perceived to bring extra costs but this is usually offset by

lower defect removal costs later


Pairing

Benefits:

• Improved design: different perspectives

• Effective training/mentoring

• Knowledge retention: if developers leave

• Improve quality: review, conformance

• Form of continuous verification


Pairing

When used to fulfill regulatory or compliance

requirement for review:

• Establish acceptance criteria to ensure the design

output meets design input requirements

• Define the qualifications of the reviewers appropriate

to the task

• Provide appropriate evidence of the review


Pairing

What about independence of review?

• Certain critical, complex, or safety related

software may need additional perspectives to

provide diversity of knowledge and experience to

ensure adequate review.

• Could be accomplished by changing pairing

assignments or supplemental reviews.


Stop the line!

Stop the line is the practice of monitoring processes to

identify abnormalities that require attention.

• Monitor the development process: burndown charts,

regression results wave, tests written/passed charts

• Make information available to team (to everyone!): helps

to ensure full engagement to own and solve problems,

connect remote teams

• Act on the information quickly


Retrospectives/Reflections

Retrospectives and reflections are the practices that encourage a

team to regularly inspect their performance and take action to be

more effective.

• Retrospectives are event driven (e.g. end of iteration, release) and

focus on the preceding iteration/release and explore what could

have been done better.

• Reflections are done periodically and focus on how the team is

functioning.

• Continuous improvement


Collective Ownership

Collective ownership is about the team taking

ownership and responsibility for the quality of

the delivered software product.

• The team owns it all, not just one aspect.

• Equal voice: any team member can address

design, implementation, or test


AAMI/TIR SW1:2012

TIR Content:

• Perspectives

• Concepts

• Practices: Topics related to product definition

and requirements documentation


Stories

When does a story have enough definition to begin

work?

• Story is a high level description of a piece of

functionality

• Must be detailed enough to allow prioritization

• Must be detailed enough to allow developer to

effectively elaborate and implement the story


Stories

• Effective prioritization helps ensure that the

highest value stories are done early in

development.

The presence of an embedded customer/

product owner can facilitate the elaboration

of the story with less documentation.


Acceptance Tests

Can the acceptance tests for stories be used for

final requirements?

• Important to produce requirements documentation that is useful

to both the team and to outsiders such as regulators, auditors, and

even other projects that might build on or interface to this project.

• Stories are generally at a high level

• If stories are updated with the results of the elaboration, could

possibly be used in requirements for the product.


Executable Requirements

Can Executable Requirements be a valid part of the

requirements definition and documentation?

• Executable requirement is an agile technique where requirements are

written in a formal language such that the test(s) for the requirement are

automatically generated.

• Challenge is to provide reviewable requirements documentation to

regulators. Ensure readability by outsider, Use English-like words for your

definition language.


V & V

How does agile help with requirements

verification and validation?

• Agile promotes the active involvement of the customer

• Pair programming allows refinement of requirements as

story is elaborated and implemented

• Demonstrations allow feedback from users on the

adequacy of requirements


AAMI/TIR SW1:2012

TIR Content:

• Perspectives

• Concepts

• Practices: Topics related to software

architecture


Evolving Architecture

Project Layer

• Big picture architecture activities are performed “just

enough” to plan releases, increments, and some stories

Story Layer

• Impacts on architecture are addressed and adjusted as

needed

Underlying process is risk management: Architecture

plays a huge role in effective risk control.


Architecture Verification

• Continuous integration and test can expose

architectural weaknesses early.

• When refactoring happens, the ease or

difficulty of doing it is a reflection of the

maturity and correctness of the architecture.


AAMI/TIR SW1:2012

TIR Content:

• Perspectives

• Concepts

• Practices: Topics related to detailed design


Detailed Design

• Not as distinct an activity as linear-flow

models might suggest.

• Planning documentation or procedures must

define how detailed design is to be

performed.


Emergent Design

• Agile encourages “Simple design” principle -> simple

enough to deliver the functionality of the story.

– Discourages complex designs intended to provide options

for future functionality that is not yet in the product.

• Refactoring is the process by which designs are re-

worked as new functionality is added. Better designs

emerge.


Design Documentation

• Well-written code along with well-written test

cases can itself be good documentation of the

detailed design – separate documentation may

not be necessary.

• Use the “hand-off test” – what verbal

information does a developer say to another

developer when handing off development?


AAMI/TIR SW1:2012

TIR Content:

• Perspectives

• Concepts

• Practices: Topics related to implementation

and unit verification


Test Driven Development

• Unit test is created first, then coding is done

to satisfy the test.

• With pair programming, authors can “test-

drive” each others software.


• Agile practices such as test-driven

development, pair programming, continuous

integration, and continuous testing align to

deliver high quality software.


AAMI/TIR SW1:2012

TIR Content:

• Perspectives

• Concepts

• Practices: Topics related to integration and

integration testing; system testing


Integration

• Continuous integration forces integration from

beginning.

• Techniques such as stubs, software simulations,

hardware emulators and test fixtures allow early

integration and progressive development.

• Integration testing at Story, increment, and

release layers.


System Test

• Agile has the potential to have test results for

many different versions, so it is important to

plan and clearly describe what testing was

done on what version, and how final testing

demonstrates that the final version has been

thoroughly tested.


Regression Testing

Agile has the disadvantage that system testing done on

the story and increment layers may not be repeated on

the final version.

– Concern: a later change may have introduced a defect.

With any incremental model, regression testing is

necessary to demonstrate that changes do not

introduce unintended consequences in software that

has already been tested.


SOUP

SOUP items may be updated throughout the

project due to normal updates, bug fixes,

security patches, etc.

SOUP updates will drive additional regression

testing.


System Testing

• System tests should be developed as the code

is developed, preferably in advance.

• Tests may not fully pass if future stories are

required.

• If code is changed in future story work, tests

must be changed accordingly.


Documenting System Testing

• Regulations and standards require the results of

system testing to be documented.

• If the team is tempted to skip the documentation

work, this should be an indication that the

documentation is too costly or lacking value –

find ways to address both of these issues (e.g.

automated testing).


Traceability

• The value of using TDD practices is that

requirements, verification, and test execution

are very closely linked, thus giving you built-in

traceability.


AAMI/TIR SW1:2012

TIR Content:

• Perspectives

• Concepts

• Practices: Topics related to software release


Software Release

• Short release cycles

• Maybe not “shippable” but “near shippable”

– Usability testing

– Controlled clinical testing


Software Release

Layered Approach

• Story - most documentation is created

• Increment - named/controlled release; v&v

activities; doc package

• Release Layer – additional validation

activities; doc package completed


Software Release

• All known residual anomalies must be

documented.

– While agile doesn’t specifically address this, the

short release cycles and continuous testing will

make anomalies visible early.


AAMI/TIR SW1:2012

TIR Content:

• Perspectives

• Concepts

• Practices: Topics related to configuration

management and change management


SW Configuration Identification

• Standards require a team to document the set

of configuration items and versions that

comprise the software system.

– Continuous integration

Establish software configuration baselines at

Increment boundaries.


SOUP CM

• Acquired software must be captured and

archived in a suitable manner.

• If SOUP items are updated, these updates

must be captured.

Establish software configuration baselines at

Increment boundaries.


Change Control

• Agile embraces change … welcomes change.

• Backlog can be management mechanism for

change control.

• Backlog enforces same prioritization as used

for new features.


AAMI/TIR SW1:2012

TIR Take-Away:

• Expectation for Tailoring … use what makes sense for your organization

• Apply the values of Agile in a way that enhances a robust QMS.

• Apply the practices of Agile within the context of an established QMS.

• Set the correct expectations by defining the lifecycle and how regulatory

requirements are fulfilled.

• Establish a robust change management system to manage change and

mitigate risks associated with rapid change.


Agenda

• Why Agile?




Confessions

“Experience is something you don’t get until

just after you need it.” – Steven Wright


Discipline Point: “Requirements” are central to everything, BUT don’t need to be at same levels (waste)


Requirements detail: Low High

“go

od

en

ou

gh

” sp

ecs

SAFETY

CLINICAL UI

Typ

ical

sit

uat

ion

EVERYTHING

• May get overly focused/weighted toward unit testing

• May omit formal technical reviews with evidence

• May not perform proper regression testing or know when

to perform regression testing

• May fail to capture integration and system level test

cases during sprints

• Self-managed teams may neglect formal testing

altogether!


V&V

Each iteration should capture unit testing

As the system grows, each iteration should also capture integration

and system level test cases as appropriate for the system interactions

Integration and system test failures should be captured in a defect

tracking system

Test and spec approvals prior to running formal tests during certain

identified iteration

All test documentation is under configuration management


Discipline Points

• While system level hazards may be known, sprint

team may be not identify software-cause hazards

• Team may equate zero-defects with risk-free

• Even if hazards are mitigated, mitigation may not be

formally documented for test

• Subsequent sprints might alter mitigation software

and inadvertently dilute the mitigation effect


Risk Mitigation

Each iteration should capture any new hazards or causes

identified and document in risk management file

Each iteration should capture any mitigations developed

in that sprint and capture in requirements documentation

Phase planning should account for reviews of the risk

management documentation

Consider a designated risk team member to participate in

selected iteration or reviews


Discipline Points

Final Thoughts

• Inspect and adapt

• Understand the intent of the regulations

• Don’t check your “brain at the door”


Questions?

[email protected]

SoftwareCPR® is a software regulatory, safety, compliance, quality, and management consulting firm. We specialize in project and regulatory crisis

recovery in addition to training, preventive action, and continuous improvement.

Our expert staff and partners are industry practitioners – most have 25+ years experience in the medical device industry at technical levels through executive

management. We provide strategic and hands-on assistance to assure the success of your software investments.

Our approach focuses on your business objectives, project and regulatory risk

management, and pragmatic approaches that are tailored to your internal culture. We work closely with your internal staff, external vendors, your FDA counsel and

FDA itself.

agile and medical device software€¦ · “agile methods could be non-compliant because of the...

Documents