dealing with auditors: helping them understand agile
TRANSCRIPT
AT12 Agile Development Concurrent Session 11/13/2014 1:30 PM
"Dealing with Auditors: Helping Them Understand Agile"
Presented by:
Steve Nunziata Independent Consultant
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com
Steve Nunziata (CSM, PMP, ACP, SAFe SPC) has more than twenty-five years in IT project management, using waterfall and agile methodologies—and numerous hybrids in between. Steve’s industry experience ranges from health care, sporting goods, transportation, and insurance. For the past ten years, he has focused on agile practices and teams, fulfilling roles such as ScrumMaster, Product Owner, agile coach, project manager, and quality assurance advisor―sometimes in the same day! Steve is very active in the San Antonio agile community, facilitating monthly meet-ups and education events. In his spare time, he enjoys playing in his classic rock band and being with his wonderful family.
Dealing with Auditors: Helping them Understand Agile
CHAOS, CONSISTENCY, CREATIVITY: A JOURNEY THROUGH AGILE AUDITABILITY
Steve Nunziata, PMP, PMI-ACP, CSM, SAFe SPC November 13th, 2014
About Steve…
PMP, ACP, CSM, SAFe SPC
EDS, Nike, Adidas, USAA
Agile Trainer & Coach
New Jersey / Oregon
Bassist Extraordinaire
Alamo Agilistas / PMI
So… Why Are We Here? Opportunity:
Educate internal auditors to evolve away from formal artifacts and accept Agile tenets of visibility and transparency to demonstrate adherence to defined Quality standards.
We will collaborate on an approach to define an Agile Risk & Control framework that can start you on your journey.
How Would You Like: A 50% - or more – reduction in project ‘paperwork’ to demonstrate adherence to compliance processes?
WATERFALL AGILE
59
30
PROJECT COMPLIANCE
ARTIFACTS
A framework for consistent application of Agile practices and ceremonies across a large – and growing – organization?
Background: My Story
Zero to Sixty (Days): Chaos to Consistency
Agenda Ch
aos Failings of
Today’s Risk Management Processes
Cons
isten
cy
Why Audit Execution Models Need to Evolve Cr
eativ
ity
Creating an Agile Auditable Framework
Managing Risk – How Important is it?
The primary goal of a business is to… stay in business.
It is therefore necessary to continually evaluate, monitor, and address threats to retain market share. Otherwise, what would happen?
Managing Risk – The Risk Management Process
Risk Identification
Risk Assessment
Risk Response Risk Review
Managing Risk – ISO 9001 Summary Part 4 – The Company must establish, document, and maintain a Quality Management System (QMS)
Part 5 – Management commitment in evidence for the QMS
Part 6 – Necessary resources must be determined & provisioned
Part 7 – Plan & Develop processes for product realization. The processes must produce documents that can be (1) reviewed for acceptance; and (2) used as proof of conformance
Part 8 – All reports of non-conformances, both of the product or the process, shall be reported upon, analyzed and lead to corrective action
Managing Risk – Risk & Control Compliance Framework
Risk Controls
Control Tests
Reporting & Review
Operational Risks
Incomplete Requirements Ineffective or Incomplete
Software Solution Poor User Experience Poor Project Execution
Plan
Formal Requirements Baseline Process
Project Execution Schedule Review
Code Peer Reviews
Evidence of Formal Signoffs
Published Meeting Minutes
Documented Decisions / Logs
Formal results of Audit published for review; opportunities for improvements noted
Auditors
Are Risk Management Processes Inherently anti-Agile?
Source: http://www.devballs.com/wp-content/uploads/2010/02/agilemanifesto.gif
SDLC & Process Audit Execution Models: Challenges
While Agile adoption and evolution has continued unabated over the past several years, traditional process audits have largely been unable to keep pace. Why might this be?
SDLC & Process Audit Execution Models
Req’s Analysis Design Build Test Deploy
Systems Development Life Cycle – Linear View
SDLC & Process Audit Execution Models
Source: http://julianeverett.wordpress.com/
Red Dotted Line: Waterfall
Blue Dotted Line: Agile
RISK
TIME Project Risk Profile – Agile & Waterfall
SDLC & Process Audit Execution Models
Daily
24 H
ours
Iteration
2-4 Weeks
Release
~3 Months
Closure
~9-12 Months
SDLC Execution – Waterfall, Incremental, & Agile
SDLC & Process Audit Execution Models
Process Audit vs. SDLC Execution Gap Analysis
Closure
~9-12 Months
Release
~3 Months
Iteration
2-4 Weeks
Daily
24 H
ours
SDLC & Process Audit Execution Models
Daily Iteration
2-4 Weeks
Release
~3 Months
Closure
SDLC and Process Audit Execution: Optimal Quality State
5 Steps to Establishing an Agile Auditable Framework
Risk Validation
Inventory Agile Practices
Create Acceptable Parameters
Determine Method of Control
Establish Operational Parameters
1
2
3
4
5
5 Steps to Evolving an Agile Auditable Framework
Risk Validation
Review and Validate the current Risk & Control Framework, ensuring traceability from Risks to Controls to Control Tests.
Operational Risk: Risk Control: Control Test:
Failure to Manage Project Risks
Risk Management Process
Evidence of a Periodic Risk Review (Risk Log)
Issue Management Process
Formal, Complete Issues Log
1
5 Steps to Evolving an Agile Auditable Framework
Inventory Agile Practices
Inventory the Agile Practices supported by the organization. Scrum practices and ceremonies provide a good start.
Match the Agile ceremonies to the list of Risks in the current
Risk & Control Framework. Can a Ceremony or Practice provide an acceptable substitute? How / Why?
2
5 Steps to Evolving an Agile Auditable Framework
Inventory Agile Practices
Introduce the Agile Practice as a Control. Could it work? Could it be effective? What would be the value of the current control set – should anything remain, or can they be dismissed?
Operational Risk: Risk Control: Control Test:
Failure to Manage Project Risks
Risk Management Process
Evidence of a Periodic Risk Review
Agile Daily Standup
2
5 Steps to Evolving an Agile Auditable Framework
Create Acceptable Parameters
Research Industry standard ‘best practices’ for the ceremonies or practices you plan on using as a Control (mitigation strategy) for the Risk. A great example is Version One’s The Agile Checklist
Create a matrix defining minimally acceptable behaviors, along with anti-patterns, and radiate the desired outcomes in a common area
3
5 Steps to Evolving an Agile Auditable Framework
Create Acceptable Parameters
Agile Ceremony: Daily Standup
Best Practice Acceptable Partial Unacceptable
Occurs 5 Days per Week
Occurs 4 Days per Week
Occurs 3 Days per Week
Occurs <3 Days per Week
3 Core Questions Addressed
3 Core Questions Addressed
<3 Core Questions Addressed
<3 Core Questions Addressed
…Your Organization?
…Your Organization?
…Your Organization?
….Your Organization?
3
5 Steps to Evolving an Agile Auditable Framework
Determine Method of Control
Does the new Control Test require someone observe an Agile Ceremony, or is there a consistent formal artifact from an Agile practice that can be viewed?
4
5 Steps to Evolving an Agile Auditable Framework
Establish Operational Parameters
Review the total number of Control Tests. How many require observation from an Auditor?
Establish the Audit cycle & reporting time (Weekly? Sprint Level? Release Level? Other..?)
Train and deploy Audit resources
Execute an Audit cycle… and report to Risk Owners Learn… and continue to evolve!
5
5 Steps to Evolving… Creativity Host a Retrospective Ceremony with some of the Agile teams to uncover:
What may be challenging teams in conforming to minimal standards?
What opportunities can they recommend to evolve to controls?
Are the audits providing value in holding roles accountable for their deliverables?
Finally – when minimal standards are easily achieved – it’s time to take the next steps in maturity, and shift the pattern.
5 Steps to Evolving - Going Beyond... Challenge: can you evolve traditional, formal artifacts into a
more Agile framework? How can you continuously improve?
Picture Source: http://agile101.wordpress.com/2009/07/27/ agile-risk-management-assessing-risks-step-2-of-4/
Positive Outcomes Better alignment of Controls and Tests to the project execution model
Real time, actionable feedback & reporting to teams and Risk owners
Scalable for future methodologies & practices
Continual quality assessments; a project can have multiple reviews
Sets a benchmark for Agile maturity across an Organization
‘Humanizes’ the Audit (not ‘check the box’) – gives teams a voice
Experience – 50% reduction in Controls… while doubling Quality
Leading – NOT lagging – metric; address problems before they manifest
Opportunity for two-way communication and learnings
Challenges Optimal model is labor intensive
Inherent subjectivity in assessments (‘Auditor Bias’)
Potential for teams to feel ‘over controlled’
Oversight and administration of the process
Communication and support for changes
Determining boundaries of adherence vs. non-adherence, and appropriate remedies
Ever-evolving process; can feel like an ‘arms race’
Common Questions Does this model Scale?
How much time per week would this require?
Isn’t this just the Scrum Master’s… or (insert role here) – job?
Could we use Pair Programming as a Control?
What is the future of Agile Quality Assurance?
Objectives Met?
Source: http://www.devballs.com/wp-content/uploads/2010/02/agilemanifesto.gif
Remember: Auditors are the Board of Health!
Questions?
Thank You!