is the qa department dead in an agile world?
DESCRIPTION
Lots of Agile pundits proclaim that the role of QA Department in an Agile world is going by the way of the stenographer pool – soon to be extinct overtaken by new ideas and technologies. WE AGREE…if by QA Department they mean the traditional QA departments using approaches like a Double-V model of verification and validation. These QA departments are already extinct in most modern Enterprise Development teams which are using Agile approaches to develop multi-platform applications, deployed on cloud, delivered through mobile, web and native, by teams located all around the world. However, there is a smarter, nimbler and multifaceted QA organization taking their place and delivering substantial value in these organizations. In this webinar, I will present some of the key facets of these successful organizations including real life examples from our varied experiences.TRANSCRIPT
IS THE QA DEPARTMENT DEAD
IN AN AGILE WORLD
Delivering better software quality on Agile projects
9/26/13
2 ©Alliance Global Services 2013
Today’s Presenter
As Vice President, Solution Engineering, Sreekanth has more
than 12 years of senior technology leadership experience and
leads Alliance’s Solution Engineering organization in developing
cutting edge solutions. He is a respected technology thought
leader with a focus in testing Agile projects, test process
automation and metrics-based development approaches and
utilizing lean techniques in software development.
Today’s Host
As Director of Marketing, Sharon Lee heads the marketing
strategy and brand messaging for Alliance. With over
12 years of experience in both digital and traditional marketing,
she is focused on the effective use of media for compelling brand
messaging and creating successful marketing programs with
measurable results that impact revenue. Sharon holds a B.A.
from the University of Pennsylvania.
3 ©Alliance Global Services 2013
AGENDA
4 ©Alliance Global Services 2013
Agenda
• The case for QA in an Agile World
• Changing role of QA in an Agile World
• Role of QA in Agile organizations
• Profile of Testers and QA Analysts for these
organizations
• Automation best practices
• Metrics and measurements
5 ©Alliance Global Services 2013
The Case for QA in an Agile World
6 ©Alliance Global Services 2013
The Promise of Agile
Develop iteratively as business evolves
Deliver working software rapidly
Flexibility to change with business needs
Incorporate end-user feedback
Highly productive technical teams
7 ©Alliance Global Services 2013
Agile Challenges
Functionality/defect treadmill
High regression defect seepage
“Epic” failures
Non-functional requirements fail
Low effective team throughput
8 ©Alliance Global Services 2013
What Agile Assumes
Communication
Collaboration
Access to Stakeholders
Cross Functional Teams
9 ©Alliance Global Services 2013
Is QA Dead in this Agile World?
Some Agile proponents did not see (or argued against)
the need for QA specialists on an assembled team.
If you use TDD, Unit Testing & “Everyone Tests their
own Stuff” do we need a dedicated QA team?
10 ©Alliance Global Services 2013
Agile Layers of Testing
Prep
Development Start Wrapper Sprint 2 or 4 Week Sprints
JumpStart 24h
24h
Harden, Docs, Release
Vis
ion
Pro
du
ct B
ackl
og
Sprint Backlog Tested, Working
Software Increment
Product X
Sprint Review Sprint
Planning
Release Planning
Sprint 1 Sprint 2 Sprint n
3 - 5 Sprints
Prototyping Reqs Testing
Unit Testing Regression
Integration Compatibility Performance
Security
Quality Assurance
Acceptance Functional
Deployment
11 ©Alliance Global Services 2013
Specialization Has Benefits
12 ©Alliance Global Services 2013
Role of QA in an Agile World
13 ©Alliance Global Services 2013
Organization empowers Health plans to transform the performance of their provider networks, while significantly reducing medical, administrative and IT costs and simplifies the design, maintenance, reimbursement and performance management of provider networks, while facilitating provider-patient collaboration.
Development Objectives:
• Rapid release of new solutions and rich new features of their core Platform into market to take advantage of increased health care IT spending in the payer market due to health insurance reform (PPACA)
• Ensure high level of customer satisfaction dependent on user experience, maintainability, scalability and platform performance
• Ensure the highest levels of security and confidential data access meeting all regulatory needs and customer specific needs
Development Organization:
• TDD approach for Developers engaged in 2-week sprint cycles
• Enterprise development technologies
• Interfaces with several in-house and 3rd party development teams and applications
• SCRUM, Kanban and Lean techniques
Organization Profile
14 ©Alliance Global Services 2013
Agile Layers of Testing
Prep
Development Start Wrapper Sprint 2 or 4 Week Sprints
JumpStart 24h
24h
Harden, Docs, Release
Vis
ion
Pro
du
ct B
ackl
og
Sprint Backlog Tested, Working
Software Increment
Product X
Sprint Review Sprint
Planning
Release Planning
Sprint 1 Sprint 2 Sprint n
3 - 5 Sprints
Prototyping Reqs Testing
Unit Testing Regression
Integration Compatibility Performance
Security
Quality Assurance
Functional
Deployment
Acceptance
15 ©Alliance Global Services 2013
Testing Activities
Activity Developers Testers
Requirements Testing
Acceptance
Unit Testing
Functional/Exploratory Testing
Regression Testing
Compatibility Testing
Performance Testing
Security Testing
Deployment Testing
16 ©Alliance Global Services 2013
Quality Assurance
Activity Developers Testers
Test Strategy
Test Planning
Process Compliance
Root Cause Analysis
Metrics & Reporting
17 ©Alliance Global Services 2013
Profile of Testers & Best Practices
18 ©Alliance Global Services 2013
Technology Transformation - 2002
19 ©Alliance Global Services 2013
Technology Transformation - 2008
20 ©Alliance Global Services 2013
Technology Transformation - 2013
21 ©Alliance Global Services 2013
Distributed Teams
22 ©Alliance Global Services 2013
Communication Realities
23 ©Alliance Global Services 2013
Skillsets for an Agile Tester
Risk Assessment
Test Strategy & Planning
Test Design & Execution
Requirements & Acceptance
Metrics
Data
Deployment
CI
Cloud
UX
24 ©Alliance Global Services 2013
Automation Best Practices
25 ©Alliance Global Services 2013
Automation Objectives
26 ©Alliance Global Services 2013
Layers of Automation Layered Automation Testing Strategy for end to end coverage, automated solutions for
each tier and testing on actual devices
UX Testing
Services & API Testing
Unit Testing
27 ©Alliance Global Services 2013
Automation - Best Practices
Automation is development
Automation Architecture
Lagging sprints
Non-functional
Integration with CI
28 ©Alliance Global Services 2013
Environments
Test Management
CI Integration
Reporting
Infrastructure
29 ©Alliance Global Services 2013
• Re-usable automated test suites with minimal maintenance effort
• Cross application E2E automated testing
• Excel based test suite
• Detailed test summary reports along with step wise description and screen shots
• Test scripts, test reports integrated with QMS
• Automated 55% of regression scenarios which includes 70% of business critical scenarios.
Test Automation Framework
30 ©Alliance Global Services 2013
Metrics
31 ©Alliance Global Services 2013
Metrics
32 ©Alliance Global Services 2013
Measuring Effectiveness
Automation initiatives need metrics reporting and analysis as an integral part of the
initiative to ensure success of the initiative.
• Metrics should capture both business level and execution level values
to ensure efficiency is accurately captured and reported
• Metrics should not be left for the end of the project to implement
• Metrics measurements allow implementation level course corrections or
expectations management easier
• Business Objectives outlined should be directly tied to the metrics
program and measured through out the initiative
• Assumptions and approximations can be made at the beginning of the
initiative
33 ©Alliance Global Services 2013
Key Metrics for Agile Projects
• Use Validation points to evaluate application’s map with requirements
• Communicate to all stakeholders the coverage provided by formal testing
Coverage Metrics
• Number of defects identified in different stages of software lifecycle
• Number of defects created in different stages of software lifecycle
Defect Identification Efficiency
• Quality of software code
• Impact of technical or architectural choices Technical Debt
• Demonstrate the confidence of the testing team in the testing efforts executed on a release with the resources provided for testing Confidence Level
34 ©Alliance Global Services 2013
Metrics & Effectiveness Measurements Detailed execution metrics will be captured to glean insights about test process, execution and
SDLC process.
35 ©Alliance Global Services 2013
36 ©Alliance Global Services 2013
Thank You Sreekanth Singaraju
VP of Solution Engineering
www.allianceglobalservices.com
37 ©Alliance Global Services 2013
Coverage Metrics
Principle
Communicate to all stakeholders the coverage provided by formal testing
Dimensions
• Use Validation points to evaluate application’s compliance with requirements
• Weight them by business criticality
Definition
Designed Coverage = Designed Validation Points / All Validation Points
(Estimated)
Executed Coverage = Executed Validation Points / All Validation Points
(Estimated)
Best Practices
• Keep coverage metrics simple
• Introduce granularity to address business needs
• Trend to show progress (or declines – great case for additional resources)
38 ©Alliance Global Services 2013
Defect Identification Efficiency
Principle
Identify how defects are identified and created. Evaluate compliance of Agile
principles.
Dimensions
• Number of defects identified in different stages of software lifecycle
• Number of defects created in different stages of software lifecycle
• Use key Software stages to draw valuable inferences
Definition
Proportion of defects identified by each stage of a lifecycle – Ex – Testing –
56%, UAT – 18%, Production - 4%, other - 22%
Best Practices
• Start with defects identified first – then introduce defects created
• Use “Other” to catch exceptions and introduce granularity incrementally
• Factor in defect severity if that information is readily available
39 ©Alliance Global Services 2013
Technical Debt
Principle
Provide a financial impact of technical choices on the overall product resulting
from technology, process and implementation choices made during
development
Dimensions
• Quality of software code
• Unresolved defects that are business or quality critical
• Impact of technical or architectural choices
Definition
Cost estimates for redressing most critical (Business perspective) defects or
technical choices
Best Practices
• Explosive- Handle and communicate with care
• Be conservative – use a very strong criteria to evaluate “criticality” • Use several available tools to evaluate software code quality
40 ©Alliance Global Services 2013
Confidence Level
Principle
Demonstrate the confidence of the testing team in the testing efforts executed
on a release with the resources provided for testing.
Informational
• Test Execution Coverage
• Expertise of resources to test the application
• Quantitative results of defects including Defect identification efficiency
Definition
Based on the evaluation of the 3 dimensions above rank each of them on a
scale of 0(Poor) – 1(High). Overall confidence =
(3.Coverage.Resources.Results)/(Coverage+Resources+Results)
Best Practices
• Be prepared with supporting facts
• Define clearly laid out criteria for ranking