cost of quality how to save money
DESCRIPTION
TRANSCRIPT
Sibgatullina Taisiya
Solution Consultant, CEE
Cost of Quality
How to Save money
2 HP Application Lifecycle Management & HP Quality Center 11.00
Part 1: Quality assurance
3 HP Application Lifecycle Management & HP Quality Center 11.00 3
Cost Of Quality
4 HP Application Lifecycle Management & HP Quality Center 11.00
Application Lifecycle model
5 HP Application Lifecycle Management & HP Quality Center 11.00
ALM
5
HP
(41.1%)
IDC Worldwide Automated Software Quality Revenue by Vendor in „09
Market Leading Solutions
IBM (Rational) 11.1%
Microsoft 5.6%
Compuware 5.3%
Borland 2.9%
Empirix 1.9%
Others (Combined)
32.1%
Source: Melinda Ballou IDC, “Worldwide Automated Software Quality 2009-2013 Forecast Update and 2008 Vendor Shares: Evolving Quality in a Challenging Market” (Doc # 219695E) October 2009
“As the market revenue leader, with a 41.1% share, HP is in a position to disproportionately influence the overall ASQ market.”
©2009 HP Confidential 7
New Gartner report just published in January 2011
HP ALM #1 AGAIN
Designed for the End-to-End Application Lifecycle
HP ALM 11.00 is a Unified Platform
Requirements Management
Requirements Management
Requirements Definition
Quality Management
Functional Performance Security
Development Management
IDE, SCM, Build
Integrations
Defect Mgmt
Secure Develop-
ment
Foundation
Traceability Reporting Process Standardization
(i.e. Waterfall, Agile)
Extensibility (Open APIs)
Inte
gra
tions
into
Str
ate
gy a
nd P
lannin
g
(PPM
, G
ov)
In
tegra
tions in
to R
un
(Deplo
yment, A
PM, ITS
M)
Unified for the Core App Lifecycle, Integrated with the Complete App Lifecycle
Application Lifecycle Management
Project Planning & Tracking Enterprise Release Management
ALM Platform
Process Standardization (i.e. Waterfall, Agile)
A single platform for managing the application lifecycle
HP Application Lifecycle Management
Dashboard
Single Data Repository Reporting Open APIs (REST)
HP Application Lifecycle Management
Release Management
Requirements Management
Defect Management
Project Planning & Tracking
Business Models
Test Lab Test Plan
HP Unified Functional Testing
Performance Optimization Application Security Testing
Business Application Validation
HP QuickTest Professional
HP Service Test
HP Performance Center
HP LoadRunner
HP QAInspect HP Business Process
Testing
Services
Service Test Management
Test Data Management
Asset Sharing & Reuse
10 HP Application Lifecycle Management & HP Quality Center 11.00
Project Planning & Tracking Track & measure project milestones and KPIs
E2E Testing
Readiness Assessment
Testing Progress
92% Req Reviewed
Req Approved
0 Defects High
Defects Med
Test Exec
Tests Passed
15
98%
92%
10 Defects High
Defects Med
Test Auth
Test Exec
Tests Passed
25
98%
89%
45 Defects High
Defects Med
Test Exec
Tests Passed
95
78%
90%
93%
Test Authoring
Integration E2E Testing
User Acceptance
Testing
Requirements Authoring
Development Test
Execution
Functional, Performance and Security
95%
85% Req Implemented
Tests Reviewed
75% Development
Progress
Requirements Progress
11
Integrated solution for defining, managing and tracking requirements
Requirements Management
– Business analyst can collaborate on application requirements without fear of overriding other‟s work
– Assess overall risk to the business through real-time visibility of test coverage and associated defects against business needs
– Assess the impact of change before changing requirements
– Easy adoption by Business Analysts
– Define and track multiple requirement types
– Establish templates to ensure higher quality requirements
– Manage requirements dependencies
– Bi-directional traceability from requirements to requirements, tests and defects
– Manage change requests through formal approval process
– Familiar rich test editor interface
– Version Control and Baselining
– Leverage existing assets in MS Word
12
Track changes made to entities in your project over time
Baselining
– Compare releases and cycles
– A snapshot of the project state at a
particular point in time
– Enable rollback of assets to key points in the
application lifecycle
– Compare baselines to assess changes
– Ability to use baselines in test execution
– Create baselines – including traceability
between assets
– Compare baselines both at high level and
drill down to specific changes including
traceability
– Report and print baselines for sign-off
– Control who can create and modify
baselines
13
Track and measure the progress and quality of releases and cycles
Release & Cycle Management
– Manage projects and assets against release
and cycles
– Track release progress
– Gain visibility into release readiness and
release quality
– Compare releases and cycles for continuous
process improvement
– Manage requirements & tests according to
application releases and dev/QA cycles
– Compare planned vs. actual progress for
each application release and cycle
– Assess release quality across multiple cycles
14
Track and measure project milestones and KPIs
Project Planning & Tracking
– Visibility into application projects progress
and status
– Visibility into key project milestones
– Identify delays or poor quality before it
impacts the schedule
– Proactively adjust plans when a project is
not on track
– Define and track project milestones
– Define and measure key performance
indicators across the application lifecycle
– Automatically updates project status against
milestones
– Drill down on KPIs for more details
15
Define, manage and track all test scripts
Test Plan
– Create test cases to adequately test the requirements
– Manage all test types (Functional, Performance, Security) in one place
– Ensure requirements coverage through traceability to requirements
– Manage change associated with requirements changes
– Design, build and manage manual and
automated test cases
– Link test cases to requirements
– Leverage test cases from MS Word and MS
Excel
– Version Control and Baselining
16
The next generation of manual testing
HP Sprinter
– Accelerate manual testing
– Improved testing accuracy
– Better communication of defects to development (accelerated defect remediation)
– Unobtrusive, easy to use user interface
– Supports exploratory testing
– Reduce time and resources by testing multiple environments at once
– Toolkit – Screen capture and annotations, measure objects position etc.
– Record user actions to support exploratory testing
– Automatic data injection
– Mirror testing
– Log defects with associated information directly into HP Quality Center/ALM
17
Track defects across the entire application lifecycle
Defect Management
– Manage, track and enforce defects across the application lifecycle
– Create defects from manually or directly from the execution of manual and automated tests
– Clearly communicate bugs to developers with context from requirements and tests
– Integrated into developers IDE
– Enforce defect lifecycle process through custom fields and workflow
– Log defects directly from test runs with test details (test step, result, screenshots, attachments) to ensure the developers can reproduce the issue
– Traceability to requirements, tests and other defects
– Alert key stakeholders through automated emails
18
Clear communication and visibility into your application projects
Reports, Graphs & Dashboard
– Make informed business decisions
– Clearly communicate to all stakeholders
– Understand the real-time status of an application
– Full trending analysis and insight into application projects
– Understand release and cycle progress
– Fully customizable project reports (export to MS Word)
– Integrated Dashboard
– Out of the box or customizable analysis reports and graphs
– Report to MS Excel
– Embed graphs and charts into web portals
– Requirements traceability matrix
©2010 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice
Development tools integration
• Traceability between requirements, code changes and defects
• Adjustable processes
• Side-tools integrations
Working with defects in various tools
Quality Center
Release & Requirements
Test Plan
Test Lab
Defects
Developer
STRATEGY/
DEMAND
Strategic
demand
• New
applications
• New services
• Application
integrations Operational
demand
Other tools
JIRA
BugZilla
StarTeam
CCE Tracker
VersionOne
Development tools integrations
• Tasktop & CollabNet
• Eclipse & Eclipse-based clients
• Visual Studio
• CollabNet
• AccuRev
• Perforce
• CVS
• ClearCase
• CollabNet
• Hudson
• Electric Cloud
• AnthillPro
TFS
HP Application Lifecycle Intelligence
SVN Jenkins
TFS
CVS Electric Cloud
SVN Hudson
Waterfall
RUP
Scrum
Hybrid
Single Platform
Test Build Req
HP ALM
TFS
The Decision Support System for ALM…
SVN Jenkins
TFS
CVS Electric Cloud
SVN Hudson
Waterfall
RUP
Scrum
Hybrid
Single Platform
Test Build Req
HP ALM
Key ALI Features
Code changes from Subversion
Code changes from TFS
New Module
Drill down
End-to-end app delivery Report
Key ALI Features
HP ALM 11 and TFS integration
QC TFS
TFS adapter for QC Synchronizer 1.4
Defects
Bugs & Reqs
Work Items
Reqs
28 HP Application Lifecycle Management & HP Quality Center 11.00
Part 2: Performance validation
Solutions for management and automation of performance optimization
HP Performance Center
User/privilege management
Infrastructure management
Central repository
Global access & collaboration
Requirements & Defects
Management Framework: Performance Center
Resource pooling & scheduling
HP Performance Center Family
Performance Testing: LoadRunner / Performance Center
VuGen Controller Load Generator Analysis Monitors
Diagnostics
J2EE .NET SOA SAP Oracle
The market leading solution for performance validation
HP LoadRunner
Enables a consistent, repeatable process to: • Emulate production
workload to test a broad range of applications
• Identify performance
bottlenecks, diagnose the root cause of issues
• Optimize application
performance before deployment
Reduce risk of deploying systems that do not meet performance requirements
Reduce cost by predicting scalability and capacity and eliminating over-deployment
Reduce time by shortening test cycle time. Accelerate delivery of high quality applications
– Sequential process takes time
– Scripting expertise needed (especially for complex apps)
– Protocol level scripts break with minor app changes
Current scripting process
The challenge: Scripting process
Record a script
Correlations Parameterize
data Add
transactions Add logic
– Many technologies, frameworks and toolkits
– Very customizable – almost no “vanilla” usage
– Dynamic, asynchronous behavior
– Client-side processing and logic
– Most testing solutions do not support all the customizations
New Web 2.0 technologies
The challenge: Scripting for web 2.0 apps
Services
Web Server
Web Services
Web Server
Web Server
Web Server
App Server
Data Server
Networks Data
Server
Cloud
App Server
EDW
Client
LoadRunner VuGen
Record an application business process into an automated script Add end-user transactions, parameterize input data, add verification checkpoints Apply user behaviors such as network connectivity or pacing
Application
Scripts
Business process
VuGen Recorder
Browser
LoadRunner Controller
Design and execute load scenarios that mimic production conditions
Emulate hundreds or thousands of concurrent users
Gather end-to-end transaction response times
Collect application, network and system data across all architectural tiers and components
Diagnostics
Monitors
Load
balancer
Web
server
App servers
Database Network Controller Load
generators
VUsers
US
Europe
The revolutionary way of scripting modern apps
INTRODUCING: LoadRunner TruClient
Scripting engine embedded within
browser
Transactions, and logic can be added while scripting
Data can be parameterized
while scripting
Automatic suggestion of automated steps
No correlations required!
Interactive, dynamic scripting!
Supports all Ajax
applications
Diagnose performance bottlenecks
Key benefits
• Monitor system performance during tests
• Diagnose performance bottlenecks as they occur
• Identify application code level problems
• Resolve issues faster
Key capabilities
• Sitescope monitoring for system performance metrics
• Diagnostics to drill down to the method level
• Data available during run time as well as for post-test analysis
Monitoring and Diagnostics
LoadRunner Performance Center 36
6 November 2011
Customize and extend the industry-standard platform
Key benefits
• Build niche/custom protocols
• Export analysis data for custom reports
• Include custom monitoring and diagnostics
Key capabilities
• Custom protocol development via a SDK
• Analysis API to export data for custom reports
• Extensible monitoring and diagnostics framework
Extensible Platform
HP Partners
--> •HP supports 46% of protocols customers need.
•New Demand = 30% of current usage
•25% of customers requested AJAX, RDP and VoIP
•Smaller demand for multiple protocols sums up to big opportunity
LoadRunner Performance Center 37 6 November 2011
Performance Requirements & Defects, Coverage reports
Performance Center 11.0
Performance Requirements
Requirement status
Requirement coverage
New PC Lifecycle User to access performance requirements and defects
Next-gen platform: based on the ALM platform
Performance Center 11.0
Performance Center
ALM Platform
Performance Testing
Performance Lab Mgmt
(scheduling)
Performance Requirements &
Defects
Performance Test
Management
Admin Console Users & license Projects CoE
Collaboration Traceability Reporting Version Control
Authentication Authorization Workflow APIs, Integrations
Topology
Performance Center 11.0
Drag & drop from list to build your
topology diagram
Define machine names and properties
Right click to set up monitors
– Timeslots
• Redone in Outlook fashion
• More information available regarding booked timeslots
• Allows for maintenance windows
Miscellaneous
Performance Center 11.0
Shows when you can start a test based on your
requirements
Timeslots
Performance Center 11.0
Gives information on used timeslots
– Timeslots
• Redone in Outlook fashion
• More information available regarding booked timeslots
• Allows for maintenance windows
Miscellaneous
Performance Center 11.0
Shows when you can start a test based on your
requirements
Timeslots
Performance Center 11.0
Gives information on used timeslots
Operational security
management
OPERATIONS
Service desk
Production monitoring
Connect to
pro
ductio
n
Go/ No Go
Performance Center – fully integrated into the application lifecycle
45 6 November 2011
A repeatable quality management process mitigates risk
45 6 November 2011
Inte
gra
te w
ith d
em
and
Functional requirements
Business requirements
Security requirements
Performance requirements
Other non-functional
requirements
REQUIREMENTS MANAGEMENT
Align with management and stakeholders
Collaborate with design and development teams
Assess and
Analyze risk
Establish testing
priorities
Create test plans
RISK-BASED TEST PLANNING
TEST MANAGEMENT AND EXECUTION
Execute security scans
Identify and customize security
policies
DEFECT MANAGEMENT
Execute functional tests
Create manual test cases
Automate regression test
cases
Execute tests, diagnose and
resolve problems
Create performance scripts and scenarios
Connect to
pro
ductio
n
STRATEGY/ DEMAND
Strategic demand
• New applications
• New services • Application
integrations
Operational demand
• Defects • Enhancements • Change requests
Enterprise Architecture and
Policies
• SOA • Security
Operational security
management
OPERATIONS
Service desk
Production monitoring
Operational security
management
OPERATIONS
Service desk
Production monitoring
STRATEGY/ DEMAND
Strategic demand
• New applications
• New services • Application
integrations
Operational demand
• Defects • Enhancements • Change requests
Enterprise Architecture and
Policies
• SOA • Security
Go/ No Go
Execute tests, diagnose and
resolve problems
Create performance scripts and scenarios
Part 3: HP UFT
Part 3.1 Functional Testing and Service Test
A Modern testing team has new needs
People
• Expand skills to test beyond the GUI
• Extend testing to more members of the team
• Increased collaboration with developers and business analysts
Processes
• Automate as early as possible
• Accommodate faster change
Technology
• Complex integrated systems
• Test GUIs, APIs, web services and vertical integration
• Test RIA, Web 2.0
Late stage testing is too expensive…
Iteration 1 Iteration 2 Iteration 3 Test
When it was injected way back here? Why wait to find a bug here?
Remember, the age old QA principle…
Time between injection
and identification
Cost to
Fix
Test & automate as early as possible
C1 C1
C2
C1
C2
C3
G2
C1
C2
G1 G1
C3
C1
C2
G1
G3
G2
C3
C1
C2
G1
… …
BP1
BP2
G3
G2
C3
C1
C2
G1
Regression test at the end of each iteration or as part of continuous build
Automate tests when available components
become stable
C GUI-less component
G GUI element
BP business process
stable
component
Modern applications have many layers Process Layer
A new transaction
Exchange
Screen 5 (Java)
Screen 6 (.NET)
SAP
Screen 3 (.NET)
Screen 4 (Ajax)
Web 2.0 Billing
Screen 1 (Ajax)
Screen 2 (Java)
End to end business process
Graphical user interface
layer
Headless Layer
Interact with modern GUI interfaces, Web 2.0
Continue the scenario to the headless layer; test the integration between the layers
Report on the quality of the entire end to end process
Component 1 (.Net API)
Component 2 (Java API)
Component 3 (web service)
Component 4 (JMS service)
Component 5 (web service)
Component 6 (REST service)
HP Software solutions Process Layer
A new transaction
Exchange
Screen 5 (Java)
Screen 6 (.NET)
SAP
Screen 3 (.NET)
Screen 4 (Ajax)
Web 2.0 Billing
Screen 1 (Ajax)
Screen 2 (Java)
End to end business process
Graphical user interface
layer
Headless Layer
Component 1 (.Net API)
Component 2 (Java API)
Component 3 (web service)
Component 4 (JMS service)
Component 5 (web service)
Component 6 (REST service)
HP Unified Functional Testing
HP Unified Functional Testing includes the ability to test the GUI layer + the Headless layer + the end to end process.
Unified Functional Testing
HP Service Test HP Functional Testing
(HP QuickTest Pro + all add-ins)
Call activities from one product to the
other
Transfer complex data structures
Integration
Unified report
What‟s new with HP Unified Functional Testing 11.0?
Report for a Unified Functional Testing scenario
Shows activities from both HP Functional Testing &
HP Service Test
Multiple views of the flow
Full traceability for easier debugging
Testing Web 2.0 GUIs
• Ajax toolkits including Dojo, GWT, YUI
• RIA such as Flex, Silverlight v2.0, v3.0, (v4.0 soon), WPF extensibility
• New support for Firefox
OOB Web technology support
• Wizard-like tool helps users extend support to more web 2.0 controls
• Partners and customers can share code created to customize support for add‟l toolkits
Community
Easy, wizard-like interface
Extends HP Functional Testing fast; uses Intellisense
Automated deployment
Free for anyone? Yes, FREE!
Opening web extensibility to the online community: The Extensibility Accelerator
HP Service Test 11.0 - new Visual Test Designer
11/6/2011
Modern, intuitive test design interface
Reusable modules improve maintainability
Visual test designer extends testing to non-developers
Flexible and extensible service processing
BPT Positioning
What is BPT?
Business Process Testing is a framework for QA organizations to build and reuse structured testing components in both manual and automated testing.
Target users
Non-technical QA personnel
Subject Matter Experts
BPT is ideal to Build frameworks to maximize reuse of testing components
Mitigate risks by aligning business requirements with end-to-end test scenarios
Part 3.2 BPT
BPT 11.0 Simpler. Faster. Better.
What‟s new in BPT 11
Significantly improved performance
Built an all new simple and intuitive user interface
Provided the ability to leverage QC 11.0 Capabilities
Enhanced Traceability via Component Criteria
Rapid Manual Testing with Sprinter Integration
BPT 11 Exponential Performance Boost
More Components = Better Performance
Number of components
Perf
orm
anc
e
Test Script view – New look & Feel
Cleaner UI showing all BPT Resources and Dependencies
Clean and Simple Iteration View
•Tree structure instead of a flat list
• Improved Iterations dialog (multi-
iterations view & Single Iterations view)
• new ‘Comments’ column for each
Component instance in the test
• Search option inside the Iterations
dialog
One-step Component Creation Process
Simplified the component creation process from three steps to just one
step
Enhanced Drag and Drop Feature
Proving the ability to drag and drop multiple test components into the test
script area
Summary of Key Enhancements and Features
• All new BPT UI built using QC web toolkit for seamless look and feel
• Ability to create a manual testing component in just one step
• Ability to drag and drop multiple components into test script pane
• Enhanced test iteration view to show different combinations
• Simple and easy to understand all new test scrip view
• Cleaner look and feel of QC resource dependencies in BPT
• Application area is a standard test resource in BPT
• BPT integration with QC Sprinter
• Enhanced traceability to map requirements to component level
• Major performance improvements with a redesigned execution
engine
Traceability & Metrics
Single Platform
Simplified Web Access
Technology Agnostic
Powerful Automation
Quality at the Center
Modular
Integrated
Heterogeneous
Best-in-class
HP Lifecycle Solutions & Operating Principles
COMPLETE LIFECYCLE
COVERAGE
PLAN
Project & Portfolio
Management
DELIVER RUN RETIRE
HP ALM
Quality Center
Perform Center
App. Security Center
Business Availability
Deployment Automation
Service Management
Records Management
Archiving
CORE LIFECYCLE EXCELLENCE
HP APPLICATION LIFECYCLE MANAGEMENT
Project Planning & Tracking Enterprise Release Management
ALM Foundation
Process standardization Reporting Extensibility
Requirements Management
Development Management
Quality Management
Definition Versioning &
Change Control Link to
IDE, SCM Defect Mgmt.
Secure Code
Function Perform Secure
SOA Systinet
Why HP?
Integrated management & automation: instead of loosely coupled point tools, HP provides a cohesive platform for lifecycle management and automation.
Technology & methodology agnostic: broadest support for heterogeneous environments (.NET, Java, SAP, Oracle, etc.) – covers more than 70 environments. Covers traditional and Agile methods in single solution.
Project ready, enterprise scale: whether for a team of ten, or an enterprise of tens of thousands, HP solutions offer proven configurability and scalability.
Complete application lifecycle: no other company offers the integrated product coverage of HP to support applications from beginning to end.