Download - Software Testing: Legacy
Software Testing:Legacy & Future
2005
Josh SoldatiHead Of Testing Services, North America
2Cognizant Confidential
Testing No Longer The Ugly Step-Child
“Once the backroom boys, the ‘invisibles’ of the software industry are striking back. Testing has assumed strategic importance in all software services companies today. Customers and users have become tired of paying top dollar for software, which either does not work, or if it does, then not according to expectations.”
- Business World India*
*Emphasis added.
3Cognizant Confidential
Software Testing: The Legacy
RequirementsInception DesignCode & Unit Test Testing
Rollout & Deployment
Testing considered a discrete step in the lifecycle – the last step.
Testing and software quality assurance not considered specialized skills or career path.
No formal processes or metrics established for software testing.
• Testing teams involved too late in the lifecycle, limiting ability to provide accurate and comprehensive test coverage
• Testing effort not adequately estimated, impacting delivery schedule and/or budget
• Developers test their own code
• Top talent moves out of software quality roles
• Unqualified resources perform and manage testing activities
• Limited ability to measure/improve either efficiency or quality of testing effort
The Legacy The Impact
4Cognizant Confidential
The Ultimate Cost: Late Defect Detection*
Requirements Design Code Unit Test Test Execution Production
Find/Fix defects during Requirements = x to 2x = $75 to $150**
Find/Fix defects during Design = 5x = $375**
Find/Fix defects during Coding = 10x = $750**
Find/Fix defects during Unit Testing = 20x = $1500**
Find/Fix defects during Test Execution = 50x = $3750**
Find/Fix defects during Production = 200x = $15000**
*Source:Gartner.
** Example cost figures assume x = 1 person hour, at a cost of $75/hr.
5Cognizant Confidential
Breaking Away From The Legacy:Cognizant’s Holistic View
Integrated Development LifecycleIntegrated Development Lifecycle
Integrated Testing Processes & Methodology
RequirementsInception DesignCode & Unit Test
TestExecution
Rollout & Deployment
Testing Infrastructure & Tools.
Testing Operating ModelTesting Operating Model
Testing OrganizationTesting Organization
6Cognizant Confidential
Cognizant’s Testing Framework:Critical Components Of Software Testing
Process• Code Promotion*• Build Acceptance• Defect Management*• Test Environment Management• Test Execution Reporting• Knowledge Management• Metrics Tracking• Delivery Assurance• Process Quality Initiatives
Process• Code Promotion*• Build Acceptance• Defect Management*• Test Environment Management• Test Execution Reporting• Knowledge Management• Metrics Tracking• Delivery Assurance• Process Quality Initiatives
Methodology• Test Planning & Estimation• Requirements Traceability*• Integration Testing• System Testing• User Acceptance Testing• Performance Testing• Regression Testing• Test Data Creation• Automation Testing• Non-Functional Testing
Methodology• Test Planning & Estimation• Requirements Traceability*• Integration Testing• System Testing• User Acceptance Testing• Performance Testing• Regression Testing• Test Data Creation• Automation Testing• Non-Functional Testing
Infrastructure• Staging environments*• Test beds• Desktops• Networks
Infrastructure• Staging environments*• Test beds• Desktops• Networks
Tools• Test Management• Performance• Automation• White Box• Requirements Management*• Templates & Other Tools
Tools• Test Management• Performance• Automation• White Box• Requirements Management*• Templates & Other Tools
Operating Model• Project Structuring• Staff Assignment• Work-load Management• Organizational SLAs
Operating Model• Project Structuring• Staff Assignment• Work-load Management• Organizational SLAs
Organization• Organization Structure• Skills Mix• Training Programs• Professional Certifications• Career Path
Organization• Organization Structure• Skills Mix• Training Programs• Professional Certifications• Career Path
* Linked to overall development methodology and lifecycle.
7Cognizant Confidential
Consolidation Of Software Testing
Project Team ,Department, orBusiness Unit
Development Team
Testers
Project Team ,Department, orBusiness Unit
Testers
Project Team ,Department, orBusiness Unit
Testers
Decentralized Software Testing
• Testing teams aligned by project, department, or business unit.
• Varied methodologies, processes, tools, infrastructure and metrics across groups.
• Limited or non-existent software testing SLAsestablished across the enterprise.
• Inefficient management of resources – people and infrastructure.
Consolidated Quality Groups
Project Team ,Department, orBusiness Unit
Project Team ,Department, orBusiness Unit
Project Team ,Department, orBusiness Unit
Development Team
Development Team
Development Team
Development Team
Development Team
Consolidated Quality Group
• CQG organization operates across the enterprise and supports individual teams/groups on a scheduled work queue basis.
• Consistent methodologies, processes, tools, infrastructure and metrics across groups.
• SLAs established and performance metrics measured throughout..
• Leverage resources across teams/groups and manager work.
Evo
luti
on
8Cognizant Confidential
The Move To Of Offshore Testing
§ Independent Verification & Validation is now offered by all major offshore service providers.
§ Represents fastest growing business within Cognizant, and an increasingly larger portion of the offshore IT spend.
§ Many organizations have piloted offshore testing efforts over the past three years and are now embracing across the enterprise.
§ Key drivers:
- Achieve cost savings
- Leverage mature processes
- Establish better controls, metrics, and SLAs
- Gain access to expertise (performance testing, automation testing, test management, etc.)
- Flexible resource management
- Opportunity for 24X7 Testing
9Cognizant Confidential
Managed Test Centers:The Marriage Of Onsite/Offshore With Centralized Quality Groups
CQGCQG
Dedicated Resource Pool(Core Team)
Rationalized:
• Processes• Methodology• Infrastructure• Tools• Operating Model• Organization
Integrated Governance Model
MTC
Functional Performance
Automation ProcessFlexible Resource Pool
(“Flex” Team)
Critical Features &Benefits
üContinuous process improvement
üCommon automation frameworks
üShared infrastructure and tools
üStandard operating model with structured SLAs
üOptimized onsite/offshore model
üRigorous metrics collection and reporting
üCommon process and methodologies
10Cognizant Confidential
Managed Test Center:An Example Scenario
The Current State Scenario
§ An organization has a 50 person testing team, comprised of 25 FTEs and 25 contractors.
§ Fully loaded costs and benefits for an organization's testing team members are $100,000/yr.
§ Annual voluntary attrition is 10%, with the average cost of replacement cost equivalent to 3 months salary and benefits..
§ The organization has implemented some automation – with about 20% of functional testing activities automated.
The Future State Scenario
§ The organization would retain 10% of it’s internal and contract resources for testing. There would be a 10% reduction in force, with the remaining 80% of internal FTEs reassigned to other tasks within the organization.
§ Approximately 70% of the Managed Test Center team would form the “core” team, while an additional 30% capacity (the “flex” team) would ramp up cyclically every four months.
§ The onsite/offshore ratio would fluctuate between 10 – 15% onsite vs. 85 – 90% offshore depending on current release cycle demands.
§ The knowledge transfer and piloting phase would last three months, with a ten person team (6 onsite, 4 offshore).
§ By the end of the third year, the Managed Test Center will have achieved an overall 15% improvement in efficiency.
§ Total resource requirements would drop year-over-year based on automation and efficiency gains.
11Cognizant Confidential
Managed Test Centers:A Three Year Labor Cost ROI Scenario
Year 1 Year 2 Year 3 TotalCurrent State
Internal Testing Labor CostsTesting Team FTE 2,500,000.00$ 2,500,000.00$ 2,500,000.00$ 7,500,000.00$
Testing Contractors 2,016,000.00$ 2,016,000.00$ 2,016,000.00$ 6,048,000.00$
Voluntary Attrition Costs 62,500.00$ 62,500.00$ 62,500.00$ 187,500.00$
TOTAL CURRENT STATE COSTS 4,578,500.00$ 4,578,500.00$ 4,578,500.00$ 13,735,500.00$
MTC Future State
Internal Testing Labor CostsTesting Team FTE 812,500.00$ 250,000.00$ 250,000.00$ 1,312,500.00$
Testing Contractors 655,200.00$ 201,600.00$ 201,600.00$ 1,058,400.00$
Voluntary Attrition Costs 20,312.50$ 6,250.00$ 6,250.00$ 32,812.50$
Internal Separation Costs 20,000.00$ -$ -$ 20,000.00$
Internal Testing Costs 1,508,012.50$ 457,850.00$ 457,850.00$ 2,423,712.50$
Cognizant Labor CostsOnsite 776,067.83$ 517,208.33$ 507,873.24$ 1,801,149.40$
Offshore 1,499,693.83$ 1,398,249.22$ 1,369,025.28$ 4,266,968.33$
Cognizant Labor Costs 2,275,761.66$ 1,915,457.54$ 1,876,898.52$ 6,068,117.73$
TOTAL MTC LABOR COSTS 3,783,774.16$ 2,373,307.54$ 2,334,748.52$ 8,491,830.23$
Labor Cost Savings
MTC Cost Savings BreakdownDirect Labor Costs 586,644.40$ 1,402,819.42$ 1,429,820.71$ 3,419,284.53$
Incremental Automation Benefits -$ 393,724.80$ 393,724.80$ 787,449.60$
Efficiency 208,081.44$ 408,648.24$ 420,205.97$ 1,036,935.65$
TOTAL ANNUAL SAVINGS ($) 794,725.84$ 2,205,192.46$ 2,243,751.48$ 5,243,669.77$ TOTAL ANNUAL SAVINGS (%) 17% 48% 49% 38%
12Cognizant Confidential
Managed Test Centers:A Three Year Labor Cost ROI Scenario
Managed Test Center: 3 Year ROI ScenarioAverage Annual Staffing Levels
0.0
10.0
20.0
30.0
40.0
50.0
60.0
Year 1 Year 2 Year 3
Res
ou
rces
Client FTEs Client Contractors MTC Onsite MTC Offshore
13Cognizant Confidential
Managed Test Center Case Study: Fortune 50 Global Conglomerate
Results• Built a dedicated 50 person CQG to operate within enterprise and project-level SLAs.• Established well defined software testing processes; developed process road maps, estimation
models, automation standards etc.• Reduced post-production defects through structured requirements analysis and full traceability to
test cases.• Reduced regression test cycle times by automating test cases using Rational Robot. • Increased visibility to testing and development activities through comprehensive metrics collection
and reporting.
SolutionEstablish an offshore Central Quality Group to operate on deliver software testing services organization structured around internal SLAs.
ChallengeFortune 50 conglomerate with decentralized, global IT operations sought to reduce operating costs while increasing efficiency and consistency within it’s software quality assurance organization. Executive leadership mandated continuous process improvement, but environment lacked processes or tools to manage.
14Cognizant Confidential
MTC Case Study:Approach & Ramp
Q1 Q2 Q3
ü Application 1
ü Application 2
ü Application 3
ü Application 4
ü Application 5
ü Application 6
ü Application 7
ü Application 8
ü Application 9
ü Application 10
ü Application 11
ü Application 12
ü Application 13
ü Application 14
ü Application 15
ü Application 16
ü Application 17
ü Application 18
ü Application 19
Q4
ü Application 20
ü Application 21
ü Application 22
CQG Case Study: Resource Ramp
05
101520253035404550
Q1 Q2 Q3 Q4
Res
ourc
es
Onsite
Offshore
CQG Case Study: Resource Ramp
05
101520253035404550
Q1 Q2 Q3 Q4
Res
ourc
es
Onsite
Offshore
Pilot Project
• Validates and tunes process model.
• Builds experience with tools.
• Provides opportunity to refine metrics.
• Builds SMEs in domain and technology.
15Cognizant Confidential
MTC Case Study: Process Model
Request for service
Next Release
Test execution
Test design and developmentTest
planning
Information gathering
Knowledge acquisition
Analysis &estimation
Test evaluation
Automation analysis
Establish regression
Concept / Requirements
Ideas / Analysis
Dev’t / Develop
Scale Up / Launch
(Base line & Automation)
Process ModelPost Launch
Feasibility / Design Use
Case Sign
Off
Test Case &
UA
T Sign O
ff
Archive Deliverables
Quality Report
Quality Plan & Requirements
Sign Off
Request for service
Next Release
Test execution
Test design and developmentTest
planning
Information gathering
Knowledge acquisition
Analysis &estimation
Test evaluation
Automation analysis
Establish regression
Concept / Requirements
Ideas / Analysis
Dev’t / Develop
Scale Up / Launch
(Base line & Automation)
Process ModelPost Launch
Feasibility / Design Use
Case Sign
Off
Test Case &
UA
T Sign O
ff
Archive Deliverables
Quality Report
Quality Plan & Requirements
Sign Off
16Cognizant Confidential
MTC Case Study: Benefits
Productivity
• 24*7 testing model leading to reduction in test cycle time
• Automated Regression Testing
• Quick ramp-up and ramp-down of resources
• Perfectly synchronized onsite-offshore setup
• One Stop shop for providing dedicated resources for various CQG technology needs
• Skilled Rational automation resources
Quality
• Assurance of Business Requirements
• Proven and unique methodology for Testing
• Combination of automation tools
• Continuous Process Improvements
• High test case coverage
• SEI – CMM Level 5 and ISO 9001:2000 certified processes
Cost
• Immediate savings in software testing costs
• Improvement in delivery of Business Requirements
• Save client costs and time by internally managing resources
• Flexible staffing based on resource load variations
• Higher cost reduction due to low onsite ratio
• Compressed time to market - early returns on IT investments
• High automation gains
17Cognizant Confidential
MTC Case Study: Automation Benefits
Key Highlights
• 56% Automation
• Rational Tools used for automation
• Traceability Matrix maintained
• Module Centric Approach followed helps to minimize maintenance effort
• Reusable components to be used across projects
• Common Standards
• Trained & Certified Professionals
• Time & Cost savings during Regression testing
Overall Automation
56%44%
Automation Manual
Effort Savings
70%
80%
90%
100%
% E
ffor
t S
avin
gs
Effort Savings
Automation Vs Manual
0
50
100
150
200
250
300
350
400
450
App 1 App 2 App 3 App 4 App 5 App 6
Per
son
Ho
urs
Manual test execution effort Automated script execution
18Cognizant Confidential
MTC Case Study: Sample Portfolio Metrics (Quarterly)
Project Effort Variation'
Schedule Variation
Defect Density (Defects/Hour)
Productivity(TC Creation)-TCPs/Hours
Productivity(TC Execution) TCPs/Hour
Load Factor
Review Efficiency
Testing Efficiency
Application 1 0.0 -12.0% 0.470 NA 2.740 1.0 0.0 90.3%Application 2 0.0 0.0% 0.028 16.567 8.283 0.4 0.0 96.5%Application 3 0.0 0.0% 0.000 15.400 8.680 0.0 0.0 100.0%Application 4 -0.1 22.0% 0.000 18.400 11.032 0.9 0.0 100.0%Application 5 0.0 0.0% 0.000 N/A 6.120 0.0 0.0 100.0%Application 6 0.0 0.0% 0.000 N/A 8.750 0.8 0.0 N/AApplication 7 0.0 0.0% 0.800 NA 0.970 0.7 0.0 15.3%Application 8 0.0 0.0% 0.014 18.000 9.815 0.4 0.0 100.0%Application 9 2.3 9.1% 0.013 7.925 20.410 0.2 0.0 92.9%Application 10 0.0 0.0% 0.003 N/A 19.732 0.7 0.0 90.9%Application 11 1.2 0.0% 0.150 N/A 9.230 1.0 0.1 90.3%Application 12 1.5 0.0% 0.152 2.243 3.180 0.3 0.1 94.0%Application 13 -0.1 -4.0% 0.043 5.538 3.512 0.8 0.1 88.9%Application 14 0.0 0.0% 0.000 NA 2.060 0.3 0.0 NAApplication 15 -0.1 0.0% 0.043 9.875 3.511 0.7 0.0 100.0%Application 16 -0.3 NA 0.163 11.111 6.897 0.2 0.1 100.0%Application 17 0.0 0.0% 0.000 NA 7.000 1.0 0.0 NAApplication 18 0.0 N/A 0.000 6.000 2.353 0.4 0.0 100.0%Application 19 0.2 0.0% 0.000 N/A 11.885 0.5 0.0 100.0%Application 20 0.3 0.0% 0.000 16.667 N/A 0.4 0.0 N/AApplication 21 -0.2 0.0% 0.088 18.500 7.873 0.2 0.0 100.0%Application 22 0.9 N/A 0.000 N/A 3.229 0.3 0.0 100.0%Application 23 0.0 N/A 0.000 N/A 12.222 0.4 0.0 100.0%
19Cognizant Confidential
Cognizant’s Testing Services
20Cognizant Confidential
Cognizant’s Testing Services
Cognizant Testing Services if a truly independent verification and validation service of Cognizant
Technology Solutions focused exclusively on supporting your software testing needs.
Cognizant Testing Services if a truly independent verification and validation service of Cognizant
Technology Solutions focused exclusively on supporting your software testing needs.
Functional Automation Process Performance
21Cognizant Confidential
Performance Performance TestingTesting
• Catch performance bottlenecks prior to production release• Identify gaps in non-functional requirements definition• Ensure system can grow to meet future demand
Cognizant Testing Services:Service Offerings
Independent Independent Functional Functional TestingTesting
• Reduced costs through effective use of outsourced resources.• Reduced testing cycles with predictable, repeatable results.• Access to deep resource pool for peak test demands.
Test Test AutomationAutomation
• Greatly reduced testing cycle times – especially regression.• Reduced costs and effort.• Supports iterative development and higher frequency of releases.
Test Test Process Process ConsultingConsulting
• Higher system quality – fewer post-production defects• Increased predictability in test effort estimation & execution• Greater efficiency and resource utilization across the enterprise
22Cognizant Confidential
Managed Test Centers:Cognizant’s Comprehensive Testing Solution Offering
CQGCQG
Dedicated CognizantResource Pool
(Core Team)
CognizantTesting ServicesCapabilities
• Processes• Methodology• Infrastructure• Tools• Operating Model• Organization
Integrated Governance Model
MTC
Functional Performance
Automation ProcessFlexible Cognizant
Resource Pool(“Flex” Team)
Critical Features &Benefits
üContinuous process improvement
üCommon automation frameworks
üShared infrastructure and tools
üStandard operating model with structured SLAs
üOptimized onsite/offshore model
üRigorous metrics collection and reporting
üCommon process and methodologies
23Cognizant Confidential
Cognizant’s Testing Services Organization Is Built Upon A Robust Foundation
The Foundation Of Cognizant’s Testing ServicesThe Foundation Of Cognizant’s Testing Services
Independent Independent Functional TestingFunctional Testing
Best of Breed Tools
State-Of-The-Art Infrastructure
Vertical Industry Depth
Skilled & Experienced Professionals
Proven Testing Methodology
Test AutomationTest Automation Test Process Test Process ConsultingConsulting
Performance Performance TestingTesting
24Cognizant Confidential
Skilled & Experienced Testing Professionals
140+110+80+60+
LoadRunnerSilk PerformerWeb loadRational
120+60+25+30+
Performance
260+300+80+280+
WinRunnerRobotQA RunSilk Test
140+125+35+50+
Automation
800+Various200+Functional
Skilled Professionals
ToolsProjects Executed
Type Of Testing
Cognizant’s Testing Services organization is built upon a team of over 1000 software
testing professionals who are dedicated to a career in testing and QA.
• Presentations in International Testing conferences
• Experience in Business Domain
• Executed over 1000 person years of testing
• Active Certifications program:
• CCP, CSTE, CSQA
• RCP (Rational), CPS (Mercury)
• Domain
Thank you2005