software testing: legacy

Software Testing:Legacy & Future

2005

Josh SoldatiHead Of Testing Services, North America

2Cognizant Confidential

Testing No Longer The Ugly Step-Child

“Once the backroom boys, the ‘invisibles’ of the software industry are striking back. Testing has assumed strategic importance in all software services companies today. Customers and users have become tired of paying top dollar for software, which either does not work, or if it does, then not according to expectations.”

- Business World India*

*Emphasis added.


Software Testing: The Legacy

RequirementsInception DesignCode & Unit Test Testing

Rollout & Deployment

Testing considered a discrete step in the lifecycle – the last step.

Testing and software quality assurance not considered specialized skills or career path.

No formal processes or metrics established for software testing.

• Testing teams involved too late in the lifecycle, limiting ability to provide accurate and comprehensive test coverage

• Testing effort not adequately estimated, impacting delivery schedule and/or budget

• Developers test their own code

• Top talent moves out of software quality roles

• Unqualified resources perform and manage testing activities

• Limited ability to measure/improve either efficiency or quality of testing effort

The Legacy The Impact


The Ultimate Cost: Late Defect Detection*

Requirements Design Code Unit Test Test Execution Production

Find/Fix defects during Requirements = x to 2x = $75 to $150**

Find/Fix defects during Design = 5x = $375**

Find/Fix defects during Coding = 10x = $750**

Find/Fix defects during Unit Testing = 20x = $1500**

Find/Fix defects during Test Execution = 50x = $3750**

Find/Fix defects during Production = 200x = $15000**

*Source:Gartner.

** Example cost figures assume x = 1 person hour, at a cost of $75/hr.


Breaking Away From The Legacy:Cognizant’s Holistic View

Integrated Development LifecycleIntegrated Development Lifecycle

Integrated Testing Processes & Methodology

RequirementsInception DesignCode & Unit Test

TestExecution

Rollout & Deployment

Testing Infrastructure & Tools.

Testing Operating ModelTesting Operating Model

Testing OrganizationTesting Organization


Cognizant’s Testing Framework:Critical Components Of Software Testing

Process• Code Promotion*• Build Acceptance• Defect Management*• Test Environment Management• Test Execution Reporting• Knowledge Management• Metrics Tracking• Delivery Assurance• Process Quality Initiatives

Process• Code Promotion*• Build Acceptance• Defect Management*• Test Environment Management• Test Execution Reporting• Knowledge Management• Metrics Tracking• Delivery Assurance• Process Quality Initiatives

Methodology• Test Planning & Estimation• Requirements Traceability*• Integration Testing• System Testing• User Acceptance Testing• Performance Testing• Regression Testing• Test Data Creation• Automation Testing• Non-Functional Testing

Methodology• Test Planning & Estimation• Requirements Traceability*• Integration Testing• System Testing• User Acceptance Testing• Performance Testing• Regression Testing• Test Data Creation• Automation Testing• Non-Functional Testing

Infrastructure• Staging environments*• Test beds• Desktops• Networks

Infrastructure• Staging environments*• Test beds• Desktops• Networks

Tools• Test Management• Performance• Automation• White Box• Requirements Management*• Templates & Other Tools

Tools• Test Management• Performance• Automation• White Box• Requirements Management*• Templates & Other Tools

Operating Model• Project Structuring• Staff Assignment• Work-load Management• Organizational SLAs

Operating Model• Project Structuring• Staff Assignment• Work-load Management• Organizational SLAs

Organization• Organization Structure• Skills Mix• Training Programs• Professional Certifications• Career Path

Organization• Organization Structure• Skills Mix• Training Programs• Professional Certifications• Career Path

* Linked to overall development methodology and lifecycle.


Consolidation Of Software Testing

Project Team ,Department, orBusiness Unit

Development Team

Testers


Testers


Testers

Decentralized Software Testing

• Testing teams aligned by project, department, or business unit.

• Varied methodologies, processes, tools, infrastructure and metrics across groups.

• Limited or non-existent software testing SLAsestablished across the enterprise.

• Inefficient management of resources – people and infrastructure.

Consolidated Quality Groups




Development Team

Development Team

Development Team

Development Team

Development Team

Consolidated Quality Group

• CQG organization operates across the enterprise and supports individual teams/groups on a scheduled work queue basis.

• Consistent methodologies, processes, tools, infrastructure and metrics across groups.

• SLAs established and performance metrics measured throughout..

• Leverage resources across teams/groups and manager work.

Evo

luti

on


The Move To Of Offshore Testing

§ Independent Verification & Validation is now offered by all major offshore service providers.

§ Represents fastest growing business within Cognizant, and an increasingly larger portion of the offshore IT spend.

§ Many organizations have piloted offshore testing efforts over the past three years and are now embracing across the enterprise.

§ Key drivers:

- Achieve cost savings

- Leverage mature processes

- Establish better controls, metrics, and SLAs

- Gain access to expertise (performance testing, automation testing, test management, etc.)

- Flexible resource management

- Opportunity for 24X7 Testing


Managed Test Centers:The Marriage Of Onsite/Offshore With Centralized Quality Groups

CQGCQG

Dedicated Resource Pool(Core Team)

Rationalized:

• Processes• Methodology• Infrastructure• Tools• Operating Model• Organization

Integrated Governance Model

MTC

Functional Performance

Automation ProcessFlexible Resource Pool

(“Flex” Team)

Critical Features &Benefits

üContinuous process improvement

üCommon automation frameworks

üShared infrastructure and tools

üStandard operating model with structured SLAs

üOptimized onsite/offshore model

üRigorous metrics collection and reporting

üCommon process and methodologies


Managed Test Center:An Example Scenario

The Current State Scenario

§ An organization has a 50 person testing team, comprised of 25 FTEs and 25 contractors.

§ Fully loaded costs and benefits for an organization's testing team members are $100,000/yr.

§ Annual voluntary attrition is 10%, with the average cost of replacement cost equivalent to 3 months salary and benefits..

§ The organization has implemented some automation – with about 20% of functional testing activities automated.

The Future State Scenario

§ The organization would retain 10% of it’s internal and contract resources for testing. There would be a 10% reduction in force, with the remaining 80% of internal FTEs reassigned to other tasks within the organization.

§ Approximately 70% of the Managed Test Center team would form the “core” team, while an additional 30% capacity (the “flex” team) would ramp up cyclically every four months.

§ The onsite/offshore ratio would fluctuate between 10 – 15% onsite vs. 85 – 90% offshore depending on current release cycle demands.

§ The knowledge transfer and piloting phase would last three months, with a ten person team (6 onsite, 4 offshore).

§ By the end of the third year, the Managed Test Center will have achieved an overall 15% improvement in efficiency.

§ Total resource requirements would drop year-over-year based on automation and efficiency gains.


Managed Test Centers:A Three Year Labor Cost ROI Scenario

Year 1 Year 2 Year 3 TotalCurrent State

Internal Testing Labor CostsTesting Team FTE 2,500,000.00$ 2,500,000.00$ 2,500,000.00$ 7,500,000.00$

Testing Contractors 2,016,000.00$ 2,016,000.00$ 2,016,000.00$ 6,048,000.00$

Voluntary Attrition Costs 62,500.00$ 62,500.00$ 62,500.00$ 187,500.00$

TOTAL CURRENT STATE COSTS 4,578,500.00$ 4,578,500.00$ 4,578,500.00$ 13,735,500.00$

MTC Future State

Internal Testing Labor CostsTesting Team FTE 812,500.00$ 250,000.00$ 250,000.00$ 1,312,500.00$

Testing Contractors 655,200.00$ 201,600.00$ 201,600.00$ 1,058,400.00$

Voluntary Attrition Costs 20,312.50$ 6,250.00$ 6,250.00$ 32,812.50$

Internal Separation Costs 20,000.00$ -$ -$ 20,000.00$

Internal Testing Costs 1,508,012.50$ 457,850.00$ 457,850.00$ 2,423,712.50$

Cognizant Labor CostsOnsite 776,067.83$ 517,208.33$ 507,873.24$ 1,801,149.40$

Offshore 1,499,693.83$ 1,398,249.22$ 1,369,025.28$ 4,266,968.33$

Cognizant Labor Costs 2,275,761.66$ 1,915,457.54$ 1,876,898.52$ 6,068,117.73$

TOTAL MTC LABOR COSTS 3,783,774.16$ 2,373,307.54$ 2,334,748.52$ 8,491,830.23$

Labor Cost Savings

MTC Cost Savings BreakdownDirect Labor Costs 586,644.40$ 1,402,819.42$ 1,429,820.71$ 3,419,284.53$

Incremental Automation Benefits -$ 393,724.80$ 393,724.80$ 787,449.60$

Efficiency 208,081.44$ 408,648.24$ 420,205.97$ 1,036,935.65$

TOTAL ANNUAL SAVINGS ($) 794,725.84$ 2,205,192.46$ 2,243,751.48$ 5,243,669.77$ TOTAL ANNUAL SAVINGS (%) 17% 48% 49% 38%


Managed Test Centers:A Three Year Labor Cost ROI Scenario

Managed Test Center: 3 Year ROI ScenarioAverage Annual Staffing Levels

0.0

10.0

20.0

30.0

40.0

50.0

60.0

Year 1 Year 2 Year 3

Res

ou

rces

Client FTEs Client Contractors MTC Onsite MTC Offshore


Managed Test Center Case Study: Fortune 50 Global Conglomerate

Results• Built a dedicated 50 person CQG to operate within enterprise and project-level SLAs.• Established well defined software testing processes; developed process road maps, estimation

models, automation standards etc.• Reduced post-production defects through structured requirements analysis and full traceability to

test cases.• Reduced regression test cycle times by automating test cases using Rational Robot. • Increased visibility to testing and development activities through comprehensive metrics collection

and reporting.

SolutionEstablish an offshore Central Quality Group to operate on deliver software testing services organization structured around internal SLAs.

ChallengeFortune 50 conglomerate with decentralized, global IT operations sought to reduce operating costs while increasing efficiency and consistency within it’s software quality assurance organization. Executive leadership mandated continuous process improvement, but environment lacked processes or tools to manage.


MTC Case Study:Approach & Ramp

Q1 Q2 Q3

ü Application 1

ü Application 2

ü Application 3

ü Application 4

ü Application 5

ü Application 6

ü Application 7

ü Application 8

ü Application 9

ü Application 10

ü Application 11

ü Application 12

ü Application 13

ü Application 14

ü Application 15

ü Application 16

ü Application 17

ü Application 18

ü Application 19

Q4

ü Application 20

ü Application 21

ü Application 22

CQG Case Study: Resource Ramp

05

101520253035404550

Q1 Q2 Q3 Q4

Res

ourc

es

Onsite

Offshore

CQG Case Study: Resource Ramp

05

101520253035404550

Q1 Q2 Q3 Q4

Res

ourc

es

Onsite

Offshore

Pilot Project

• Validates and tunes process model.

• Builds experience with tools.

• Provides opportunity to refine metrics.

• Builds SMEs in domain and technology.


MTC Case Study: Process Model

Request for service

Next Release

Test execution

Test design and developmentTest

planning

Information gathering

Knowledge acquisition

Analysis &estimation

Test evaluation

Automation analysis

Establish regression

Concept / Requirements

Ideas / Analysis

Dev’t / Develop

Scale Up / Launch

(Base line & Automation)

Process ModelPost Launch

Feasibility / Design Use

Case Sign

Off

Test Case &

UA

T Sign O

ff

Archive Deliverables

Quality Report

Quality Plan & Requirements

Sign Off

Request for service

Next Release

Test execution

Test design and developmentTest

planning

Information gathering

Knowledge acquisition

Analysis &estimation

Test evaluation

Automation analysis

Establish regression

Concept / Requirements

Ideas / Analysis

Dev’t / Develop

Scale Up / Launch

(Base line & Automation)

Process ModelPost Launch

Feasibility / Design Use

Case Sign

Off

Test Case &

UA

T Sign O

ff

Archive Deliverables

Quality Report

Quality Plan & Requirements

Sign Off


MTC Case Study: Benefits

Productivity

• 24*7 testing model leading to reduction in test cycle time

• Automated Regression Testing

• Quick ramp-up and ramp-down of resources

• Perfectly synchronized onsite-offshore setup

• One Stop shop for providing dedicated resources for various CQG technology needs

• Skilled Rational automation resources

Quality

• Assurance of Business Requirements

• Proven and unique methodology for Testing

• Combination of automation tools

• Continuous Process Improvements

• High test case coverage

• SEI – CMM Level 5 and ISO 9001:2000 certified processes

Cost

• Immediate savings in software testing costs

• Improvement in delivery of Business Requirements

• Save client costs and time by internally managing resources

• Flexible staffing based on resource load variations

• Higher cost reduction due to low onsite ratio

• Compressed time to market - early returns on IT investments

• High automation gains


MTC Case Study: Automation Benefits

Key Highlights

• 56% Automation

• Rational Tools used for automation

• Traceability Matrix maintained

• Module Centric Approach followed helps to minimize maintenance effort

• Reusable components to be used across projects

• Common Standards

• Trained & Certified Professionals

• Time & Cost savings during Regression testing

Overall Automation

56%44%

Automation Manual

Effort Savings

70%

80%

90%

100%

% E

ffor

t S

avin

gs

Effort Savings

Automation Vs Manual

0

50

100

150

200

250

300

350

400

450

App 1 App 2 App 3 App 4 App 5 App 6

Per

son

Ho

urs

Manual test execution effort Automated script execution


MTC Case Study: Sample Portfolio Metrics (Quarterly)

Project Effort Variation'

Schedule Variation

Defect Density (Defects/Hour)

Productivity(TC Creation)-TCPs/Hours

Productivity(TC Execution) TCPs/Hour

Load Factor

Review Efficiency

Testing Efficiency

Application 1 0.0 -12.0% 0.470 NA 2.740 1.0 0.0 90.3%Application 2 0.0 0.0% 0.028 16.567 8.283 0.4 0.0 96.5%Application 3 0.0 0.0% 0.000 15.400 8.680 0.0 0.0 100.0%Application 4 -0.1 22.0% 0.000 18.400 11.032 0.9 0.0 100.0%Application 5 0.0 0.0% 0.000 N/A 6.120 0.0 0.0 100.0%Application 6 0.0 0.0% 0.000 N/A 8.750 0.8 0.0 N/AApplication 7 0.0 0.0% 0.800 NA 0.970 0.7 0.0 15.3%Application 8 0.0 0.0% 0.014 18.000 9.815 0.4 0.0 100.0%Application 9 2.3 9.1% 0.013 7.925 20.410 0.2 0.0 92.9%Application 10 0.0 0.0% 0.003 N/A 19.732 0.7 0.0 90.9%Application 11 1.2 0.0% 0.150 N/A 9.230 1.0 0.1 90.3%Application 12 1.5 0.0% 0.152 2.243 3.180 0.3 0.1 94.0%Application 13 -0.1 -4.0% 0.043 5.538 3.512 0.8 0.1 88.9%Application 14 0.0 0.0% 0.000 NA 2.060 0.3 0.0 NAApplication 15 -0.1 0.0% 0.043 9.875 3.511 0.7 0.0 100.0%Application 16 -0.3 NA 0.163 11.111 6.897 0.2 0.1 100.0%Application 17 0.0 0.0% 0.000 NA 7.000 1.0 0.0 NAApplication 18 0.0 N/A 0.000 6.000 2.353 0.4 0.0 100.0%Application 19 0.2 0.0% 0.000 N/A 11.885 0.5 0.0 100.0%Application 20 0.3 0.0% 0.000 16.667 N/A 0.4 0.0 N/AApplication 21 -0.2 0.0% 0.088 18.500 7.873 0.2 0.0 100.0%Application 22 0.9 N/A 0.000 N/A 3.229 0.3 0.0 100.0%Application 23 0.0 N/A 0.000 N/A 12.222 0.4 0.0 100.0%


Cognizant’s Testing Services


Cognizant’s Testing Services

Cognizant Testing Services if a truly independent verification and validation service of Cognizant

Technology Solutions focused exclusively on supporting your software testing needs.

Cognizant Testing Services if a truly independent verification and validation service of Cognizant

Technology Solutions focused exclusively on supporting your software testing needs.

Functional Automation Process Performance


Performance Performance TestingTesting

• Catch performance bottlenecks prior to production release• Identify gaps in non-functional requirements definition• Ensure system can grow to meet future demand

Cognizant Testing Services:Service Offerings

Independent Independent Functional Functional TestingTesting

• Reduced costs through effective use of outsourced resources.• Reduced testing cycles with predictable, repeatable results.• Access to deep resource pool for peak test demands.

Test Test AutomationAutomation

• Greatly reduced testing cycle times – especially regression.• Reduced costs and effort.• Supports iterative development and higher frequency of releases.

Test Test Process Process ConsultingConsulting

• Higher system quality – fewer post-production defects• Increased predictability in test effort estimation & execution• Greater efficiency and resource utilization across the enterprise


Managed Test Centers:Cognizant’s Comprehensive Testing Solution Offering

CQGCQG

Dedicated CognizantResource Pool

(Core Team)

CognizantTesting ServicesCapabilities

• Processes• Methodology• Infrastructure• Tools• Operating Model• Organization

Integrated Governance Model

MTC

Functional Performance

Automation ProcessFlexible Cognizant

Resource Pool(“Flex” Team)

Critical Features &Benefits

üContinuous process improvement

üCommon automation frameworks

üShared infrastructure and tools

üStandard operating model with structured SLAs

üOptimized onsite/offshore model

üRigorous metrics collection and reporting

üCommon process and methodologies


Cognizant’s Testing Services Organization Is Built Upon A Robust Foundation

The Foundation Of Cognizant’s Testing ServicesThe Foundation Of Cognizant’s Testing Services

Independent Independent Functional TestingFunctional Testing

Best of Breed Tools

State-Of-The-Art Infrastructure

Vertical Industry Depth

Skilled & Experienced Professionals

Proven Testing Methodology

Test AutomationTest Automation Test Process Test Process ConsultingConsulting

Performance Performance TestingTesting


Skilled & Experienced Testing Professionals

140+110+80+60+

LoadRunnerSilk PerformerWeb loadRational

120+60+25+30+

Performance

260+300+80+280+

WinRunnerRobotQA RunSilk Test

140+125+35+50+

Automation

800+Various200+Functional

Skilled Professionals

ToolsProjects Executed

Type Of Testing

Cognizant’s Testing Services organization is built upon a team of over 1000 software

testing professionals who are dedicated to a career in testing and QA.

• Presentations in International Testing conferences

• Experience in Business Domain

• Executed over 1000 person years of testing

• Active Certifications program:

• CCP, CSTE, CSQA

• RCP (Rational), CPS (Mercury)

• Domain

Thank you2005

software testing: legacy

Documents