ife as a evel 5 test organization - software testing & qa ...€¦ · star ‘99 east -...
TRANSCRIPT
P R E S E N T A T I O N
International Conference On
Software Testing, Analysis & ReviewMAY 10-14, 1999 • ORLANDO, FL
PresentationPaperBio W11
Wednesday, May 12, 19993:15PM
LIFE AS A CMM LEVEL5 TEST ORGANIZATION
John RomanakTelcordia Technologies, Inc.
Software Quality AssuranceIndependent Testing
“Life as a CMM Level 5 Test Organization”May 12, 1999
John N. RomanakExecutive Director
Software Quality Assurance Independent Testing(732) 699-8444
Copyright © 1999 Telcordia Technologies, Inc. Used by permission only.
Life as a CMM Level 5 Test Organization - 2
STAR ‘99 EAST - Bellcore Independent Testing
Bellcore, with its long history of making innovative
telecommunications technology work, is now
Telcordia Technologies
Our name has changed, but our tradition of high
standards and exceptional performance continues.
Life as a CMM Level 5 Test Organization - 3
“Quality, timeliness, cost-effectiveness”
. . . You cannot have all three
STAR ‘99 EAST - Bellcore Independent Testing
Bellcore has proven otherwise!
Life as a CMM Level 5 Test Organization - 4
STAR ‘99 EAST - Bellcore Independent Testing
Corporate Facts• Fast Facts About Bellcore: A Wholly Owned Subsidiary of
Science Applications International Corporation (SAIC)
• Bellcore is a leading provider of Communications Software,
Engineering and Consulting Services based on World-Class
Research
Corporate Statistics• 1998 Revenues exceeded $1 billion
• Employs 6,000
Who is Bellcore?
Life as a CMM Level 5 Test Organization - 5
STAR ‘99 EAST - Bellcore Independent Testing
Key Facts• Approximately 3500 employees are part of Software Quality System
• CMM Level 3 and ISO9000 Certified
• 482 Technical Quality Engineers
• 164 Major Releases Shipped
• 214 Million Lines of Code Tested
• “Best in Class” Cumulative Field Fault results
• “Best in Class” Test Cost Efficiency results
Software Quality System Profile
Life as a CMM Level 5 Test Organization - 6
47.3
43.5
40.2
35.8
28.5
15.4
9.2 8.57.2
4.63.3
0
10
20
30
40
50
4Q93 2Q94 4Q94 2Q95 4Q95 2Q96 4Q96 2Q97 4Q97 2Q98 4Q98
Industry Best In Class
Fau
lts/
KF
P
STAR ‘99 EAST - Bellcore Independent Testing
QMO Compliance
ISO 9001 Certification (Phase I)
ISO 9001 Certification
(All Business Units)
CMM Level 3
Formation ofIndependent TestOrganization (5/93)
Field Fault Density- KFP
Source:Software Systems Metrics Operations Report
Life as a CMM Level 5 Test Organization - 7
709
220
12795
0
100
200
300
400
500
600
700
800
Jan Feb Mar Apr May June July Aug Sep Oct Nov Dec
1995
1996
1997
1998
Source: Customer Service Center Metrics Database
STAR ‘99 EAST - Bellcore Independent Testing
Data for 1995, 1996 and 1997 is displayed as provided at the end of each year and not retroactively adjusted
Sev1/2 Maintenance Requests
Life as a CMM Level 5 Test Organization - 8
Source: Bellcore Corporate Survey
60%65%
80%85%
91%95%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1992 1993 1994 1995 1996 1997
Positive Trend in Customer Satisfaction
Customer SatisfactionSTAR ‘99 EAST - Bellcore Independent Testing
ISO 9001 Registered
CMM Level 3 Rating
Life as a CMM Level 5 Test Organization - 9
STAR ‘99 EAST - Bellcore Independent Testing Start the Quality Journey
June, 1989: Why start:
“Mindset Change”
Independent Test - Clean Separation of System Test from Design & Development
Poor Relationship
With Development
Testing Provides
Low Value
Low Test Automation
Low Customer
Satisfaction
Product Vs.Solution Focus
Life as a CMM Level 5 Test Organization - 10
STAR ‘99 EAST - Bellcore Independent Testing The Journey Continues
May, 1993: “Corporate Organizational Change”
Independent Test Launched - Software Systems Institutionalizes Clean Separation of System Test from Development.
Test Management Vision Launched
• What Would the World Class Testing Organization Look Like?
• How Would the World Class Testing Organization Operate?
• What Would it be Like to be Part of the World Class
Testing Organization?
• What Would the Results Look Like?
Life as a CMM Level 5 Test Organization - 11
STAR ‘99 EAST - Bellcore Independent Testing The Journey Advances
April, 1995: “Cultural Realization”:
Management Change - Corporate Alignment
Strict Quality Discipline - QMO
• Testing is a Key Component Throughout the Process
• Early Testing is Critical to Success = “Prevention Oriented”
Life as a CMM Level 5 Test Organization - 12
• Alignment with Business Unit Strategies
• Test Strategies Driven by Product Life Cycle
• Tactical Implementation of Business Model Goals
• Focus on Cost Reduction, Cost Avoidance, Time-to-Market
STAR ‘99 EAST - Bellcore Independent Testing
Know Your Products - Manage Them “Fundamentally” Different
SQA Testing Business Model
“Advance the Leadership”
Manage Test like a Business
Life as a CMM Level 5 Test Organization - 13
STAR ‘99 EAST - Bellcore Independent Testing Strong Software Quality Assurance Structure
October, 1997: “Software Quality Assurance Alignment”
Corporate alignment of Quality Operation Center with new President & COO
SQA Web Site
• Internal Access by Corporate Officers
• Business Operations Council Review
• Overall Quality Assessment - “Quality Conscience”
Life as a CMM Level 5 Test Organization - 14
The Software Systems Quality Assurance Testing Organizations,provide independent centralized test services for Bellcore’s
Software Systems Applications.
STAR ‘99 EAST - Bellcore Independent Testing
TelcordiaTechnologies
Shipped on Time
SQA Home
Business Models
Documentation
Meeting Info
Products
QA Automation Arch.
Solution Status
Product Status
Year 2000
SQA Web Site - Home Page
Qual. & Oper. Center
Metrics
Life as a CMM Level 5 Test Organization - 15
STAR ‘99 EAST - Bellcore Independent Testing
10200 Software Quality Assurance TestingProduct Releases in Development - Product Status Summary
May 12, 1999(Corporate VP)
Department 10230 (J.N. Romanak - ED)
VP - GM
Business Unit
Rel # Ship Date
Y PRODUCT 1 1.1 6/30/99
Y PRODUCT 2 1.2.3 7/1/99
R PRODUCT 3 4.0 7/29/99
Y PRODUCT 4 2.2 6/30/99
Y PRODUCT 5 2.2.2 10/31/99
Y PRODUCT 6 2.3 6/30/99
G PRODUCT 7 2.4 10/31/99
G PRODUCT 8 3.1.2 9/30/99
G PRODUCT 9 4.1 10/31/99
SQA Web Site - Product Status
Life as a CMM Level 5 Test Organization - 16
STAR ‘99 EAST - Bellcore Independent Testing SQA Web Site - Detailed Metrics
HISTORICAL IN PROGRESS
Estimate Current
Release # 1.0 2.0 3.0 4.0
Contractual Ship date 07/29/99
Planned Ship Date 07/29/99
Actual Ship Date
RELEASE PROFILE
Assessment
Key Points
PT PROFILE
FAULT PROFILE
Milestones Plan Forecast Actual
Life as a CMM Level 5 Test Organization - 17
STAR ‘99 EAST - Bellcore Independent Testing
• Soar with your Strengths√ Technical Staff
• Advance the Leadership√ “The Management”
• Dare to Soar√ Rapid Growth
Reset the Test Management Vision - ‘98
Business Growth↑ 204% Increase in Products Tested↑ 267% Increase in Deployed Code↑ 145% Increase in Major Releases Shipped Quality Improvement↑ 306% Increase in Test Automation↑ 93% Improvement in Field Fault Density/KFP↑ 81% Improvement in Field Fault Density/MLOC
Cost Efficiency↓ 61% Decrease in Test Cost Per Line of Code
Tested
Impact to the Corporation
Life as a CMM Level 5 Test Organization - 18
* Field Fault Density
* Sev 1/2 Maintenance Requests
• Test Automation
• PT/MUT Test Faults
• Removal Efficiency
• Total Lines of Code Supported
• Pre-Shipment Fault Density
• Major Releases Shipped
• Product Test Fault Density
• Test Cost Efficiency
Measures of Progress - Blue Chart
“The Yardstick”
STAR ‘99 EAST - Bellcore Independent TestingProven Quality Results
Future
• Automatic Test Case Generation
• Test Coverage
• Strong Partnership with Applied Research
Life as a CMM Level 5 Test Organization - 19
• Industry Leader in Quality Process and Discipline
• Strong Quality Champion and Catalyst
• SQA Testing Business Model
• Solution/Integration Testing Focus
• Industry Leader in SQA Testing Measures of Progress Results
• State of the Art Test Tools
• “Best In Class” Independent Centralized Testing Operation
“Soar with your Strengths...”
STAR ‘99 EAST - Bellcore Independent TestingProven Strengths - Distinctive Competence Differentiators
Life as a CMM Level 5 Test Organization - 20
STAR ‘99 EAST - Bellcore Independent Testing
Quality Is Our Differentiator
“You can have all three”
Quality, Timeliness, Cost Effectiveness
“Life as a CMM Level 5 Test Organization”
The Differentiator
Life as a CMM Level 5 Test OrganizationCopyright 1999 Telcordia Technologies, Inc. Used by permission only.
1
Bellcore, with its long history of making innovativetelecommunications technology work, is now
Telcordia Technologies
Our name has changed, but our tradition of high standardsand exceptional performance continues.
________________________________________________
STAR’99EAST Bellcore Paper
“Life as a CMM Level 5 Test Organization”
Introduction:
Quality, timeliness, cost-effectiveness – for years software professionals have
believed you cannot have all three. Bellcore has proven otherwise. Bellcore’s
approach to Software Quality has made us a leading provider of World-Class software.
Bellcore’s Software Development organization numbers about 3500 employees, with a
portfolio of 150 operations support systems and over 110 Million Lines Of Code
(MLOC). Positioning Bellcore for SEI CMM Level Five in 1999.
Testing is needed now more than ever and testing is now more complex than
ever. Testing is not a revenue-producing activity, but rather a revenue-preserving one.
However, testing cannot be an add-on activity, after software development is complete.
It must be integrated throughout the development lifecycle in order to be cost-effective.
A more effective testing capability leads directly to increased customer satisfaction, as
systems work and work right the first time. A more efficient testing capability leads
directly to increased Quality, as test cost savings increase so does customer
profitability and satisfaction.
Life as a CMM Level 5 Test OrganizationCopyright 1999 Telcordia Technologies, Inc. Used by permission only.
2
Goals and Objectives:
• Journey through Development and Test Culture paradigm changes
• Understand how to Manage Test like a Business
• Experience what it’s like to be a Corporate Independent SQA Testing Centeraligning directly to the President of the Company
• Explore Bellcore’s sophisticated SQA Web interface for overall SoftwareQuality Assessment, which can be accessed by executives to view eachsystem’s overall quality “health” status based upon proven process anddiscipline
• Proven Measures of Progress historical metrics that serve as Bellcore’sdistinctive competence differentiation in being an Industry Leader inIndependent SQA Testing operations
Meeting the Challenge for Test Excellence:
The quality of software testing is normally inconsistent, with different organizations
performing testing in different ways. Although corporate quality models can exist, with
clearly defined milestones and deliverables, the processes used to create deliverables
are not always detailed and the model is not always used by all organizations. While
many excellent test processes and knowledgeable people exist within the industry, this
knowledge is not applied consistently to achieve predictable, consistent testing
outcomes.
Key issues encountered in the Industry with its current testing environment are:
• Unclear ownership of the testing discipline in the company
• General lack of esteem in which testers are generally held compared toother software development professionals
• Inconsistent levels of involvement of users in the test planning process, andtesters in the planning and software development efforts
Life as a CMM Level 5 Test OrganizationCopyright 1999 Telcordia Technologies, Inc. Used by permission only.
3
• Ineffective integration of software testing throughout the softwaredevelopment life cycle
• Proliferation of locally controlled testing facilities and testing within theproduction environment
• Lack of a centralized program to define, develop and maintain test skills.
Bellcore’s testing challenge is for a robust, mature testing practice to ensure that
corporate software quality goals are achieved consistently and predictably. This will
require a corporate-wide testing methodology, tools, training and guidance, as well as
infrastructure support for the initiative. Key goals of this practice are:
• Clear demonstration of management support for testing
• Well-defined test processes used by trained individuals
• Testing efforts that are aligned with business goals
• Testing measurements to ensure adequate test quality, performance and
accountability
• High level of quality and overall customer satisfaction for testing.
Bellcore clearly understands testing vision and software quality assurance challenges
that lie ahead. There are many similarities between software industry’s current situation
and its challenges and Bellcore’s experiences with software development. Often,
testing of software products is an afterthought - a requirement for sign-off or at best an
opportunity to find the most obvious of the software “bugs” prior to production. The
focus is typically on defect removal instead of on defect prevention. It is much more
cost effective to avoid “bugs” before coding is completed than to test, identify and
correct software anomalies after coding is completed. Bellcore’s solution for software
quality assurance testing is based on the premise that the software testers are involved
throughout the Software Development Lifecycle. From the development of software
testing strategies during the software sales process to the production implementation at
a customer site, the Testers are always valued members of the software development
team. The Testers provide important insights on the testability of the requirements, the
Life as a CMM Level 5 Test OrganizationCopyright 1999 Telcordia Technologies, Inc. Used by permission only.
4
usability of the designs, and the overall test strategy required to fully test the feature,
product or solution at hand. Since the testers typically have the most “hands-on”
experience (next to the actual customer), they bring highly valuable customer insights
to the team that will help to improve overall customer satisfaction. The involvement of
the Independent Test Staff throughout the Software Development Lifecycle will also
help to raise the overall technical expertise of the Test staff as they increase their
understanding of the product and the overall end-customer requirements and usage.
Experience Bellcore’s Journey for Test Excellence:
Bellcore has been an industry leader in testing techniques and quality assurance
applications. Bellcore has developed a unique testing program and methodology that
has led to a level of product and process interoperability that is unmatched in the
industry. Bellcore’s Software Systems Organization has received industry recognition
in two major quality arenas:
♦ In October 1996, Bellcore’s ISO 9001 Certification was expanded to include all of
Software Systems. The quality system was examined for conformance to the
twenty elements required by ISO through a random sampling across a wide
spectrum of projects. With no open non-conformities at the end of the audit, we
were recommended for immediate certification.
♦ In December 1996, Bellcore Software Systems successfully completed the
Software Engineering Institute’s Capability Maturity Model (CMM) assessment and
achieved a Level 3 rating with significant strengths noted in some key process
areas at higher levels. Only 11% of organizations assessed have been rated as
Level 3 or higher. In terms of organizational size and breadth of scope, Bellcore
Software Systems was the largest assessment ever performed.
Bellcore’s software quality journey for “Best in Class” Independent Testing began in the
1992/1993 timeframe. At that time, only 30% of our testing was automated. Testing
was viewed as providing low value and the relationship between the testing and
development organizations was poor at best.
Life as a CMM Level 5 Test OrganizationCopyright 1999 Telcordia Technologies, Inc. Used by permission only.
5
Bellcore’s approach to Independent Software Quality Assurance Testing, a clean
separation of system test from design and development, has been a key component of
this success. Bellcore's testing methodology specifies the ownership of tasks and
responsibilities and also the process of obtaining agreement and buy-in across
organizations. In addition, goals are concise and quantitative and linked to Bellcore’s
appraisal process. Bellcore's testing methodology requires that quality is everyone's
responsibility across the organization. Bellcore's methodology specifies the roles of
each organization and of each participant. Deliverables by all organizations involved in
the system development are reviewed and signed off according to the criteria set by the
team. Each test activity has roles and participants clearly identified. Bellcore’s strategy
includes methods for determining the number of staff, level of expertise and particular
job functions required for a software project/product based on where in the software
lifecycle the project/product falls. In general, Bellcore’s methodology includes 4 primary
job functions: Test Project Manager (responsible for initiatives related to the Quality
Assurance Test function), Test Manager (responsible for the overall test activities for
the product/project that his/her staff supports and the management of this staff), Lead
Tester (responsible for the technical aspects of the testing and for mentoring the
testers) and Tester (responsible for detailed test planning and execution).
It is a common behavioral characteristic of software development organizations to staff
the software testing function with under-qualified personnel. Most often, talented
market hires are directed to the software development shop before consideration is
given to the software testing organization. It is not uncommon for a Test Organization
to report to the same management as the Requirements and Development staff. This
structure lends itself to the redirection of qualified technical staff to work on systems
engineering and/or development efforts instead of software testing. The first step in
the evolution of improving software quality is to improve the quality of the test staff.
The key here is what Bellcore has done with the formation of an independent test
organization, so there is a clean separation of system test from design and
development. Make it known within the company that only “the best” will work in the
Independent Testing Organization. Make sure this Test Organization has the best
Life as a CMM Level 5 Test OrganizationCopyright 1999 Telcordia Technologies, Inc. Used by permission only.
6
equipment – not “hand me downs” from Development. Hire highly technical,
experienced people. Seed the Test
Organization with a handful of stars from inside and outside the company to increase
the credibility of the Test Organization and to act as role models, mentors, and trainers
for the less technical staff. Do not allow poor performers to transfer to the Independent
Testing Organization. Reward the test staff when they succeed. Make it clear that
Testing is a profession with a clear career path. Set up clear job descriptions for each
level of staff with titles that are comparable to those of the Development Organization.
Make sure the Training Organization provides clear training tracks and courses for
professional testers. Provide opportunities for the test staff to attend outside training
and Test conferences in order to raise awareness that Testing is a valuable career
choice for software professionals.
Bellcore’s experienced test managers tailor and implement comprehensive testing
solutions, while developing an overall test strategy and plan to support its Software
Systems’ Business. By following a well defined Test Strategy, we are able to
implement a corporate-wide testing methodology, tools, training and guidance. Testing
efforts will be aligned with business goals, and progress will be measured to ensure
adequate test quality, performance and accountability, with a focus on quality and
overall customer satisfaction. Bellcore’s business of software testing is based on the
maturity of the software product or solution being tested. For example, a product’s test
automation strategy will be very different for an emerging product than for an
established product. The focus of the automation strategy is cost reduction, cost
avoidance, and time to market. In addition, Bellcore’s Independent Testing operations
moves the decision-making and budget responsibilities for each product’s test function
to the line manager of the test group. The Test Manager is responsible for test cost
estimation, test budget tracking, and personnel. Also, the test manager and the
individual testers are responsible for signing the test readiness checklist and signoff
sheet for each product release. Bellcore additionally calls for key learning sessions
after each phase of the software process and the Independent Testing Organization
provides valuable insights for each of those. In addition, our Test Organization uses a
Life as a CMM Level 5 Test OrganizationCopyright 1999 Telcordia Technologies, Inc. Used by permission only.
7
full range of in-process test metrics to manage each testing phase, and to determine
areas for improvement going forward. These metrics are produced at the product level
and rolled up to the corporate level with intermediate levels as necessary. The
corporate level metrics are used to
compare and contrast Bellcore’s performance against software industry trends and to
track overall progress as a company. Individual product metrics are used to determine
the items to be entered in each product’s Quality Improvement Plan and to measure the
product’s progress in achieving these planned improvements.
In addition to metrics, the Bellcore Independent Testing Organization incorporates the
use of a SQA Testing Project Manager. The Project Manager is responsible for guiding
the Independent Testing Management team in identifying key areas for improvement
for the Test Function as opposed to the individual Products, and for identifying and
implementing the steps necessary to move the Testing and overall Quality to best in
class in the industry. Bellcore has dramatically increased the number of test cases
executed per tester in its own testing organization by the effective use of test
automation. Test tools play a critical role in an organization's overall quality program.
Tools make it possible to improve the efficiency and effectiveness of the testing
process. When used prudently, test tools can increase the rate of defect discovery and
decrease test cycle times. However, successfully introducing tools into an organization
requires an overall automation program. The program must be based on (1) goals with
realistic expectations for return on investment, (2) an organizational structure with
strong management support and (3) technology built with sound software engineering
discipline. Experience has shown that automation goals should not be set uniformly
across all application teams. Goals for automating should be established based on the
following factors: application lifecycle, application life-span, relationship to critical
business work-flows, and platform strategy. Developing automated scripts at the wrong
time or for the wrong reasons can be much more expensive than manual testing.
Therefore, the automation program should be initiated with specific and realistic
expectations for the return on investment.
Life as a CMM Level 5 Test OrganizationCopyright 1999 Telcordia Technologies, Inc. Used by permission only.
8
Metrics are critical to the successful implementation of a quality testing organization for
several reasons. First of all, metrics give all members of the team visibility into the
project as well as advance warning of problems. In addition, metrics allow the
organization to track its progress on quality improvement and celebrate success. Some
of the metrics that Bellcore uses successfully as its “yardstick” are:
• Field Fault Density
• Severity 1 and 2 Maintenance Requests
• Test Automation
• System Test/Multi-Unit Test Faults Ratio
• Removal Efficiency
• Pre-shipment Fault Density
• Major Releases Shipped
• Test Cost Efficiency
In addition, Bellcore has established a considerable in-process metrics program. This
program allows for the Management and Staff of the product teams to set goals and
evaluate their progress against the goals on a consistent basis. In-process metrics can
be used to determine if additional Management intervention is needed when products
fail to meet these in-process goals. Bellcore has considerable experience in
implementing and managing realistic jeopardy action plans that will guide a product
back onto schedule. Bellcore has found that this set of metrics provides valuable insight
into areas for improvement and the level of product quality.
Bellcore's testing methodology clearly states organizational testing goals, in both
subjective measures and metrics. Goals are formed “top-down” and clearly support the
overall corporate goals. Quarterly corporate test and quality goals are measured and
the corresponding graphs and charts are strategically displayed in all locations so that
each member of the product team (not only the test organization) is aware of the
current status. In particular, the test metrics of effectiveness (number of MRs found by
Life as a CMM Level 5 Test OrganizationCopyright 1999 Telcordia Technologies, Inc. Used by permission only.
9
testing, number of MRs found in deployment, etc.) and efficiency (percent of test cases
automated tracked to the business model for automation, etc.) combine to clarify each
testing organization’s goals.
Bellcore’s Proven Results:
Since implementation of our Quality Process and our Independent Software Quality
Assurance (SQA) Testing Organization, we have reduced the number of Severity 1 and
2 troubles our customers experience by 82% from 1995 to 1997, and, 51% through
1998, saving tens of millions of dollars for both our customers and ourselves, and
improving our overall customer satisfaction from 60% to 95%.
To put these results in context, from 1993 to 1998, Bellcore’s Software Systems
organization experienced the following growth in its business:
• 204% increase in products tested
• 267% increase in deployed code
• 145% increase in major releases shipped
In terms of quality improvement, Bellcore’s results are:
• 306% increase in test automation
• 93% improvement in Cumulative Field Faults/KFP
• 81% improvement in Cumulative Field Faults/MLOC
From a cost efficiency standpoint, Bellcore has achieved a 61% decrease in Test Cost
per Line of Code Tested over the same period.
Life as a CMM Level 5 Test OrganizationCopyright 1999 Telcordia Technologies, Inc. Used by permission only.
10
Bellcore's proven testing methodology involves culture, process, discipline, and
organizational change. Bellcore acknowledges the difficulty to introduce and
implement this new process. However, this methodology will lead to institutionalizing
test quality assurance, with measurable Quality Improvement.
Bellcore’s Testing Paradigm Change:
Bellcore has created a world class software development quality assurance testing
organization. We know the pitfalls and success factors, especially for an organization
of the size and complexity of Bellcore’s. We know what it takes to make such an
enormous paradigm change to where software testing is a valued part of software
development, software testers are highly critical members of the product/project team,
and where the focus shifts from error detection to error prevention. We have a proven
model for job and career satisfaction for testers and test managers that enables hiring
and retaining excellent staff in Testing.
Benefits of Bellcore’s Independent Testing Organization to the business include:
• Shortening implementation time for launching internal testing programs
• Assuring high quality results by working with a world class process
• Reducing costs of recruiting and retaining top personnel
• Directing investment to the right equipment and tools
• Giving personnel more time to focus on developing their own expertise andcapabilities
• Improved planning and budgeting
• Exceeding customers’ expectations by starting testing earlier so customers’ servicesget to market on time
• Achieving “Best in Class” measurable Quality Improvement and Metrics.
Life as a CMM Level 5 Test OrganizationCopyright 1999 Telcordia Technologies, Inc. Used by permission only.
11
In Closing:
Bellcore’s SQA Independent Testing Organization is now industry-recognized as best in
class. We have presented numerous papers describing our SQA Testing approach at
industry conferences such as STAR, Software Quality week, EuroSTAR and ISSRE. In
the last 3 years, we have provided workshops on Independent SQA Testing and have
shared our success story for many large corporations.
References:
Special thanks goes to Anita Sauer, Executive Director, and Chris Spuler, Senior
Director, for their experience and insights in shaping key points in this paper.
John N. RomanakJohn N. Romanak – Senior Executive Director - Software Quality Assurance IndependentTesting for Telcordia's Software Development organization. John leads SQA Testing ProjectManagement, Contract/Proposal Management for Elective Testing Services,Testing/Automation Architecture and Applied Research - Testing Paradigm advancement.
Thirty years’ experience in Software Systems, with the past seven years concentrated inbuilding a world class independent testing organization at Telcordia. Telcordia’s SoftwareDevelopment organization numbers about 3,500 employees, and is ISO9000 and SEI CMMLevel Three certified. Telcordia’s quality journey began in 1993, when testing was viewedas providing low value and there was a very poor relationship between testing anddevelopment. Telcordia’s SQA Independent Testing organization is now industry-recognized as best in class. Quality, timeliness, cost-effectiveness – for years softwareprofessionals have believed you cannot have all three. Telcordia has proven otherwise.John Romanak has given workshops on Independent SQA Testing and has sharedTelcordia’s success story with many large corporations.