control model testing
DESCRIPTION
TRANSCRIPT
![Page 1: Control model testing](https://reader034.vdocument.in/reader034/viewer/2022052321/54590aefb1af9f33608b5342/html5/thumbnails/1.jpg)
Matthew Sul l ivan Scot t Barber Sof tware Test Profess ionals Conference Fal l 2011
MANAGING RISK FOR SOFTWARE PRODUCTS
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
![Page 2: Control model testing](https://reader034.vdocument.in/reader034/viewer/2022052321/54590aefb1af9f33608b5342/html5/thumbnails/2.jpg)
“STATE OF THE S/W TESTING PRACTICE”
• Find bugs (identify risks) OR • Check for compliance (V&V) “Role” of
QA/Testing
• Appears undervalued, BUT •Doesn’t provide nearly the value it
could “Value” of
QA/Testing
• Business goals & value propositions • Business risks & risk controls • Executive information needs
QA/Testing is “out of sync” with
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
![Page 3: Control model testing](https://reader034.vdocument.in/reader034/viewer/2022052321/54590aefb1af9f33608b5342/html5/thumbnails/3.jpg)
“THE UNDER-INFORMED DIRECTING THE UNDER-TRAINED TO DO THE UNIMPORTANT”
Artifacts (the Unimportant)
•Bugs no one wants to fix •Metrics no one
understands •Documents no one
reads
Testers (the Untrained)
•Don’t know what the executives need, SO
•They do what they are asked to
Executives (the Uninformed):
•Don’t know how to ask for what they need, SO
•They ask for what they know
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
![Page 4: Control model testing](https://reader034.vdocument.in/reader034/viewer/2022052321/54590aefb1af9f33608b5342/html5/thumbnails/4.jpg)
IMPROVING THE SITUATION (PART 1)
Focus on: •Delivering business value •Reducing business risk
At every business layer, identify & balance: •Responsibility •Accountability
Get your superiors to read Ch 16:Rightsizing the Cost of Testing: Tips for Executives of How to Reduce the Cost of Software Testing; CRC Press 2011
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
![Page 5: Control model testing](https://reader034.vdocument.in/reader034/viewer/2022052321/54590aefb1af9f33608b5342/html5/thumbnails/5.jpg)
IMPROVING THE SITUATION (PART 2)
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
![Page 6: Control model testing](https://reader034.vdocument.in/reader034/viewer/2022052321/54590aefb1af9f33608b5342/html5/thumbnails/6.jpg)
Businesses reduce allocation of resources to testing because of a perception of diminished value.
FEELING UNDER SIEGE?
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
![Page 7: Control model testing](https://reader034.vdocument.in/reader034/viewer/2022052321/54590aefb1af9f33608b5342/html5/thumbnails/7.jpg)
WHAT DIMINISHES VALUE FOR TESTING?
1. Lack of insight into future
2. Redundancy 3. Specification
blocks 4. Lack of
independence 5. Scope
constraint Copyright © 2011 PerfTestPlus, Inc. All rights
reserved.
![Page 8: Control model testing](https://reader034.vdocument.in/reader034/viewer/2022052321/54590aefb1af9f33608b5342/html5/thumbnails/8.jpg)
LACK OF INSIGHT INTO THE FUTURE
Why didn’t this come up in
testing!
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
![Page 9: Control model testing](https://reader034.vdocument.in/reader034/viewer/2022052321/54590aefb1af9f33608b5342/html5/thumbnails/9.jpg)
REDUNDANCY
Sign here, and then sign the next box attesting to
the authenticity of the previous signature.
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
![Page 10: Control model testing](https://reader034.vdocument.in/reader034/viewer/2022052321/54590aefb1af9f33608b5342/html5/thumbnails/10.jpg)
SPECIFICATION BLOCK
Honestly I’d love to start testing today, but first I need detailed requirements. VERY
detailed requirements
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
![Page 11: Control model testing](https://reader034.vdocument.in/reader034/viewer/2022052321/54590aefb1af9f33608b5342/html5/thumbnails/11.jpg)
LACK OF INDEPENDENCE
Its not fun being the captain’s “no-
man”.
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
![Page 12: Control model testing](https://reader034.vdocument.in/reader034/viewer/2022052321/54590aefb1af9f33608b5342/html5/thumbnails/12.jpg)
SCOPE CONSTRAINT
Someone else was supposed to be watching
for icebergs.
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
![Page 13: Control model testing](https://reader034.vdocument.in/reader034/viewer/2022052321/54590aefb1af9f33608b5342/html5/thumbnails/13.jpg)
REQUIREMENT-DRIVEN APPROACH
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
![Page 14: Control model testing](https://reader034.vdocument.in/reader034/viewer/2022052321/54590aefb1af9f33608b5342/html5/thumbnails/14.jpg)
The purpose of testing is to reduce uncertainty about the future impact of technology.
THE MEANING OF LIFE (FOR TESTERS)
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
![Page 15: Control model testing](https://reader034.vdocument.in/reader034/viewer/2022052321/54590aefb1af9f33608b5342/html5/thumbnails/15.jpg)
ALTERNATIVE APPROACH
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
![Page 16: Control model testing](https://reader034.vdocument.in/reader034/viewer/2022052321/54590aefb1af9f33608b5342/html5/thumbnails/16.jpg)
RISK AS A COMMON LANGUAGE
Risk
Security
Functional
Performance Usability
Compliance
Whether explicitly or implicitly, all forms of testing revolve around the reduction and management of risk.
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
![Page 17: Control model testing](https://reader034.vdocument.in/reader034/viewer/2022052321/54590aefb1af9f33608b5342/html5/thumbnails/17.jpg)
To effectively manage risk, you must effectively manage knowledge.
THE SECRET TO MANAGING RISK
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
![Page 18: Control model testing](https://reader034.vdocument.in/reader034/viewer/2022052321/54590aefb1af9f33608b5342/html5/thumbnails/18.jpg)
Control Model Testing is a business-aligned approach to software testing that derives “test cases” from knowledge models of the system based on a risk-based taxonomy .
WHAT IS CONTROL MODEL TESTING?
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
![Page 19: Control model testing](https://reader034.vdocument.in/reader034/viewer/2022052321/54590aefb1af9f33608b5342/html5/thumbnails/19.jpg)
WHAT IS OUR TAXONOMY BASED UPON?
COSO Enterprise Risk
Management Integrated Framework
The Open Group Technical Standard on
Risk Taxonomy
PerfTest Plus Taxonomy Extensions for
Control Model Testing
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
![Page 20: Control model testing](https://reader034.vdocument.in/reader034/viewer/2022052321/54590aefb1af9f33608b5342/html5/thumbnails/20.jpg)
WHAT ARE THE BASIC ENTITIES?
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
![Page 21: Control model testing](https://reader034.vdocument.in/reader034/viewer/2022052321/54590aefb1af9f33608b5342/html5/thumbnails/21.jpg)
THE OPEN GROUP’S RISK ASSESSMENT FRAMEWORK
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
![Page 22: Control model testing](https://reader034.vdocument.in/reader034/viewer/2022052321/54590aefb1af9f33608b5342/html5/thumbnails/22.jpg)
Business •Financial •Legal •Brand or Reputation
Product •Security •Performance •Usability •Other Qualities
Project •Budget •Schedule •Communication
RISK LAYERS
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
![Page 23: Control model testing](https://reader034.vdocument.in/reader034/viewer/2022052321/54590aefb1af9f33608b5342/html5/thumbnails/23.jpg)
UNADDRESSED RISK
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
![Page 24: Control model testing](https://reader034.vdocument.in/reader034/viewer/2022052321/54590aefb1af9f33608b5342/html5/thumbnails/24.jpg)
Controls prevent or mitigate risk which may impact business objectives. Control Model Testing helps identify and assess these controls.
HOW CAN TESTS ADDRESS THREATS AND LEVEL OF RISK?
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
![Page 25: Control model testing](https://reader034.vdocument.in/reader034/viewer/2022052321/54590aefb1af9f33608b5342/html5/thumbnails/25.jpg)
Systems • Firewalls • Encryption • Load Balancing
Preferences • Settings • Security and Access Model
Policies • Code Standards • Monitor and Response • HR
TYPES OF CONTROLS
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
![Page 26: Control model testing](https://reader034.vdocument.in/reader034/viewer/2022052321/54590aefb1af9f33608b5342/html5/thumbnails/26.jpg)
Development • Development and Test Tools • Code standards • Software components
Implementation • Checklists • Installation scripts
Maintenance • Alerts and Triggers • SOPs • Configuration Management
CONTROLS CONTEXT
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
![Page 27: Control model testing](https://reader034.vdocument.in/reader034/viewer/2022052321/54590aefb1af9f33608b5342/html5/thumbnails/27.jpg)
“SAMSARIC” TEST LIFECYCLE
Knowledge
Effort
Analyze
Assess
Evaluate
Report
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
![Page 28: Control model testing](https://reader034.vdocument.in/reader034/viewer/2022052321/54590aefb1af9f33608b5342/html5/thumbnails/28.jpg)
Examine • System • Users • Environment
Identify • Objectives • Processes • Threats • Controls
Output • Initial Control Model
ANALYSIS
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
![Page 29: Control model testing](https://reader034.vdocument.in/reader034/viewer/2022052321/54590aefb1af9f33608b5342/html5/thumbnails/29.jpg)
INITIAL CONTROL MODEL
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
![Page 30: Control model testing](https://reader034.vdocument.in/reader034/viewer/2022052321/54590aefb1af9f33608b5342/html5/thumbnails/30.jpg)
Activities •Identify authorities •Solicit opinions •Evaluate exposure •Determine impact
Outcomes •Risk assessment •Assessed Control Model •Test plan
ASSESSMENT
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
![Page 31: Control model testing](https://reader034.vdocument.in/reader034/viewer/2022052321/54590aefb1af9f33608b5342/html5/thumbnails/31.jpg)
ASSESSED CONTROL MODEL
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
![Page 32: Control model testing](https://reader034.vdocument.in/reader034/viewer/2022052321/54590aefb1af9f33608b5342/html5/thumbnails/32.jpg)
Activities •Execute planned and
derivative tests • Identify discrepancies •Determine capability
Outcomes •Tested Control Model •Test results • Issues /
recommendations
EVALUATION
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
![Page 33: Control model testing](https://reader034.vdocument.in/reader034/viewer/2022052321/54590aefb1af9f33608b5342/html5/thumbnails/33.jpg)
EXECUTED CONTROL MODEL
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
![Page 34: Control model testing](https://reader034.vdocument.in/reader034/viewer/2022052321/54590aefb1af9f33608b5342/html5/thumbnails/34.jpg)
Activities •Communicate •Recommend •Respond
Outcomes •Implementation plan •Knowledgebase update •Confirmation of or
revisions to test plan
REPORTING
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
![Page 35: Control model testing](https://reader034.vdocument.in/reader034/viewer/2022052321/54590aefb1af9f33608b5342/html5/thumbnails/35.jpg)
Leader
Manager
Coordinator
Tester
THE FOUR ROLES IN CONTROL MODEL TESTING
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
![Page 36: Control model testing](https://reader034.vdocument.in/reader034/viewer/2022052321/54590aefb1af9f33608b5342/html5/thumbnails/36.jpg)
Responsibilities: •Representation •Roadmaps
Interests • Information • Certainty
Talents • Communication • Vision
Typical Business Titles •Director of Testing or Quality Assurance • Chief Audit Officer (or Assistant to..) • Principle Consultant
LEADER
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
![Page 37: Control model testing](https://reader034.vdocument.in/reader034/viewer/2022052321/54590aefb1af9f33608b5342/html5/thumbnails/37.jpg)
Responsibilities: •Organizing •Developing
Interests •Capability •Consistency
Talents •Understanding •Motivating
Typical Business Titles •Test Manager
MANAGER
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
![Page 38: Control model testing](https://reader034.vdocument.in/reader034/viewer/2022052321/54590aefb1af9f33608b5342/html5/thumbnails/38.jpg)
Responsibilities • Planning •Oversight
Interests • Successful outcome • Thoroughness
Talents • Teamwork • Attention
Typical Business Titles • Test or QA Lead or Senior • Analyst or Engineer Level 2 or 3 •Manager 1
COORDINATOR
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
![Page 39: Control model testing](https://reader034.vdocument.in/reader034/viewer/2022052321/54590aefb1af9f33608b5342/html5/thumbnails/39.jpg)
Responsibilities •Execution •Analysis
Interests •Discovery •Experimentation
Talents •Curiosity •Skepticism
Typical Business Titles •Test or QA Analyst or Engineer •Analyst or Engineer Level 1 or 2
TESTER
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
![Page 40: Control model testing](https://reader034.vdocument.in/reader034/viewer/2022052321/54590aefb1af9f33608b5342/html5/thumbnails/40.jpg)
Business
Test Leader
Product
Test Manager
Project
Test Coordinator Tester
RISK LAYERS AND ROLES
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
![Page 41: Control model testing](https://reader034.vdocument.in/reader034/viewer/2022052321/54590aefb1af9f33608b5342/html5/thumbnails/41.jpg)
Testing should be an indispensible advisor for leadership Testing should not be a convenience or scapegoat for
development All types of testing revolve around risk management The key to managing risk is managing knowledge Testing needs to be a learning discipline in the context of risk
taxonomy The test process should be a continuous cycle reducing effort
through increased knowledge Testing roles should correlate to management or risk, not
resources
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
SUMMARY
![Page 42: Control model testing](https://reader034.vdocument.in/reader034/viewer/2022052321/54590aefb1af9f33608b5342/html5/thumbnails/42.jpg)
[email protected] [email protected]
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
QUESTIONS?
![Page 43: Control model testing](https://reader034.vdocument.in/reader034/viewer/2022052321/54590aefb1af9f33608b5342/html5/thumbnails/43.jpg)
The Open Group (http://www3.opengroup.org/): Risk Taxonomy Technical Standard - https://www2.opengroup.org/ogsys/jsp/publications/PublicationDetails.jsp?publicationid=12156
The Committee of Sponsoring Organizations of the Treadway Commission, or COSO (http://www.coso.org/)
Enterprise Risk Management-Integrated Framework - http://www.coso.org/ERM-IntegratedFramework.htm
PerfTestPlus, Inc. (http://www.perftestplus.com/) Control-Model Testing – (http://www.perftestplus.com/control-model-testing) Rightsizing the Cost of Testing: Tips for Executives of How to Reduce the Cost of Software Testing; CRC Press 2011
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
RECOURCES
![Page 44: Control model testing](https://reader034.vdocument.in/reader034/viewer/2022052321/54590aefb1af9f33608b5342/html5/thumbnails/44.jpg)
Matthew Sullivan Quality Control Engineer CCH TeamMate Wolters Kluwer
Test and Support Engineer for PricewaterhouseCoopers for 10 years
Extensive experience in audit and risk management industry
Specialist in testing Microsoft .NET, MS SQL Server, and Lotus Notes applications
MS in Sof tware Engineering from Regis University
Scott Barber CTO, PerfTestPlus, Inc Widely regarded exper tise in
per formance. Contributor to:
Performance Testing Guidance for Web Applications– Microsoft Press
Beautiful Testing - O’Reilly Press How to Reduce the Cost of Testing -
Taylor and Francis
Executive Director of the Association for Sof tware Testing
Co-Founder of the Workshop of Per formance and Reliabil ity
ABOUT US
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.