techniques for validating the security quality of infrastructure software john d. mcgregor...
TRANSCRIPT
Outline
Motivation Proposed strategy Detailed actions Conclusion
Theme – Life Cycle Threats
There is not a means for automated testing of large software, both static and mobile code, to detect, identify malicious code, sleeper codes, and exploitable vulnerabilities and to determine and understand the potential impact on the life-cycle of the codes. Current testing approaches are largely manual rather than automated.
CSIIR Workshop Themes document
A Recommended Strategy
Software security vulnerabilities are often caused by defective specification, design, and implementation.
…require that software be designed with security at the very heart of the design process…
Establish a security verification and validation program to evaluate different software development processes and practices for effectiveness in producing secure software.
Certify those processes demonstrated to be effective for producing secure software.
Security Across the Software Development Lifecycle Task Force
Recommended Practices
Statistical testing - Usage based testing permits valid statistical estimation of quality with respect to all the executions not tested and tends to find any remaining high-failure-rate defects early.
Production testing - Two strategies are testing security functionality with standard functional testing techniques, and risk-based security testing based on attack patterns and threat models. A good security test plan (with traceability back to requirements) uses both strategies.
Process models - organizations can use the goals and attributes defined in process models as high-level guides for defining and improving their management and engineering processes in the ways they feel are most appropriate for them.
Security Across the Software Development Lifecycle Task Force
SEI
The security of a software-intensive system is directly related to the quality of its software1.
Over 90% of software security incidents are caused by attackers exploiting known software defects.
Analysis of 45 e-business applications showed that 70% of security defects were design defects.
Experienced and capable software engineers inject, on average, one defect every nine lines of code.
A one million line of code systems typically contains 1,000-5,000 defects when shipped.
1 http://www.sei.cmu.edu/tsp/tsp-security.html
A Final Source
“One of the key things that developers can do to help secure their systems is to write code that can withstand attack and use security features properly.”
Defend Your Code with Top Ten Security Tips Every Developer Must Know 8 out of 10 tips are directly programming issues
http://msdn.microsoft.com/security/securecode/default.aspx?pull=/msdnmag/issues/02/09/securitytips/default.aspx
Basic Assumptions
Testing early is cheaper than testing later. System test of a system with j modules and k
experiments (iterations or different implementations for a given interface) results in the following cost of testing
tj ckkjT ]1)1[(),(
Where ct is the cost per test. Module-level tests with j modules and k
experiments results in
tjkckjT ),(Baldwin and Clark. Design Rules The Power of Modularity, volume 1, MIT
Press, 2000.
Basic assumptions - 2
Design Design Inspection
Code Development Test
Acceptance Production
IBM 1 1.5 60 100
Boehm 1 15 to 40 30 to 70 40 to 1000
Remus 1 20 82
Ackerman 1 2 to 10
Russell 1 2 to 4 30
Though estimates vary, the cost of removing defect increases dramatically later in the life cycle.
Premise
Our premise is that poorly written software will contain more vulnerabilities than well written software where the security quality attribute is a design driver.
Current views of security often take a defensive approach. Some of the security infrastructure even adds to the security risk due to the complexity it adds to the product.
We propose an offensive approach in which security is a key design driver and a priority throughout the development process.
Context
A chain of quality should be threaded through the entire process so that validation is most effective and efficient.
SystemsEngineering
Class NameResponsibility Collaborator
Super Class
RequirementsUse Case Name:Use:
User:Frequency:See Also:
Use Case Modeling
DomainExpert
DomainAnalysis
ArchitecturalDesign
ClassDevelopment
Incremental Integration
and System TestingApplication
Analysis
Client
Use Case Name:Use:
User:Frequency:See Also:
Application Use Cases
View
ModelCon
trolle
r
Class Specification
ClassDesign
ClassDerivation
ClassReuse
Implementation Refinement
TestingClass Delivery
Architectureinspection
Clusterdesign inspection
Analysis model inspection
Analysismodel inspection
Requirementsmodel inspection
A Proposed Strategy
Develop method engineering tactics and guidelines that enhance the security quality of the software through improved processes.
Structure architecture evaluation techniques to focus on security by searching for static security patterns.
Discover and capture test patterns that correspond to dynamic security patterns.
Develop focused test techniques to effectively explore security test patterns while reducing the test suite size.
Create a defect model for security that can be used to predict types and number of security vulnerabilities in scientific codes.
Action – Develop method engineering techniques
Method engineers create custom-made processes to help a project achieve specific goals.
The goal of being “secure” needs to be operationalized so that these engineers can assemble methods in ways that enhance the security of the software product built using their processes.
This task would involve extending the Software Process Engineering MetaModel1 (SPEM) standard to define security-specific constructs. The model would be automated using existing tools such as MetaEdit+.
1 Software Process Engineering Metamodel, version 1.1, Object Management Group (OMG), 2005.
Action – Develop method engineering techniques - 2
The security-oriented method fragments would be suitable to integrate into processes defined in the context of the SPEM.
Process audits could evaluate the strength of the security aspect of the process, once it is explicitly embedded in the process, just as other qualities are validated.
Deliverables: A process definition guide that would show how to design security-centric development process fragments, an assembly guide, and an example process.
Action – Create architecture security analysis
Architects have techniques that they apply to an architecture in order to improve the behavior of the architecture with respect to classes of security threats1.
Assuring system survivability requires showing that the system architecture is adequately resilient to likely patterns of attack2.
One approach to architecture design is to identify quality attributes for the architecture and make design decisions that enhance the desirable qualities and degrade the less desirable ones.
Architecture evaluation techniques such as the architecture trade-off analysis method (ATAM) can be used to focus architecture evaluations on the security quality through security-specific scenarios.
One aspect of this task would be to develop a set of architecture-level security scenarios that would guide the evaluation of an architecture for the security quality.
1 Security and Survivability Reasoning Frameworks and Architectural Design Tactics. CMU/SEI-2004-TN-022
2 Architectural Refinement for the Design of Survivable Systems CMU/SEI-2001-TN-008
Action – Create architecture security analysis - 2
This evaluation can be automated if the architects use a formal architecture description language (ADL).
Our current implementation uses the Architectural Analysis and Design Language (AADL) from the Society of Automotive Engineers (SAE) and Eclipse plug-ins.
A second part of the task could be to identify standard mechanisms for representing security in an architecture description so that security patterns could be automatically recognized.
Deliverables: A set of validation scenarios and specific architecture tactics for recognizing security vulnerabilities at the architecture level
Current implementation as Eclipse plug-in for AADL
Action – Create architecture security analysis - 3
This analysis could also be conducted on pre/post-test basis
Use an architecture re-construction technique such as OAR1 to extract the “as built” architecture from the implementation.
Compare to the “as designed” architecture to determine if changes have occurred
1 Options Analysis for Reengineering (OAR): A Method for Mining Legacy Assets CMU/SEI-2001-TN-013
Action – Discover test patterns for security
Test patterns are solutions to testing problems in context. They often correspond to design patterns.
Test patterns are language independent Content of a test pattern1
Problem - Description of pattern to be tested Context - Special testing conditions Forces - What types of faults are we looking
for? Solution - Test case selection strategy that
tests the interactions among the components that implement the pattern
Example – A sample implementation
1 John McGregor and David Sykes Practical Guide to Testing Object-Oriented Software, Addison-Wesley, 2001.
Action – Discover test patterns for security - 2
Problem: The synchronous communication between two objects is modified to be asynchronous by adding a callback object. How should this be tested?
ClientClient
ServerServer
ClientClient
ServerServer
CallbackCallback
Action – Discover test patterns for security - 3
Context An intermediate object forwards messages
Forces Possible to intermingle successive messages/responses Temporal considerations must be added to the test case
Solution Construct tests that exercise the callback in a variety of states Construct multiple clients, submit multiple requests through multiple
callback objects Build test cases that submit a second message prior to receiving the
response from the first message
Action – Discover test patterns for security - 4
In this task, security vulnerabilities and standard designs to correct them would be studied.
The standard designs would lead to a catalog of specific test patterns.
Users of the test patterns could implement them in whatever language was in use.
Deliverables: test pattern catalog
Task force identified security patterns
1. Make the Client Invisible2. Target Programs That Write to
Privileged OS Resources3. Use a User-Supplied Configuration
File to Run Commands That Elevate Privilege
4. Make Use of Configuration File Search Paths
5. Direct Access to Executable Files6. Embedding Scripts within Scripts7. Leverage Executable Code in Non-
executable Files8. Argument Injection9. Command Delimiters10. Multiple Parsers and Double
Escapes
11. User-Supplied Variable Passed to File System Calls
12. Postfix NULL Terminator13. Postfix, Null Terminate, and
Backslash14. Relative Path Traversal15. Client-Controlled Environment
Variables16. User-Supplied Global Variables
(DEBUG=1, PHP Globals, and So Forth)
17. Session ID, Resource ID, and Blind Trust
18. Analog In-Band Switching Signals (aka “Blue Boxing”)
19. Attack Pattern Fragment: Manipulating Terminal Devices
20. Simple Script Injection
49 security patterns have been identified. Here are a few.
Security Across the Software Development Lifecycle Task Force
Action – Develop focused test techniques - 1
Operational profiles establish a relative frequency of operations for a particular type of user or application.
Test cases are chosen with the same relative frequency to mimic actual use or to mimic attack scenarios.
This narrows the range of values to cover in test.
User Operation Relative frequency
User1
Op1 .2
Op2 .6
Op3 .2
User2
Op1 .4
Op2 .3
Op3 .3
Action – Develop focused test techniques - 2
Testing all possible values even once, much less all possible combinations of values, is usually impossible.
Combinatorial test designs can systematically sample test values to ensure the maximum coverage with minimum test cases.
Orthogonal array testing is one combinatorial approach.
OATS tool - 1
Three variables, each with three possible values, would require 27 test cases for all combinations coverage. Database
driverJDBC1.0 JDBC2.0 ODBC1.0
browser Netscape IE5.0 Firefox
# of web servers
5 50 200
OATS tool - 2
OATS reduces this to 9 test cases, testing pair-wise but remains approximately 96% as effective as “all combinations” coverage at finding defects.
1 JDBC1.0 Netscape 200
2 JDBC1.0 IE5.0 50
3 JDBC1.0 FoxFire 5
4 JDBC2.0 Netscape 50
5 JDBC2.0 IE5.0 5
6 JDBC2.0 FoxFire 200
7 ODBC1.0 Netscape 5
8 ODBC1.0 IE5.0 200
9 ODBC1.0 FoxFire 50
Action – Develop focused test techniques - 3
Pair-wise value combinations can reduce the size of the test suite dramatically
These techniques have been applied in domains such as telecommunications.
In this task we would develop an industrial strength tool that allows the design of combinatorial test suites.
Action – Develop focused test techniques - 4
This technique is used in conjunction with traditional functional and structural strategies for test case selection.
Functional tests ensure that all specified functions are correct and structural tests ensure that all feasible paths are valid.
The result is confidence that the software does everything it is supposed to and nothing that it Is not supposed to.
Combinatorial techniques increase the coverage of large software products without significant increases in test effort.
This approach allows for risk-based test selections. Deliverables: software tool that uses combinatorial
techniques and risk-based techniques to define effective test cases.
Action – Develop a security defect model for scientific codes
If you just found the 73rd defect in your 50,000 LOC program, do you feel good about it?
With a validated defect model you would at least know how to feel.
You would know approximately how many defects to expect and what types of defects to look for.
Action – Develop a security defect model for scientific codes - 2
Using the relative frequency and estimated size, estimates can be made of the number of defects and the test case selection process be more precisely directed.
Security defect type
Relative frequency
per KLOCStatic queue sizes
60%
HTTP cookies 30%Argument injection
10%
Total 100%
Action – Develop a security defect model for scientific codes - 3
In this task we would develop a defect model framework for scientific codes using historic data.
The framework would be packaged with a specialization method for tailoring the framework to a specific development context.
This would allow us to provide statistical estimates of the number of defects remaining in a software product and to provide a confidence interval on statistics such as reliability.
Deliverables: defect model
Conclusion
No single action or algorithm will produce secure software. There must be a chain of quality activities.
The outlined strategy places security quality gates at several places in the software product development life cycle.
SystemsEngineering
Class NameResponsibility Collaborator
Super Class
RequirementsUse Case Name:Use:
User:Frequency:See Also:
Use Case Modeling
DomainExpert
DomainAnalysis
ArchitecturalDesign
ClassDevelopment
Incremental Integration
and System TestingApplication
Analysis
Client
Use Case Name:Use:
User:Frequency:See Also:
Application Use Cases
View
Model
Contro
ller
Class Specification
ClassDesign
ClassDerivation
ClassReuse
Implementation Refinement
TestingClass Delivery
Architectureinspection
Clusterdesign inspection
Analysis model inspection
Analysismodel inspection
Requirementsmodel inspection
Conclusion - 2
Implementing any part of this strategy will improve the security of the software products produced.
Develop method engineering tactics and guidelines that enhance the security quality of the software through improved processes.
Structure architecture evaluation techniques to focus on security by searching for static security patterns.
Discover and capture test patterns that correspond to dynamic security patterns.
Instrument combinatorial test techniques to effectively explore security test patterns while reducing the test suite size.
Create a defect model for security that can be used to predict types and number of security vulnerabilities in a given product.