real world software assurance test suite€¦ · test suite - cwes for c programs weakness type...
TRANSCRIPT
Real World Software Assurance Test Suite:
STONESOUPCharles Oliveira/SAMATE
Guest Researcher at Software and Systems Division, IT Laboratory
NIST
Outline- Introduction
- STONESOUP program
- Test suite
- Test case sample
- TEXAS usage
- Documents and reports
2
Introduction - SOUP
libssl libxml
libpq ...
Open source libs
Java/Spring
PHP/Zend
C++/Boost ...
Frameworks
Apache
Postgres
Drivers
...
Standalone apps
Application
Is this Software Of Unknown Provenance
(SOUP) safe?
3rd party software
3
STONESOUP program
Securely Taking On New Executable
Software Of Uncertain Provenance
(STONESOUP) http://www.iarpa.gov/index.php/research-programs/stonesoup
4
STONESOUP programThe goal of STONESOUP program was to eliminate the effects of vulnerabilities in
software applications by:
- extending the scope and capability of approaches for analysis, confinement, and
diversification;
- addressing a wide range of security vulnerabilities within the same framework;
- integrating approaches to leverage the strengths and weaknesses of each;
- adding no more than 10% running time slowdown.
5
STONESOUP programPhase 1
Neutralize 75% of vulnerabilities of 2 weakness types in 10k SLOC programs
Phase 2
Neutralize 80%+ of vulnerabilities of 4 weakness types in 100k SLOC programs
Phase 3
Neutralize 90%+ of vulnerabilities of 6 weakness types in 500k SLOC programs
Phase 3 performers were those that made significant
progress in Phase 2 as measured by the program metrics.
The three teams and the names of their developmental
tools are:
Kestrel Institute - VIBRANCE (video)
Columbia University - Minestrone
Grammatech - PEASOUP
6
STONESOUP program - PerformersSTONESOUP performers neutralize vulnerabilities in:
7
STONESOUP program - Test & Evaluation System- Test & Evaluation eXecution and Analysis System (TEXAS) was designed
and developed to test Performer technology
- Developed by STONESOUP team
- Command Line Interface (CLI) to run and evaluate tests cases
- Communication API to interact to Performer’s tools
8
Test suite - Base programs
637477
637
638636
637
480476
637
478
380JTree
380GNU Tree
160
479
380
GNU Grep
478
9
Number of test cases per base program in red circles
Test suite - CWEs for C programsWeakness type CWEs (56)
Concurrency handling(765) 363 367 412 414 543 609 663 764 765 820 821 833 831 828 479
Injection(701) 078 088 089
Number handling(725) 190 191 194 195 196 197 369 682 839
Resource drains(733) 400 459 674 774 789 834 835 401 771 773 775
Memory corruption(965) 120 124 126 127 129 134 170 415 416 590 761 785 805 806 822 824 843
Null pointer(693) 476
10
Test suite - CWEs for C programsWeakness type CWEs (56)
Concurrency handling(765) 363 367 412 414 543 609 663 764 765 820 821 833 831 828 479
Injection(701) 078 088 089
Number handling(725) 190 191 194 195 196 197 369 682 839
Resource drains(733) 400 459 674 774 789 834 835 401 771 773 775
Memory corruption(965) 120 124 126 127 129 134 170 415 416 590 761 785 805 806 822 824 843
Null pointer(693) 476
11
CWE-363: Race Condition Enabling Link Following (2.8)CWE-367: Time-of-check Time-of-use (TOCTOU) Race Condition (2.8)CWE-412: Unrestricted Externally Accessible Lock (2.8)CWE-414: Missing Lock Check (2.8)CWE-543: Use of Singleton Pattern Without Synchronization in a Multithreaded Context (2.8)CWE-609: Double-Checked Locking (2.8)CWE-663: Use of a Non-reentrant Function in a Concurrent Context (2.8)CWE-764: Multiple Locks of a Critical Resource (2.8)CWE-765: Multiple Unlocks of a Critical Resource (2.8)CWE-820: Missing Synchronization (2.8)CWE-821: Incorrect Synchronization (2.8)CWE-833: Deadlock (2.8)CWE-831: Signal Handler Function Associated with Multiple Signals (2.8)CWE-828: Signal Handler with Functionality that is not Asynchronous-Safe (2.8)CWE-479: Signal Handler Use of a Non-reentrant Function (2.8)
Test suite - CWEs for C programsWeakness type CWEs (56)
Concurrency handling(765) 363 367 412 414 543 609 663 764 765 820 821 833 831 828 479
Injection(701) 078 088 089
Number handling(725) 190 191 194 195 196 197 369 682 839
Resource drains(733) 400 459 674 774 789 834 835 401 771 773 775
Memory corruption(965) 120 124 126 127 129 134 170 415 416 590 761 785 805 806 822 824 843
Null pointer(693) 476
12
CWE-078: OS Command Injection (2.8)CWE-088: Argument Injection or Modification (2.8)CWE-089: SQL Injection (2.8)
Test suite - CWEs for C programsWeakness type CWEs (56)
Concurrency handling(765) 363 367 412 414 543 609 663 764 765 820 821 833 831 828 479
Injection(701) 078 088 089
Number handling(725) 190 191 194 195 196 197 369 682 839
Resource drains(733) 400 459 674 774 789 834 835 401 771 773 775
Memory corruption(965) 120 124 126 127 129 134 170 415 416 590 761 785 805 806 822 824 843
Null pointer(693) 476
13
CWE-190: Integer Overflow or Wraparound (2.8)CWE-191: Integer Underflow (Wrap or Wraparound) (2.8)CWE-194: Unexpected Sign Extension (2.8)CWE-195: Signed to Unsigned Conversion Error (2.8)CWE-196: Unsigned to Signed Conversion Error (2.8)CWE-197: Numeric Truncation Error (2.8)CWE-369: Divide By Zero (2.8)CWE-682: Incorrect Calculation (2.8)CWE-839: Numeric Range Comparison Without Minimum Check (2.8)
Test suite - CWEs for C programsWeakness type CWEs (56)
Concurrency handling(765) 363 367 412 414 543 609 663 764 765 820 821 833 831 828 479
Injection(701) 078 088 089
Number handling(725) 190 191 194 195 196 197 369 682 839
Resource drains(733) 400 459 674 774 789 834 835 401 771 773 775
Memory corruption(965) 120 124 126 127 129 134 170 415 416 590 761 785 805 806 822 824 843
Null pointer(693) 476
14
CWE-400: Resource Exhaustion (2.8)CWE-459: Incomplete Cleanup (2.8)CWE-674: Uncontrolled Recursion (2.8)CWE-774: Allocation of File Descriptors or Handles Without Limits or Throttling (2.8)CWE-789: Uncontrolled Memory Allocation (2.8)CWE-834: Excessive Iteration (2.8)CWE-835: Infinite Loop (2.8)CWE-401: Memory Leak (2.8)CWE-771: Missing Reference to Active Allocated Resource (2.8)CWE-773: Missing Reference to Active File Descriptor or Handle (2.8)CWE-775: Missing Release of File Descriptor or Handle after Effective Lifetime (2.8)
Test suite - CWEs for C programsWeakness type CWEs (56)
Concurrency handling(765) 363 367 412 414 543 609 663 764 765 820 821 833 831 828 479
Injection(701) 078 088 089
Number handling(725) 190 191 194 195 196 197 369 682 839
Resource drains(733) 400 459 674 774 789 834 835 401 771 773 775
Memory corruption(965) 120 124 126 127 129 134 170 415 416 590 761 785 805 806 822 824 843
Null pointer(693) 476
15
CWE-120: Classic Buffer Overflow (2.8)CWE-124: Buffer Underflow (2.8)CWE-126: Buffer Over-read (2.8)CWE-127: Buffer Under-read (2.8)CWE-129: Improper Validation of Array IndexCWE-134: Uncontrolled Format String (2.8)CWE-170: Improper Null Termination (2.8)CWE-415: Double Free (2.8)CWE-416: Use After Free (2.8)
CWE-590: Free of Memory not on the Heap (2.8)CWE-761: Free of Pointer not at Start of Buffer (2.8)CWE-785: Use of Path Manipulation Function without Maximum-sized Buffer (2.8)CWE-805: Buffer Access with Incorrect Length Value (2.8)CWE-806: Buffer Access Using Size of Source Buffer (2.8)CWE-822: Untrusted Pointer Dereference (2.8)CWE-824: Access of Uninitialized Pointer (2.8)CWE-843: Type Confusion (2.8)
Test suite - CWEs for C programsWeakness type CWEs (56)
Concurrency handling(765) 363 367 412 414 543 609 663 764 765 820 821 833 831 828 479
Injection(701) 078 088 089
Number handling(725) 190 191 194 195 196 197 369 682 839
Resource drains(733) 400 459 674 774 789 834 835 401 771 773 775
Memory corruption(965) 120 124 126 127 129 134 170 415 416 590 761 785 805 806 822 824 843
Null pointer(693) 476
16
CWE-476: NULL Pointer Dereference (2.8)
Test suite - CWEs for Java programsWeakness type CWEs (50)
Concurrency handling(568) 363 367 412 414 543 609 663 764 765 820 821 833 832 567 572
Injection(526) 078 088 089 564
Number handling(532) 190 191 194 195 196 197 369 839
Resource drains(532) 400 459 674 774 789 834 835
Error handling(532) 209 248 252 253 390 391 460 584
Tainted data(498) 023 036 041 606
17
Test suite - CWEs for Java programsWeakness type CWEs (50)
Concurrency handling(568) 363 367 412 414 543 609 663 764 765 820 821 833 832 567 572
Injection(526) 078 088 089 564
Number handling(532) 190 191 194 195 196 197 369 839
Resource drains(532) 400 459 674 774 789 834 835
Error handling(532) 209 248 252 253 390 391 460 584
Tainted data(498) 023 036 041 606
18
CWE-363: Race Condition Enabling Link Following (2.8)CWE-367: Time-of-check Time-of-use (TOCTOU) Race Condition (2.8)CWE-412: Unrestricted Externally Accessible Lock (2.8)CWE-414: Missing Lock Check (2.8)CWE-543: Use of Singleton Pattern Without Synchronization in a Multithreaded Context (2.8)CWE-609: Double-Checked Locking (2.8)CWE-663: Use of a Non-reentrant Function in a Concurrent Context (2.8)CWE-764: Multiple Locks of a Critical Resource (2.8)CWE-765: Multiple Unlocks of a Critical Resource (2.8)CWE-820: Missing Synchronization (2.8)CWE-821: Incorrect Synchronization (2.8)CWE-833: Deadlock (2.8)CWE-832: Unlock of a Resource that is not Locked (2.8)CWE-567: Unsynchronized Access to Shared Data in a Multithreaded Context (2.8)CWE-572: Call to Thread run() instead of start() (2.8)
Test suite - CWEs for Java programsWeakness type CWEs (50)
Concurrency handling(568) 363 367 412 414 543 609 663 764 765 820 821 833 832 567 572
Injection(526) 078 088 089 564
Number handling(532) 190 191 194 195 196 197 369 839
Resource drains(532) 400 459 674 774 789 834 835
Error handling(532) 209 248 252 253 390 391 460 584
Tainted data(498) 023 036 041 606
19
CWE-078: OS Command Injection (2.8)CWE-088: Argument Injection or Modification (2.8)CWE-089: SQL Injection (2.8)CWE-564: SQL Injection: Hibernate (2.8)
Test suite - CWEs for Java programsWeakness type CWEs (50)
Concurrency handling(568) 363 367 412 414 543 609 663 764 765 820 821 833 832 567 572
Injection(526) 078 088 089 564
Number handling(532) 190 191 194 195 196 197 369 839
Resource drains(532) 400 459 674 774 789 834 835
Error handling(532) 209 248 252 253 390 391 460 584
Tainted data(498) 023 036 041 606
20
CWE-190: Integer Overflow or Wraparound (2.8)CWE-191: Integer Underflow (Wrap or Wraparound) (2.8)CWE-194: Unexpected Sign Extension (2.8)CWE-195: Signed to Unsigned Conversion Error (2.8)CWE-196: Unsigned to Signed Conversion Error (2.8)CWE-197: Numeric Truncation Error (2.8)CWE-369: Divide By Zero (2.8)CWE-839: Numeric Range Comparison Without Minimum Check (2.8)
Test suite - CWEs for Java programsWeakness type CWEs (50)
Concurrency handling(568) 363 367 412 414 543 609 663 764 765 820 821 833 832 567 572
Injection(526) 078 088 089 564
Number handling(532) 190 191 194 195 196 197 369 839
Resource drains(532) 400 459 674 774 789 834 835
Error handling(532) 209 248 252 253 390 391 460 584
Tainted data(498) 023 036 041 606
21
CWE-400: Resource Exhaustion (2.8)CWE-459: Incomplete Cleanup (2.8)CWE-674: Uncontrolled Recursion (2.8)CWE-774: Allocation of File Descriptors or Handles Without Limits or Throttling (2.8)CWE-789: Uncontrolled Memory Allocation (2.8)CWE-834: Excessive Iteration (2.8)CWE-835: Infinite Loop (2.8)
Test suite - CWEs for Java programsWeakness type CWEs (50)
Concurrency handling(568) 363 367 412 414 543 609 663 764 765 820 821 833 832 567 572
Injection(526) 078 088 089 564
Number handling(532) 190 191 194 195 196 197 369 839
Resource drains(532) 400 459 674 774 789 834 835
Error handling(532) 209 248 252 253 390 391 460 584
Tainted data(498) 023 036 041 606
22
CWE-209: Information Exposure Through an Error Message (2.8)CWE-248: Uncaught Exception (2.8)CWE-252: Unchecked Return Value (2.8)CWE-253: Incorrect Check of Function Return Value (2.8)CWE-390: Detection of Error Condition Without Action (2.8)CWE-391: Unchecked Error Condition (2.8)CWE-460: Improper Cleanup on Thrown Exception (2.8)CWE-584: Return Inside Finally Block (2.8)
Test suite - CWEs for Java programsWeakness type CWEs (50)
Concurrency handling(568) 363 367 412 414 543 609 663 764 765 820 821 833 832 567 572
Injection(526) 078 088 089 564
Number handling(532) 190 191 194 195 196 197 369 839
Resource drains(532) 400 459 674 774 789 834 835
Error handling(532) 209 248 252 253 390 391 460 584
Tainted data(498) 023 036 041 606
23
CWE-023: Relative Path Traversal (2.8)CWE-036: Absolute Path Traversal (2.8)CWE-041: Improper Resolution of Path Equivalence (2.8)CWE-606: Unchecked Input for Loop Condition (2.8)
Test suite - Base programs- Total of 7770 test cases which generates ~240GB compressed!!!
- The STONESOUP Test and Evaluation team (T&E) used 277 independent virtual
machines simultaneously on Amazon Web Services between April and December
2014 for performers to run the test cases.
- The NIST VM is 22GB and contains test cases patched from the base program
- The strategy was to patch the test cases, distributing ‘.diff’ files instead of whole
copies of each base program
24
Test suite - Virtual Machine (VMware)- Download (11x2GB) at http://samate.nist.gov/SARD/testsuite.php#standalone
- OS: Ubuntu 12.04
- CPU: 4 VCPU recommended
- Memory: 4GB (8GB recommended)
- Storage: 59GB Total / 41GB Used / 16GB Available
- Inside NIST_TT_VM folder there is a document with login and password for the VM
- Important directories:
- /opt/stonesoup: contains the entire NIST STONESOUP package including scripts and documents
- /opt/share: contains a TEXAS installation, test cases (diffs), base programs all their dependencies
- Performers’ tools are not in the VM
25
Test case sample (from the virtual machine)- Pick a test case in /opt/share/testcases/
- Chosen: C-C476C-SUBV-03-ST01-DT05-DF10-CF22-01
26
Test case sample (from the virtual machine)- Pick a test case in /opt/share/testcases/
- Chosen: C-C476C-SUBV-03-ST01-DT05-DF10-CF22-01
C or Java
27
Test case sample (from the virtual machine)- Pick a test case in /opt/share/testcases/
- Chosen: C-C476C-SUBV-03-ST01-DT05-DF10-CF22-01
CWE-476: NULL Pointer Dereference
28
Test case sample (from the virtual machine)- Pick a test case in /opt/share/testcases/
- Chosen: C-C476C-SUBV-03-ST01-DT05-DF10-CF22-01
29
Algorithmic variant: refined CWEs mapped to a code snippet previously defined by T&E team
Test case sample (from the virtual machine)- Pick a test case in /opt/share/testcases/
- Chosen: C-C476C-SUBV-03-ST01-DT05-DF10-CF22-01
Base program:
CMUD Coffee MUD CTRE GNU Tree FFMP FFMpeg GIMP Gimp
GREP GNU Grep OSSL OpenSSL PSQL Postgres SUBV Apache Subversion
WIRE Wireshark ELAS Elastic Search JMET Apache JMeter JENA Apache Jena
JTRE Java Tree LENY ApacheLenya LUCE Apache
Lucene POIX Apache POI
30
Test case sample (from the virtual machine)- Pick a test case in /opt/share/testcases/
- Chosen: C-C476C-SUBV-03-ST01-DT05-DF10-CF22-01
Injection point: represent specific locations in the base program that are guaranteed
to be executed given the defined I/O pairs. Identifiers reference different injection points in each base program.
31
Test case sample (from the virtual machine)- Pick a test case in /opt/share/testcases/
- Chosen: C-C476C-SUBV-03-ST01-DT05-DF10-CF22-01
Taint source:
01 ENVIRONMENT_VARIABLE
02 FILE_CONTENTS
03 SOCKET
04 SHARED_MEMORY
32
Test case sample (from the virtual machine)- Pick a test case in /opt/share/testcases/
- Chosen: C-C476C-SUBV-03-ST01-DT05-DF10-CF22-01
Data type:
01 ARRAY 05 STRUCT
02 SIMPLE 06 TYPEDEF
03 VOID_POINTER 07 UNION
04 HEAP_POINTER
33
Test case sample (from the virtual machine)- Pick a test case in /opt/share/testcases/
- Chosen: C-C476C-SUBV-03-ST01-DT05-DF10-CF22-01
Data flow:
01 ADDRESS_ALIAS_1 11 BASIC
05 ADDRESS_AS_CONSTANT 12 VAR_ARG_LIST
06 ADDRESS_AS_FUNCTION_RETURN_VALUE 17 BUFFER_ADDRESS_POINTER
10 INDEX_ALIAS_50 18 JAVA_GENERICS
34
Test case sample (from the virtual machine)- Pick a test case in /opt/share/testcases/
- Chosen: C-C476C-SUBV-03-ST01-DT05-DF10-CF22-01
Control flow:
01 INTERCLASS_1 18 POINTER_TO_FUNCTION
08 INTERFILE_1 19 RECURSIVE
12 INTERPROCEDURAL_1 22 MACROS
16 INTERRUPT 26 FUNCT_INVOC_OVERLOAD
35
Test case sample (from the virtual machine)- Pick a test case in /opt/share/testcases/
- Chosen: C-C476C-SUBV-03-ST01-DT05-DF10-CF22-01
Unique increment:increment in case of multiple test cases are sharing the same
parameters aforementioned.
36
Test case sample (from the virtual machine)- Pick a test case in /opt/share/testcases/
- Chosen: C-C476C-SUBV-03-ST01-DT05-DF10-CF22-01
- Browsing the test case
- install/: this test case installation files
- scripts/: specific scripts to manage running process
- src/: the entire base program + files seeded with intentional weaknesses
- testData/: input data which will [and won’t] trigger the seeded weakness
- testOutput/: matching output data for each input data
- C-C476C-SUBV-03-ST01-DT05-DF10-CF22-01.xml: TEXAS “makefile”
- C-C476C-SUBV-03-ST01-DT05-DF10-CF22-01.yaml: benign and exploiting inputs
37
Stage 1: standard compilation
Stage 2: compilation with performer
technology
TEXAS usage
I/O Pairs
Analysis/Compilation
Execution
Scoring
The source code or binary of a program is scanned looking for
CWE code patterns and applying diversification techniques to
harden the resulting binary. The output of the Analysis phase is a
binary executable.
The Execution step is run for each I/O, and involves actually
invoking the binary created in the Analyze step with known
inputs. Performer technology may also monitor the execution of
the binary to look for execution patterns indicative of an attack in
progress or software vulnerability.
Scoring executed immediately after the Execution step and looks at
the environment for the known outputs defined in the metadata
for the given I/O pair that was executed.
38
Analysis/Compilation
Execution
Scoring
Documents & ReportsMain STONESOUP documents provided at SARD website:
- Test and evaluation phase 3 final report
- Performers’ reports
- Weaknesses documentation
- Test cases creation guide
- TEXAS user guides
Visit: http://samate.nist.gov/SARD/around.php
39