tool output integration framework · 2016-09-15 · cppcheck • splint • rats … toif...

©2016 Lockheed Martin Aeronautics Company

Tool Output Integration Framework Enhanced Static Code Analysis Identifying Critical Vulnerabilities in Code

Dr. Ben Calloni, P.E., CISSP, CEH Principle Investigator Fellow Software Security OMG’s Cyber Risk Summit September 14, 2016


Overview

• Purpose of the Study − Background

• Previous Lockheed Martin Aeronautics SCA studies • NSA SCA Test Results

− Software Languages Covered • Test Case Coverage

− Common Weakness Enumerations (CWE) by MITRE − NSA Standardized Test Suites (Juliet)

• Tool Capability Analysis Results − 2012 Tool Study – Tool A and Tool C − TOIF Results C/C++

• Summary of Findings • Conclusion

Chart 1

[email protected]

September 14, 2016 OMG Cyber Risk Summit


Purpose

• Not Dynamic Analysis (Testing while executing the code) • Not Penetration Testing

• Incorporate Static Code Analysis within the larger Trusted

Software Development Process − Solid SwE based on

• Requirements • Design • Secure Coding Standards • System Testing and Evaluation

• Make the SCA execution as seamless as the software compile and build “DURING DEVELOPMENT”!

• Eliminate as many security coding flaws at the point of creation.

Chart 2 September 14, 2016 OMG Cyber Risk Summit


Security Definition

Cyber Vulnerability (CISSP BoK) 1. A flaw* (aka weakness) exists in the

system 2. Attacker has access to the flaw, and 3. Attacker has capability to exploit the flaw

• Examples − Lack of security patches − Lack of current virus definitions − Software Bug − Lax physical security

Basic definition of Vulnerability • refers to the inability to withstand the effects of a hostile

environment • open to attack or damage Defenders can only control these!

*e.g. Buffer Overflow is still on SANS Top 25 (#3). Industry has known and discussed since 1988!

September 14, 2016 OMG Cyber Risk Summit Chart 3


Test Case Coverage

• Common Weakness Enumerations (CWE) by MITRE

Chart 4

CWE™ International in scope and free for public use, CWE provides a unified, measurable set of software weaknesses that is enabling more effective discussion, description, selection, and use of software security tools and services that can find these weaknesses in source code and operational systems as well as better understanding and management of software weaknesses related to architecture and design.

• Juliet Test Suite from National Security Agency

The Juliet Test Suite is an aggregation of test cases developed by the National Security Agency (NSA) Center for Assured Software (CAS) specifically for use in testing static analysis tools. It is intended for anyone who wishes to use the test cases for their own testing purposes, or who would like to have a better understanding of how test cases were created. The Juliet Test Suite is comprised of C/C++ and Java test cases. Version 1.1 of the C/C++ test suite contains examples for 119 different CWEs and contains 57,099 test cases. Version 1.1.1 of the Java test suite contains examples of 113 different CWEs and contains 23,957 test cases.



CWE Example

Chart 5

CWE-119: Improper Restriction of Operations within the Bounds of a Memory Buffer Description Summary The software performs operations on a memory buffer, but it can read from or write to a memory location that is outside of the intended boundary of the buffer. Extended Description Certain languages allow direct addressing of memory locations and do not automatically ensure that these locations are valid for the memory buffer that is being referenced. This can cause read or write operations to be performed on memory locations that may be associated with other variables, data structures, or internal program data. As a result, an attacker may be able to execute arbitrary code, alter the intended control flow, read sensitive information, or cause the system to crash.


http://cwe.mitre.org/index.html


National Security Agency CAS Studies (2010 Report)

Chart 6

BlackHat 2011 Kris Britton https://www.youtube.com/watch?v=g0UL2Nam5hE



NSA Data

Presented by NSA at OMG Meeting in DC, Mar 2012



NSA Reported Coverage Chart C/C++ Weakness Classes



NSA Study Conclusions



Lockheed Martin Aeronautics Study



Lockheed Martin Aeronautics SCA Study (2011)

• Current Tool Inefficiency (based on NIST test cases) − Tool D* − Tool E*

Chart 12

32%

68%

SCA Coverage C/C++ Tool D Undetected

23%

77%

SCA Coverage Java Tool E Undetected

C/C++ used on Air Vehicle Java used on Ground Systems

*Tools D and E are not the same as the NSA tools D / E. September 14, 2016 OMG Cyber Risk Summit


C/C++ Coverage Lockheed Martin Study (2012)

Chart 13

Determine which 2 tools in combination provides broadest coverage September 14, 2016 OMG Cyber Risk Summit


2012 SCA Tool Study Results

• Commercial Off-the-Shelf (COTS) − Tool A* − Tool B* − Tool C*

• Tool A and Tool C provided 60.2% coverage of the C/C++ Juliet test cases − This particular pairing of the 2 provided the best coverage.

Chart 14

*Tools A, B, and C are not the same as the NSA tools A, B, C. September 14, 2016 OMG Cyber Risk Summit


Coverage based on 2012 COTS Study (60.2%)

Chart 15

Tool A Tool C 15

23

36

114 134

190

195

197

222 223 252

226

242

244

247

253

256

259 272

304

319

321

327

328

338

364

367

369

390

396

400 401

404 415

416

457

459

467

468 476

478

480

481

482

483

489

500

506

534

562

563

570

571

590

591

605

606

617

672

675

676

680

685

688 690

758 761

762

773

789

835

843

121

122

123

124

126

127

188

191

204

273

284

325

374

377

391

392

398

440

469

475

484

78

194

196

426

427

464

479

510

511

535

546

547

560

561

587

588

620

665

666

674

775

780

785

832

366

The CWE ID’s shown are those in the Juliet test cases. September 14, 2016 OMG Cyber Risk Summit


Tool A

Tool B coverage / non coverage

Chart 16

Tool C 15

23

36

114 134

190

195

197

222 223 252

226

242

244

247

253

256

259 272

304

319

321

327

328

338

364

367

369

390

396

400 401

404 415

416

457

459

467

468 476

478

480

481

482

483

489

500

506

534

562

563

570

571

590

591

605

606

617

672

675

676

680

685

688 690

758 761

762

773

789

835

843

121

122

123

124

126

127

188

191

204

273

284

325

374

377

391

392

398

440

469

475

484

78

194

196

426

427

464

479

510

511

535

546

547

560

561

587

588

620

665

666

674

775

780

785

832

Tool B found 6 additional, did not report 32 (False Negatives).

366



Observations of Lockheed Martin 2012 SCA Study

• Each COTS product’s output was formatted differently − Analyzing the data was labor intensive!

• Developers would need to be trained on each individual tool • Combining the two COTS products gave slightly more coverage

− The COTS tools tend to look for the same weaknesses



Approach (2013-14)

• Reviewed NSA Tool Capability Report, Static Analysis Tools for C, C++, and Java (March 24, 2011) (FOUO)

• Reviewed available SCA tool reports from NIST sponsored Static Analysis Tool Exposition (SATE) program

• Acquired Open Source Tool Output Integration Framework tool • Used validated C++ test cases

− NSA Juliet Test Suite v1.1



Tool Output Integration Framework - TOIF

• TOIF initially developed in 2012 • in partnership of two companies (Data Access Technologies and

KDM Analytics) • Under DHS SBIR program with SBIR Topic Number: H-SB09.2-004

Software Testing and Vulnerability Analysis. • Goal: release TOIF technology as open source.

• Furthermore, KDM Analytics productized TOIF and make it available through: • open source • commercially available open source

− Maintenance updates, OSS License indemnification, trusted delivery

• commercially integrated with Blade Threat/Risk Analyzer (KDM Analytics proprietary solution)



TOIF Architecture – High Level

Chart 20

TOIF Adaptor(s)

• CppCheck • Splint • Rats

…

TOIF Assimilator

Defect Generator Tool(s)

CppCheck Splint Rats

Eclipse (for SwEs)

KDM BLADE

(for SSEs)

TOIF Report View

• Open Source Tools (C/C++) included with TOIF − CppCheck 1.4 − Splint 3.1.2 − RATS 2.3

There are 2 Java Open Source Tools shipped with TOIF FindBugs and Jlint



TOIF Flaw Reporting



TOIF Coverage (NSA Weakness Classes)

Chart 22

0

2

4

6

8

10

12

14

16

18

20

TOIF Combined Coverage

3 Open Source SCATools

Total CWEs



Cppcheck Splint

RATS

Coverage Based on TOIF Study (61.3%)

Chart 23

15

23

36

134

190

197

223 226

242

244

247

256

259

272

328

338

364

367

369

390

396

400

401

404

416

457

467

468

476

478

480

481

483

489

500

506

534

563

570

571

605

617

675

676

680

761

762

773 789

835 843

121

122

123

126

127

188

204

273

284

325

366

374

377

391

392

440

469

484

78

196

426

427

464

479

510

511

535

546

560

561

587

588

620

666 674

780

832

124 366

114

195

253

304

319 327

459

482

590

591 606

672

685

688 690

758

191

194

785

397

398

222

475

547

775

321

415

252

665

562



TOIF Analysis

• TOIF provided 61.3% coverage of the C/C++ Juliet test cases • TOIF integrates the output from multiple SCA tools into an

Eclipse based view − TOIF significantly reduced the amount of time it takes review

the results from the SCA tools’ output. • TOIF uses Eclipse to display SCA tools output

− Eclipse tool is already in use at Aero

Chart 24

Additional adaptors could expand the CWE coverage.

• C/C++ − Sparse − Uno − BLAST − Fragma-C

• Java − Checkstyle − Sonar − PMD



Benefits of TOIF

• TOIF has the flexibility of using Open Source SCA tools − Reduces acquisition/licensing cost of SCA tools − Reduces SCA training costs − Increases coverage of finding flaws / defects − Reduces manpower required in safety critical / security relevant

code reviews (focus on only defects not covered by TOIF) − Improve software quality by incorporating SCA tool analysis

during development phase • Simplifies the effort for the developer to incorporate flaw

remediation − Reduce software rework (hence costs) late in the development

process



Conclusion

• TOIF is the better solution for performing Static Code Analysis to assist the developer − Single execution point − Normalized and standardized output format − Eclipse based – developers will not have to trained on another tool

• TOIF provides a mechanism to “tune out” false positives on individual adapted SCA tools.

• LM Aeronautics Company can add additional tests / tune existing ones in the OSS SCA tools to cover CWE’s relevant to Avionics.

• Adapt additional SCA tools to TOIF to expand CWE coverage for Developers, System Security Engineering teams and C&A efforts.

• Substantial reduction in licensing costs.

Chart 26

Specific capabilities of individual commercial tools should be licensed on a case by case SSE requirement.



TOIF References

• Presentation on original TOIF project − http://www.dhs.gov/sites/default/files/publications/csd-edwin-

seidewitz-data-access-technologies.pdf • Additional DHS SBIR funding to enhance TOIF

− https://www.sbir.gov/sbirsearch/detail/402647 • KDM Site about TOIF

− http://www.kdmanalytics.com/toif/index.html



Questions


tool output integration framework · 2016-09-15 · cppcheck • splint • rats … toif...

Documents