struggles at the frontiers: persistent pursuit of …...1 struggles at the frontiers: persistent...

1Struggles at the Frontiers: Persistent Pursuit of Software Assurance in the Development and Sustainment of Defense Systems© 2017 Carnegie Mellon University

[Distribution Statement A] This material has been approved for public release and unlimited distribution

1

Software Solutions Symposium 2017

Software Engineering Institute

Carnegie Mellon University

Pittsburgh, PA 15213

Struggles at the Frontiers: Persistent Pursuit of Software Assurance in the Development and Sustainment of Defense Systems© 2017 Carnegie Mellon University

[[Distribution Statement A] This material has been approved for public release and unlimited distribution

Struggles at the Frontiers: Persistent Pursuit of Software Assurance in the Development and Sustainment of Defense Systems

Dr. Kenneth E. Nidiffer



2


Copyright 2017 Carnegie Mellon University

Copyright 2017 Carnegie Mellon University

This material is based upon work funded and supported by the Department of Defense under Contract No. FA8721-05-C-0003 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center.

Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the United States Department of Defense.

NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN “AS-IS” BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT.

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution.

This material may be reproduced in its entirety, without modification, and freely distributed in written or electronic form without requesting formal permission. Permission is required for any other use. Requests for permission should be directed to the Software Engineering Institute at [email protected].

Carnegie Mellon® and CERT® are registered marks of Carnegie Mellon University.

TSPSM

DM-0004299



3


Source: SEI Increasing globalization

Software Enabled Systems are Today’s Strategic Resource

Dr. Bill Scherlis*

“Software is the building material for modern society”

ManualLabor

Water

Steam

Oil

Software

//upload.wikimedia.org/wikipedia/commons/1/18/AndersonMill.jpg

//upload.wikimedia.org/wikipedia/commons/1/18/AndersonMill.jpg



4


SEI Works in Several Core Technical Areas

Autonomy & Counter-

Autonomy

C4ISR Mission Assurance

Cyber Missions

Data Modeling & Analytics

Human-Machine

Interactions

Software & Information Assurance

Systems Verification &

Validation

Combine to Provide Software and Cyber Capabilities



5


Context

• Definition*: The level of confidence that software functions as intended and is free of vulnerabilities, either intentionally or unintentionally designed or inserted as part of the software throughout the life cycle.

• Perspective: Persistent pursuit of software quality assurance is a constant struggle

• Future: Software enabled systems underpin the cyber environment, enabling explorations into new frontiers

Source: SEI

*Source: DoDI 5200.44 Protection of Mission Critical Functions to Achieve TrustedSystems and Networks (TSN), and 2013 NDAA S933



6


Software Assurance is a Constant Purpose, and Software-Enabled Systems are Moving Targets

• Constant purpose

– Software assurance provides the required level of confidence that software functions as intended (and no more) and is free of vulnerabilities, either intentionally or unintentionally designed or inserted in software, throughout the lifecycle*

• Moving target

– The changing and expanding role that software plays in cyberspace means that the development of software-enabled systems must continue to evolve while we pursue software quality

*Source: DoDI 5200.44 Protection of Mission Critical Functions to Achieve Trusted

Systems and Networks (TSN), and 2013 NDAA S933



7


1. Increasing complexity of software assurance

2. Satisfying unique operational mission and business needs

3. Solving the SDLC* software tool chasm

4. Addressing limited spectrum availability

5. Handling the expanding code base

6. Understanding attack patterns, vulnerabilities, and weaknesses

7. Increasing vulnerabilities

8. Designing-in software quality over the lifecycle

9. Reducing technical debt

10. Working in the infancy of software engineering discipline

Software Assurance Struggles at the Frontiers

* SDLC: System Development Life Cycle



8


Increasing Complexity of Software AssuranceFirst Line of Defense in Software Assurance is the Application (Software) Layer

84% of breaches exploit

vulnerabilities in the Application1

1. Clark, Tim, Most cyber Attacks Occur from this Common Vulnerability, Forbes. 03-10-2015

2. Feiman, Joseph, Maverick Research: Stop Protecting Your Apps; It’s Time for Apps to Protect Themselves, Gartner. 09-25-2014. G00269825

Yet, funding for IT defense vs.

software assurance is 23-to-12



9


SEI Technical ThemesSEI Technologies Address DoD Software Assurance Challenges

Focus Areas:

• the behavior as built: architecture evaluation, code analysis, security patterns, etc.

• the behavior during evolution: system monitoring and self-adaptation, etc.

• the affordability: cost estimation, sustainment work, etc.

• the quality: defect modeling, architectural fault modeling, requirements engineering, etc.

• the mission success: acquisition support, acquisition dynamics, acquisition and architecture alignment, etc.

In the broad sense as defined by the DoD challenges and underscored by the SEI Technical Insight.



10


Satisfying Unique Operational Mission and Business NeedsPractical Answers for a Complicated World – Intelligent Integrated Solutions

Source: SEI

http://www.bing.com/images/search?q=facebook+logo&id=B1BB872E048BBAC42E643B605530577BD4E9FAB3&FORM=IQFRBA

http://www.bing.com/images/search?q=facebook+logo&id=B1BB872E048BBAC42E643B605530577BD4E9FAB3&FORM=IQFRBA



11


Addressing Limited Spectrum AvailabilitySpectrum-based Technologies - Key to DoD Mission and Commercial Business Needs

DoD Mission

• Increasing spectrum needs

Commercial Business

• Increasing spectrum needs

Exploding requirements for information and mobility!CAGR = compound annual growth rate Source: Kenneth R. Turner, Dep. Director, Spectrum Policy and International Engagements, DoD Chief Information Officer



12


0 100 200 300 400 500 600

Boeing 787, total flight software

Android (upper estimate)

Linux 3.1 (recent version, 2013)

Apache Open Office

F-35 Fighter

Microsoft Office (2001)

Windows 2000 (2000)

Microsoft Office for Mac (2006)

Symbian

Windows 7

Windows XP (2001)

Microsoft Office (2013)

Large Hadron Collider

Microsoft Visual Studio 2012

Windows Vista (2007)

Facebook (without backend code)

US Army's Future Combat System

Debian 5.0 codebase

Mac OS X 10.4

Software in typical new car, 2013

Debian 5.0 (all software in package)

Healthcare.gov

Size of Codebase (SLOC)

Handling the Expanding Code Base

Software Is Dramatically Expanding with Limited Natural Governance

Source: David McCandless, Information is Beautiful, 21 September 2016 web retrieval

Millions of Source Lines of Code

Google = 2,000 MSLOC



13


Weakness

Weakness

Weakness

Weakness/Vulnerability

Reported

Asset

Attack

Impact

Item

Item

Item

Attack

Attack

Function

Asset

Impact

Impact

knownthreatactors

attack patterns(CAPEC)

weaknesses& vulnerabilities

(CWE/CVE)

actions*

technicalimpacts

operational impacts

• “Actions” include architecture choices; design choices; added security functions, activities, and processes; physical decomposition choices; static and dynamic code assessments; design reviews; dynamic testing; and pen testing.

• Vulnerability is the intersection of three elements: a system susceptibility or flaw, attacker access to the flaw, and attacker capability toexploit the flaw.

System & System Security

EngineeringTrades

Understanding Attack Patterns, Vulnerabilities, and Weaknesses

water floodinghatch door in

sub

close door sub fills

w/water sub sinks

Source: Bob Martin, MITRE



14


Increasing VulnerabilitiesReported Common Vulnerabilities and Exposures (CVE)

Source: Dr. Robert A. Martin, MITRE Corporation, August 2015

© 2013 MITRE

CVE 1999 to 2001to 2015

CVEs(reported, publicly known vulnerabilities and exposures with patches)

VULNERABILITIES

WEAKNESSES

CWEs(characterized, discoverable, possibly exploitable weaknesses with mitigations)

Zero-Day Vulnerabilities(previously unmitigated weaknesses that are exploited with little or no warning)

Uncharacterized Weaknesses

Unreported or undiscovered Vulnerabilities



15


Designing-in System Acquisition Quality Over the LifecycleContinuous Engineering Over The Life Cycle

Full Rate Production Decision Review

CA

Technology Maturation & Risk

Reduction

Engineering & Manufacturing Development

Production and Deployment

Operation and Sustainment

SFR CDR TRR SVRASR

Materiel Solution Analysis

MDD

IOT&EOTRR

Requirements Decision

Developmental RFP Decision

PDR

B

AOA ICD

Criticality Analysis

RFP

ThreatVuln

Analysis

Understand SwA Risks

Design InspectionSecure Coding

PracticesCode Inspection

Test Coverage Metrics

Monitor Threats/ Attacks

Upgrade SwADesign Principles

Respond to IncidentsDefects

Pen Test

Software assurance must be baked-in throughout entire system lifecycle

IdentifyCodeFlaws

IdentifyTestGaps

IOC FOC

VulnRoot

Cause

EffectiveThreat

Response

CyberSituationalAwareness

CyberHardenedSystems

SoundArch &Design

MoreSecureCode

BetterTest

Suites

SwACounter-Measures Fewer

ProcessVuln

EliminateDesignFlaws

SRR

Source: OSD/DASD/SE

Distribution Statement A – Approved for public release. Distribution is unlimited.



16


SEI Supporting JFAC* Pilot Program Focused on Solving the SDLC Software Assurance Tool Chasm

• The JFAC is a federation of DoD organizations that have a shared interest in

promoting software and hardware assurance in defense acquisition programs,

systems, and supporting activities.

• SEI provides experience SEI technical staff to support the dynamic and

evolving needs of the JFAC, such as the SDLC software assurance tool chasm

*JFAC: D0D Joint Federated Assurance Center



17


17

Source: PERFORMANCE OF THEDEFENSE ACQUISITION SYSTEM2015ANNUAL REPORTSept 16, 2015

Reducing Technical DebtEngineering Assurance into the Fabric of Programs

Right Direction



18


Reducing Technical DebtAircraft Software Development and Rework Cost

Reference: U.S. Air Force Scientific Advisory Board. Sustaining Air Force Aging Aircraftinto the 21st Century (SAB-TR-11-01). U.S. Air Force, 2011.



19


Working in the Infancy of the Software Engineering DisciplineIntegrate Software and Systems Engineering with a Focus on Cyber

Physical Science Bioscience Computer/Software/Cyber Science

Origins/History Begun in antiquity Begun in antiquity Mid-20th century

Enduring Laws Laws are foundational to

furthering exploration in the

science

Laws are foundational to

furthering exploration in the

science

Only mathematical laws have proven

foundational to computation

Framework of

Scientific Study

Four main areas: astronomy,

physics, chemistry, and

earth sciences

Science of dealing with

health maintenance and

disease prevention and

treatment

• Several areas of study: computer

science, software/systems

engineering, IT, HCI, social

dynamics, AI

• All nodes are attached to and rely

on a netted system

R&D and Launch

Cycle

10–20 years 10–20 years Significantly compressed; solution

time to market must happen very

quickly

Source: SEIHCI: human–computer interaction; AI: artificial intelligence



20


1. Increasing Complexity of Software Assurance

2. Satisfying unique operational mission and business needs

3. Solving the SDLC Software Tool Chasm

4. Addressing limited spectrum availability

5. Handling the expanding code base

6. Understanding attack patterns, vulnerabilities, and weaknesses

7. Increasing vulnerabilities

8. Designing-in software quality over the lifecycle

9. Reducing Technical Debt

10. Working in the infancy of software engineering discipline

Software Assurance Struggles at the Frontiers



21


SEI Technologies Address DoD Software Assurance Challenges

Today’s Presentations of SEI’s Contributions

• Software Vulnerabilities Track

- Relationship Between Design Flaws and Software Vulnerabilities

- Predicting Software Bug Closure Rates and Reliability Growth with

Weibull Modeling

- Using Malware Analysis to Identify Overlooked Security

Requirements

• Modernization and Sustainment Track

- Case Study - Successful Government Roadmap Modernization

Project

- Panel - Software Sustainment: Continuous Engineering to Deliver

Warfighter Capability

- Methodology for the Cost Benefit Analysis of a Large Scale Multi-

phasic Software Enterprise Migration



22


Questions?



23


Contact Information

Dr. Kenneth E. Nidiffer, Director of Strategic Plans

for Government Programs

Software Engineering Institute, Carnegie Mellon University

Office: + 1 703-247-1387

Fax: + 1 703-908-9235

Email: [email protected]



24


References

Gallagher, B.; Nidiffer, K. & Saga, R. The Ordered Process for Improving Agile

Engineering Outcomes, CrossTalk, Nov/Dec 2016

JFAC Collaboration Portal https://intelshare.intelink.gov/sites/jfac/

SEI Training: Manage Cybersecurity Risk Across the Lifecycle - The CERT Software

Assurance Framework (SAF) Assessment

SEI Training: Architecture Tradeoff Analysis Method (ATAM)

SEI Training: Security Requirements Engineering Using the SQUARE Method

SEI Training: Team Software Process (TSP)

Source Code Analysis Laboratory (SCALe), SEI Technical Note, CMU/SEI-2012-TN-

013, 2013

Quantifying Uncertainty in Early Lifecycle Cost Estimation (QUELCE), Technical

Report, CMU/SEI-2011-TR-026, 2011

Alberts, C.; Woody, C.; & Dorofee, A. Evaluating Security Risks using Mission

Threads(CMU/SEI-2014-TN-025), 2014.

https://intelshare.intelink.gov/sites/jfac/

struggles at the frontiers: persistent pursuit of …...1 struggles at the frontiers: persistent...

Documents