cybersecurity in defense: measuring risk and the aaf · “risk matrices should not be used for...

Think Differently: Cybersecurity and the

Adaptive Acquisition Framework

Securing the DoD Supply ChainCybersecurity Maturity Model Certification

Ms. Katie ArringtonChief Information Security Officer for Acquisition

2

UNCLASSIFIED

UNCLASSIFIED

CMMC Model Structure

3

17 Capability Domains (v1.0) Capabilities are assessed for Practice and Process Maturity

DISTRIBUTION A. Approved for public release

UNCLASSIFIED

4

LEVEL 1BASIC CYBER HYGIENE

LEVEL 2INTERMEDIATE CYBER HYGIENE

LEVEL 3GOOD CYBER HYGIENE

LEVEL 4PROACTIVE

LEVEL 5ADVANCED / PROGRESSIVE

17 PRACTICES

Demonstrate compliance with Federal Acquisition Regulation (FAR) 48 CFR 52.204-21

72 PRACTICES

Comply with the FAR

Perform a select subset of 48 practices from the NIST SP 800-171 r1

Perform an additional 7 practices to support intermediate cyber hygiene

130 PRACTICES

Comply with the FAR

Perform all 110 practices from the NIST SP 800-171 r1

Perform an additional 20 practices to support good cyber hygiene

152 PRACTICES

Comply with the FAR


Perform a select subset of 13 practices from Draft NIST SP 800-171B

Perform an additional 29 practices to demonstrate a proactive cybersecurity program

171 PRACTICES

Comply with the FAR


Perform a select subset of 17 practices from Draft NIST SP 800-171B

Perform an additional 40 practices to demonstrate an advanced cybersecurity program

CMMC Practice Progression


Reduces risk of Advanced Persistent Threats (APTs)

UNCLASSIFIED

5

LEVEL 1PERFORMED

LEVEL 2DOCUMENTED

LEVEL 3MANAGED

LEVEL 4REVIEWED

LEVEL 5OPTIMIZING

0 PROCESSES

Select practices are documented where required

2 PROCESSES

Each practice is documented, including Level 1 practices

A policy exists that includes all activities

3 PROCESSES

Each practice is documented


Adherence is verified through Examine or Test

A plan exists, is maintained, and resourced that includes all activities (includes mission, goals, project plan, resourcing, training needed, and involvement of relevant stakeholders)

4 PROCESSES




A plan exists that includes all activities

Activities are reviewed and measured for effectiveness (results of the review is shared with higher level management and for issue resolution)

5 PROCESSES




A plan exists that includes all activities

Activities are reviewed and measured for effectiveness

There is a standardized, documented approach across all applicable organizational units

CMMC Maturity Process Progression


UNCLASSIFIED

CMMC Model Evolution v0.4 to v0.5 to v0.6 to v0.7 to v1.0

6

380

859

316

59 9

219

44 9

17343 9

17143 5 35

11592 96

4233

78 83 85

3717

58 56 62

2617

55 59

26 1617

55 58

26 15

Level 1 Level 2 Level 3 Level 4 Level 5

Practices by Level

40

19

26

1621 21

17

41

913

5

17

8

36

16 17

45

13

39

1720

16 18

7

16

30

9 10

3

128

27

16 16

40

12

34

5

15

5

15

0

16 18

7 94 6 4

1511

5

35

15

26

3

14

511

0

1114

6 82

6 4

128

3

27

13

26

2

14

511

0

11 13

6 82

6 4

128

3

27

13

AC AM AA AT CM CG IDA IR MA MP PS PP RE RM SAS SA SCP SII

Practices by Domain

V0.4 V0.5 V0.6 V0.7 V1.0


UNCLASSIFIED

• CMMC Model leverages multiple sources and references– CMMC Level 1 only includes practices from FAR Clause 52.204-21

– CMMC Levels 4 and 5 do not include QTY 16 practices from Draft NIST SP 800-171B because of implementation complexity/constraints and/or cost

CMMC Model v1.0: Source Counts

7

CMMC Model v1.0: Number of Practices per Source

* Note: QTY 15 safeguarding requirements from FAR clause 52.204-21 correspond to QTY 17 security requirements from NIST SP 800-171r1, and in turn, QTY 17 practices in CMMC

CMMC Level

Total Number Practices per CMMC Level

Source

48 CFR 52.204-21

NIST SP 800-171r1

Draft NISTSP 800-171B Other

Level 1 17 17 * 17 - -

Level 2 55 - 48 - 7

Level 3 58 - 45 - 13

Level 4 26 - - 13 13

Level 5 15 - - 4 11

Total 171 17* 110 17 44

Excluded - - - 16 -


UNCLASSIFIED

Grant Certification

Conduct Certification

Certificate

Update

Internet Accessible Lookup

Advance to Level

Options:1. Internal2. SVC Provider3. Partner

Source Selection

(Go/No-Go)

RFP Award

Self-Evaluate

Companies Create

DatabaseEst. PMO

Office

ACQ Review

RFI “Level x”& Date

Develop Model

CMMC Concept

CMMC REQT

PMRequiring Activity

Select Certifier

CertifierDevelop

Accreditation Body REQT.

Est. MOU Accrd. Body

BID

Verify CMMC Level

FindCertifier

Document Cert

Accreditation BodyCMMC Gov’tGov’t PMCertifierCompany

SRM Database

Sr. Advisory Council

Beginwork

Accrd. BodyIOC

CMMC Implementation Flow

BeginWork

Accrd. BodyIOC

Market Place

CMMCCertificateDatabase

Create Databas

e

8DISTRIBUTION A. Approved for public release

UNCLASSIFIED

CMMC Accreditation Body Activities

9

Accreditation Body (AB) Manager

Training Accreditation

Credentialing

Infrastructure

(Support Systems)

• Train Individuals

• Train Organizations

• Train Instructors

• Knowledge Store

• Market Place • Artifact Store• Records

Mgmt.

• Grant C3PAO accreditations

• Audit C3PAO• Process Complaints

• Grant Individual credentials

• Certifiers• Accredited

Certifiers

• Coordinate w/ CMMC PMO and CMMC Advisory Council

• Dispute resolution• Capture metrics• Integrate and coordinate

functional areas

Assessment Operations

• Technical Appeals• Quality Control• Manage

Assessment Tool• Publish CMMC

Certificates

Populated and accessible by DoD

systems

CMMCDatabase


UNCLASSIFIED

CMMC Draft Schedule: CY20

10

Q2FY20 Q3FY20 Q4FY20 Q1FY21

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

Rulemaking

CMMC Roll-Out Plan

DoDI 5000.02 Cybersecurity Enclosure

CMMC Implementation Pathfinder(s) with Subset of DIB Sector

CMMC Accreditation Body (AB)

CMMC Databases & Infrastructure

CMMC AB Training• Train the trainers • CMMC classes for assessors

Draft CMMC Training Material to CMMC Accreditation Body (AB)

Defense Acquisition University (DAU) CMMC Training

CMMC Assessment Guides

CMMC Model

DAR Council Meeting Goal: Complete Rulemaking Process

Initial RFIs with CMMC Requirement

Initial RFPs with CMMC Requirement

* Depends upon Rulemaking

Complete reviews and approval

Complete & Release v1.0

Update & Refine CMMC Assessment Guides

Deliver Levels 4-5 to CMMC ABDeliver Levels 1-3 to CMMC AB

Deliver Draft TrainingCMMC 101 Coordinate and Conduct Training Pathfinder with CMMC AB

Deliver Draft TrainingCMMC Levels 1-3

Deliver Draft TrainingCMMC Levels 4-5

Establish CMMC AB Board

Sign MOU (TBD) Marketplace (TBD)

Certification process for candidate CMMC Third Party Assessment Organizations (C3PAOs)

Initial Planning Database/Infrastructure Pathfinder

Initial Beta Testing

Initiate Training for CMMC 101, Levels 1-3

Initiate Training for CMMC Levels 4-5

Refine Draft Training & Conduct Training Pathfinder with DoD

Initiate Training for CMMC 101, Levels 1-3Initiate Training for CMMC Levels 4-

5

* Depends upon Rulemaking

Potential update based on rulemaking process (TBD)

Pathfinder(s)Initial Planning

UNCLASSIFIED

Projected CMMC Roll-Out

11

Total Number of Prime Contractors and Sub-Contractors with CMMC RequirementFY21 FY22 FY23 FY24 FY25

Level 1 895 4,490 14,981 28,714 28,709

Level 2 149 748 2,497 4,786 4,785

Level 3 448 2,245 7,490 14,357 14,355

Level 4 4 8 16 24 28

Level 5 4 8 16 24 28

Total 1,500 7,500 25,000 47,905 47,905

Total Number of Contracts with CMMC RequirementFY21 FY22 FY23 FY24 FY25

15 75 250 479 479

• OUSD(A&S) will work with Services and Agencies to identify candidate programs that will have the CMMC requirement during FY21-FY25 phased roll-out

• All new DoD contracts will contain the CMMC requirement starting in FY26

UNCLASSIFIED


12

https://www.acq.osd.mil/cmmc/index.html


UNCLASSIFIED

Douglas Hubbard Author and founder of Hubba rd Decis ion Resea rch (HDR)

Hubbard Decision Research2 South 410 Canterbury CtGlen Ellyn, Illinois 60137

www.hubbardresearch.com

© Hubbard Decision Research, 2020

How to Measure Anything in

Cybersecurity Risk

What is your single biggest risk in cybersecurity?

How you assess cybersecurity risk.

15

Presenter

Presentation Notes

Follow-up Questions: How do we know what works? How can we start to improve risk assessment?

Summarizing Research on Risk Matrices

◇ “Risk Matrices should not be used for decisions of any consequence.”￭ Bickel et al. “The Risk of Using Risk

Matrices”, Society of Petroleum Engineers, 2014

◇ “…they ca n be ‘w orse tha n useless’”￭ Tony Cox “W ha t’s w rong w ith Risk

Ma trices” inves tiga tes va rious ma thema tica l consequences of ordina l sca les on a ma trix.

16

Like

lihoo

d

Impact

“The first principle is that you must not fool yourself, and you are the easiest person to fool.”

—Richard P. Feynman

17

Analysis Placebo An apparently “structured” method will increase

confidence in estimates and decisions even when measured performance is the same or worse.

What Works?

Research Results:

Simple statistical models beat human experts in a wide variety of estimates

and forecasts.

18

Paul Meehl assessed 150 studies comparing experts to statistical models in many fields (sports,

prognosis of liver disease, etc.).

Philip Tetlock tracked a total of over 82,000 forecasts from 284

political experts in a 20 year study covering elections, policy effects, wars, the economy and

more.

What Quantitative Risk Can Look Like

19

Inherent Risk

Risk Appetite

Residual Risk

Loss Exceedance Curve

Probability-Weighted Average

Return on ControlControl 1 846%Control 2 131%Control 3 15%Control 4 -45%

How much risk do we have? Is the risk acceptable? How should I reduce risk?

A Simple “One-For-One Substitution”

Each “Dot” on a risk matrix can be better represented as a row on a table like this.

The inputs are used in a Monte Carlo simulation in Excel.

The output can then be represented as a Loss Exceedance Curve.

20

Examples can be found at www.howtomeasureanything.com

A Path to Improvement

21

Even without new data, some subjective estimation methods are better than others.

To improve estimates, you have more data than you think

You need less data than you think

A very mathematical corollary from super advanced statistics:

If you know almost nothing, almost anything will tell you something

LCDR Ryan HilgerUSN

Big IdeaIn an increasingly complex threat environment,

a cquis ition profess iona ls need: ◇ a better unders ta nding of their progra m’s

a tta ck surfa ce a nd ◇ a different w a y to conceptua lize risk a nd

mea sures to mitiga te it

All statements are made in a personal capacity and do not represent the views of Navy Strategic Systems Programs, the Department of the Navy, or the Department of Defense

Expanding on the Big Idea ◇ A complete decomposition of the problem statement unlocks new

options￭ More alternatives than technological solutions!￭ Understand what information would cause you to change the

program cost/schedule/performance baseline◇ A well-grounded understanding of the theory behind the system is

crucial to effective operations￭ Overall, most seem to lack sufficient understanding of probability,

statistics, and data curation to effectively implement this paradigm◇ Healthy skepticism for even long-standing tools and processes improves

outcomes


Takeaways◇ Develop. Your. People. They are our most important asset. Treat them as

such.◇ Multiple options exist to get help:

￭ GSA Schedule contracts￭ Small Business Innovation Research contracts￭ Simplified Acquisition contracts

◇ It is worth a bit of program budget to learn!


Timothy DenmanCybersecurity Learning Director Defense Acquis ition Univers ity

27

“Adaptability” - the ability to effectively react to circumstances to reduce risk to cost, performance, and schedule

“Adaptive acquisition” • includes exposing outcome-based requirements to as broad a

marketplace of solution providers as possible; • benchmarking of best existing capability; • reactively adapting system design to take advantage of existing mature

technology; • and streamlining engineering, programmatic, and procurement

bureaucracy

• This begins with understanding risks and producing measurable and testable objectives.

Cybersecurity should be at the center of adaptive acquisition – it cannot be an afterthought

Presenter

Presentation Notes

In the context of acquisition, “adaptability” means to reduce risk to cost, performance, and schedule, by reacting to circumstances

Adaptive Acquisition Framework DoDD 5000.01: The Defense Acquisition SystemDoDI 5000.02: Operation of the Adaptive Acquisition Framework

Tenets of the Defense Acquisition System1. Simplify Acquisition Policy2. Tailor Acquisition Approaches3. Empower Program Managers

4. Data Driven Analysis5. Active Risk Management6. Emphasize Sustainment

< 5 years

< 1 yearCybe

rsec

urity

PathSelection

Defense Business Systems

Middle Tierof

Acquisition

Acquisition of Services

Major Capability

Acquisition

OPE

RATI

ON

S AN

D SU

STAI

NM

ENT

Business Capability Acquisition Cycle

SoftwareAcquisition

RapidPrototyping

CapabilityNeed

IdentificationSolutio

nAnalysi

s

FunctionalRequirements and

Acquisition Planning

AcquisitionTesting and Deployment

CapabilitySupport

Plan

ning

Phas

e I1 I2…

MVP MVCR Rn

OD

Rapid Fielding

10

MaterialSolutionAnalysis

TechnologyMaturation

and Risk Reduction

Engineering and ManufacturingDevelopment

Production andDeployment

MDD MS A MS B MS C IOC FOC

ATP ATP ATP ATP

In In InExecution Phase

OD

< 5 years

UrgentCapability

Acquisition < 2 years

DD

1Formthe

Team

2ReviewCurrentStrategy

3PerformMarket

Research

4Define

Require-ments

5Develop

AcquisitionStrategy

6ExecuteStrategy

7Manage

Performance

PLAN DEVELOP EXECUTE

Legend:ATP: Authority to ProceedDD: Disposition DecisionFOC: Full Operational CapabilityI: IterationIOC: Initial Operational CapabilityMDD: Material Development DecisionMS: MilestoneMVP: Minimum Viable ProductMVCR: Minimum Viable Capability ReleaseOD: Outcome DeterminationR: Release

Version 3.2

https://www.dau.edu/tools/t/DAU-Glossary

Cybersecurity and DOT&E (FY18 Annual Report)DOT&E identified five improvement a rea s to ena ble cyber defenders to do their jobs w ell:

◇ Scope the ta sk by defining the key cyber terra in, opera tiona l miss ions , ta sks , a nd expecta tions .

◇ Foster unity of effort a mongst pa rticipa nts tha t ha ve different roles (offens ive, defens ive) a nd responsibilities (interna l a nd externa l to a ss igned key cyber terra in).

◇ Know the key cyber terra in, opera tiona l concepts , a nd a va ila ble tools .◇ Ma tch tools a nd skills to the opera tiona l ta sks , miss ions , a nd key

cyber terra in.◇ Pra ctice a nd tra in in opera tiona lly representa tive conditions a ga ins t

rea lis tic cyber-a tta cks .DOD missions and systems remain at risk from adversarial cyber operations. Significant improvements are being made, … BUT they are NOT outpacing the growing capabilities of potential adversaries.29

DAU learning offerings on Cybersecurity ◇ Expertise

￭ 9 full-time cybersecurity professors￭ 6 intermittent cybersecurity professors￭ 6 locations spanning 4 time zones

◇ Customized Training Program DFARS CDI and Cybersecurity Maturity Model Certification (CMMC) v 1.0

◇ 24/7 Learning: www.dau.edu￭ Online Courses ￭ Cybersecurity Community of Practice￭ On-the-Job Tools: Cybersecurity & Acq. Lifecycle Integration Tool (CALIT)

◇ Courses￭ Cyber Training Range￭ Capture the Flag and outreach events

◇ Consulting initiatives with all major Services and Cyber Table Top facilitation

◇ Credentialing: cybersecurity credential courses beginning in Spring 2020

◇ Townhalls, Workshops, Rapid Deployment Training

Delivering world-class cybersecurity training and consultingcontributing to a decisive edge for our warfighters30

http://www.dau.edu/

32

Q & A

33

Upcoming Events

• Feb. 26: @4:00-5:30 pm ET @ The Garden: Actionable Inclusion Panel • Mar. 11: @4:00-5:00 pm ET @ The Garden: Suicide Prevention in the

DON

Follow the NavalX Eventbrite page for future events

Stay connected with NavalX

34

Upcoming Events

Feb. 12: DAU Webcast: Design Thinking on the Job Feb. 20: 2020 DAU Acquisition Update Feb. 26: DAU Webcast: Best Practices for Contracting for DMSMS

Management

DAU Events Calendar: https://www.dau.edu/EventsDAU webcasts: https://www.dau.edu/p/dau-webcasts

Stay connected with DAU

https://www.dau.edu/Events

https://www.dau.edu/p/dau-webcasts

cybersecurity in defense: measuring risk and the aaf · “risk matrices should not be used for...

Documents