risk-based continuous systems of systems data integrity...

30
Risk-based Continuous Systems of Systems Data Integrity Auditing of IoT Dr. Byron Mattingly Thursday: 1:45p-2:30p 12 October 2017 26th Annual ASQ Audit Division Conference: The Intercontinental Hotel, Addison, Texas October 12 - 13, 2017

Upload: others

Post on 20-Aug-2020

2 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Risk-based Continuous Systems of Systems Data Integrity ...rube.asq.org/audit/2017/10/auditing/rick-based-continuous-systems-… · Risk-based Continuous Systems of Systems Data Integrity

Risk-based Continuous Systems of

Systems Data Integrity Auditing of IoT

Dr. Byron Mattingly

Thursday: 1:45p-2:30p

12 October 2017

26th Annual ASQ Audit Division Conference:

The Intercontinental Hotel, Addison, TexasOctober 12 - 13, 2017

Page 2: Risk-based Continuous Systems of Systems Data Integrity ...rube.asq.org/audit/2017/10/auditing/rick-based-continuous-systems-… · Risk-based Continuous Systems of Systems Data Integrity

Byron Mattingly, PhD, MPH, MBAhttp://www.linkedin.com/in/byronmattingly

ASQ: CBA, CMQ/OE, CQA, CRE, CQE, CSQE, CSSBBANSI-ASQ ISO 17025 Certified Lead AssessorHL7 Certified Control SpecialistHIMSS CPHIMS, PMI-ACP, PMP

ASQ Software Division Chair

The views expressed in this presentation are my own and do not reflect the views

of my employer or other organizations with which I am or have been affiliated.

All cited Trademarks are the property of their respective owners.

26th Annual ASQ Audit Division Conference:

The Intercontinental Hotel, Addison, TexasOctober 12 - 13, 2017

Page 3: Risk-based Continuous Systems of Systems Data Integrity ...rube.asq.org/audit/2017/10/auditing/rick-based-continuous-systems-… · Risk-based Continuous Systems of Systems Data Integrity

Abstract

October 12 - 13, 201726th Annual ASQ Audit Division Conference: The Intercontinental Hotel, Addison, Texas

As the cost of micro-electromechanical sensors (including gyroscopes, accelerometers and

pressure sensors) has dropped by more than 80% in the past years, the number of connectable

things equipped with these sensors continues to undergo explosive growth, thus boosting the

expansion of information due to mobile, cloud, social and other forces. In 2020, it is estimated

that the "internet of things" (things that will input data themselves without being entirely

dependent on people for information) will reach 30b and that total amount of data in the digital

universe has increased from 130 exabytes in 2005, to 8.6 zettabytes (=8.6 x 1000 exabytes) in

2015 to more than 40 zettabytes in 2020.

Increasingly, then, "things" considered as complex adaptive systems--systems of "agents" in a

network acting in parallel--are entering a rich but ever-expanding ecology at the "edge of chaos",

regions in a phase transition zone far from equilibrium between stability and chaos. Such

systems offer new challenges for the quality professional seeking to control and mitigate risk over

and above multi-agent “systems of systems” because of properties like hierarchical self-

organization, emergent but non-linear behavior, and self-similarity.

This session examines how refactoring (complex adaptive) systems of (complex adaptive)

systems of the Internet of Things (IoT) partitioned by risk can be used to continuously audit data

integrity flows similar to block-chaining techniques used to establish non-reputable financial audit

trails. In particular, specific suggestions for risk controls applied to the chain of custody of data

audit trails can be used to develop pipe-like message passing guidelines for an I/O semantics

that is protocol-independent thereby minimizing unintended interactions and side-effects that

may lead to, for instance, patient harm in the case of mobile medical devices

Page 4: Risk-based Continuous Systems of Systems Data Integrity ...rube.asq.org/audit/2017/10/auditing/rick-based-continuous-systems-… · Risk-based Continuous Systems of Systems Data Integrity

Expected Takeaways

Reimagining continuously auditing systems of systems in

terms of the Boyd OODA Paradigm.

Better understanding of integrating risk controls to follow

critical chain of custody data audit trails in complex

software ecologies such as mobility medical apps in the

clinical and non-clinical environments that deploy IoT

architectures

Better understanding of how continuously auditing and

monitoring residual risks can significantly improve the

data integrity of a complex adaptive system.

26th Annual ASQ Audit Division Conference:

The Intercontinental Hotel, Addison, TexasOctober 12 - 13, 2017

Page 5: Risk-based Continuous Systems of Systems Data Integrity ...rube.asq.org/audit/2017/10/auditing/rick-based-continuous-systems-… · Risk-based Continuous Systems of Systems Data Integrity

Harvey unloaded 33 trillion gallons of water in the U.S.

October 12 - 13, 201726th Annual ASQ Audit Division Conference: The Intercontinental Hotel, Addison, Texas

Saturday evening update, Sept. 2, 2017:

The overwhelming majority of Harvey’s rains are

over, and we have a new calculation for the total

volume of water it dispensed on U.S. soil: 33 trillion

gallons. This number incorporates the rainfall not

only in Texas and Louisiana, but also in Tennessee

and Kentucky, which also experienced torrents.

Source: https://www.washingtonpost.com/news/capital-weather-gang/wp/2017/08/30/harvey-has-unloaded-24-5-trillion-gallons-of-water-on-texas-and-

louisiana/?utm_term=.4705b5c245ba

Page 6: Risk-based Continuous Systems of Systems Data Integrity ...rube.asq.org/audit/2017/10/auditing/rick-based-continuous-systems-… · Risk-based Continuous Systems of Systems Data Integrity

Harvey unloaded 33 trillion gallons of water in the U.S.

October 12 - 13, 201726th Annual ASQ Audit Division Conference: The Intercontinental Hotel, Addison, Texas

Source: https://www.washingtonpost.com/news/capital-weather-gang/wp/2017/08/30/harvey-has-unloaded-24-5-trillion-gallons-of-water-on-texas-and-

louisiana/?utm_term=.4705b5c245ba

33 x 1012 gals = 29.97 cubic miles

U.S. = 3.797 million square miles

Equivalent rainfall of ½ inch across U.S.

Image Source (9/22/2017): https://nhd.usgs.gov/NHD_Medium_Resolution.html

Page 7: Risk-based Continuous Systems of Systems Data Integrity ...rube.asq.org/audit/2017/10/auditing/rick-based-continuous-systems-… · Risk-based Continuous Systems of Systems Data Integrity

If one teaspoon = one byte of data . . .

October 12 - 13, 201726th Annual ASQ Audit Division Conference: The Intercontinental Hotel, Addison, Texas

757,500 gallons of water per second

768 “bytes” (teaspoons) to a gallon

582 MB/s of data flowing @ 1 teaspoon / byte

Source (8/22/2015): http://www.niagarafallsstatepark.com/Amazing-Facts.aspx

Image Source (8/22/2015): https://en.wikipedia.org/wiki/Niagara_Falls

582 MB/s

Page 8: Risk-based Continuous Systems of Systems Data Integrity ...rube.asq.org/audit/2017/10/auditing/rick-based-continuous-systems-… · Risk-based Continuous Systems of Systems Data Integrity

How big is 44 ZB (= 44 x 1021) bytes of data?

October 12 - 13, 201726th Annual ASQ Audit Division Conference: The Intercontinental Hotel, Addison, Texas

Source: https://www.emc.com/leadership/digital-universe/index.htm

Page 9: Risk-based Continuous Systems of Systems Data Integrity ...rube.asq.org/audit/2017/10/auditing/rick-based-continuous-systems-… · Risk-based Continuous Systems of Systems Data Integrity

How big is 44 ZB (= 44 x 1021) bytes of data?

October 12 - 13, 201726th Annual ASQ Audit Division Conference: The Intercontinental Hotel, Addison, Texas

1 Zettabyte is 1021 bytes = 1 Trillion GB

44 ZB = 5.73x1019 gallons of water

Source (8/22/2015): http://www.niagarafallsstatepark.com/Amazing-Facts.aspx

Image Source (8/22/2015): https://en.wikipedia.org/wiki/Niagara_Falls

2.4M years

Page 10: Risk-based Continuous Systems of Systems Data Integrity ...rube.asq.org/audit/2017/10/auditing/rick-based-continuous-systems-… · Risk-based Continuous Systems of Systems Data Integrity

Remote Patient Monitoring Market to Top

$26 Billion by 2018: Big Data or HUGE Data?

http://www.meddeviceonline.com/doc/cutting-through-the-remote-patient-monitoring-hype-0001

“According to a recent

report by market research

firm IDC, the IoT currently

comprises some 20 billion

connected “things” — all of

them collecting, sharing,

and/or using data — and

that number is expected to

approach 30 billion by

2020.”

26th Annual ASQ Audit Division Conference:

The Intercontinental Hotel, Addison, TexasOctober 12 - 13, 2017

Page 11: Risk-based Continuous Systems of Systems Data Integrity ...rube.asq.org/audit/2017/10/auditing/rick-based-continuous-systems-… · Risk-based Continuous Systems of Systems Data Integrity

11

“There’s Plenty of Room at the Bottom”

—Richard Feynman (APS Meeting, Dec 29, 1959)

Sources (8/21/2015): http:////www.semi.org/en/IndustrySegments/EmergingMarkets/CTR_038029

http://www.digikey.com/en/articles/techzone/2012/jul/a-designers-guide-to-mems-sensors

http://www.bosch.com/en/com/products_services/industry_trade/sensors_mems_consumer_electronics/sensors-mems-for-consumer-electronics.php

Gyroscope ~ 90¢> 10x Price Drop

in 5 years!

MEMS Market Grows

as Prices Decline

Page 12: Risk-based Continuous Systems of Systems Data Integrity ...rube.asq.org/audit/2017/10/auditing/rick-based-continuous-systems-… · Risk-based Continuous Systems of Systems Data Integrity

Old Software Never Dies

Image source: http://geekandpoke.typepad.com/geekandpoke/2007/11/old-software-li.html

//COBUCLG JOB CLASS=A,MSGCLASS=A,MSGLEVEL=(1,1)

//HELOWRLD EXEC COBUCLG,PARM.COB='MAP,LIST,LET'

//COB.SYSIN DD *

001 IDENTIFICATION DIVISION.

002 PROGRAM-ID. 'HELLO'.

003 ENVIRONMENT DIVISION.

004 CONFIGURATION SECTION.

005 SOURCE-COMPUTER. IBM-360.

006 OBJECT-COMPUTER. IBM-360.

0065 SPECIAL-NAMES.

0066 CONSOLE IS CNSL.

007 DATA DIVISION.

008 WORKING-STORAGE SECTION.

009 77 HELLO-CONST PIC X(12) VALUE 'HELLO, WORLD'.

075 PROCEDURE DIVISION.

090 000-DISPLAY.

100 DISPLAY HELLO-CONST UPON CNSL.

110 STOP RUN.

//LKED.SYSLIB DD DSNAME=SYS1.COBLIB,DISP=SHR

// DD DSNAME=SYS1.LINKLIB,DISP=SHR

//GO.SYSPRINT DD SYSOUT=A

//

Code source: http://en.wikipedia.org/wiki/COBOL

26th Annual ASQ Audit Division Conference:

The Intercontinental Hotel, Addison, TexasOctober 12 - 13, 2017

Page 13: Risk-based Continuous Systems of Systems Data Integrity ...rube.asq.org/audit/2017/10/auditing/rick-based-continuous-systems-… · Risk-based Continuous Systems of Systems Data Integrity

Code listing for the Apollo Guidance

Computer (AGC) program

October 12 - 13, 201726th Annual ASQ Audit Division Conference: The Intercontinental Hotel, Addison, Texas

Source: http://authors.library.caltech.edu/5456/1/hrst.mit.edu/hrs/apollo/public/archive/1701.pdf

Image source: https://www.nasa.gov/mission_pages/apollo/apollo11.html

> 3M parts / 700k components / 20k contractors

Page 14: Risk-based Continuous Systems of Systems Data Integrity ...rube.asq.org/audit/2017/10/auditing/rick-based-continuous-systems-… · Risk-based Continuous Systems of Systems Data Integrity

Image Source (8/24/2015): http://www.boeing.com/commercial/777/

Boeing 777 = 3M parts from 500 suppliers

Source (8/25/2015):

http://www.usatoday.com/story/money/business/2014/03/30/why-a-boeing-777-costs-320-million-dollars/7063805/

26th Annual ASQ Audit Division Conference:

The Intercontinental Hotel, Addison, TexasOctober 12 - 13, 2017

Page 15: Risk-based Continuous Systems of Systems Data Integrity ...rube.asq.org/audit/2017/10/auditing/rick-based-continuous-systems-… · Risk-based Continuous Systems of Systems Data Integrity

Model Based Systems Engineering with

Interface Control Documents (ICDs)

ICD

:A

:A IF

:Subsystem 1

:A IF :B IF

:Subsystem 2

:B

:B IF

req_1

req_2

req_n

:A/B IF

Risk = Severity x

Probability(hazard)

Example risk tools:

PHA, FTA, FMEA (use, process, design)

4 Pillars of MBSE:

1. Structure

2. Behavior

3. Requirements

4. Parametrics

Start Here: http://www.omgsysml.org/

26th Annual ASQ Audit Division Conference:

The Intercontinental Hotel, Addison, TexasOctober 12 - 13, 2017

Page 16: Risk-based Continuous Systems of Systems Data Integrity ...rube.asq.org/audit/2017/10/auditing/rick-based-continuous-systems-… · Risk-based Continuous Systems of Systems Data Integrity

Meir “Manny” Lehman’s Law

“As an evolving program is

continually changed, its complexity,

reflecting deteriorating structure,

increases unless work is done to

maintain or reduce it.”—Meir Manny Lehman, 1980

Source: http://en.wikipedia.org/wiki/Technical_debt

26th Annual ASQ Audit Division Conference:

The Intercontinental Hotel, Addison, TexasOctober 12 - 13, 2017

Page 17: Risk-based Continuous Systems of Systems Data Integrity ...rube.asq.org/audit/2017/10/auditing/rick-based-continuous-systems-… · Risk-based Continuous Systems of Systems Data Integrity

Boyd OODA for Validation

Observations Decision

(Hypothesis)

Action

(Test)

Cultural

Traditions

Genetic

Heritage

New

Information Previous

Experience

Analyses &

SynthesisFeed

ForwardFeed

Forward

Implicit

Guidance

& Control

Implicit

Guidance

& Control

Unfolding

Interaction

With

EnvironmentUnfolding

Interaction

With

EnvironmentFeedback

Feedback

Outside

Information

Unfolding

Circumstances

Observe Orient Decide Act

From “The Essence of Winning and Losing,”

J. R. Boyd, January 1996

http://www.d-n-i.net

Feed

Forward

26th Annual ASQ Audit Division Conference:

The Intercontinental Hotel, Addison, TexasOctober 12 - 13, 2017

Page 18: Risk-based Continuous Systems of Systems Data Integrity ...rube.asq.org/audit/2017/10/auditing/rick-based-continuous-systems-… · Risk-based Continuous Systems of Systems Data Integrity

Feedback Loops Reduce Uncertainty

and Manage Change

Build

Test

Measure

Metric

Learn

Hypo-thesis

• OODA Loops

• Shewhart / Deming

Cycles (PDCA)

• Six Sigma (DMAIC)

• Lean Manufacturing

• Lean Startup

• Lean UX

26th Annual ASQ Audit Division Conference:

The Intercontinental Hotel, Addison, TexasOctober 12 - 13, 2017

Page 19: Risk-based Continuous Systems of Systems Data Integrity ...rube.asq.org/audit/2017/10/auditing/rick-based-continuous-systems-… · Risk-based Continuous Systems of Systems Data Integrity

Agile Approach

Develop, Test

& Feedback

Develop, Test

& Feedback

Develop, Test

& Feedback

. . . Release

Project Setup

26th Annual ASQ Audit Division Conference:

The Intercontinental Hotel, Addison, TexasOctober 12 - 13, 2017

Page 20: Risk-based Continuous Systems of Systems Data Integrity ...rube.asq.org/audit/2017/10/auditing/rick-based-continuous-systems-… · Risk-based Continuous Systems of Systems Data Integrity

Verification Driven Agile Development

From this:Time

Buginjection Bugdetected Bugfound Bugfixed

Tdetect Tfind Tfix

… to thisTime

Buginjection Bugfound Bugfixed

Td Tfind Tfix

26th Annual ASQ Audit Division Conference:

The Intercontinental Hotel, Addison, TexasOctober 12 - 13, 2017

Page 21: Risk-based Continuous Systems of Systems Data Integrity ...rube.asq.org/audit/2017/10/auditing/rick-based-continuous-systems-… · Risk-based Continuous Systems of Systems Data Integrity

Example Automated Verification Testing System

High Level DesignTC = Test Case

Goal: 90-95% TCs Automated

Device Under TestTest Automation HostTest Automation Client

TC1 TC2 . . . TCN

Test Management

System

Python Keyword

Library

Python

RESTful Web

Services

Test Harness

(Switch box, patient

simulators, etc.)

C++ / Java

Test Agent

Clinical Software

Test Controller: PC or Linux SBC

26th Annual ASQ Audit Division Conference:

The Intercontinental Hotel, Addison, TexasOctober 12 - 13, 2017

Page 22: Risk-based Continuous Systems of Systems Data Integrity ...rube.asq.org/audit/2017/10/auditing/rick-based-continuous-systems-… · Risk-based Continuous Systems of Systems Data Integrity

Partition System of Systems by Risk“The greatest complexities arise exactly at boundaries. . . .The lesson of

boundaries is hard even for systems thinkers to get. There is no single,

legitimate boundary to draw around a system.”—Donella H. Meadows, Thinking in Systems, pp. 95, 97

Image Source: http://en.wikipedia.org/wiki/List_of_tectonic_plates

Limits to Growth (1972)

• Non-linear interactions

between systems

• Exponential depletion:

y = (1/r) x ln ((r x s) + 1)

where:y = years left;

r = continuous growth rate

s = static reserve

R = reserve;

C = (annual) consumption

26th Annual ASQ Audit Division Conference:

The Intercontinental Hotel, Addison, TexasOctober 12 - 13, 2017

Page 23: Risk-based Continuous Systems of Systems Data Integrity ...rube.asq.org/audit/2017/10/auditing/rick-based-continuous-systems-… · Risk-based Continuous Systems of Systems Data Integrity

What is re-factoring?

Code refactoring is a “disciplined

technique for restructuring an existing

body of code, altering its internal

structure without changing its external

behavior,” undertaken in order to

improve some of the nonfunctional

attributes of the software.Source: http://en.wikipedia.org/wiki/Code_refactoring

26th Annual ASQ Audit Division Conference:

The Intercontinental Hotel, Addison, TexasOctober 12 - 13, 2017

Page 24: Risk-based Continuous Systems of Systems Data Integrity ...rube.asq.org/audit/2017/10/auditing/rick-based-continuous-systems-… · Risk-based Continuous Systems of Systems Data Integrity

What is Regulatory Debt?/1

“Decisions made to defer necessary

risk management and control

throughout a software development

lifecycle may result in regulatory

debt.”

26th Annual ASQ Audit Division Conference:

The Intercontinental Hotel, Addison, TexasOctober 12 - 13, 2017

Page 25: Risk-based Continuous Systems of Systems Data Integrity ...rube.asq.org/audit/2017/10/auditing/rick-based-continuous-systems-… · Risk-based Continuous Systems of Systems Data Integrity

• Technical Debt Refactor

• Regulatory Debt Risk Control

(esp. Refactor into “System of Systems”)

What is Regulatory Debt?/2

How to Pay Down the Debt:

“The computing scientist’s main challenge is not to get

confused by the complexities of his own making.”

—E. W. Dijkstra

26th Annual ASQ Audit Division Conference:

The Intercontinental Hotel, Addison, TexasOctober 12 - 13, 2017

Page 26: Risk-based Continuous Systems of Systems Data Integrity ...rube.asq.org/audit/2017/10/auditing/rick-based-continuous-systems-… · Risk-based Continuous Systems of Systems Data Integrity

ISO 9001:2015—Control of Information

26th Annual ASQ Audit Division Conference:

The Intercontinental Hotel, Addison, TexasOctober 12 - 13, 2017

7.5.3 Control of documented information

7.5.3.1 Documented information required by the quality management

system and by this International Standard shall be controlled to ensure:

a) it is available and suitable for use, where and when it is needed;

b) it is adequately protected (e.g. from loss of confidentiality, improper

use, or loss of integrity).

7.5.3.2 For the control of documented information, the organization shall

address the following activities, as applicable:

a) distribution, access, retrieval and use;

b) storage and preservation, including preservation of legibility;

c) control of changes (e.g. version control);

d) retention and disposition.

Page 27: Risk-based Continuous Systems of Systems Data Integrity ...rube.asq.org/audit/2017/10/auditing/rick-based-continuous-systems-… · Risk-based Continuous Systems of Systems Data Integrity

21 CFR 11 Compliance

26th Annual ASQ Audit Division Conference:

The Intercontinental Hotel, Addison, TexasOctober 12 - 13, 2017

Electronic Records

Any digital information that a computer system can

create, modify, maintain, archive, retrieved or

distribute

Key Elements

• Who created the electronic record?

• When was the record created?

• Do audit trails track changes (and deletions)

throughout the file retention time?

• Can only authorized personnel and entities use

the system?

Page 28: Risk-based Continuous Systems of Systems Data Integrity ...rube.asq.org/audit/2017/10/auditing/rick-based-continuous-systems-… · Risk-based Continuous Systems of Systems Data Integrity

What is data integrity?

26th Annual ASQ Audit Division Conference:

The Intercontinental Hotel, Addison, TexasOctober 12 - 13, 2017

ALCOA

• Attributable -- the originator of the data authorized /

authenticated

• Legible -- non-reputable traceable audit trail can be

reproduced / viewed (by data consumer)

• Contemporaneous -- data time stamped when

recorded

• Original record -- "true copy" preserves integrity (and

meaning!) of record (including subsequent activity--

cf. WHO)

• Accurate -- data are correct, truthful, valid and

reliable

Page 29: Risk-based Continuous Systems of Systems Data Integrity ...rube.asq.org/audit/2017/10/auditing/rick-based-continuous-systems-… · Risk-based Continuous Systems of Systems Data Integrity

Blockchains

October 12 - 13, 201726th Annual ASQ Audit Division Conference: The Intercontinental Hotel, Addison, Texas

Source: https://en.wikipedia.org/wiki/Blockchain

“Blockchains are secure by design and are an example of a distributed computing

system with high Byzantine fault tolerance.”

Image Source: https://bitcoin.org/en/

Page 30: Risk-based Continuous Systems of Systems Data Integrity ...rube.asq.org/audit/2017/10/auditing/rick-based-continuous-systems-… · Risk-based Continuous Systems of Systems Data Integrity

Questions?

October 12 - 13, 201726th Annual ASQ Audit Division Conference: The Intercontinental Hotel, Addison, Texas

Thank You!