risk and uncertainty...risk and uncertainty stephen n. luko united technologies aerospace systems...

RISK and Uncertainty

Stephen N. Luko United Technologies Aerospace Systems

Hartford section ASQ, Short Course, October, 2017

1

S. Luko, ASQ presentation, Oct. 2017

RISK and Uncertainty

Stephen N. Luko is an industrial Statistician with 38 years of experience currently employed by United Technologies Aerospace Systems where he is a technical Fellow in Risk, Reliability and Statistics. He is a Fellow of ASQ and a certified Quality and Reliability Engineer through ASQ. He is also Fellow of ASTM International and is past national chair of committee E11 on Quality and Statistics. Steve has taught college level courses in Mathematics and Statistics for 29 years, has numerous publications in the technical literature, and has given many industrial short courses on topics related to Quality, Reliability, Risk and Statistics. In 2009 Steve won the Shainin Medal from ASQ for work he had done on a certain type of sampling issue where multiple streams are involved; in 2012 he received the Harold F. Dodge award from ASTM for leadership achievements and standards development on committee E11.

Instructor brief bio: Stephen N. Luko

2


Today

This short course presents and summarizes some of the essential characteristics of “Uncertainty” and “Risk”; and shows where and how these terms apply in real world applications. We also consider examples, standards and some aspects in managing Risk.

3


Process

We’ll keep this quite informal. Ask your questions whenever they occur to you and feel free to comment and share experiences. We’ll take a full hour for lunch and several short breaks. Don’t forget to network – we’re all here for the same reason.

4


Topics

5


1. Overview of Uncertainty and Risk

2. Illustrations along the way

3. Managing Risk

4. Measurement Uncertainty

5. Uncertainty Budget

6. Probabilistic risk analysis

7. Workshop – manage your risks

8. Standards

9. Conclusion

Begin with “Uncertainty”

A psychological state of mind that may be characterized in varying degrees by caution, indecision, anxiety, hesitation, delay, mistrust, skepticism, ambiguity, fear, trepidation, “wonder” and other emotions.

The opposite of or lack of certainty (absolute knowledge). Results are only “probable” at best, particularly in the short term.

The absence of, or a reduced state 0f knowledge regarding the outcomes of future (also past and present) states or events.

6



Applies in somewhat different ways to a wide variety of differing fields including: physics and engineering, economics, finance and investment, insurance, psychology, sociology and world events, philosophy, statistics and information science, medicine, history and general “news”.

Uses: In general, statistics and uncertainty concepts are used more and more in real world applications.

The relation to Quality should be obvious: “Uncertainty is a always a debit to quality” (S. Luko)

7


Begin with “Uncertainty” 8



But generally uncertainty relates to statements about:

1. Future events or states given present evidence

2. Past events or states give current historical data 3. Present events or states given present available evidence.

9


Uncertainty – Examples of Use

Heisenberg uncertainty principle (Physics)

Measurement uncertainty (statistics, metrology); The “propagation of uncertainty”; Uncertainty Budget

About product performance – Quality.

About natural events (e.g. weather/geological related)

About what we hear, read about or what is reported

Uncertainty about past events

Uncertainty in Markets - economics

Uncertainty in medicine (epidemiology)

“We live in uncertain times”

“Moral Uncertainty”

10


Measurement Uncertainty

Let X be the value of something measured. Let T be the true (idealized and unrealized value of the thing measured).

We are always allowed to ask: “How far apart might X be from T?”

This is the “delta” between reality and the actual practice or observation.

Note that the value of T is generally unknown; but “delta” might be studied under some assumed conditions of uncertainty.

11


Type A or B Measurement Uncertainty

Type A uncertainty is statistical, meaning methods that employ probability and statistical principles directly with data/measurements.

Type B uncertainty involves anything else, a-priori; such as assigning a probability distribution to a variable or by the use of “expert” advice in a problem; or by knowledge of historical trends in some way.

12


Uncertainty “Budget”

Some companies measure their uncertainty using an “Uncertainty Budget”

This is a formal way to quantify uncertainties in your process/product measurements – originally developed by NIST.

The “budget” allocates type A and B uncertainties in columnar format and uses the coverage factor and associated confidence (we’re being really brief on this) . In the end the uncertainty in a reported measurement is of the form y±uC where u is a global uncertainty expressed at some confidence C.

13


Uncertainty “Budget”, example template

14


Type A (Statistical) Uncertainty

Gage R&R is the typical type A: “R”=repeatability. “R”=Reproducibility.

Bias “P&B”; replaced the older “accuracy”

Consistency and Stability

Linearity

Resolution

Other sources of variance

15


Statistical Uncertainty - Methods

Simple Gage R&R: short form; long form

Bias: delta between a sample average and a standard

Consistency: constancy of variance

Stability: measurement process in control

Linearity: change in bias over operational values

Resolution: smallest detectable degree of object value measured.

Other sources of variance: “gages”; materials; facilities or labs, test methods.

16



Some quarters use a term called “Intermediate Precision”: This term originated in ASTM.

intermediate precision, n—the closeness of agreement between test results obtained under specified intermediate precision conditions. E177

intermediate precision conditions, n—conditions under which test results are obtained with the same test method using test units or test specimens (see Practice E691, 10.3) taken at random from a single quantity of material that is as nearly homogeneous as possible, and with changing conditions such as operator, measuring equipment, location within the laboratory, and time. E177

E691 – ASTM standard concerning Interlaboratory studies.

17



Also … the terms repeatability conditions and reproducibility conditions are found within the ASTM E11 suite of standards.

repeatability conditions, n—conditions where independent test results are obtained with the same method on identical test items in the same laboratory by the same operator using the same equipment within short intervals of time. E177

reproducibility conditions, n—conditions where test results are obtained with the same method on identical test items in different laboratories with different operators using different equipment. E177

E177: Use of the Terms Precision and Bias in ASTM Test Methods

18



Resolution in a measurement system: two types.

Mechanical resolution – strictly a function of the hardware of the system.

Statistical resolution – includes effect due to mechanical resolution as well as other MSA components (gage R&R etc.)

Effective resolution – the actual resolution resulting from the assessment of mechanical and statistical effects.

19



SN ratio – if σt is the standard deviation of the true object and if σr is the measurement error standard deviation, then SN= σt/σr is the signal to noise ratio (popular among electrical engineers)

Discrimination ratio: “D” measures the number of “distinct product categories”.

Or approximately

20


1 + 2

= D2r

2t

r

t

σ

1.414σ = D

The formula measures or estimates the number of approximate 97% confidence intervals (based on σ𝑟) that just fit within the spread or distribution of pure part dimensions. A 97% confidence interval is an interval centered on a single measurement that would contain the actual (pure) dimension represented by that measurement, 97% of the time.


21


Table 3, EVALUATING THE S/N RATIO

D Analysis and Interpretation

1 100% noise

2 Go/No-go capability

3 Low-grade variable data; increases control chart errors.

4 Improved variable data

5 Good variable data

D=1 means that all parts give the same response measurement or reading. No actual part variability can be discerned. Consequently the gage or measurement system is of little use. High D ratios indicate a high precision gage or measurement system. Table 3 gives some guidelines in evaluating and interpreting the discrimination ratio.


Juran’s Rule of “10’s”… “The instrument should be able to divide the tolerance into about 10 parts…”,

22


Let y be a measurement, X the true (pure) part measure, and e the measurement error. The measurement y is expressed as: Y=X+e. X and e are independent; and each has a variance. Suppose the error standard deviation is some fraction of the standard deviation of measurements (y). Use the fraction 1/k. With efficiency defined as the ratio of the standard deviation of X to the standard deviation of Y, the formula on the left results.

%100/11% 2kEff

k %Eff

1 0.000

2 86.603

3 94.281

4 96.825

5 97.980

10 99.499

20 99.875

50 99.980

100 99.995

Type B – Uncertainty Examples

Assume the measurement error standard deviation or reliability is just some value.

Assume a conservative POD (probability of detection) for an inspection process

Assume 10% new accounts and a 2.5% loss of accounts.

Assume a distribution (uniform, normal, etc.) for a variable in a complex analysis.

Assume 30% of the units in the field have the issue.

Others

23


Standards Concerning Uncertainty

Evaluation of measurement data – Guide to the expression of uncertainty in measurement, JCGM 100:2008 (GUM 1995 with minor corrections); published by NIST – very technical.

ASTM E2655, Reporting Uncertainty of Test Results and Use of the Term Measurement Uncertainty in ASTM Test Methods

24



ASTM E2554, Practice for Estimating and Monitoring the Uncertainty of Test Results of a Test Method Using Control Chart Techniques.

ASTM E691, Practice for Conducting an Interlaboratory Study to Determine the Precision of a Test Method

ASTM E2655, Reporting Uncertainty of Test Results and Use of the Term Measurement Uncertainty in ASTM Test Methods

25



ASTM E2782, Standard Guide for Measurement Systems Analysis (MSA)

ASTM E2282, Standard Guide for Defining the Test Result of a Test Method

SAE AS13003, Measurement Systems Analysis Requirements for the Aero Engine Supply Chain

A2LA G104, Guide for Estimation of Measurement Uncertainty In Testing

26



A very popular manual among automotive suppliers: AIAG, MSA reference Manual (currently 4th ed.)

ASTM E2282, Standard Guide for Defining the Test Result of a Test Method

SAE AS13003, Measurement Systems Analysis Requirements for the Aero Engine Supply Chain

A2LA G104, Guide for Estimation of Measurement Uncertainty In Testing

Many others

27


Risk

The concept of “Risk” is closely associated with “Uncertainty”, Reliability and statistical thinking.

“Risk - The effect of uncertainty on objectives”, ISO Guide 73 on Risk Vocabulary.

“The potential for an unwanted outcome resulting from an incident, event, or occurrence, as determined by its likelihood and the associated consequences”, Department of Homeland Security, Risk Lexicon.

Also ANSI Z690.1-2011 – this standard is identical with the ISO version (Guide 73).

28


Risk

Risk in ISO 9001:2015 – “ …. the effect of uncertainty on an expected result.”

1. An effect is a deviation from the expected – positive or negative.

2. Risk is about what could happen and what the effect of this happening might be

3. Risk also considers how likely it is.

Risk based thinking – compare this to “statistical thinking”; compare the definition to ISO Guide 73.

29


Risk

“Risk is an expression of possible loss over a specific period of time or number of operational cycles. It may be indicated by the probability of an accident times the damage in dollars, lives, and / or operating units.” FAA Advisory Circular 39-8.

30


RISK in ISO Guide 73

“The effect of Uncertainty on Objectives”

An Objective is Generally what is desired or planned.

NOTE 1: An effect is a deviation from the expected— positive and/or negative.

NOTE 2: Objectives can have different aspects (such as financial, health and safety, and environmental goals) and can apply at different levels (such as strategic, organization-wide, project, product and process).

31



Three important components: 1) what can happen (events); 2) how often (probability, statistical); 3) with what consequence (loss/cost).

Some organizations emphasize one more than the other; others emphasize two together more than the remaining. Everything is context dependent.

“context” means the specific application in a specific industry, business or social arena.

32



Likelihood – Chance of something happening, ISO Guide 73.

NOTE 1: In risk management terminology, the word ‘‘likelihood’’ is used to refer to the chance of something happening, whether defined, measured or determined objectively or subjectively, qualitatively or quantitatively, and described using general terms or mathematically [such as a probability or a frequency over a given time period].

33


RISK as Related to Statistics

Statistics are just the numerical representatives of what has happened and how often. Recall type A uncertainty is data based.

Several key concepts: mean, rate, proportion, standard deviation and standard error.

34


RISK as Related to Statistics

“Expected Value” means - on the average, the mean value, the center, the “typical value”: These are used as estimates of what can happen - Generally estimated by ordinary statistics.

Standard deviation, “sigma”, variance, standard error, margin of error, range or spread, “scatter”, mean square error: All of these are measures of uncertainty in some way.

35


RISK Management

Section 3 of The ISO Guide 73, terminology document concerns the broad topic of the risk management process and makes up the bulk of the terms in this standard.

ISO 31000-2009, Risk Management – Principles and Guidelines. Also, ANSI Z690.2-2011. This document and the ISO document are identical.

36


RISK Management

Framework – section 4

Mandate and Commitment

Risk Management Policy

Accountability

Integration into organizational processes

Resources

Internal and External Communication

Implementation

Monitoring, Review, Improvement

37


RISK Management

Process – section 5

General Policy or basic organizational directives – highest level of management

Establishing the context – How is “risk” defined in your organization? What is considered a risk? What are the levels of Risk?

Risk identification – how do we know when we have a risk situation in our organization?

38


RISK Management

Process – section 5

Risk assessment, analysis and evaluation – this is the quantification of risk; the very specifics of the situation being worked.

Risk Treatment and plans for – what to we do about it? Can it be mitigated?

Communication – who needs to know about it?

Ongoing monitoring and review – lets watch it carefully.

Recording (write it down: documentation)

39


RISK Management

Essentially, risk assessment tries to answer the following: 1) What can happen? How often (probability); With what consequences (cost/loss); Why did it happen (root cause)?; How is it treated operationally?

Additionally, what are the factors that mitigate the consequences of the risk or reduce the probability of future occurrence and/or losses.

“Assessment” is the broader management concept; “analysis” refers to the detailed tools of mathematics, engineering, science and data.

40


RISK Assessment Techniques

ISO 31010-2009 or ANSI Z690.3-2011. These documents are identical.

The longest of the 3 companion standards on risk.

Catalogs of techniques

Compares techniques in differing situations

41



Scope, References and Terminology sections

Three short sections on: assessment concepts, assessment process, and selection of Techniques

The longest section is the Annexes, which is a catalog on Risk Assessment Techniques and comparative tables for various scenarios. There are two of these

42



Annex A contains several tables that classify techniques by several categories – applicability (not at all, applicable, strongly applicable)

In addition, types of assessment techniques are classified as to their relevance (to resources and capability, to uncertainty and to complexity)

Illustration – the standards

43


RISK and SAFETY

Aerospace and transportation

Medical practice

Homeland security/Social context

Financial markets and personal finance

Business survival/loss

Natural events

Other

44


RISK Example as Used in Aerospace

How a standard risk assessment/analysis is done. (manufacturing/business context)

A risk issue is identified – could be from field experience, from manufacturing/quality records, from engineering/expert considerations, other.

A target population is identified – also called “population at risk”.

Obtain data from the target population – this can take many forms but often it is time to failure/event or proportion for the identified failure mode.

45



Note - target population is really a sample of objects that might have been available in theory.

Estimate the distribution of failure time using a statistical technique – foremost of which is a probability plot (examples next slide).

For example there may be other (many) aspects that vary randomly in some way and that need to be characterized/studied.

Distributions that might be used: Normal, lognormal, Weibull, exponential, Extreme value, gamma, beta (continuous). Binomial, Poisson, geometric, hypergeometric (discrete)…. Others.

46



47



The estimated distribution becomes the “model” that is used for the risk prediction.

Many other examples of this use: Motor vehicle or aerospace systems or components; more generally, product life (such as for warrantee purposes).

Product reliability data is often used or is similar.

Use the model to predict what should have happened right now given the state of the current target population and the model as we have estimated it.

48



If F(t) is the estimated cumulative distribution function (cdf) then the prediction of the NOW risk is estimated in theory using all n data items (failures and suspensions) as :

This is usually modified for bias and implemented as:

49



If the NOW risk is seriously different from the observed events, then either “calibrate” the model by adjusting a parameter (usually the mean or median), or rethink the model; also, examine the data.

If calibrated, use the model to estimate future events that are expected to occur among the fielded (unfailed) assets.

Have to consider utilization (monthly hrs. of operation) and redundancy (units per Aircraft).

Note: if no events are observed we usually calibrate to 1 event, NOW.

50



The resulting forecast is called a “do nothing risk”.

This is an expected number of “base” events (and perhaps a measure of the uncertainty of the estimate) that can be used in a number of ways to portray risk (e.g. an event rate, events per month etc.)

Hazard ratios: Factor(s) applied to the final risk result that takes you from the base event to the next level of severity (i.e. conditional probability). For example, not all fuel leaks lead to a on board fire; and not all fires lead to major events or loss of life.

51



52


EXAMPLE: Using a population at risk of 297 assets (units fielded) with various times accrued. Using a population model derived from data and assumptions. And resulting in a forecast (at left for 2 years).


Risk mitigation: When “do nothing risk” is high, or above some “threshold” requirement, some action has to be taken.

Ground or recall, inspect and repair/replace at some interval, repair/replace/retrofit according to a plan; burn in, scrap, etc.

53



Caveats

Sometimes we assume some aspect of the model, particularly when zero events have occurred.

Sometimes we have to estimate fleet times (failures and/or suspensions); fleet size.

There might be fleet growth, WIP and inventory to consider.

Type of failure mode is important (random, wear out, infant mortality – the “bathtub curve”)

We might use engineering judgment on some aspect of the modeling.

54



Non standard risk assessments

Complex scenarios often involving: combination of continuous and discrete (attribute) distributions; multivariate factors; Monte Carlo simulation; Bayesian methods; Complex systems analysis; complex sampling scenarios.

Demand: Something is needed but only periodically or intermittently, randomly and often rarely. But when it is needed it is really needed. For example a fire extinguisher, an emergency brake.

55


RISK and Quality

Generally, higher risks mean a debit to long term quality and integrity.

Management needs to look at “risks” in their organizations – customer retention, customer perception, product performance and reliability all carry risk of loss or degradation.

More generally, “Risk” and “uncertainty” are relatively (new) attributes of overall Quality that need management attention.

56


Thoughts and reflections – RISK mitigation

Good planning and preparedness

Monitoring and review

Redundancy in its many forms

Design margin (a form of redundancy)

Quality tools such as check sheets and other mistake proofing methods.

Appropriate training

Understanding uncertainty

Having Good data!

57


Thoughts and reflections – RISK enablers

Be careful with Lean (many manifestations)

Be careful with Multitasking (many manifestations)

Always consider margin

Poor or too little data; wrong analysis

Too much emphasis on the expert.

“Bad” assumptions

“Too little too late”.

Poor communications

58


Bibliography

ANSI/ASSE Z690.1–2011. (2011). Vocabulary for Risk Management; Washington, D.C.: American National Standards Institute.

ANSI/ASSE Z690.2–2011. (2011). Risk Management Principles and Guidelines. Washington, D.C.: American National Standards Institute.

ANSI/ASSE Z690.3–2011. (2011). Risk Assessment Techniques; Washington, D.C.: American National Standards Institute.

Federal Aviation Administration. (2003). Advisory Circular 39–8, Washington, D.C.: Federal Aviation Administration.

ISO Guide 73. (2009), Risk Management Terminology, Geneva, Switzerland: International Organization for Standardization (ISO).

ISO 31000-2009, Risk Management – Principles and Guidelines, Geneva, Switzerland: International Organization for Standardization (ISO).

U.S. Department of Homeland Security (2010), DHS Risk Lexicon, Washington, D.C.: U.S. Department of Homeland Security.

Luko, Stephen N., (2013), Risk Management Terminology, Quality Engineering, 25:3, 292-297

Luko, Stephen N., (2013), Risk Management Principles and Guidelines, Quality Engineering, 25:450–453, 2013

59


RISK & Uncertainty

Completion of Introduction

Q&A

60


risk and uncertainty...risk and uncertainty stephen n. luko united technologies aerospace systems...

Documents