adam evans & kristian cruickshank, nova systems - developing uav safety cases

43
http://www.novasystems.com.au Experience Knowledge Independence Developing UAV Safety Cases UAV Triple Zero Summit Mr Adam Evans Mr Kristian Cruickshank 1

Upload: informa-australia

Post on 23-Aug-2014

223 views

Category:

Law


11 download

DESCRIPTION

Adam Evans & Kristian Cruickshank, Nova Systems delivered the presentation at the 2014 UAV Triple Zero Summit. The 2014 UAV Triple Zero Summit had a theme and focus on ‘Mobilising and Regulating UAVs in Australian Emergency Response’. It drew on government policy, current legislation and privacy protocol in establishing an informed analysis of the current and future scope surrounding the utilization of unmanned systems in this sector. For more information about the event, please visit: http://www.informa.com.au/uavtriplezero14

TRANSCRIPT

Page 1: Adam Evans & Kristian Cruickshank, Nova Systems - Developing UAV safety cases

http://www.novasystems.com.au Experience Knowledge Independence

Developing UAV Safety Cases

UAV Triple Zero Summit

Mr Adam Evans

Mr Kristian Cruickshank

1

Page 2: Adam Evans & Kristian Cruickshank, Nova Systems - Developing UAV safety cases

http://www.novasystems.com.au Experience Knowledge Independence

Overview

2

Nova’s Background

What is a Safety Case?

When is a Safety Case Required?

UAV Safety Case Paradigm

Safety Case Process

Levels of Acceptable Risk

Emergency Services Risk Context

UAV Operation Risk Analysis

Treating Unacceptable Risk

UAV Safety Management Systems

Consolidating the Safety Case

Page 3: Adam Evans & Kristian Cruickshank, Nova Systems - Developing UAV safety cases

http://www.novasystems.com.au Experience Knowledge Independence

Terminology

UAV vs UAS vs RPA vs RP vs RPAS

Throughout this presentation:

UAV = RPA

UAS = RPAS

UAV Controller = RP

3

Page 4: Adam Evans & Kristian Cruickshank, Nova Systems - Developing UAV safety cases

http://www.novasystems.com.au Experience Knowledge Independence

Origins in Defence T&E

Involved in all significant ADF UAS projects to date:

Heron

Shadow 200

Aerial Targets

Nova contracted by ADF to develop UAV regulatory framework

Specialists in Military and Civil Airworthiness, inclusive of operational and technical risk management

Aeronautical Engineers Australia specialists in Civil Airworthiness and CASRs

Practitioners in various aerospace engineering and operational domains

Nova’s Background

4

Page 5: Adam Evans & Kristian Cruickshank, Nova Systems - Developing UAV safety cases

http://www.novasystems.com.au Experience Knowledge Independence

What is a Safety Case?

Broad Definition:

A structured argument of compiled evidence demonstrating that a system is acceptably safe

No CASA definition for UAV Safety Case

CASA Airworthiness Circular for Aerodromes:

“A documented body of evidence that provides a demonstrated and valid argument that a system is adequately safe for a given application and environment over its lifetime” (AC 139-16(1))

Propose that the definition used in AC 139-16(1) is suitable for UAVs

5

Page 6: Adam Evans & Kristian Cruickshank, Nova Systems - Developing UAV safety cases

http://www.novasystems.com.au Experience Knowledge Independence

What is a Safety Case?

Elements of a UAV Safety Case Adequate Level of Safety. Benchmark is ‘acceptable’* level of risk posed to the general public.

Given Application and Environment. Safety case must define the types of UAV operations and the environmental factors present in those operations

Statement of Operating Intent (SOI) or Concept of Operations (CONOPS) or equivalent

Key environmental factors are – population densities, physical environment, airspace category.

Lifetime. UAV context may lessen the importance of this element – possibly more ‘disposable’ than most aircraft? Still requires consideration.

* ‘Acceptable’ may vary depending on a given emergency services scenario

6

Page 7: Adam Evans & Kristian Cruickshank, Nova Systems - Developing UAV safety cases

http://www.novasystems.com.au Experience Knowledge Independence

What is a Safety Case?

Elements of a UAV Safety Case (cont)

System. Unmanned Aerial System plus the Safety Management System or equivalent implemented.

Demonstrated Argument. Logical, valid, and defensible argument constructed from applicable body of evidence.

No specific CASA guidance on what the argument must consider

Experience with Military UAS provides a reasonable basis for considerations

7

Page 8: Adam Evans & Kristian Cruickshank, Nova Systems - Developing UAV safety cases

http://www.novasystems.com.au Experience Knowledge Independence

When is a Safety Case Required?

Implied by NPRM1309OS (regulations and guidance not published yet)

Intent of once-off Area Approval is the same as a safety case, but safety case can be enduring

Operation of Large UAV (> 150kg)

Operating outside of Standard Operating Conditions

Over Populous Areas

Beyond Visual Line Of Sight

Greater than 400ft

Other than Class G airspace

Closer than 3NM from aerodrome

8

Page 9: Adam Evans & Kristian Cruickshank, Nova Systems - Developing UAV safety cases

http://www.novasystems.com.au Experience Knowledge Independence

When is a Safety Case Required?

Put Simply:

UAV OPERATIONS THAT WOULD BE OF MOST BENEFIT TO EMERGENCY SERVICES!

9

Page 10: Adam Evans & Kristian Cruickshank, Nova Systems - Developing UAV safety cases

http://www.novasystems.com.au Experience Knowledge Independence

Likely Scenarios for Safety Case

Search and Rescue

BVLOS, Over Populous Areas, Above 400ft

Fire Spotting

Restricted Airspace?

Police Tactical Operations

BVLOS, Over Populous Areas, Controlled Airspace

Natural Disasters

BVLOS, Over Populous Areas, Above 400ft, Launching from Aerodromes

Others?

10

Page 11: Adam Evans & Kristian Cruickshank, Nova Systems - Developing UAV safety cases

http://www.novasystems.com.au Experience Knowledge Independence

UAV Safety Case Paradigm

Different approach than regular aircraft – Why?

Aircraft Type Certification and Operational Management Regulations established and industry complies

UAV origins – Hobby and Military

No internationally recognised Type Certification Requirements established

‘Risk Management Approach’ instead of a ‘Compliance to Standards’ approach

11

Page 12: Adam Evans & Kristian Cruickshank, Nova Systems - Developing UAV safety cases

http://www.novasystems.com.au Experience Knowledge Independence

UAV Safety Case Paradigm

The Future Safety Paradigm – Establishing Compliance with Technical Airworthiness Requirements

Confidence in Integrity of System Design

Confidence in Quality of Manufacture

Design of Maintenance Schemes that maintain aircraft reliability

Same process as normal Aircraft

Challenges with ‘The Future’ Cost

Establishing requirements for different UAV categories (Small, Medium, Large, Commuter?)

Detect and Avoid + more

12

Page 13: Adam Evans & Kristian Cruickshank, Nova Systems - Developing UAV safety cases

http://www.novasystems.com.au Experience Knowledge Independence

UAV Safety Case Paradigm

The Current Safety Paradigm – Technical and Operational Risk Management

Defining Acceptable Levels of Risk to Public

Determine worst credible Consequence of UAV accident

Determine Probability of worst credible Consequence occurring

Reliability of UAV (hardware reliability combined with integrity of software) – if possible to determine

Probability of fatality/injury given impact

Population density + more

Technical and Operational risk treatments

Plus ‘normal’ aircraft requirements (maintenance, flight operations system, Safety Management System, etc)

13

Page 14: Adam Evans & Kristian Cruickshank, Nova Systems - Developing UAV safety cases

http://www.novasystems.com.au Experience Knowledge Independence

UAV Safety Case Process

14

Develop SOI / CONOPS

Define Acceptable

Levels of Risk

System Safety Assessment

Compare Risk to Acceptable

Levels

Risk Acceptable

?

Develop Risk Mitigations

Consolidate Safety Case

Operational, maintenance,

design, SOI change, etc

Yes

No

Evidence

SOI, Acceptable Risk, UAV design,

Maintenance System, Safety Management System, Operators

Manual, OEM Documentation

Page 15: Adam Evans & Kristian Cruickshank, Nova Systems - Developing UAV safety cases

http://www.novasystems.com.au Experience Knowledge Independence

Statement of Operating Intent

Analogous to Concept of Operations

Derived from Military Context

Defines types of operations and informs risk assessment process

15

Page 16: Adam Evans & Kristian Cruickshank, Nova Systems - Developing UAV safety cases

http://www.novasystems.com.au Experience Knowledge Independence

Statement of Operating Intent

Key Aspects

Role. Function(s) or purpose(s) assigned to system – SAR, Fire Spotting, Surveillance, etc.

Tasks are a sub-element of Role. Tasks to be conducted under a given role.

Environment. Totality of surroundings/conditions of operations (airspace, areas of operation, physical environment, etc)

Flight Envelope. Defines outermost boundary of flight conditions for UAV to remain airworthy.

Flight Usage Spectrum. Flight Profiles for each task/role, frequency of profiles, Rate of Effort, etc.

16

Page 17: Adam Evans & Kristian Cruickshank, Nova Systems - Developing UAV safety cases

http://www.novasystems.com.au Experience Knowledge Independence

Safety Case Process

17

Develop SOI / CONOPS

Define Acceptable

Levels of Risk

System Safety Assessment

Compare Risk to Acceptable

Levels

Risk Acceptable

?

Develop Risk Mitigations

Consolidate Safety Case

Operational, maintenance,

design, SOI change, etc

Yes

No

Evidence

SOI, Acceptable Risk, UAV design,

Maintenance System, Safety Management System, Operators

Manual, OEM Documentation

Page 18: Adam Evans & Kristian Cruickshank, Nova Systems - Developing UAV safety cases

http://www.novasystems.com.au Experience Knowledge Independence

Safety Targets – Example Maximum acceptable Individual probability of fatality or serious injury to the General Public: 1 X10-7 per flight hour

Maximum acceptable Collective fatality expectation to the General Public: 1000 X10-6 (1x10-3) per annum OR 5x10-7 per flight hour

Maximum acceptable Individual probability of fatality or serious injury to the Mission Personnel: 1 X10-6 per flight hour

Maximum acceptable Collective fatality expectation to the Mission Personnel: 10000 X10-6 (1 x10-2) per annum OR 1 X10-5 per flight hour

18

Page 19: Adam Evans & Kristian Cruickshank, Nova Systems - Developing UAV safety cases

http://www.novasystems.com.au Experience Knowledge Independence

Defining Levels of Acceptable Risk

What ‘level of safety’, integrity or reliability do we need to operate a 20kg UAV in an sparsely populated rural environment?

What if the operation is attempting to prevent an assault?

What if the operation is attempting to prevent a homicide?

What if the operation is attempting to prevent multiple homicides

What if the aircraft has sufficient range to fly into densely populated area?

19

Page 20: Adam Evans & Kristian Cruickshank, Nova Systems - Developing UAV safety cases

http://www.novasystems.com.au Experience Knowledge Independence

Emergency Service Risk Context

May be quite simple to balance risk

When exposing the public to risks, the basis for determining the risk as acceptable must be able to stand up to public scrutiny

20

Public risk benefit

from UAV operation

Public risk exposure

without UAV operation

Page 21: Adam Evans & Kristian Cruickshank, Nova Systems - Developing UAV safety cases

http://www.novasystems.com.au Experience Knowledge Independence

Example safety target

Homicide Assault

Probable 1x10-3 1x10-4

Likely 1x10-4 1x10-5

Unlikely 1x10-5 1x10-7

Rare 1x10-7 1x10-9

Page 22: Adam Evans & Kristian Cruickshank, Nova Systems - Developing UAV safety cases

http://www.novasystems.com.au Experience Knowledge Independence

Explanation of Table

UAV operations to prevent a Homicide are reasonable if the risk to the general public is less than 1x10-3 and it is determined that the assailant will Probably commit the crime.

UAV operations to prevent 5 Assaults are reasonable if the risk to the general public is less than 5x10-7 and it is determined that the assailant could, but is Unlikely commit the crime.

22

Page 23: Adam Evans & Kristian Cruickshank, Nova Systems - Developing UAV safety cases

http://www.novasystems.com.au Experience Knowledge Independence

Safety Case Process

23

Develop SOI / CONOPS

Define Acceptable

Levels of Risk

System Safety Assessment

Compare Risk to Acceptable

Levels

Risk Acceptable

?

Develop Risk Mitigations

Consolidate Safety Case

Operational, maintenance,

design, SOI change, etc

Yes

No

Evidence

SOI, Acceptable Risk, UAV design,

Maintenance System, Safety Management System, Operators

Manual, OEM Documentation

Page 24: Adam Evans & Kristian Cruickshank, Nova Systems - Developing UAV safety cases

http://www.novasystems.com.au Experience Knowledge Independence

24

System Safety Assessment

Unrecoverable Failure Rate

Unrecoverable Failure Rate unknown? Fault tree analysis to identify safety critical systems

Engines/Navigation/Airframe/Autopilot/etc

Various techniques to assess overall reliability / integrity of design

Consider existing Operational Mechanisms

Use Casualty/Fatality Expectation Rate Analysis to quantify risks to personnel

End Product is Unmitigated Risk

Software reliability?

Page 25: Adam Evans & Kristian Cruickshank, Nova Systems - Developing UAV safety cases

http://www.novasystems.com.au Experience Knowledge Independence

FTA for Military UAS operations – Air Vehicle Escape

25

Page 26: Adam Evans & Kristian Cruickshank, Nova Systems - Developing UAV safety cases

http://www.novasystems.com.au Experience Knowledge Independence

Casualty Expectation Methodologies

Once Unrecoverable Failure Rates for the vehicle are known

Used to determine Collective and Individual Risks to General Public and Mission Essential Personnel

Based on population densities and Lethal Area of Vehicle

26

Page 27: Adam Evans & Kristian Cruickshank, Nova Systems - Developing UAV safety cases

http://www.novasystems.com.au Experience Knowledge Independence

Casualty Expectation

CE = λ x PCasualty|Strike x PStrike|Impact x PImpact

CE – Casualty Expectation (collective risk)

λ – Reliability

PImpact – Probability of high energy crash given a failure

PStrike|Impact - Probability of striking an individual

PCasualty|Strike - Probability of killing someone

27

Page 28: Adam Evans & Kristian Cruickshank, Nova Systems - Developing UAV safety cases

http://www.novasystems.com.au Experience Knowledge Independence

Casualty Expectation

Pimpact

Difficult to determine (e.g. reliability of FTS)

Substantial computational resources

Integrate over all possible crash locations

Pstrike/impact

Exposure time

Population density

Lethal Area of Vehicle

28

Page 29: Adam Evans & Kristian Cruickshank, Nova Systems - Developing UAV safety cases

http://www.novasystems.com.au Experience Knowledge Independence

Casualty Expectation

Pcasualty/strike

Depends on debris KE and explosive energy in the Air Vehicle

Requires analysis of various materials in Air Vehicle

CASA paper assists

Page 30: Adam Evans & Kristian Cruickshank, Nova Systems - Developing UAV safety cases

http://www.novasystems.com.au Experience Knowledge Independence

Risk Comparison

30

Develop SOI / CONOPS

Define Acceptable

Levels of Risk

System Safety Assessment

Compare Risk to Acceptable

Levels

Risk Acceptable

?

Develop Risk Mitigations

Consolidate Safety Case

Operational, maintenance,

design, SOI change, etc

Yes

No

Evidence

SOI, Acceptable Risk, UAV design,

Maintenance System, Safety Management System, Operators

Manual, OEM Documentation

Page 31: Adam Evans & Kristian Cruickshank, Nova Systems - Developing UAV safety cases

http://www.novasystems.com.au Experience Knowledge Independence

Risk Comparison

CE = λ x PCasualty|Strike x PStrike|Impact x Pimpact

31

Public risk benefit

from UAV operation

Public risk exposure

without UAV operation

Prevent possible homicide (1x10-4)

Page 32: Adam Evans & Kristian Cruickshank, Nova Systems - Developing UAV safety cases

http://www.novasystems.com.au Experience Knowledge Independence

Risk Mitigation

32

Develop SOI / CONOPS

Define Acceptable

Levels of Risk

System Safety Assessment

Compare Risk to Acceptable

Levels

Risk Acceptable

?

Develop Risk Mitigations

Consolidate Safety Case

Operational, maintenance,

design, SOI change, etc

Yes

No

Evidence

SOI, Acceptable Risk, UAV design,

Maintenance System, Safety Management System, Operators

Manual, OEM Documentation

Page 33: Adam Evans & Kristian Cruickshank, Nova Systems - Developing UAV safety cases

http://www.novasystems.com.au Experience Knowledge Independence

Risk Mitigation

Develop Risk Treatments if Acceptable threshold exceeded

Operational Treatments:

Restrict range of UAV

Extended VLOS

Operations only up to (X) population density

Etc

Technical Treatments:

Different UAV

OEM redesign (datalink reliability)

33

Page 34: Adam Evans & Kristian Cruickshank, Nova Systems - Developing UAV safety cases

http://www.novasystems.com.au Experience Knowledge Independence

Minimising Risk (below acceptable?)

Emergency services have a duty of care to minimise the risk to the public

Further work could be done in order to identify risk levels at the front line

Flight plans could be optimised to reduce risk to public

Aircraft type or configuration selected to reduce risk

Possibility for assumed clearance for flight if specific criteria is satisfied (as specified by Operations Manual/Safety Management System)

34

Page 35: Adam Evans & Kristian Cruickshank, Nova Systems - Developing UAV safety cases

http://www.novasystems.com.au Experience Knowledge Independence

UAV Safety Management Systems

Likely that specific risk mitigating processes or techniques will need to be enacted during operations

If there is an ongoing need to identify and treat risks, or an ongoing Operational Risk Management process – this will form a large portion of the UAV Safety Management System

Best place for these to be documented and enforced (forming part of the safety case) will be in a Safety Management System

35

Page 36: Adam Evans & Kristian Cruickshank, Nova Systems - Developing UAV safety cases

http://www.novasystems.com.au Experience Knowledge Independence

UAV Safety Management System

Ongoing Risk Assessments may include:

Mission Planning Processes and Tools

Onsite Risk Assessments

Particularly relevant to Emergency Services Risk Context

Risk Assessment and Treatment on repairs or maintenance

More?

36

Page 37: Adam Evans & Kristian Cruickshank, Nova Systems - Developing UAV safety cases

http://www.novasystems.com.au Experience Knowledge Independence

UAV Safety Management Systems

No specific requirement in CASR 101 (or NPRM1309OS)

However, intent of identifying and managing safety risks associated with UAVs is applicable

SMS Gap Analysis Tool provided by CASA

Operations Manual and other corporate plans/procedures may be sufficient without a dedicated SMS for most operations

Likely that most Operations Manuals would include these considerations, but a dedicated SMS may be advisable for large UAVs and non-standard operating conditions

37

Page 38: Adam Evans & Kristian Cruickshank, Nova Systems - Developing UAV safety cases

http://www.novasystems.com.au Experience Knowledge Independence

Consolidating the Safety Case

38

Develop SOI / CONOPS

Define Acceptable

Levels of Risk

System Safety Assessment

Compare Risk to Acceptable

Levels

Risk Acceptable

?

Develop Risk Mitigations

Consolidate Safety Case

Operational, maintenance,

design, SOI change, etc

Yes

No

Evidence

SOI, Acceptable Risk, UAV design,

Maintenance System, Safety Management System, Operators

Manual, OEM Documentation

Page 39: Adam Evans & Kristian Cruickshank, Nova Systems - Developing UAV safety cases

http://www.novasystems.com.au Experience Knowledge Independence

Consolidating the Safety Case

Don’t make it too scenario dependent

What if you haven’t thought of all scenarios?

Flexibility where operational functions and risks remain valid

Structured argument

Outline the process

SOI/CONOPS

Justify Risk Acceptability (where did safety targets come from?)

Describe Risk Mitigations (where necessary, show that they’ve been incorporated into design/operations)

39

Page 40: Adam Evans & Kristian Cruickshank, Nova Systems - Developing UAV safety cases

http://www.novasystems.com.au Experience Knowledge Independence

Consolidating the Safety Case

Various techniques for ‘Structuring’ argument (Goal Structured Notation is a good method)

40

Safe UAV Operation

SO

I/CO

NO

PS

Safety Target(s)

System Reliability

/ Integrity

System Limitations

Op Risk Manage

Ops/Maint Processes

System Safety

Assess

Page 41: Adam Evans & Kristian Cruickshank, Nova Systems - Developing UAV safety cases

http://www.novasystems.com.au Experience Knowledge Independence

Consolidating the Safety Case

If residual risk is unacceptable – Talk to CASA

In some cases the risk may simply be too high

Back to the drawing board

The UAV you intended to use may not be the answer

UAVs may not be the answer

Finally – Submit the Safety Case!

Hopefully this is not the first time CASA has seen it...

Get them involved from the start for planned ‘high’ risk operations

41

Page 42: Adam Evans & Kristian Cruickshank, Nova Systems - Developing UAV safety cases

http://www.novasystems.com.au Experience Knowledge Independence

Questions?

42

Page 43: Adam Evans & Kristian Cruickshank, Nova Systems - Developing UAV safety cases

http://www.novasystems.com.au Experience Knowledge Independence

Developing UAV Safety Cases

UAV Triple Zero Summit

Mr Adam Evans http://www.linkedin.com/pub/adam-evans/12/850/b73

Mr Kristian Cruickshank http://www.linkedin.com/pub/kristian-cruickshank/34/922/ba4

43