measuring safety performance - an analyst’s perspective
TRANSCRIPT
![Page 1: Measuring Safety Performance - An Analyst’s Perspective](https://reader038.vdocument.in/reader038/viewer/2022102322/5480048ab4af9f11798b4632/html5/thumbnails/1.jpg)
Measuring Safety PerformanceAn Analyst’s Perspective
By: Indunil Weerasinghe28th September, 2013
www.linkedin.com/in/indunilmwwww.slideshare.net/walk_the_safety_talk
![Page 2: Measuring Safety Performance - An Analyst’s Perspective](https://reader038.vdocument.in/reader038/viewer/2022102322/5480048ab4af9f11798b4632/html5/thumbnails/2.jpg)
Measuring Safety PerformanceAn Analyst’s Perspective 2
What are we trying to achieve?
• In Australia:– Civil Aviation Order 82.5 lists AOC
holder’s obligations for HCRPT *:
* HCRPT = High Capacity Regular Public Transport
![Page 3: Measuring Safety Performance - An Analyst’s Perspective](https://reader038.vdocument.in/reader038/viewer/2022102322/5480048ab4af9f11798b4632/html5/thumbnails/3.jpg)
What are we trying to achieve?
• Internationally:– ICAO framework– Safety Management Manual (SMM) Doc 9859
Measuring Safety PerformanceAn Analyst’s Perspective 3
![Page 4: Measuring Safety Performance - An Analyst’s Perspective](https://reader038.vdocument.in/reader038/viewer/2022102322/5480048ab4af9f11798b4632/html5/thumbnails/4.jpg)
What are we trying to achieve?
• A key element of an effective SMS is to measure safety performance
• Collect, analyse and report safety data
• Objective: to facilitate safety assurance
Measuring Safety PerformanceAn Analyst’s Perspective 4
![Page 5: Measuring Safety Performance - An Analyst’s Perspective](https://reader038.vdocument.in/reader038/viewer/2022102322/5480048ab4af9f11798b4632/html5/thumbnails/5.jpg)
Process
•Ensure data quality / integrity
•Extract reactive (lagging) indicators
•Extract proactive (leading) indicators
•Develop predictive analyses
Measuring Safety PerformanceAn Analyst’s Perspective 5
![Page 6: Measuring Safety Performance - An Analyst’s Perspective](https://reader038.vdocument.in/reader038/viewer/2022102322/5480048ab4af9f11798b4632/html5/thumbnails/6.jpg)
Quality of data
• Data should be:– Valid, complete, consistent, and accurate
• Will always need some clean up, but...• Not feasible to clean it to perfection!
Measuring Safety PerformanceAn Analyst’s Perspective 6
![Page 7: Measuring Safety Performance - An Analyst’s Perspective](https://reader038.vdocument.in/reader038/viewer/2022102322/5480048ab4af9f11798b4632/html5/thumbnails/7.jpg)
Quality of dataBe pragmatic:
Be mindful of the limitations of your data,but make the most of what you have available!
Measuring Safety PerformanceAn Analyst’s Perspective 7
![Page 8: Measuring Safety Performance - An Analyst’s Perspective](https://reader038.vdocument.in/reader038/viewer/2022102322/5480048ab4af9f11798b4632/html5/thumbnails/8.jpg)
Three Levels of Analysis
3.Predictive
2. Proactive
1. Reactive
Measuring Safety PerformanceAn Analyst’s Perspective 8
![Page 9: Measuring Safety Performance - An Analyst’s Perspective](https://reader038.vdocument.in/reader038/viewer/2022102322/5480048ab4af9f11798b4632/html5/thumbnails/9.jpg)
Three Levels of Analysis
• To demonstrate:
– A safety data set was analysed
– Not true operational data (“mock” data used for presentation purposes only)
Measuring Safety PerformanceAn Analyst’s Perspective 9
![Page 10: Measuring Safety Performance - An Analyst’s Perspective](https://reader038.vdocument.in/reader038/viewer/2022102322/5480048ab4af9f11798b4632/html5/thumbnails/10.jpg)
REACTIVE INDICATORSBasic analyses to support reporting and reactive management
Measuring Safety PerformanceAn Analyst’s Perspective 10
![Page 11: Measuring Safety Performance - An Analyst’s Perspective](https://reader038.vdocument.in/reader038/viewer/2022102322/5480048ab4af9f11798b4632/html5/thumbnails/11.jpg)
What are “Reactive” Indicators
• Also referred to as “Lagging” Indicators
• Basic measurements that will sum up the recent past
• Can be expressed in terms of simple graphs and charts
• For example....
Measuring Safety PerformanceAn Analyst’s Perspective 11
![Page 12: Measuring Safety Performance - An Analyst’s Perspective](https://reader038.vdocument.in/reader038/viewer/2022102322/5480048ab4af9f11798b4632/html5/thumbnails/12.jpg)
Examples of Reactive Indicators
Mar-12 Apr-12 May-12 Jun-12 Jul-12 Aug-12 Sep-12 Oct-12 Nov-12 Dec-12 Jan-13 Feb-13 Mar-130
10
20
30
40
50
60
70
80
90
Incidents (per month) - Whole Fleet
Fleet TotalB777A380E190
?RATE
orQTY
?
Measuring Safety PerformanceAn Analyst’s Perspective 12
![Page 13: Measuring Safety Performance - An Analyst’s Perspective](https://reader038.vdocument.in/reader038/viewer/2022102322/5480048ab4af9f11798b4632/html5/thumbnails/13.jpg)
Examples of Reactive Indicators• Measuring RATE:
– E.g. Incidents per 1000FH– Communicates a relative measure of risk– Allows for comparison
• E.g. between a/c types, or across timelines
• Measuring QUANTITY:– Communicates an absolute measure of risk– Provides an understanding of the operational impact
• E.g. To determine resources required for incident investigation
Measuring Safety PerformanceAn Analyst’s Perspective 13
![Page 14: Measuring Safety Performance - An Analyst’s Perspective](https://reader038.vdocument.in/reader038/viewer/2022102322/5480048ab4af9f11798b4632/html5/thumbnails/14.jpg)
Examples of Reactive Indicators
35.3%
29.3%
13.2%
6.4%
5.8%2.4%
2.2% 1.6% 1.0% 2.7%
Incidents by Incident Type
Wildlife
Weather
ATS Deficiency
Laser Light
Aerodrome Related
Other Aircraft Failure to Comply
Forecast Deficiency
Other Operating Environment
Wake Turbulence
Other
Measuring Safety PerformanceAn Analyst’s Perspective 14
![Page 15: Measuring Safety Performance - An Analyst’s Perspective](https://reader038.vdocument.in/reader038/viewer/2022102322/5480048ab4af9f11798b4632/html5/thumbnails/15.jpg)
Examples of Reactive Indicators
Oct-10 Nov-10 Dec-10 Jan-11 Feb-11 Mar-11 Apr-11 May-11 Jun-11 Jul-11 Aug-11 Sep-11 Oct-11 Nov-11 Dec-110
10
20
30
40
50
60
Seasonal Variation of Weather Related Incidents
Drops off towards the end of the year (Oct/Nov/Dec)
Measuring Safety PerformanceAn Analyst’s Perspective 15
![Page 16: Measuring Safety Performance - An Analyst’s Perspective](https://reader038.vdocument.in/reader038/viewer/2022102322/5480048ab4af9f11798b4632/html5/thumbnails/16.jpg)
PROACTIVE INDICATORSBasic statistical analyses to support proactive management
Measuring Safety PerformanceAn Analyst’s Perspective 16
![Page 17: Measuring Safety Performance - An Analyst’s Perspective](https://reader038.vdocument.in/reader038/viewer/2022102322/5480048ab4af9f11798b4632/html5/thumbnails/17.jpg)
What are “Proactive” Indicators• Classified as a “Leading” Indicator
• Forecasting the near future based on trends
• Aims to flag problem areas before they manifest
• From ICAO’s SMM (Rev 3):– Uses “standard deviation” to set up “alert levels”
Measuring Safety PerformanceAn Analyst’s Perspective 17
![Page 18: Measuring Safety Performance - An Analyst’s Perspective](https://reader038.vdocument.in/reader038/viewer/2022102322/5480048ab4af9f11798b4632/html5/thumbnails/18.jpg)
Setting the Alert Levels• Calculate monthly mean
• Let’s call this μ
• Calculate the std dev. over the same period
• Let’s call this σ
• Calculations could be done using 12 or 24 month data
Measuring Safety PerformanceAn Analyst’s Perspective 18
![Page 19: Measuring Safety Performance - An Analyst’s Perspective](https://reader038.vdocument.in/reader038/viewer/2022102322/5480048ab4af9f11798b4632/html5/thumbnails/19.jpg)
When are alerts triggered?
• Reading above (μ + 3σ) in any month
• Reading above (μ + 2σ) in two consecutive months
• Reading above (μ + σ) in three consecutive months
Measuring Safety PerformanceAn Analyst’s Perspective 19
![Page 20: Measuring Safety Performance - An Analyst’s Perspective](https://reader038.vdocument.in/reader038/viewer/2022102322/5480048ab4af9f11798b4632/html5/thumbnails/20.jpg)
When are alerts triggered?
Jul-12 Aug-12 Sep-12 Oct-12 Nov-12 Dec-12 Jan-13 Feb-13 Mar-13 Apr-13 May-13 Jun-1315
17
19
21
23
25
27
29
31
33
35
Incidents Last YE's μ μ + σ μ + 2σ μ + 3σ
Measuring Safety PerformanceAn Analyst’s Perspective 20
![Page 21: Measuring Safety Performance - An Analyst’s Perspective](https://reader038.vdocument.in/reader038/viewer/2022102322/5480048ab4af9f11798b4632/html5/thumbnails/21.jpg)
PREDICTIVE INDICATORSAdvanced statistical analysis to support predictive management
Measuring Safety PerformanceAn Analyst’s Perspective 21
![Page 22: Measuring Safety Performance - An Analyst’s Perspective](https://reader038.vdocument.in/reader038/viewer/2022102322/5480048ab4af9f11798b4632/html5/thumbnails/22.jpg)
What are “Predictive” Indicators
• Also classified as a “Leading” Indicator• Less guidance available• Calls for a more creative approach to the
analysis
• This is where understanding your data and knowing how to make the most of it will really pay off!
Measuring Safety PerformanceAn Analyst’s Perspective 22
![Page 23: Measuring Safety Performance - An Analyst’s Perspective](https://reader038.vdocument.in/reader038/viewer/2022102322/5480048ab4af9f11798b4632/html5/thumbnails/23.jpg)
An example of a Predictive Indicator
• The following statistical test and its application won’t necessarily lie true of all data sets
• It’s just an example to get you thinking about how you could further analyse your data
Measuring Safety PerformanceAn Analyst’s Perspective 23
![Page 24: Measuring Safety Performance - An Analyst’s Perspective](https://reader038.vdocument.in/reader038/viewer/2022102322/5480048ab4af9f11798b4632/html5/thumbnails/24.jpg)
The Chi square test
• Null hypothesis (H0):– Aircraft type category are
INDEPENDENT variables
• Chi square test on H0 yielded a P score of 0%
• Conclusion - Reject the null:• There is a highly, statistically significant
dependency
Measuring Safety PerformanceAn Analyst’s Perspective 24
![Page 25: Measuring Safety Performance - An Analyst’s Perspective](https://reader038.vdocument.in/reader038/viewer/2022102322/5480048ab4af9f11798b4632/html5/thumbnails/25.jpg)
Taking away the “Stats Speak”
• A given aircraft type is more vulnerable to incidents of a certain nature
• Opportunity to predict and investigate problem areas regardless of “alert levels”:
– For example:• Risk profile to show high risk areas
Measuring Safety PerformanceAn Analyst’s Perspective 25
![Page 26: Measuring Safety Performance - An Analyst’s Perspective](https://reader038.vdocument.in/reader038/viewer/2022102322/5480048ab4af9f11798b4632/html5/thumbnails/26.jpg)
What do we mean by a “Risk profile”?
• What if we were to compare...
Actual frequency of a particular type of
incident on a particular aircraft
Theoretical frequency if the incident were to happen independently
of aircraft type
Measuring Safety PerformanceAn Analyst’s Perspective 26
![Page 27: Measuring Safety Performance - An Analyst’s Perspective](https://reader038.vdocument.in/reader038/viewer/2022102322/5480048ab4af9f11798b4632/html5/thumbnails/27.jpg)
A380 Risk profile “hot spots”
• This tells us “VHF Comms” incidents are 7x more likely on this a/c in my fleet than if a/c and incident types were independent
• Flags up a high risk area, regardless of an alert level being breached
• Warrants a root-cause analysis:– Is there an equipment reliability issue? Pilot training? Procedural issue?
Measuring Safety PerformanceAn Analyst’s Perspective 27
![Page 28: Measuring Safety Performance - An Analyst’s Perspective](https://reader038.vdocument.in/reader038/viewer/2022102322/5480048ab4af9f11798b4632/html5/thumbnails/28.jpg)
IN CONCLUSION...
Measuring Safety PerformanceAn Analyst’s Perspective 28
![Page 29: Measuring Safety Performance - An Analyst’s Perspective](https://reader038.vdocument.in/reader038/viewer/2022102322/5480048ab4af9f11798b4632/html5/thumbnails/29.jpg)
Keep in mind....All these examples are simply that... Just fake data designed to convey an
idea.
Don’t apply the conclusion!Don’t even apply the method!
Just apply the thinking!Measuring Safety Performance
An Analyst’s Perspective 29
![Page 30: Measuring Safety Performance - An Analyst’s Perspective](https://reader038.vdocument.in/reader038/viewer/2022102322/5480048ab4af9f11798b4632/html5/thumbnails/30.jpg)
OpportunitiesChallenges
1. Minimal guidance on “predictive indicators”
As SMS’s mature and enter latter phases of implementation, this is an area that will benefit from increasing industry & academic collaboration.
Potential for significant value addition to the “safety assurance” process
Measuring Safety PerformanceAn Analyst’s Perspective 30
![Page 31: Measuring Safety Performance - An Analyst’s Perspective](https://reader038.vdocument.in/reader038/viewer/2022102322/5480048ab4af9f11798b4632/html5/thumbnails/31.jpg)
Opportunities
2. Operational safety data tells you whether you are achieving the desired safety outcomes, but...
... do you know if those outcomes are being achieved in the most efficient, least costly manner?
Measuring Safety PerformanceAn Analyst’s Perspective 31
![Page 32: Measuring Safety Performance - An Analyst’s Perspective](https://reader038.vdocument.in/reader038/viewer/2022102322/5480048ab4af9f11798b4632/html5/thumbnails/32.jpg)
Opportunities A “balanced scorecard” approach to safety
metrics would complement the existing leading/lagging indicators
Better return on your safety investment
Incentivise the desired safety behaviours
Nurture a generative safety attitudes
Cultivate a continuous improvement culture
Measuring Safety PerformanceAn Analyst’s Perspective 32
![Page 33: Measuring Safety Performance - An Analyst’s Perspective](https://reader038.vdocument.in/reader038/viewer/2022102322/5480048ab4af9f11798b4632/html5/thumbnails/33.jpg)
• Thanks for reading, and remember...
• The aggregate safety of our industry lies in our hands, so...
• ... the more we innovate and share our ideas, the better the outcome we can achieve.
• Please feel free to contact me:– On LinkedIn at www.linkedin.com/in/indunilmw– Or email me at [email protected]
Measuring Safety PerformanceAn Analyst’s Perspective 33
![Page 34: Measuring Safety Performance - An Analyst’s Perspective](https://reader038.vdocument.in/reader038/viewer/2022102322/5480048ab4af9f11798b4632/html5/thumbnails/34.jpg)
34
Download these Resources• ICAO’s Safety Management Manual (Doc 9859 Rev 3)
• Robert S Kaplan’s paper from the Harvard Business School on the Balanced Scoredcard
• Testing for Independence using the “Chi Square” Test, from Berkeley University
• Australian Legislative Instrument: Civil Aviation Order 82.5
Measuring Safety PerformanceAn Analyst’s Perspective