embraer risk assessment “three flags”...

EMBRAER – Safety Review Board

Embraer Risk Assessment

“Three Flags” Method

Maximilian Kleinubing – B.S.

Flight Test Engineer

Embraer Gavião Peixoto

Email: maximilian.kleinubing@embraer.com.br

EMBRAER – Safety Review Board

EMBRAER – Safety Review Board

Embraer Risk AssessmentEmbraer Risk Assessment

“Three Flags” Method

EMBRAER – Safety Review Board• Introduction

• Objective

• References

• Flight Envelope Analysis

• Test Point Execution Analysis

• System Failures Analysis• System Failures Analysis

• Risk Management

• Conclusions

EMBRAER – Safety Review Board

Introduction

Risk Assessment methodology has a key impact on safety during flight tests.

It is considered that the best methodology will be the one that is concise,

effective and has the ability to unveil the hazardseffective and has the ability to unveil the hazardsinvolved on the tests.

EMBRAER – Safety Review Board

Objective

• The goal of this presentation is to show how Embraer performs its Flight Test Risk how Embraer performs its Flight Test Risk Assessments at the present day.

EMBRAER – Safety Review Board

References

• Embraer ENS 00650

• FAA Order 4040.26• FAA Order 4040.26

EMBRAER – Safety Review Board

• Why Three Flags ?

EMBRAER – Safety Review Board

Flight EnvelopeTest Point

ExecutionSystem Failures

ACJ 1309 evaluation

EMBRAER FLIGHT TEST RISK ASSESSMENT METHOD

Operational Envelope

Design

Three Flags Method

ACJ 1309 evaluation

combined with flight

test effects to each

failure

Sequence of Assessment

Limited

Screen Factor Method

Each analysis is independent from each other.

EMBRAER – Safety Review Board

Flight Envelope

EMBRAER – Safety Review Board

Flight Envelope Analysis

FLIGHT ENVELOPE RISK CLASSIFICATION

OPERATIONAL ENVELOPE LOW

LIMITED ENVELOPE MEDIUMLIMITED ENVELOPE MEDIUM

DESIGN ENVELOPE MEDIUM

SCREEN FACTOR HIGH

EMBRAER – Safety Review Board

“Operational Envelope”: Envelope flown by operational

pilots. Normally is defined by protection systems and

placards.

“Design Envelope”: Envelope established by design

reports and larger than the operational envelope.

DEFINITIONS

reports and larger than the operational envelope.

“Limited Envelope”: Envelope that was limited during

the developmental flight testing due to engineering

problems encountered.

“Screen Factor”: Aims to detect if the a flight test is high

risk based on lack of sufficient experience, limited

engineering models and critical damages potential.

EMBRAER – Safety Review Board“Screen Factor” Application to High Risk Detection

THREE FLAGS

Screen Factor Tool

I – Detect Expertise on the Proposed Tests

Previous tests are sufficient to predict a safe behavior of the new proposed tests ?

II – Detect Limitations of Modeling Tools

Best available modeling tools are sufficient to predict a safe behavior for

the new proposed tests?

III – Detect Type of Possible Effects of the Proposed Test

Hazardous or catastrophic effects might result from the proposed tests if

predictions are incorrect?

EMBRAER – Safety Review BoardTest Point Execution

EMBRAER – Safety Review BoardTest Point Execution

• Considering that you might NOT have performed the maneuver ever before, the

pilot/FTE can count only with:

• Flight Experience

• Expertise on the aircraft

• Expertise on that type of maneuver• Expertise on that type of maneuver

• Lessons Learned

• It intends to DETECT if certain risks are presented during the maneuver execution.

Because of that some people might consider it to be vague.

• It intends to be as simple as possible and was designed ONLY for test point risk

evaluation.

EMBRAER – Safety Review Board“Three Flags” Application to Test Point Execution

THREE FLAGS

Test Point Execution SCALE

I – Detect training or gradual approach necessities

Training

Try-outs

Gradual Approach

Can the lack or

necessity of these

affect Safety ?

Is Needed ? And

II – Detect Errors Tolerances

Test Tolerances

Positioning TolerancesDo

Affect safety if extrapolated or disregarded ?

AndCan be Considered to be tight ?

III – Detect Recovering or Discontinuing possibilities

Recovering

DiscontinuingWhen The maneuver, is there a probable chance to get

into unsafe situation ?

EMBRAER – Safety Review Board

System Failures

EMBRAER – Safety Review Board

System Failures Analysis

• Based on AC 1309 safety assessment used by Embraer.

• Takes credit of the System Safety Assessment reports for all • Takes credit of the System Safety Assessment reports for all maneuvers analyzed.

• Takes credit of the System Safety Assessment reports to obtain failure probabilities.

• Flight test crew must define failures effects.

EMBRAER – Safety Review Board

FIN

AL P

RO

BA

BIL

ITY

HIGHLY PROBABLE

P > 10e-3LOW MEDIUM HIGH UNACCEPTABLE UNACCEPTABLE

PROBABLE

10e-3 > P > 10e-5LOW LOW MEDIUM HIGH UNACCEPTABLE

REMOTE

10e-5 > P > 10e-7LOW LOW LOW MEDIUM HIGH

EXTREMALLY

REMOTE LOW LOW LOW LOW MEDIUM

TABLE – SYSTEM FAILURES EVALUATION

FIN

AL P

RO

BA

BIL

ITY

REMOTE

10e-7 > P > 10e-9

LOW LOW LOW LOW MEDIUM

IMPROBABLE

P < 10e-9LOW LOW LOW LOW LOW

NO SAFETY

EFFECTMINOR MAJOR HAZARDOUS CATASTROPHIC

FAILURE EFFECT

EMBRAER – Safety Review Board

Flight EnvelopeTest Point

ExecutionSystem Failures

ACJ 1309 evaluation

EMBRAER FLIGHT TEST RISK ASSESSMENT METHOD

Operational Envelope

Design

Three Flags Method

ACJ 1309 evaluation

combined with flight

test effects to each

failure

Sequence of Assessment

Limited

Screen Factor Method

Each analysis is independent from each other.

EMBRAER – Safety Review Board

Risk Management

EMBRAER – Safety Review Board

Risk Management

• VMCG

• HAZARD: Landing gear collapse.

• CAUSE(S): Shimmy and ground/flight loads;

• RISK MINIMIZATION:

-As many as you want.

-Emphasize main hazards of the test.

-To agree upon Minimizing and Mitigation procedures.

Ex.

• RISK MINIMIZATION:1. According to the theoretical predictions the new caster is conservative

in relation to shimmy. However, the new landing gear shimmy

characteristics were not simulated. A ground test with and without the

steering spring will be performed to access the shimmy and control

characteristics of the new nose landing gear.

2. The landing gear loads predicted are 1% higher than in previous

design.

• RISK MITIGATION1. Firefighters standing by.

2. Ambulance standing by.

and Mitigation procedures.

EMBRAER – Safety Review Board

GENERAL OBSERVATIONS – METEOROLOGICAL CONDITIONS

Risk ClassificationMETEOROLOGICAL CONDITION

(TEST AREA AND DESTINATION/ALTERNATIVE)

LOW VMC/IMC (according to test requirements)

MEDIUM VMC/IMC (according to test requirements)

HIGH VMC (necessary)

1ST FLIGHT VMC (necessary)

EMBRAER – Safety Review BoardGENERAL OBSERVATIONS – CREW MINIMUM REQUIREMENTS

ClassificationExperience

(years)

Total Flight

Experience

(flight hours)

Rest between

same type of

flight (hours)

Number of

Flights on the

same day

(same

classification)

Minimum

Pilot Crew

Parte 23

Minimum Pilot

Crew

Parte 25

LOW ≥ 1 ≥ 1.000 ≥ 2 ≤ 3 1 PPRA(1) 1 PPRA(1)

MEDIUM ≥ 2 ≥ 1.000 ≥ 4 ≤ 2 1 PPRA(1) 2 PPRA(1)

HIGH ≥ 5 ≥ 1.500 ≥ 12 ≤ 1 2 PPRA(1) 2 PPRA(1)

Pilots

HIGH ≥ 5 ≥ 1.500 ≥ 12 ≤ 1 2 PPRA 2 PPRA

1st Flight ≥ 10 ≥ 2.000 ≥ 12 ≤ 1 2 PPRA(1) 2 PPRA(1)

ClassificationExperience

(years)

Total Flight

Experience (flight

hours)

Rest between

same type of

flight (hours)

Number of

Flights on the

same day (same

classification)

FTE(3) Rate

LOW - ≥ 10 ≥ 2 ≤ 3 C

MEDIUM ≥ 1 ≥ 100 ≥ 4 ≤ 2 B

HIGH ≥ 3 ≥ 200 ≥ 12 ≤ 1 A

1stststst Flight ≥ 5 ≥ 500 ≥ 12 ≤ 1 A*

FTEs

EMBRAER – Safety Review Board

• The Risk Assessment method is been used for only 6 months so it is

VERY NEW.

CONCLUSIONS

• Search for inconsistencies during application of the method.

• Nothing can substitute the FTE/TP discussions and analysis before the

actual flight test.

• The Risk Assessment method intends to be a trustworthy guide for

flight test preparation.

EMBRAER – Safety Review Board

EMBRAER – Safety Review Board

Embraer Risk Assessment

“Three Flags” Method

Maximilian Kleinubing – B.S.

Flight Test Engineer

Embraer Gavião Peixoto

email: maximilian.kleinubing@embraer.com.br

THANK YOU !

EMBRAER – Safety Review Board

Example

High Speed

Characteristics

Risk Assessment

(abstract)

Version: - DEN/GFT – 22 / 07 / 2009

EMBRAER – Safety Review BoardRisk Assessment: High Speed Characteristics

EMBRAER – Safety Review Board

Risk Assessment

• The Risk Assessment here presented was based on the Embraer

Normative System ENS-000650 Rev. 5 (Gerenciamento de Riscos em

Ensaio em Vôo).

The risk will be evaluated in the following items:

• System Failure

• Flight Envelope

• Test Point Execution

EMBRAER – Safety Review Board

• Risk Classification:

MANEUVER SF TPE FE RATEFINAL RATE FINAL RATE FINAL RATE FINAL RATE

AFTER AFTER AFTER AFTER MITIGATIONMITIGATIONMITIGATIONMITIGATION

Vdf-Mdf Determination L M M M M

CG Shift L L M M M

Inadvertent Speed Increase L M M M M

Lateral Gust Upset L L M M M

Risk Assessment: High Speed Characteristics

Lateral Gust Upset L L M M M

Longitudinal Gust Upset L L M M M

Two Axis Gust Upset L M M M M

Leveling Off From Climb L L M M M

Descent From Mach Airspeed Limit Altitude L L M M M

Speed Brake Deployment (up to Vfc / Mfc) L M M M M

Speed Brake Deployment (Vdf / Mdf) L H H H H

Roll Capability (up to Vfc / Mfc) L M M M M

Roll Capability (Vdf / Mdf) L H H H H

Speed Excursion to VDF/MDF L M M M M

EMBRAER – Safety Review Board

High Speed Characteristics

Hazard: Speed excursion beyond Vd/Md

Cause: Excessive pitch down associated to loss of longitudinal control effectiveness

Minimizing risk procedures:1. Flutter Flight Test Campaign already performed up to Vd/Md.

Emergency procedures:Emergency procedures:1. Retard both engines thrust levers to IDLE;

2. Speed brake (If Speed Brake Deployment test already executed);

3. Apply primary longitudinal control (elevator) nose up;

4. Apply secondary longitudinal control (pitch trim) nose up;

5. Return for landing.

EMBRAER – Safety Review Board

High Speed Characteristics

Hazard: Speed excursion beyond Vd/Md

Cause: Loss or Misleading Airspeed Information

Minimizing risk procedures:1. Crew must monitor and compare different airspeed sources;

Emergency procedures:Emergency procedures:1. Retard both engines thrust levers to IDLE;

2. Apply primary longitudinal control (elevator) nose up;

3. Apply secondary longitudinal control (pitch trim) nose up;

4. Return for landing.

EMBRAER – Safety Review Board

Conclusions:

MANEUVER SF TPE FE RATEFINAL RATE FINAL RATE FINAL RATE FINAL RATE

AFTER AFTER AFTER AFTER MITIGATIONMITIGATIONMITIGATIONMITIGATION

Vdf-Mdf Determination L M M M M

CG Shift L L M M M

Inadvertent Speed Increase L M M M M

Lateral Gust Upset L L M M M

Risk Assessment: High Speed Characteristics

Lateral Gust Upset L L M M M

Longitudinal Gust Upset L L M M M

Two Axis Gust Upset L L M M M

Leveling Off From Climb L L M M M

Descent From Mach Airspeed Limit Altitude L L M M M

Speed Brake Deployment (up to Vfc / Mfc) L M M M M

Speed Brake Deployment (Vdf / Mdf) L H H H H

Roll Capability (up to Vfc / Mfc) L M M M M

Roll Capability (Vdf / Mdf) L H H H H

Speed Excursion to VDF/MDF L M M M M

EMBRAER – Safety Review Board

Conclusions:

1. According to the Risk Assessment here presented, the risk involved in the Roll

Capability and Speed Brake Deployment at Vdf / Mdf was considered HIGH

2. The pilot and FTE must:

CR PILOT/FTEExperience

(years in the

Flight Test Experience

Rest Time between

Number of Flight in the

PILOT / FTE

Risk Assessment: High Speed Characteristics

3. VFR (Daylight) – Finish High Risk Test Points 1h before sunset.

4. The Roll Capability and Speed Brake Deployment at Vdf / Mdf requires: Ambulance (Medical Support) and SAR.

CR PILOT/FTE (years in the

activity)Experience

(hours)

between Flights

Flight in the same day

(ENS-004757)

HIGHPILOT ≥ 5 ≥ 1.500 ≥ 12 ≤ 1 2 PPRA

FTE ≥ 3 ≥ 200 ≥ 12 ≤ 1 A

EMBRAER – Safety Review Board

Conclusions (cont.):

1. According to the Risk Assessment here presented, the risk involved in the High Speed

Characteristics Campaign (except Roll Capability and Speed Brake Deployment at Vdf

/ Mdf ) was considered MEDIUM

2. The pilot and FTE must:

Experience Flight Test Rest Time Number of PILOT / FTE

Risk Assessment: High Speed Characteristics

3. VFR or IFR, according to test requirement and procedure during recovery of the

maneuver.

4. The Campaign does not require special support (Firefighters, Chase and SAR).

CR PILOT/FTEExperience

(years in the

activity)

Flight Test Experience

(hours)

Rest Time between Flights

Number of Flight in the same day

PILOT / FTE (ENS-004757)

MEDIUMPILOT ≥ 2 ≥ 1.000 ≥ 4 ≤ 2 1 PPRA

FTE ≥ 1 ≥ 100 ≥ 4 ≤ 2 B

EMBRAER – Safety Review Board

System Failure Risk Classification:

MANEUVERMANEUVERMANEUVERMANEUVER BASIC EVENTBASIC EVENTBASIC EVENTBASIC EVENT FAILUREFAILUREFAILUREFAILURERATERATERATERATE

FAILUREFAILUREFAILUREFAILUREEFFECTEFFECTEFFECTEFFECT

RISKRISKRISKRISKASSESSMENTASSESSMENTASSESSMENTASSESSMENT

Total loss of airspeed information 1.00E-07 HAZARDOUS LOW

Misleading display of airspeed information on

both primary flight displays1.00E-07 HAZARDOUS LOW

Loss of airspeed information on primary flight

displays1.00E-05 MAJOR LOW

All Maneuvers

displays

Misleading display of airspeed information on

one primary flight display1.00E-05 MAJOR LOW

Pitch trim runaway beyond timer limits 5.11E-12 CATASTROPHIC LOW

Pitch trim runaway within timer authority 3.90E-07 MAJOR LOW

Total Loss of pitch trim Function N/A MINOR LOW

Asymmetric Uncommanded spoiler panels

deployment beyond monitor limits 6.83E-10 CATASTROPHIC LOW

Symmetric Uncommanded spoiler panels

deployment beyond monitor limits 3.42E-10 MAJOR LOW

Asymmetric spoiler panel upfloat N/A MAJOR LOW

EMBRAER – Safety Review Board

Test Point Execution Risk Classification - Three Flags:

MANEUVERMANEUVERMANEUVERMANEUVERQUESTIONSQUESTIONSQUESTIONSQUESTIONS

FLAGSFLAGSFLAGSFLAGS RISK RISK RISK RISK ASSESSMENTASSESSMENTASSESSMENTASSESSMENT1111 2222 3333

Vdf-Mdf Determination - 2 MEDIUM

CG Shift - - - 0 LOW

Inadvertent Speed Increase - 2 MEDIUM

Lateral Gust Upset - - 1 LOWLateral Gust Upset - - 1 LOW

Longitudinal Gust Upset - - 1 LOW

Two Axis Gust Upset - - 1 LOW

Leveling Off From Climb - - - 0 LOW

Descent From Mach Airspeed Limit Altitude - - 1 LOW

Speed Brake Deployment (up to Vfc / Mfc) - 2 MEDIUM

Speed Brake Deployment (Vdf / Mdf) 3 HIGH

Roll Capability (up to Vfc / Mfc) - 2 MEDIUM

Roll Capability (Vdf / Mdf) 3 HIGH

Speed Excursion to VDF/MDF - 2 MEDIUM

EMBRAER – Safety Review Board

Flight Envelope Risk Classification: Screen Factor

MANEUVERMANEUVERMANEUVERMANEUVERQUESTIONSQUESTIONSQUESTIONSQUESTIONS

FLAGSFLAGSFLAGSFLAGS RISK RISK RISK RISK ASSESSMENTASSESSMENTASSESSMENTASSESSMENT1111 2222 3333

Vdf-Mdf Determination - - 1 NOT HIGH

CG Shift - - - 0 NOT HIGH

Inadvertent Speed Increase - - 1 NOT HIGH

Lateral Gust Upset - - 1 NOT HIGHLateral Gust Upset - - 1 NOT HIGH

Longitudinal Gust Upset - - 1 NOT HIGH

Two Axis Gust Upset - - 1 NOT HIGH

Leveling Off From Climb - - - 0 NOT HIGH

Descent From Mach Airspeed Limit Altitude - - 1 NOT HIGH

Speed Brake Deployment (up to Vfc / Mfc) - 2 NOT HIGH

Speed Brake Deployment (Vdf / Mdf) 3 HIGH

Roll Capability (up to Vfc / Mfc) - - 1 NOT HIGH

Roll Capability (Vdf / Mdf) 2 HIGH

Speed Excursion to VDF/MDF - - 1 NOT HIGH

EMBRAER – Safety Review Board

Flight Envelope Risk Classification:

TEST POINTTEST POINTTEST POINTTEST POINT FLIGHT FLIGHT FLIGHT FLIGHT ENVELOPEENVELOPEENVELOPEENVELOPE

RISK RISK RISK RISK CLASSIFICATIONCLASSIFICATIONCLASSIFICATIONCLASSIFICATION

Vdf-Mdf Determination DESIGN MEDIUM

CG Shift DESIGN MEDIUM

Inadvertent Speed Increase DESIGN MEDIUM

Lateral Gust Upset DESIGN MEDIUMLateral Gust Upset DESIGN MEDIUM

Longitudinal Gust Upset DESIGN MEDIUM

Two Axis Gust Upset DESIGN MEDIUM

Leveling Off From Climb DESIGN MEDIUM

Descent From Mach Airspeed Limit Altitude DESIGN MEDIUM

Speed Brake Deployment (up to Vfc / Mfc) DESIGN MEDIUM

Speed Brake Deployment (Vdf / Mdf) 3 FLAGS HIGH

Roll Capability (up to Vfc / Mfc) DESIGN MEDIUM

Roll Capability (Vdf / Mdf) 3 FLAGS HIGH

Speed Excursion to VDF/MDF DESIGN MEDIUM

EMBRAER – Safety Review Board

System Failure:

FIN

AL

PR

OB

AB

ILIT

Y

HIGHLY

PROBABLELOW MEDIUM HIGH UNACCEPTABLE UNACCEPTABLE

PROBABLE LOW LOW MEDIUM HIGH UNACCEPTABLE

REMOTE LOW LOW LOW MEDIUM HIGH

EXTREMALLY

REMOTELOW LOW LOW LOW MEDIUM

IMPROBABLE LOW LOW LOW LOW LOW

NO SAFETY

EFFECTMINOR MAJOR HAZARDOUS CATASTROPHIC

FAILURE EFFECT

EMBRAER – Safety Review Board

Embraer Risk Assessment

“Three Flags” Method

Maximilian Kleinubing – B.S.

Flight Test Engineer

Embraer Gavião Peixoto

email: maximilian.kleinubing@embraer.com.br

THANK YOU !

EMBRAER – Safety Review BoardExample:

Test Point – Maximum Dive Speed Demonstrated in Flight (Vdf) Determination

Aircraft: Choose an aircraft you are currently flying.

Configuration: UP/0

Initial Altitude: 15.000 ±1000 ft

Trim Speed: Vmo

“Three Flags” Application to Test Point Execution“Three Flags” Application to Test Point Execution

Trim Speed: Vmo

Minimum Recovery Altitude: 10.000 ft

1. TRIM aircraft at Vmo/Mmo during 10 seconds;

2. ESTABLISH a pitch attitude 7.5° (pushover maneuver Nz=0.5g) below the trimmed

attitude and maintain during 20 seconds. Do not reduce thrust;

3. PERFORM a pull up maneuver with load factor 1.5g. After start pull up, THRUST

REDUCTION may be used, if required.