introduction to irriis testing platform irriis mit conference rome 8 february 2007 claudio...
TRANSCRIPT
![Page 1: Introduction to IRRIIS testing platform IRRIIS MIT Conference ROME 8 February 2007 Claudio Balducelli](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c765503460f9492a6a3/html5/thumbnails/1.jpg)
IRRIIS- FP6-2005–IST-4
EC - LOGO
Introduction to IRRIIS testing platform
IRRIIS MIT Conference ROME 8 February 2007
Claudio Balducelli
![Page 2: Introduction to IRRIIS testing platform IRRIIS MIT Conference ROME 8 February 2007 Claudio Balducelli](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c765503460f9492a6a3/html5/thumbnails/2.jpg)
IRRIIS
Summary
Design a testing environment for MIT Modelling and running attack and fault
behaviours Testing strategies for MIT components Proposed test-bed configuration Conclusions
![Page 3: Introduction to IRRIIS testing platform IRRIIS MIT Conference ROME 8 February 2007 Claudio Balducelli](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c765503460f9492a6a3/html5/thumbnails/3.jpg)
IRRIIS
Target Infrastructures
Models
Vulnerabilities of the Target
Infrastructures
Fault/attack Scenarios Generation
Models of faults & attacksUse domain
knowledge
Considervulnerabilities
Design a testing environment for MIT
![Page 4: Introduction to IRRIIS testing platform IRRIIS MIT Conference ROME 8 February 2007 Claudio Balducelli](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c765503460f9492a6a3/html5/thumbnails/4.jpg)
IRRIIS
Meaning of attacks and faults
Attacks: A disturbance of the LCCI generated by eventscoming from outside the LCCI
Faults: A disturbance of the LCCI generated by eventscoming from the components that are part of theLCCI
Definition of the meaning of attacks and faults
![Page 5: Introduction to IRRIIS testing platform IRRIIS MIT Conference ROME 8 February 2007 Claudio Balducelli](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c765503460f9492a6a3/html5/thumbnails/5.jpg)
IRRIIS
Meaning of attacks and faults
Attacks:
Natural disaster (earthquake, flood, etc)
Premeditated terrorist attack
Cyber attacks (cyber-intrusion)
Operator errors
………….….
Faults:
Physical component failure (aging, stress, etc.)
Software component failure (bug, wrong istal. etc)
Wrong component activation
………….….
![Page 6: Introduction to IRRIIS testing platform IRRIIS MIT Conference ROME 8 February 2007 Claudio Balducelli](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c765503460f9492a6a3/html5/thumbnails/6.jpg)
IRRIIS
Normal behavior & fault behavior in SimCIP
Activationevent
t1
Start Comp. 1
Comp. 1
Start Comp. 2
Comp. 2End
Start Comp. 3
t2 Comp. 3End
Activationevent
Normal behavior consists in an initial state and a sequence of events represented in form of a petri net oriented graph
![Page 7: Introduction to IRRIIS testing platform IRRIIS MIT Conference ROME 8 February 2007 Claudio Balducelli](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c765503460f9492a6a3/html5/thumbnails/7.jpg)
IRRIIS
Initiatingevent
t1
Failure of Comp. 1
t2
Failure ofComp. 2
t3
RestartComp. 1
t4 t5
Loss ofservice 2
Normal behavior & fault behavior in SimCIP
Loss ofService 1
Fault behavior may be represented in a similar way
Fault eventsIn LCCI-1
Failure ofComp. 2
t6
Failure of Comp. 1
t7
Fault eventsIn LCCI-2
![Page 8: Introduction to IRRIIS testing platform IRRIIS MIT Conference ROME 8 February 2007 Claudio Balducelli](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c765503460f9492a6a3/html5/thumbnails/8.jpg)
IRRIIS
For a certain LCCI normal behaviors are well known and their number is
limited the number and the combinations of fault behaviors are
very high and not always known in advance how to design fault behaviors? how to select fault behaviors? utilisation of a model based on attack/fault trees seem
useful to formalise and manage the knowledge needed to generate attack/fault behaviour
Normal behavior & fault behavior in SimCIP
![Page 9: Introduction to IRRIIS testing platform IRRIIS MIT Conference ROME 8 February 2007 Claudio Balducelli](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c765503460f9492a6a3/html5/thumbnails/9.jpg)
IRRIIS
G0
A1 A2 A2
The root of the tree (G) represents an event that could significantly harm the infrastructure’s mission.
The terminal leafs (A) of the tree represent the actions to execute for reaching the high level goals
Every path in the attack tree represents a unique type of attack
Goal G0AND A1 A2 A3
Goal G0OR A1 A2 A3
The attack trees could be visualized also in textual form
G0
A1 A2 A3
Every node could be decomposed inside lower level nodes using <AND>, <XOR> and <OR> decomposition types
AND
OR
Modelling attack knowledge attack/fault trees
![Page 10: Introduction to IRRIIS testing platform IRRIIS MIT Conference ROME 8 February 2007 Claudio Balducelli](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c765503460f9492a6a3/html5/thumbnails/10.jpg)
IRRIIS
G0
S1 A2 S2
A3 A4 A5 A6
The tree generate the following two attack patterns
<A3, A2, A5, A6>
<A4, A2, A5, A6>
The “terminal leafs” of the tree (A1..An) represent the actions steps needed to execute the attack
The “intermediate nodes” (S1..Sn) represent the steps in which a decision has to be taken
The attack tree generates attack patterns (attack behaviors), composed by sequences of actions.
Attack goal
Modelling attack knowledge attack/fault trees
![Page 11: Introduction to IRRIIS testing platform IRRIIS MIT Conference ROME 8 February 2007 Claudio Balducelli](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c765503460f9492a6a3/html5/thumbnails/11.jpg)
IRRIIS
TE
S1 C2 S3
C11 C12 C31 C32
The tree generate the following two
fault patterns
<C11, C2, C31, C32>
<C12, C1, C31, C32>
The “terminal leafs” of the tree (C..) represent the elementary failures of the single components of LCCI.
The “intermediate nodes” (S…) represent failures of subsystems or services for which the components contribute
The fault tree generates fault patterns (fault behaviors), composed by sequences of elementary failures.
Top event Fault trees
Modelling attack knowledge attack/fault trees
![Page 12: Introduction to IRRIIS testing platform IRRIIS MIT Conference ROME 8 February 2007 Claudio Balducelli](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c765503460f9492a6a3/html5/thumbnails/12.jpg)
IRRIIS
And gate
Or gateOR gate
AND gate
Example of attack tree to model an attack in a local area network (tree structure)
The reference model take in account the:Fault Tree Handbook ofUS Nuclear Regulatory Commission
![Page 13: Introduction to IRRIIS testing platform IRRIIS MIT Conference ROME 8 February 2007 Claudio Balducelli](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c765503460f9492a6a3/html5/thumbnails/13.jpg)
IRRIIS
And gate
Or gateOR gate
AND gate
Example of attack tree to model an attack in a local area network (tree structure)
Verify theaccessibility to a subnet
![Page 14: Introduction to IRRIIS testing platform IRRIIS MIT Conference ROME 8 February 2007 Claudio Balducelli](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c765503460f9492a6a3/html5/thumbnails/14.jpg)
IRRIIS
And gate
Or gateOR gate
AND gate
Example of attack tree to model an attack in a local area network (tree structure)
Discover the target locations & addresses
![Page 15: Introduction to IRRIIS testing platform IRRIIS MIT Conference ROME 8 February 2007 Claudio Balducelli](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c765503460f9492a6a3/html5/thumbnails/15.jpg)
IRRIIS
And gate
Or gateOR gate
AND gate
Example of attack tree to model an attack in a local area network (tree structure)
Make sniffing activity or damages
![Page 16: Introduction to IRRIIS testing platform IRRIIS MIT Conference ROME 8 February 2007 Claudio Balducelli](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c765503460f9492a6a3/html5/thumbnails/16.jpg)
IRRIIS
And gate
Or gateOR gate
AND gate
Example of attack tree to model an attack in a local area network (tree structure)
Generated behaviours table------------------------------------------------------------------------------------------------Attack behaviour 0 <A1, A2, A4, A5, A6, A7, A8 >
Attack behaviour 1 <A1, A2, A4, A5, A6, A7, A9>
Attack behaviour 2 <A1, A2, A4, A5, A6, A7, A10>
Attack behaviour 3 <A1, A2, A4, A5, A6, A7, A11>
Attack behaviour 4 <A1, A3, A4, A5, A6, A7, A8 >
Attack behaviour 5 <A1, A3, A4, A5, A6, A7, A9>
Attack behaviour 6 <A1, A3, A4, A5, A6, A7, A10>
Attack behaviour 7 <A1, A3, A4, A5, A6, A7, A11> ------------------------------------------------------------------------------------------------
![Page 17: Introduction to IRRIIS testing platform IRRIIS MIT Conference ROME 8 February 2007 Claudio Balducelli](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c765503460f9492a6a3/html5/thumbnails/17.jpg)
IRRIIS
Example of attack tree to model an attack:associating difficulties to the actions
OR gate
AND gate
0.8
0.9 0.2
0.950.95
0.95
0.30.6
0.2
0.80.8
0.0 = maximum difficulty1.0 = minimum difficulty
Generated behaviours table ordered by action difficulties------------------------------------------------------------------------------------------------Attack behaviour 0 <A1, A2, A4, A5, A6, A7, A8 > with 0,39 of difficulty
Attack behaviour 2 <A1, A2, A4, A5, A6, A7, A10> with 0,24 of difficulty
Attack behaviour 1 <A1, A2, A4, A5, A6, A7, A9> with 0.12 of difficulty
Attack behaviour 3 <A1, A2, A4, A5, A6, A7, A11> with 0.08 of difficulty
Attack behaviour 4 <A1, A3, A4, A5, A6, A7, A8 > with 0.08 of difficulty
Attack behaviour 6 <A1, A3, A4, A5, A6, A7, A10> with 0.05 of difficulty
Attack behaviour 5 <A1, A3, A4, A5, A6, A7, A9> with 0.03 of difficulty
Attack behaviour 7 <A1, A3, A4, A5, A6, A7, A11> with 0.02 of difficulty------------------------------------------------------------------------------------------------
![Page 18: Introduction to IRRIIS testing platform IRRIIS MIT Conference ROME 8 February 2007 Claudio Balducelli](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c765503460f9492a6a3/html5/thumbnails/18.jpg)
IRRIIS
Macro scenarios:
how to compose attack and fault trees
Attack tree
Fault tree
Attack tree
Wait for malfunction
![Page 19: Introduction to IRRIIS testing platform IRRIIS MIT Conference ROME 8 February 2007 Claudio Balducelli](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c765503460f9492a6a3/html5/thumbnails/19.jpg)
IRRIIS
Composite attack and fault behavior
t1
BasicAction 0
t2
BasicAction 2
Final Action 0
t3 t4
Final Action 1
Networkmalfunction
BasicEvent 0
Attackbehavior
Attackbehavior
Faultbehavior
Attack escalation
![Page 20: Introduction to IRRIIS testing platform IRRIIS MIT Conference ROME 8 February 2007 Claudio Balducelli](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c765503460f9492a6a3/html5/thumbnails/20.jpg)
IRRIIS
Testing MIT components (meaning)
REQUIREMENTS:
Risk Ass. (1) - The Risk estimator assessment of cascading and escalating effects shall be performed in near real-time.Risk Ass. (2) - The Risk estimator assessment of cascading and escalating effects shall be performed in a predictive way.Risk Ass. (3) - The Risk estimator shall estimate immediate risk to the LCCI.Risk Ass. (4) - The Risk estimator may estimate expected risk to the LCCI.Risk Ass. (5) - The Risk estimator shall estimate potential cascading effects.
Objective of the TEST: validate the requirements
Risk Ass. (1) - OK
Risk Ass. (2) - OK
Risk Ass. (3) - OK
Risk Ass. (4) - NOT OK
Risk Ass. (5) - NOT OK
![Page 21: Introduction to IRRIIS testing platform IRRIIS MIT Conference ROME 8 February 2007 Claudio Balducelli](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c765503460f9492a6a3/html5/thumbnails/21.jpg)
IRRIIS
Testing MIT components (meaning)
One of the main objective of the MIT components test inside SimCIP simulated environment is the evaluation of the rate of false/true alarms.
The second is to evaluate how much the rate of false alarms may be acceptable for the LCCIs operators
![Page 22: Introduction to IRRIIS testing platform IRRIIS MIT Conference ROME 8 February 2007 Claudio Balducelli](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c765503460f9492a6a3/html5/thumbnails/22.jpg)
IRRIIS
Detecting interdependency alarms
Real statesPredicted states
Alarm No Alarm
P(Alarm) A B P(No Alarm) C D
A = Number of alarm states correctly predicted
D = Number of no alarm states correctly predicted
B = Number of no alarm states predicted as true (FALSE POSITIVE)
C = Number of alarm states not predicted (FALSE NEGATIVE)
The goal is: max(A + D), min(B + C)
Evaluation Table
![Page 23: Introduction to IRRIIS testing platform IRRIIS MIT Conference ROME 8 February 2007 Claudio Balducelli](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c765503460f9492a6a3/html5/thumbnails/23.jpg)
IRRIIS
Detecting interdependency alarms
Real statesPredicted states
Alarm No Alarm
P(Alarm) A B P(No Alarm) C D
Fn = C / ( C + D )Observed False Negative Ratio (FNR)
Fp = B / ( A + B )Observed False Positive Ratio (FPR)
![Page 24: Introduction to IRRIIS testing platform IRRIIS MIT Conference ROME 8 February 2007 Claudio Balducelli](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c765503460f9492a6a3/html5/thumbnails/24.jpg)
IRRIIS
Be not afraid to discover false alarms during the tests. This is the tests objective!!
In many cases false alarms could be simply reduced tuning the “sensitivity” level of a MIT component.
To evaluate true/false alarms ratio is not sufficient a single attack/fault behavior. Many alternative behaviors are needed!!
Logging facilities are very important during experimentations, are the tests results must be archived and documented
Detecting interdependency alarms
![Page 25: Introduction to IRRIIS testing platform IRRIIS MIT Conference ROME 8 February 2007 Claudio Balducelli](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c765503460f9492a6a3/html5/thumbnails/25.jpg)
IRRIIS
Proposed testing strategy
IRRIIS testing operator
Attack/Fault tree editor
Design or modify a scenario
tree
GA
S1 A2 S2
A3 A4 A5 A6
<A3, A2, A5, A6>
<A4, A2, A5, A6>
<A3, A2, A5, A6>
<A4, A2, A5, A6>
Fault behaviors
editor
Generate & modify fault behaviors,
insert timing information etc
Documentation console
View logsEdit test
documents
Logs
Test documen
ts
Fault behavior execution
Execute behavious,
sets monitors
Attacks/faultsexecution in
SimCIP
Test designentry point
Test designexit point
Testdesign
![Page 26: Introduction to IRRIIS testing platform IRRIIS MIT Conference ROME 8 February 2007 Claudio Balducelli](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c765503460f9492a6a3/html5/thumbnails/26.jpg)
IRRIIS
Proposed testing strategy
IRRIIS testing operator
Attack/Fault tree editor
Design or modify a scenario
tree
GA
S1 A2 S2
A3 A4 A5 A6
<A3, A2, A5, A6>
<A4, A2, A5, A6>
<A3, A2, A5, A6>
<A4, A2, A5, A6>
Fault behaviors
editor
Generate & modify fault behaviors,
insert timing information etc
Documentation console
View logsEdit test
documents
Logs
Test documen
ts
Fault behavior execution
Execute behavious,
sets monitors
Attacks/faultsexecution in
SimCIP
Test execution entry point
Test execution exit point
Fasttesting
![Page 27: Introduction to IRRIIS testing platform IRRIIS MIT Conference ROME 8 February 2007 Claudio Balducelli](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c765503460f9492a6a3/html5/thumbnails/27.jpg)
IRRIIS
Proposed testing strategy
IRRIIS testing operator
Attack/Fault tree editor
Design or modify a scenario
tree
GA
S1 A2 S2
A3 A4 A5 A6
<A3, A2, A5, A6>
<A4, A2, A5, A6>
<A3, A2, A5, A6>
<A4, A2, A5, A6>
Fault behaviors
editor
Generate & modify fault behaviors,
insert timing information etc
Documentation console
View logsEdit test
documents
Logs
Test documen
ts
Fault behavior execution
Execute behavious,
sets monitors
Attacks/faultsexecution in
SimCIP
Test entry point
Test exit point
Exhaustivetesting
![Page 28: Introduction to IRRIIS testing platform IRRIIS MIT Conference ROME 8 February 2007 Claudio Balducelli](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c765503460f9492a6a3/html5/thumbnails/28.jpg)
IRRIIS
Physical TESTBED Configurations
LAMPSSys RTI
GUI Logger
To
ol 1
Electricity
SimulatorLCCI
Data
Com
Simulator
To
ol 2
Agent / Scenario
Behaviours
An
alysis 1
An
alysis 2Fault /
Attack
Tool
MITA
nalysis 3
SimCIP Architecture
![Page 29: Introduction to IRRIIS testing platform IRRIIS MIT Conference ROME 8 February 2007 Claudio Balducelli](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c765503460f9492a6a3/html5/thumbnails/29.jpg)
IRRIIS
Physical TESTBED Configurations
GUILogger
LAMPSSys RTI
Agent / Scenario
Behaviours
Electricity
Simulator
Com
Simulator
LCCI Electricity
Data Base
Tool 1
Tool 2
Analysis 1, 2, 3 ..
LCCI Telecom
Data Base
Simple SimCIPconfiguration
![Page 30: Introduction to IRRIIS testing platform IRRIIS MIT Conference ROME 8 February 2007 Claudio Balducelli](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c765503460f9492a6a3/html5/thumbnails/30.jpg)
IRRIIS
Physical TESTBED Configuration
LAMPSSys RTI
Agent / Scenario
Behaviours
Electricity
Simulator
Com
Simulator
LCCI Electricity
Data Base
Fault /Attack
Tool
Tool 1
Tool 2
Analysis 1, 2, 3 ..
LCCI Telecom
Data Base
SimCIPfor testing
attacks and faults without MIT
GUILogger
![Page 31: Introduction to IRRIIS testing platform IRRIIS MIT Conference ROME 8 February 2007 Claudio Balducelli](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c765503460f9492a6a3/html5/thumbnails/31.jpg)
IRRIIS
Physical TESTBED Configuration
GUILogger
LAMPSSys RTI
Agent / Scenario
Behaviours
Electricity
Simulator
Com
Simulator
LCCI Electricity
Data Base
LCCI Telecom
Data Base
MT communicationElectricity Add-on Telecom Add-on
SimCIPfor testing MIT with normal
behaviors(detect false positive alarms)
![Page 32: Introduction to IRRIIS testing platform IRRIIS MIT Conference ROME 8 February 2007 Claudio Balducelli](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c765503460f9492a6a3/html5/thumbnails/32.jpg)
IRRIIS
Physical TESTBED Configuration
GUILogger
LAMPSSys RTI
Agent / Scenario
Behaviours
Electricity
Simulator
Com
Simulator
LCCI Electricity
Data Base
LCCI Telecom
Data Base
MT communicationElectricity Add-on Telecom Add-on
SimCIP for testing MIT in presence of
attacks/faults (detect false negative alarms)
Fault /Attack
Tool
Tool 1
Tool 2
Analysis 1, 2, 3 ..
![Page 33: Introduction to IRRIIS testing platform IRRIIS MIT Conference ROME 8 February 2007 Claudio Balducelli](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c765503460f9492a6a3/html5/thumbnails/33.jpg)
IRRIIS
Conclusions
Testing of MIT components will be a continuous and iterative process
It is necessary to distinguish between the fast tests of the more simple requirements and the exhaustive test process aimed to evaluate the MIT efficiency in detecting interdependency alarms
Test designing, reports logging/archiving in a standard way and with the support of a common tool, will help to have sets of comparable tests also if produced in different SimCIP installations.
The testing environment will be one of the major a research product of the project, where experimentation may continue also after the end of the project.
QUESTIONS?