fault tolerant technology multi domain computer...
TRANSCRIPT
Fault Tolerant Technology
Multi Domain Computer
Architecture
Flight Software Workshop 12/4/2018 ©Fault Tolerant Technology [email protected] 1
What if there were a
computer that:
No Viruses or malware No supply chains infections Recover from a CPU failure Worry free software upgrades Secure computer communications
Flight Software Workshop 12/4/2018 ©Fault Tolerant Technology [email protected] 2
Fault Tolerant Technology No application code changes System built using COTS parts Ran applications faster Used same power Cost about the same
Flight Software Workshop 12/4/2018 ©Fault Tolerant Technology [email protected] 3
Multi Domain Architecture
Flight Software Workshop 12/4/2018 ©Fault Tolerant Technology [email protected] 4
Multi Domain Architecture
How does it do that?
Flight Software Workshop 12/4/2018 ©Fault Tolerant Technology [email protected] 5
Single Domain Architecture
Control
Unit
Input Output
Accumulator
Arithmetic Logic
Unit
Memory
Mem-1 Mem-2 Mem-3 Mem-4 Mem-5 Mem-6
CPU
Flight Software Workshop 12/4/2018 ©Fault Tolerant Technology [email protected] 6
Multi Domain Architecture
Virtualized Address Space - SwitchProcess
Address Virtualization
Control
cPU-1 cPU-2
DID Memory
DID
Data & Instruction
PAD
Process Scheduling
Address Space
Mem-1 Mem-2 Mem-3 Mem-4 Mem-5 Mem-6
KAC
Input
Output
Flight Software Workshop 12/4/2018 ©Fault Tolerant Technology [email protected] 7
Two Factor
Communications
7. Application layer
6. Presentation layer
5. Session layer
4. Transport layer
3. Network layer
2. Data link layer
1. Physical layer
Multi Domain Architecture
Flight Software Workshop 12/4/2018 ©Fault Tolerant Technology [email protected] 8
Transportation Layer
Segmentation and Reassembling: A message is divided into segments; each
segment contains sequence number, which enables this layer in reassembling the
message. Message is reassembled correctly upon arrival at the destination and replaces
packets which were lost in transmission
Flight Software Workshop 12/4/2018 ©Fault Tolerant Technology [email protected] 9
Two Factor Communication
Virtualized Address Space - Switch
Process
Address
Virtualization
Control
cPU-1 cPU-2
DID Memory
Mem-1 Mem-2 Mem-3 Mem-4 Mem-5 Mem-6
KAC
Input
Output
Virtualized Address Space - Switch
Process
Address
Virtualization
Control
cPU-1 cPU-2
DID Memory
Mem-1 Mem-2 Mem-3 Mem-4 Mem-5 Mem-6
KAC
Input
Output
Cloud
Cloud
Flight Software Workshop 12/4/2018 ©Fault Tolerant Technology [email protected] 10
Rule and Role Based
Access Control Start
Is address space in the
databaseIs access
authorizedExecution fails
End process
Add dynamic
access rights
Load process
access table
Start process/fork
Execution failsDoes process have file
system access rights
Historic
Behavior
MDA RBAC
Decision
NO YESExecution fails
NO
NO
YES
YES
Flight Software Workshop 12/4/2018 ©Fault Tolerant Technology [email protected] 11
Fault Tolerant Technology
W W
PAD
Controller
KAC
0 0
cPU-1 cPU-2
Mem-1 Mem-2 Mem-3 Mem-4 Mem-5
Process PC CPU Memory/Notes
USB
NETWORK
usb intrp - usb-1
usb
n-io
n-io
n-io
auto - kill
intrp - net-1
auto - ok
run down ok
n-io run icmp ok
Cloud
PRS TABLE
Flight Software Workshop 12/4/2018 ©Fault Tolerant Technology [email protected] 12
Fault Tolerant Technology
W W
PAD
Controller
KAC
0 0
cPU-1 cPU-2
Mem-1 Mem-2 Mem-3 Mem-4 Mem-5
Process PC CPU Memory/Notes
Up 1 cPU-1 Mem-1 Mem-3
USB
NETWORK
usb intrp - usb-1
usb
n-io
n-io
n-io
auto - kill
intrp - net-1
auto - ok
run up ok
n-io run icmp ok
Cloud
PRS TABLE
Flight Software Workshop 12/4/2018 ©Fault Tolerant Technology [email protected] 13
Fault Tolerant Technology
0 W
PAD
Controller
KAC
0 0
cPU-1 cPU-2
Mem-1 Mem-2 Mem-3 Mem-4 Mem-5
Process PC CPU Memory/Notes
Up 1 cPU-1 Mem-1 Mem-3
USB
NETWORK
usb intrp - usb-1
usb
n-io
n-io
n-io
auto - kill
intrp - net-1
auto - ok
run up ok
n-io run icmp ok
Cloud
PRS TABLE
Flight Software Workshop 12/4/2018 ©Fault Tolerant Technology [email protected] 14
Fault Tolerant Technology
0 + 1 -> A net-1
PAD
Controller
KAC
0 0
cPU-1 cPU-2
Mem-1 Mem-2 Mem-3 Mem-4 Mem-5
Process PC CPU Memory/Notes
Up
net-1
3 cPU-1 Mem-1 Mem-3
1 cPU-2
USB
NETWORK
usb intrp - usb-1
usb
n-io
n-io
n-io
auto - kill
intrp - net-1
auto - ok
run up ok
n-io run icmp ok
Cloud
PRS TABLE
Flight Software Workshop 12/4/2018 ©Fault Tolerant Technology [email protected] 15
Fault Tolerant Technology
0 + 1 -> A net-1
PAD
Controller
KAC
0 0
cPU-1 cPU-2
Mem-1 Mem-2 Mem-3 Mem-4 Mem-5
Process PC CPU Memory/Notes
Up
net-1
7 cPU-1 Mem-1 Mem-3
1 cPU-2
USB
NETWORK
usb intrp - usb-1
usb
n-io
n-io
n-io
auto - kill
intrp - net-1
auto - ok
run up ok
n-io run icmp ok
Cloud
PRS TABLE
Flight Software Workshop 12/4/2018 ©Fault Tolerant Technology [email protected] 16
Fault Tolerant Technology
0 + 1 -> A net-1
PAD
Controller
KAC
0 0
cPU-1 cPU-2
Mem-1 Mem-2 Mem-3 Mem-4 Mem-5
Process PC CPU Memory/Notes
Up
net-1
3 cPU-1 Mem-1 Mem-3
1 cPU-2
USB
NETWORK
usb intrp - usb-1
usb
n-io
n-io
n-io
auto - kill
intrp - net-1
auto - ok
run up ok
n-io run icmp ok
Cloud
PRS TABLE
Flight Software Workshop 12/4/2018 ©Fault Tolerant Technology [email protected] 17
Fault Tolerant Technology
1 -> M icmp
PAD
Controller
KAC
1 1
cPU-1 cPU-2
Mem-1 Mem-2 Mem-3 Mem-4 Mem-5
Process PC CPU Memory/Notes
Up
icmp
10 cPU-1 Mem-1 Mem-3
4 cPU-2
USB
NETWORK
usb intrp - usb-1
usb
n-io
n-io
n-io
auto - kill
intrp - net-1
auto - ok
run up ok
n-io run icmp ok
Cloud
PRS TABLE
Flight Software Workshop 12/4/2018 ©Fault Tolerant Technology [email protected] 18
Fault Tolerant Technology
1 + 1 -> AICMP execute
Backoriface
PAD
Controller
KAC
1 1
cPU-1 cPU-2
Mem-1 Mem-2 Mem-3 Mem-4 Mem-5
Process PC CPU Memory/Notes
Up
icmp
13 cPU-1 Mem-1 Mem-3
8 cPU-2 Icmp can’t find BO entry
Kill process
USB
NETWORK
usb intrp - usb-1
usb
n-io
n-io
n-io
auto - kill
intrp - net-1
auto - ok
run up ok
n-io run icmp ok
Cloud
PRS TABLE
Flight Software Workshop 12/4/2018 ©Fault Tolerant Technology [email protected] 19
Fault Tolerant Technology
A =? 2 -
PAD
Controller
KAC
1 1
cPU-1 cPU-2
Mem-1 Mem-2 Mem-3 Mem-4 Mem-5
Process PC CPU Memory/Notes
Up 18 cPU-1 Mem-1 Mem-3
USB
NETWORK
usb intrp - usb-1
usb
n-io
n-io
n-io
auto - kill
intrp - net-1
auto - ok
run up ok
n-io run icmp ok
Cloud
PRS TABLE
Flight Software Workshop 12/4/2018 ©Fault Tolerant Technology [email protected] 20
Multi Domain Architecture
It should enable users to use ICMP messages for administrative purposes.
• It should allow large size of ICMP so users can file out if the network can carry large size of data packets
• It should be able to handle encrypted packets.
• It should be able to prevent personal machines that are not behind powerful state preserving firewalls from being used as DDOS slaves.
Flight Software Workshop 12/4/2018 ©Fault Tolerant Technology [email protected] 21
Simultaneous Failures
Hardware Failure
Virus Problem
Greaco-Latin Square
Fault Tolerance
Flight Software Workshop 12/4/2018 ©Fault Tolerant Technology [email protected] 22
Graeco Latin Square Analysis
Flight Software Workshop 12/4/2018 ©Fault Tolerant Technology [email protected] 23
PAD
ControllerKAC
cPU-0 cPU-2
Proc PCCPU Memory/Notes
Input
Output
frk run Pit-a kill
frk
frk
n-io
run Pit-b kill
run Pit-c kill
auto - ok
cPU-1 cPU-3
Mem-2
Mem-1
Mem-0P-0
P-1
Display - 0
Display - 1
Mem-3
P-2
P-0
P-1
P-2
CPU-3CPU-2CPU-0
Process Time
P-0
P-1
P-2
CPU-3CPU-2CPU-0
Hardware
Display - 2
Graeco-Latin Square Graeco-Latin Square
Graeco Latin Square Analysis
Flight Software Workshop 12/4/2018 ©Fault Tolerant Technology [email protected] 24
PAD
ControllerKAC
cPU-0 cPU-2
Proc PCCPU Memory/NotesPit-a 1 cPU-0 P-0,Mem-0,D-0
Input
Output
cPU-1 cPU-3
Mem-3
Mem-2
Mem-1
Mem-0P-0
P-1
P-2
P-0
P-1
P-2
CPU-3CPU-2CPU-0
Process Time
P-0
P-1
P-2
CPU-3CPU-2CPU-0
Hardware
Display - 0
Display - 1
Display - 2
Graeco-Latin Square Graeco-Latin Square
frk run Pit-a kill
frk
frk
n-io
run Pit-b kill
run Pit-c kill
auto - ok
Graeco Latin Square Analysis
Flight Software Workshop 12/4/2018 ©Fault Tolerant Technology [email protected] 25
PAD
ControllerKAC
cPU-0 cPU-2
Proc PCCPU Memory/NotesPit-a
Pit-b
5 cPU-0 P-0,Mem-0, D-0
1 cPU-2 P-1,Mem-1, D-1
Input
Output
cPU-1 cPU-3
Mem-3
Mem-2
Mem-1
Mem-0P-0
P-1
P-2
P-0
P-1
P-2
CPU-3CPU-2CPU-0
Process TIme
P-0
P-1
P-2
CPU-3CPU-2CPU-0
Hardware
Display - 0
Display - 1
Display - 2
Graeco-Latin Square Graeco-Latin Square
frk run Pit-a kill
frk
frk
n-io
run Pit-b kill
run Pit-c kill
auto - ok
Graeco Latin Square Analysis
Flight Software Workshop 12/4/2018 ©Fault Tolerant Technology [email protected] 26
PAD
ControllerKAC
cPU-0 cPU-2
Proc PCCPU Memory/NotesPit-a
Pit-b
Pit-c
12 cPU-0 P-0,Mem-0, D-0
5 cPU-2 P-1,Mem-1, D-1
1 cPU-3 P-2,Mem-2,D-2
Input
Output
cPU-1 cPU-3
Mem-3
Mem-2
Mem-1
Mem-0P-0
P-1
P-2
P-0
P-1
P-2
CPU-3CPU-2CPU-0
Process Time
P-0
P-1
P-2
CPU-3CPU-2CPU-0
Hardware
Display - 0
Display - 1
Display - 2
Graeco-Latin SquareGraeco-Latin Square
frk run Pit-a kill
frk
frk
n-io
run Pit-b kill
run Pit-c kill
auto - ok
Graeco Latin Square Analysis
Flight Software Workshop 12/4/2018 ©Fault Tolerant Technology [email protected] 27
PAD
ControllerKAC
cPU-0 cPU-2
Proc PCCPU Memory/NotesPit-a
Pit-b
Pit-c
22 cPU-0 P-0,Mem-0,D-0
22 cPU-2 P-1,Mem-1,D-1
5 cPU-3 P-2,Mem-2,D-2
Input
Output
cPU-1 cPU-3
265
Mem-3
Mem-2
Mem-1
Mem-0P-0
P-1
P-2
52P-0
P-1
P-2
CPU-3CPU-2CPU-0
Process Time
265P-0
P-1
P-2
CPU-3CPU-2CPU-0
Hardware
Display - 0
Display - 1
Display - 2
Graeco-Latin SquareGraeco-Latin Square
frk run Pit-a kill
frk
frk
n-io
run Pit-b kill
run Pit-c kill
auto - ok
Graeco Latin Square Analysis
Flight Software Workshop 12/4/2018 ©Fault Tolerant Technology [email protected] 28
PAD
ControllerKAC
cPU-0 cPU-2
Proc PCCPU Memory/Notes
Pit-b
Pit-c
32 cPU-2 P1,Mem-1,D1
12 cPU-3 P2,Mem-2,D2
Input
Output
cPU-1 cPU-3
265
265
Mem-3
Mem-2
Mem-1
Mem-0P-0
P-1
P-2
52P-0
155P-1
P-2
CPU-3CPU-2CPU-0
Process Time
265P-0
265P-1
P-2
CPU-3CPU-2CPU-0
Hardware
Display - 0
Display - 1
Display - 2
Graeco-Latin SquareGraeco-Latin Square
frk run Pit-a kill
frk
frk
n-io
run Pit-b kill
run Pit-c kill
auto - ok
Graeco Latin Square Analysis
Flight Software Workshop 12/4/2018 ©Fault Tolerant Technology [email protected] 29
PAD
ControllerKAC
cPU-0 cPU-2
Proc PCCPU Memory/NotesPit-b
Pit-c
1 cPU-0 P-2, Mem-0,D-2
22 cPU-3 P-2,Mem-2,D-2
Input
Output
cPU-1 cPU-3
265
265
93Mem-3
Mem-2
Mem-1
Mem-0P-0
P-1
P-2
52P-0
155P-1
52P-2
CPU-3CPU-2CPU-0
Process Time
265P-0
265P-1
93P-2
CPU-3CPU-2CPU-0
Hardware
Display - 0
Display - 1
Display - 2
Graeco-Latin SquareGraeco-Latin Square
frk run Pit-a kill
frk
frk
n-io
run Pit-b kill
run Pit-c kill
auto - ok
Graeco Latin Square Analysis
Flight Software Workshop 12/4/2018 ©Fault Tolerant Technology [email protected] 30
PAD
ControllerKAC
cPU-0 cPU-2
Proc PCCPU Memory/NotesPit-b
Pit-c
5 cPU-0 P-2,Mem-0,D-2
1 cPU-2 P-0,Mem-2,D-0
Input
Output
cPU-1 cPU-3
265
265
93Mem-3
Mem-2
Mem-1
Mem-0P-0
P-1
P-2
52P-0
155P-1
52P-2
CPU-3CPU-2CPU-0
Process Time
265P-0
265P-1
93P-2
CPU-3CPU-2CPU-0
Hardware
Display - 0
Display - 1
Display - 2
Graeco-Latin SquareGraeco-Latin Square
frk run Pit-a kill
frk
frk
n-io
run Pit-b kill
run Pit-c kill
auto - ok
Graeco Latin Square Analysis
Flight Software Workshop 12/4/2018 ©Fault Tolerant Technology [email protected] 31
PAD
ControllerKAC
cPU-0 cPU-2
Proc PCCPU Memory/NotesPit-b
Pit-c
Pit-a
22 cPU-0 P-2,Mem-0,D-2
5 cPU-2 P-0,Mem-2,D-0
1 cPU-3 P-1,Mem-1,D1
Input
Output
cPU-1 cPU-3
265
265
93Mem-3
Mem-2
Mem-1
Mem-0P-0
P-1
P-2
52P-0
155P-1
52P-2
CPU-3CPU-2CPU-0
Process Time
265P-0
265P-1
93P-2
CPU-3CPU-2CPU-0
Hardware
Display - 0
Display - 1
Display - 2
Graeco-Latin SquareGraeco-Latin Square
frk run Pit-a kill
frk
frk
n-io
run Pit-b kill
run Pit-c kill
auto - ok
Graeco Latin Square Analysis
Flight Software Workshop 12/4/2018 ©Fault Tolerant Technology [email protected] 32
PAD
ControllerKAC
cPU-0 cPU-2
Proc PCCPU Memory/NotesPit-b
Pit-c
Pit-a
32 cPU-2 P-2,Mem0,D-2
12 cPU-0 P-0,Mem2,D-0
5 cPU-3 P-1,Mem1,D-1
Input
Output
cPU-1 cPU-3
265
265
93Mem-3
Mem-2
Mem-1
Mem-0P-0
P-1
P-2
52P-0
155P-1
52155P-2
CPU-3CPU-2CPU-0
Process Time
265P-0
265P-1
9393P-2
CPU-3CPU-2CPU-0
Hardware
Display - 0
Display - 1
Display - 2
Graeco-Latin SquareGraeco-Latin Square
frk run Pit-a kill
frk
frk
n-io
run Pit-b kill
run Pit-c kill
auto - ok
Graeco Latin Square Analysis
Flight Software Workshop 12/4/2018 ©Fault Tolerant Technology [email protected] 33
PAD
ControllerKAC
cPU-0 cPU-2
Proc PCCPU Memory/Notes
Pit-c
Pit-a
22 cPU-2 P-0,Mem2,D-0
12 cPU-3 P-1,Mem1,D-1
Input
Output
cPU-1 cPU-3
265
265
93Mem-3
Mem-2
Mem-1
Mem-0P-0
P-1
P-2
5252P-0
155P-1
52155P-2
CPU-3CPU-2CPU-0
Process Time
265265P-0
265P-1
9393P-2
CPU-3CPU-2CPU-0
Hardware
Display - 0
Display - 1
Display - 2
Graeco-Latin SquareGraeco-Latin Square
frk run Pit-a kill
frk
frk
n-io
run Pit-b kill
run Pit-c kill
auto - ok
Graeco Latin Square Analysis
Flight Software Workshop 12/4/2018 ©Fault Tolerant Technology [email protected] 34
PAD
ControllerKAC
cPU-0 cPU-2
Proc PCCPU Memory/NotesPit-c
Pit-a
1 cPU-0 P-1,Mem-2,D-1
22 cPU-3 P-2,Mem-0,D-2
Input
Output
cPU-1 cPU-3
265
265
93Mem-3
Mem-2
Mem-1
Mem-0P-0
P-1
P-2
5252P-0
52155P-1
52155P-2
CPU-3CPU-2CPU-0
Process Time
265265P-0
265265P-1
9393P-2
CPU-3CPU-2CPU-0
Hardware
Display - 0
Display - 1
Display - 2
Graeco-Latin SquareGraeco-Latin Square
frk run Pit-a kill
frk
frk
n-io
run Pit-b kill
run Pit-c kill
auto - ok
Graeco Latin Square Analysis
Flight Software Workshop 12/4/2018 ©Fault Tolerant Technology [email protected] 35
PAD
ControllerKAC
cPU-0 cPU-2
Proc PCCPU Memory/NotesPit-c
Pit-a
5 cPU-0 P-1, Mem2,D-1
1 cPU-2 P-2,Mem0,D-2
Input
Output
cPU-1 cPU-3
265
265
93Mem-3
Mem-2
Mem-1
Mem-0P-0
P-1
P-2
5252P-0
52155P-1
52155P-2
CPU-3CPU-2CPU-0
Process Time
265265P-0
265265P-1
9393P-2
CPU-3CPU-2CPU-0
Hardware
Display - 0
Display - 1
Display - 2
Graeco-Latin SquareGraeco-Latin Square
frk run Pit-a kill
frk
frk
n-io
run Pit-b kill
run Pit-c kill
auto - ok
Graeco Latin Square Analysis
Flight Software Workshop 12/4/2018 ©Fault Tolerant Technology [email protected] 36
PAD
ControllerKAC
cPU-0 cPU-2
Proc PCCPU Memory/NotesPit-c
Pit-a
Pit-b
12 cPU-0 P-1,Mem-2,D-1
5 cPU-2 P-2,Mem-0,D-2
1 cPU-3 P-0,Mem-1,D-0
Input
Output
cPU-1 cPU-3
265
265
93Mem-3
Mem-2
Mem-1
Mem-0P-0
P-1
P-2
5252P-0
52155P-1
52155P-2
CPU-3CPU-2CPU-0
Process Time
265265P-0
265265P-1
9393P-2
CPU-3CPU-2CPU-0
Hardware
Display - 0
Display - 1
Display - 2
Graeco-Latin SquareGraeco-Latin Square
frk run Pit-a kill
frk
frk
n-io
run Pit-b kill
run Pit-c kill
auto - ok
Graeco Latin Square Analysis
Flight Software Workshop 12/4/2018 ©Fault Tolerant Technology [email protected] 37
PAD
ControllerKAC
cPU-0 cPU-2
Proc PCCPU Memory/NotesPit-c
Pit-a
Pit-b
22 cPU-0 P-1,Mem-2,D-1
12 cPU-2 P-2,Mem-0,D-3
5 cPU-3 P-0,Mem-1,D-0
Input
Output
cPU-1 cPU-3
265
265
93Mem-3
Mem-2
Mem-1
Mem-0P-0
P-1
P-2
5252P-0
5215552P-1
52155P-2
CPU-3CPU-2CPU-0
Process Time
265265P-0
265265265P-1
9393P-2
CPU-3CPU-2CPU-0
Hardware
Display - 0
Display - 1
Display - 2
Graeco-Latin SquareGraeco-Latin Square
frk run Pit-a kill
frk
frk
n-io
run Pit-b kill
run Pit-c kill
auto - ‘
Graeco Latin Square Analysis
Flight Software Workshop 12/4/2018 ©Fault Tolerant Technology [email protected] 38
PAD
ControllerKAC
cPU-0 cPU-2
Proc PCCPU Memory/NotesPit-c
Pit-a
Pit-b
22 cPU-0 P-1,Mem-2,D-1
12 cPU-2 P-2,Mem-0,D-3
5 cPU-3 P-0,Mem-1,D-0
Input
Output
cPU-1 cPU-3
265
265
93Mem-3
Mem-2
Mem-1
Mem-0P-0
P-1
P-2
5252P-0
5215552P-1
52155P-2
CPU-3CPU-2CPU-0
Process Time
265265P-0
265265265P-1
9393P-2
CPU-3CPU-2CPU-0
Hardware
Display - 0
Display - 1
Display - 2
Graeco-Latin SquareGraeco-Latin Square
frk run Pit-a kill
frk
frk
n-io
run Pit-b kill
run Pit-c kill
auto - ‘
Graeco Latin Square Analysis
Flight Software Workshop 12/4/2018 ©Fault Tolerant Technology [email protected] 39
PAD
ControllerKAC
CPU-0 CPU-2
Proc PCCPU Memory/Notes
Pit-b 32 cPU-1 P-0,Mem1,P-2
Input
Output
CPU-1 CPU-3
265
265
93Mem-3
Mem-2
Mem-1
Mem-0P-0
P-1
P-2
1555252P-0
5215552P-1
5252155P-2
CPU-3CPU-2CPU-0
Process Time
265265265P-0
265265265P-1
939393P-2
CPU-3CPU-2CPU-0
Hardware
Display - 0
Display - 1
Display - 2
Graeco-Latin SquareGraeco-Latin Square
OutputMem-2
frk run Pit-a kill
frk
frk
n-io
run Pit-b kill
run Pit-c kill
auto - ok
Graeco Latin Square Analysis
Flight Software Workshop 12/4/2018 ©Fault Tolerant Technology [email protected] 40
PAD
ControllerKAC
CPU-0 CPU-2
Proc PCCPU Memory/Notes
Input
Output
CPU-1 CPU-3
265
265
93Mem-3
Mem-2
Mem-1
Mem-0P-0
P-1
P-2
1555252P-0
5215552P-1
5252155P-2
CPU-3CPU-2CPU-0
Process Time
265265265P-0
265265265P-1
939393P-2
CPU-3CPU-2CPU-0
Hardware
Display - 0
Display - 1
Display - 2
Graeco-Latin SquareGraeco-Latin Square
frk run Pit-a kill
frk
frk
n-io
run Pit-b kill
run Pit-c kill
auto - ok
Multi Domain Architecture
Flight Software Workshop 12/4/2018 ©Fault Tolerant Technology [email protected] 41