thoughts and ideas on distributed security
Post on 03-Jan-2016
41 Views
Preview:
DESCRIPTION
TRANSCRIPT
Thoughts and ideas on distributed security
Dropletdroplet@kernelchina.org
Topics• None-CP design• CP-based design• RTC (run-to-completion) • Event-driven• Parallel vs pipeline • Message and RPC• Locking and IPC• Resource management• Session affinity• Reorder
P1 IOC1
IOC2
IOC3
SPU1
SPU2
SPU3
P2P3
hash1
hash2
hash3
None-CP design
None-CP design highlight
• Intelligence on IOC– IOC distributes packet by hashing
• Simple load balance– Performance depends on traffic pattern
• Need resource synchronization between SPUs– Static allocation on SPUs (Inefficient)– Dynamic allocation on SPUs (Need central
resource management point or resource synchronization between SPUs)
P1 IOC1
IOC2
IOC3
SPU1
SPU2
SPU3
P2P3
CP-based design
CPfirst path
fast path
CP-based design highlight
• Intelligence on CP• CP is responsible for– Session distribution and load balance– Resource management
• CP is the bottle neck?
RTC (run-to-completion)
• FT is RTC (run-to-completion)– It's a big loop, no yield, no scheduling– Watchdog will be triggered if a FT hold CPU 20 seconds
on Australia
• Methods for FT break and follow up– Timer events (used by session scan, HA cold sync etc…)– User events (used by session SZ etc…)
• Timer interrupt would interrupt FT every 10 milliseconds?
Event-driven
• FT is event-driven– FT is polling DPQ– Run callback registered for different DPQ– Locking on queues– Locking between events
Parallel vs pipeline
• All threads run in parallel• For dedicate path, different stage connected
by queues, different threads run in pipeline
ParallelQueue1
Thread1
Queue2Thread2
Queue3Thread3
Multi threads on different queues
Queue4
Thread4
Thread5
Thread6
Multi threads on the same queue
PipelineQueue1 Queue2 Queue3
Thread1 Thread1 Thread1 Thread1
Same thread on different stages
Queue1 Queue2 Queue3Thread1 Thread2 Thread3 Thread4
Different threads on different stages
Parallel vs pipeline• Small and simple tasks– Pros
• Cache friendly
– Cons• Latency• Complicate design and programming
• Big and complicate tasks– Pros
• Easy design and programming
– Cons• Unpredictable runtime
Message and RPC
• Message between IOC/CP/SPU– Kind of RPC– Reliable?– Serialization?– Message type/Message size– Message driven state machine
Locking and IPC
• Spinlock and spinlock enhancement– Spinlock• Busy waiting (no sleep, no timeout)• Get lock in random order• Same priority for all lockers
– Lock order– Lock reentrance
• IPC between FTs– User event
Spinlock enhancement• Read/Write lock– Concurrent read, exclusive write
• Serialization lock– Assign a ticket to locker
• Sequence lock– Writer can interrupt reader
• TryLock– Timeout on a period of time
• RCU– Copy on read, very complicate
Resource management
• Central management (Resource on CP)– Session/resource handled by all threads– Session and resource are handled by different threads– Session CP/Resource CP
• Sparse management (Resource on SPUs)– Static allocation with dynamic allocation
• Resource allocation– Allocate one resource per request– Allocate batch of resources per request
session request
CP threads
resource request
SPU threads
CP session threads
session request
resource request
CP resource threads
Central management
session/resource request
CP threads
Need CPU power?
Waste if there is no resource
request!
Memory issue!One SPU is
special!
Sparse managementSPU1/Resource bundle1 SPU2/Resource bundle2
SPU3/Resource bundle3
Local allocation Local allocation
Local allocation
Remote allocation
Remote allocation Remote allocation
Sparse management
• NUMA?• Remote allocation performance is not predictable • Static resource partition may not fit with SPU session
capacity• Complicate remote allocation mechanism
Resource allocation
Resource repository
Resource client
Request Reply
Allocate one resource per request
Resource repository
Resource client
Allocate batch of resources per request
Request Reply
Local allocation
Resource allocation• Allocate one resource per request– Cons
• Too many messages between resource client and resource repository
– Pros• Simple
• Allocate batch of resources per request– Cons
• Need resource management function on resource client and resource repository
– Pros• Reduce messages between resource client and resource repository
Session affinity
• Typical case– The packets for one session can be handled by any
threads– The threads can handle packets for any session
• Session affinity– One thread handle packets for the session exclusively– Different threads can handle different sessions
simultaneously– Serialized state machine
Session affinity
LBTAssign seq and queue
DPQ
FT
FT
Queue and return
Queue and return
Session queue
FTProcessing packet
FTLBT
Session queue
Serialization by sequence
Serialization by locking and event
Session affinity
• Serialization by sequence– Cons
• Waste
– Pros• LBT simple
• Serialization by locking and event– Cons
• LBT complex
– Pros• Efficient
Reorder
DPQ
FT
FT
FT
Out of order processing
LBTIn order
Tag sequence
POTIn order
Untag sequence
• Global ordering: Tag sequence on receiving • Session based ordering: Tag sequence on matching session • LBT and POT is single thread
Summary
• CP is necessary• Real time/ Multi core programming model• Locking is complicate• Resource management is difficult• Application needs to be multi thread aware• Order is a mandatory requirement
THANK YOU
top related