eleos: exit-less os services for sgx enclaves · 2017-05-18 · 22 may@systor' 2017 meni...
TRANSCRIPT
Eleos: Exit-Less OS Services for SGX Enclaves
Meni OrenbachMarina MinkinPavel Lifshits
Mark Silberstein
Accelerated Computing Systems Lab
Haifa, Israel
22 May@Systor' 2017 Meni Orenbach, Technion 2
What do we do?Improve performance: I/O intensive & memory demanding SGX enclaves
Why?Cost of SGX execution for these applications is high
How?In-enclave System Calls & User Managed Virtual Memory
ResultsEleos vs vanilla SGX
2x Throughput: memcached & face verification serversEven for 5x available enclave memory
Available for Linux, Windows*
(*) Without Eleos, these applications crash in Windows enclaves
22 May@Systor' 2017 Meni Orenbach, Technion 3
● Background● Motivation● Overhead analysis● Eleos design● Evaluation
22 May@Systor' 2017 Meni Orenbach, Technion 4
SGX enclaves are already here!
● Secured execution environment● Reversed sandbox● Small TCB● Private code & data
– Confidentiality
– Integrity
– Freshness
● Only CPU is trusted
Operating system
ApplicationEnclave Enclave
22 May@Systor' 2017 Meni Orenbach, Technion 5
SGX enclaves are already here!
● Secured execution environment● Reversed sandbox● Small TCB● Private code & data
– Confidentiality
– Integrity
– Freshness
● Only CPU is trusted
Operating system
ApplicationEnclave Enclave
22 May@Systor' 2017 Meni Orenbach, Technion 6
SGX enclaves are already here!
● Secured execution environment● Reversed sandbox● Small TCB● Private code & data
– Confidentiality
– Integrity
– Freshness
● Only CPU is trusted
Operating system
ApplicationEnclave Enclave
22 May@Systor' 2017 Meni Orenbach, Technion 7
SGX enclaves are already here!
● Secured execution environment● Reversed sandbox● Small TCB● Private code & data
– Confidentiality
– Integrity
– Freshness
● Only CPU is trusted
Operating system
ApplicationEnclave Enclave
22 May@Systor' 2017 Meni Orenbach, Technion 8
SGX enclaves are already here!
● Secured execution environment● Reversed sandbox● Small TCB● Private code & data
– Confidentiality
– Integrity
– Freshness
● Only CPU is trusted
Operating system
ApplicationEnclave Enclave
22 May@Systor' 2017 Meni Orenbach, Technion 9
SGX enclaves are already here!
● Secured execution environment● Reversed sandbox● Small TCB● Private code & data
– Confidentiality
– Integrity
– Freshness
● Only CPU is trusted
Operating system
ApplicationEnclave Enclave
Lets look atHow to secure server applications with enclaves
22 May@Systor' 2017 Meni Orenbach, Technion 10
Background: Lifetime of a secured server
Untrusted (Host & OS) Trusted (Enclave)
22 May@Systor' 2017 Meni Orenbach, Technion 11
Untrusted memoryUnsecured access
Background: Lifetime of a secured server
Untrusted (Host & OS) Trusted (Enclave)
22 May@Systor' 2017 Meni Orenbach, Technion 12
Untrusted memoryUnsecured access
Background: Lifetime of a secured server
Untrusted (Host & OS) Trusted (Enclave)
Dedicated SGX memLimited to: 128 MB
Secured access
22 May@Systor' 2017 Meni Orenbach, Technion 13
Wait for networkrequests
Background: Lifetime of a secured server
Hostapp
Untrusted (Host & OS) Trusted (Enclave)
22 May@Systor' 2017 Meni Orenbach, Technion 14
Wait for networkrequests
Background: Lifetime of a secured server
Hostapp
Untrusted (Host & OS) Trusted (Enclave)
22 May@Systor' 2017 Meni Orenbach, Technion 15
Wait for networkrequests
Background: Lifetime of a secured server
Decrypt requests
Enter enclave
Hostapp
Untrusted (Host & OS) Trusted (Enclave)
22 May@Systor' 2017 Meni Orenbach, Technion 16
Wait for networkrequests
Background: Lifetime of a secured server
Decrypt requests
Enter enclave
Process requests
Hostapp
Untrusted (Host & OS) Trusted (Enclave)
22 May@Systor' 2017 Meni Orenbach, Technion 17
Wait for networkrequests
Background: Lifetime of a secured server
Decrypt requests
Enter enclave
Process requests
Hostapp
Encrypt responses
Untrusted (Host & OS) Trusted (Enclave)
22 May@Systor' 2017 Meni Orenbach, Technion 18
Send responses
Wait for networkrequests
Background: Lifetime of a secured server
Decrypt requests
Enter enclave
Process requests
Exit enclave
Hostapp
Encrypt responses
Untrusted (Host & OS) Trusted (Enclave)
22 May@Systor' 2017 Meni Orenbach, Technion 19
SGX enclaves should be fast
● ISA extensions● Implemented in HW & Firmware● Same CPU HW● In-cache execution suffers no overheads
22 May@Systor' 2017 Meni Orenbach, Technion 20
SGX enclaves should be fast
● ISA extensions● Implemented in HW & Firmware● Same CPU HW● In-cache execution suffers no overheads
However...
22 May@Systor' 2017 Meni Orenbach, Technion 21
Executing a Key-Value Store in enclave is slower
22 May@Systor' 2017 Meni Orenbach, Technion 22
64 MB 512 MB0
5
10
15
20
25
30
35
40
Memory footprint
Executing a Key-Value Store in enclave is slower
Throughput: Slowdown factor
11X
34X
22 May@Systor' 2017 Meni Orenbach, Technion 23
64 MB 512 MB0
5
10
15
20
25
30
35
40
Memory footprint
Executing a Key-Value Store in enclave is slower
Throughput: Slowdown factor
11X
34X
Crashesin Windows
22 May@Systor' 2017 Meni Orenbach, Technion 24
● Background● Motivation● Overhead analysis● Eleos design● Evaluation
22 May@Systor' 2017 Meni Orenbach, Technion 25
Send responses
Wait for networkrequests
Overhead analysis
Enter enclave
Exit enclave
Hostapp
Untrusted (Host & OS) Trusted (Enclave)
Decrypt requests 150 cycles/32B
Process requests *100 cycles/32B
Encrypt responses *150 cycles/32B
22 May@Systor' 2017 Meni Orenbach, Technion 26
Overhead analysis
Enter enclave
Exit enclave
Hostapp
Untrusted (Host & OS) Trusted (Enclave)
Send responses
Wait for networkrequests
Enter enclave
Exit enclave
Hostapp Decrypt requests
150 cycles/32B
Process requests *100 cycles/32B
Encrypt responses *150 cycles/32B
~3,300cycles
22 May@Systor' 2017 Meni Orenbach, Technion 27
Overhead analysis
Enter enclave
Exit enclave
Hostapp
Untrusted (Host & OS) Trusted (Enclave)
Send responses
Wait for networkrequests
Enter enclave
Exit enclave
Hostapp Decrypt requests
150 cycles/32B
Process requests*100 cycles/32B
Encrypt responses*150 cycles/32B
~3,300cycles
~3,800cycles
22 May@Systor' 2017 Meni Orenbach, Technion 28
Overhead analysis
Enter enclave
Exit enclave
Hostapp
Untrusted (Host & OS) Trusted (Enclave)
Send responses
Wait for networkrequests
Enter enclave
Exit enclave
Hostapp Decrypt requests
150 cycles/32B
Process requests *100 cycles/32B
Encrypt responses *150 cycles/32B
~3,300cycles
~3,800cycles
Exits causes indirect costs:1.5X – 5X slower execution
FlexSC [OSDI'10] syscall analysis
22 May@Systor' 2017 Meni Orenbach, Technion 29
Overhead analysis
Enter enclave
Exit enclave
Hostapp
Untrusted (Host & OS) Trusted (Enclave)
Send responses
Wait for networkrequests
Enter enclave
Exit enclave
Hostapp Decrypt requests
150 cycles/32B
Process requests *100 cycles/32B
Encrypt responses *150 cycles/32B
~3,300cycles
~3,800cycles
Exits causes indirect costs:1.5X – 5X slower execution
FlexSC [OSDI'10] syscall analysis
22 May@Systor' 2017 Meni Orenbach, Technion 30
Eleos does better!
64 MB 512 MB0
5
10
15
20
25
30
35
40SGX Eleos
Memory footprint
3.5x
5x
Throughput: Slowdown factor
22 May@Systor' 2017 Meni Orenbach, Technion 31
Eleos does better!
64 MB 512 MB0
5
10
15
20
25
30
35
40SGX Eleos
Memory footprint
3.5x
5x
How does Eleos achieve this?
Throughput: Slowdown factor
22 May@Systor' 2017 Meni Orenbach, Technion 32
Eleos: Exit-less services
Exit-less system calls with RPC infrastructure
Exit-less SGX paging
22 May@Systor' 2017 Meni Orenbach, Technion 33
Eleos: Exit-less services
Exit-less system calls with RPC infrastructure
Exit-less SGX paging
22 May@Systor' 2017 Meni Orenbach, Technion 34
Background: SGX paging
System mem
SGX mem
Dedicated memoryEnclave code & data
Limited to 128 MB
22 May@Systor' 2017 Meni Orenbach, Technion 35
Background: SGX paging
System memsecret_foo():...*p = 1; SGX mem
EnclaveTrusted
Untrusted
22 May@Systor' 2017 Meni Orenbach, Technion 36
Background: SGX paging
System memsecret_foo():...*p = 1; SGX mem
HardwareAddress translation
EnclaveTrusted
Untrusted
22 May@Systor' 2017 Meni Orenbach, Technion 37
Background: SGX paging
System memsecret_foo():...*p = 1;
Encrypted
SGX mem
Page table
HardwareAddress translation
EnclaveTrusted
Untrusted
22 May@Systor' 2017 Meni Orenbach, Technion 38
Background: SGX paging
System memsecret_foo():...*p = 1;
Encrypted
SGX mem
Page table
HardwareAddress translation
Swapped-out
EnclaveTrusted
Untrusted
22 May@Systor' 2017 Meni Orenbach, Technion 39
Background: SGX paging
System mem
Faulthandler
secret_foo():...*p = 1;
Encrypted
SGX mem
Page table
HardwareAddress translation
Swapped-out
EnclaveTrusted
UntrustedSGX-driver
22 May@Systor' 2017 Meni Orenbach, Technion 40
Background: SGX paging
System mem
Faulthandler
secret_foo():...*p = 1;
Encrypted
Integrityvalidation
Decrypted
SGX mem
Page table
HardwareAddress translation
Swapped-out
EnclaveTrusted
UntrustedSGX-driver
22 May@Systor' 2017 Meni Orenbach, Technion 41
Background: SGX paging
System mem
Faulthandler
secret_foo():...*p = 1;*(++p) = 2;
Encrypted
Decrypted
SGX mem
Page table
HardwareAddress translation
EnclaveTrusted
SGX driverUntrusted
22 May@Systor' 2017 Meni Orenbach, Technion 42
Background: SGX paging
System mem
Faulthandler
secret_foo():...*p = 1;*(++p) = 2;
Encrypted
Decrypted
SGX mem
Page table
HardwareAddress translation
EnclaveTrusted
SGX driverUntrusted
Fast path
22 May@Systor' 2017 Meni Orenbach, Technion 43
Background: SGX paging
System mem
Faulthandler
secret_foo():...*p = 1;*(++p) = 2;
Encrypted
Decrypted
SGX mem
Page table
HardwareAddress translation
EnclaveTrusted
SGX driverUntrusted
Since SGX memory is smallpaging is not as rare as in native applications
What are the overheads?
Fast path
22 May@Systor' 2017 Meni Orenbach, Technion 44
Background: SGX paging
System mem
Faulthandler
secret_foo():...*p = 1;*(++p) = 2;
Encrypted
Decrypted
SGX mem
Page table
HardwareAddress translation
EnclaveTrusted
SGX driverUntrusted
22 May@Systor' 2017 Meni Orenbach, Technion 45
SGX paging overheads
System mem
Faulthandler
secret_foo():...*p = 1;*(++p) = 2;
Encrypted
Decrypted
SGX mem
Page table
HardwareAddress translation
EnclaveTrusted
SGX driverUntrusted
Enclaveresume
Enclaveexit
22 May@Systor' 2017 Meni Orenbach, Technion 46
SGX paging overheads
System mem
Faulthandler
secret_foo():...*p = 1;*(++p) = 2;
Encrypted
Decrypted
SGX mem
Page table
HardwareAddress translation
EnclaveTrusted
SGX driverUntrusted
Enclaveresume
Enclaveexit
Indirect costs
22 May@Systor' 2017 Meni Orenbach, Technion 47
SGX paging overheads
System mem
Faulthandler
secret_foo():...*p = 1;*(++p) = 2;
Encrypted
Decrypted
SGX mem
Page table
HardwareAddress translation
EnclaveTrusted
SGX driverUntrusted
Enclaveresume
Enclaveexit
Overaheads: Untrusted softwaremanages enclave memory
Indirect costs
22 May@Systor' 2017 Meni Orenbach, Technion 48
SGX paging overheads
System mem
Faulthandler
secret_foo():...*p = 1;*(++p) = 2;
Encrypted
Decrypted
SGX mem
Page table
HardwareAddress translation
EnclaveTrusted
SGX driverUntrusted
Enclaveresume
Enclaveexit
Overaheads: Untrusted softwaremanages enclave memory
Indirect costs
22 May@Systor' 2017 Meni Orenbach, Technion 49
Wanted: In-enclave virtual memory management
No more exits!
22 May@Systor' 2017 Meni Orenbach, Technion 50
Ideal in-enclave VM management
System mem
Faulthandler
secret_foo():...*p = 1;*(++p) = 2;
SGX mem
Page table
HardwareAddress translation
EnclaveTrusted
SGX driverUntrusted
HardwareAddress translation
22 May@Systor' 2017 Meni Orenbach, Technion 51
Ideal in-enclave VM management
System mem
Faulthandler
secret_foo():...*p = 1;*(++p) = 2;
SGX mem
Page table
HardwareAddress translation
EnclaveTrusted
HardwareAddress translation
22 May@Systor' 2017 Meni Orenbach, Technion 52
Ideal in-enclave VM management
System mem
Faulthandler
secret_foo():...*p = 1;*(++p) = 2;
SGX mem
Page table
HardwareAddress translation
EnclaveTrusted
HardwareAddress translation
No availablehardware
22 May@Systor' 2017 Meni Orenbach, Technion 53
Ideal in-enclave VM management
System mem
Faulthandler
secret_foo():...*p = 1;*(++p) = 2;
SGX mem
Page table
HardwareAddress translation
EnclaveTrusted
SoftwareAddress translation
22 May@Systor' 2017 Meni Orenbach, Technion 54
SUVM: Secured user-space VM
System mem
Faulthandler
secret_foo():s_ptr<int> p = suvm_malloc(1024);...*p = 1; SGX mem
Page table
EnclaveTrusted
SoftwareAddress translation
22 May@Systor' 2017 Meni Orenbach, Technion 55
SUVM: Secured user-space VM
System mem
Faulthandler
secret_foo():s_ptr<int> p = suvm_malloc(1024);...*p = 1; SGX mem
Page table
EnclaveTrusted
SoftwareAddress translation
Template class: SecuredPointer.
22 May@Systor' 2017 Meni Orenbach, Technion 56
SUVM: Secured user-space VM
System mem
Faulthandler
secret_foo():s_ptr<int> p = suvm_malloc(1024);...*p = 1;
Encrypted
SGX mem
Page table
EnclaveTrusted
SoftwareAddress translation
Template class: SecuredPointer.
22 May@Systor' 2017 Meni Orenbach, Technion 57
SUVM: Secured user-space VM
System mem
Faulthandler
secret_foo():s_ptr<int> p = suvm_malloc(1024);...*p = 1;
Encrypted
SGX mem
Page table
EnclaveTrusted
SoftwareAddress translation
Template class: SecuredPointer.
Swapped-out
22 May@Systor' 2017 Meni Orenbach, Technion 58
SUVM: Secured user-space VM
System mem
Faulthandler
secret_foo():s_ptr<int> p = suvm_malloc(1024);...*p = 1;
Encrypted
SGX mem
Page table
EnclaveTrusted
SoftwareAddress translation
Template class: SecuredPointer.
Swapped-out
22 May@Systor' 2017 Meni Orenbach, Technion 59
SUVM: Secured user-space VM
System mem
Faulthandler
secret_foo():s_ptr<int> p = suvm_malloc(1024);...*p = 1;
Encrypted
Decrypted
SGX mem
Page table
EnclaveTrusted
SoftwareAddress translation
Template class: SecuredPointer.
Swapped-out
Integrityvalidation
22 May@Systor' 2017 Meni Orenbach, Technion 60
SUVM: Secured user-space VM
System mem
Faulthandler
secret_foo():s_ptr<int> p = suvm_malloc(1024);...*p = 1;
Encrypted
Decrypted
SGX mem
Page table
EnclaveTrusted
SoftwareAddress translation
Template class: SecuredPointer.
Swapped-out
Integrityvalidation
Control pathin-enclave
22 May@Systor' 2017 Meni Orenbach, Technion 61
SUVM: Secured user-space VM
System mem
Faulthandler
secret_foo():s_ptr<int> p = suvm_malloc(1024);...*p = 1;*(++p) = 2;
Encrypted
Decrypted
SGX mem
Page table
EnclaveTrusted
SoftwareAddress translation
22 May@Systor' 2017 Meni Orenbach, Technion 62
SUVM: Secured user-space VM
System mem
Faulthandler
secret_foo():s_ptr<int> p = suvm_malloc(1024);...*p = 1;*(++p) = 2;
Encrypted
Decrypted
SGX mem
Page table
EnclaveTrusted
SoftwareAddress translation
22 May@Systor' 2017 Meni Orenbach, Technion 63
SUVM: Secured user-space VM
System mem
Faulthandler
secret_foo():s_ptr<int> p = suvm_malloc(1024);...*p = 1;*(++p) = 2;
Encrypted
Decrypted
SGX mem
Page table
EnclaveTrusted
SoftwareAddress translation
Fast pathNo page table
Lookup!
22 May@Systor' 2017 Meni Orenbach, Technion 64
Wait...Software based VM management?
Based on software address translation on GPUs, ActivePointers [ISCA'2016]
22 May@Systor' 2017 Meni Orenbach, Technion 65
SUVM key contributions
● Multi-threaded
Compared to SGX:
Fast path: up to 20% overheads
Slow path: Eliminates costs of exits
1 Thread 4 ThreadsREAD 5.5x 7xWRITE 3.5x 5.9x
Throughput speedup
22 May@Systor' 2017 Meni Orenbach, Technion 66
Software address translation offers new optimizations
● Customized page size● Customized eviction policy● Multi-enclave memory coordination● Write-back only dirty pages● Sub-page direct access to backing store
22 May@Systor' 2017 Meni Orenbach, Technion 67
Software address translation offers new optimizations
● Customized page size● Customized eviction policy● Multi-enclave memory coordination● Write-back only dirty pages● Sub-page direct access to backing store
Virtual Machineballooning
22 May@Systor' 2017 Meni Orenbach, Technion 68
Software address translation offers new optimizations
● Customized page size● Customized eviction policy● Multi-enclave memory coordination● Write-back only dirty pages● Sub-page direct access to backing store
Virtual Machineballooning
22 May@Systor' 2017 Meni Orenbach, Technion 69
● Background● Motivation● Overhead analysis● Eleos design● Evaluation
22 May@Systor' 2017 Meni Orenbach, Technion 70
Biometric Identity checking server
Face verificationserver
Workloadgenerator
?=
450MB DB(5X SGX mem)
+ ID
10Gb NIC
22 May@Systor' 2017 Meni Orenbach, Technion 71
1 2 40
0.5
1
1.5
2
2.5
3
3.5
Eleos Native
Server threads
Biometric Identity validating serverSpeedup compared to vanilla SGX
22 May@Systor' 2017 Meni Orenbach, Technion 72
1 2 40
0.5
1
1.5
2
2.5
3
3.5
Eleos Native
Server threads
Biometric Identity validating serverSpeedup compared to vanilla SGX
22 May@Systor' 2017 Meni Orenbach, Technion 73
1 2 40
0.5
1
1.5
2
2.5
3
3.5
Eleos Native
Server threads
Biometric Identity validating serverSpeedup compared to vanilla SGX
Eleos scales better than vanilla-SGX:Saves inter-processor-interrupts
22 May@Systor' 2017 Meni Orenbach, Technion 74
1 2 40
0.5
1
1.5
2
2.5
3
3.5
Eleos Native
Server threads
Biometric Identity validating serverSpeedup compared to vanilla SGX
Eleos scales better than vanilla-SGX:Saves inter-processor-interrupts
Saturate 10Gb network
22 May@Systor' 2017 Meni Orenbach, Technion 75
Memcached
WorkloadGenerator(memaslap)
GET( )
~75 LOC modificationfor SUVM
MemcachedGraphene LibOS [Eurosys'2014]
500MB DB(5.5X SGX mem)
10Gb NIC
22 May@Systor' 2017 Meni Orenbach, Technion 76
1 Thread 4 Threads0
0.5
1
1.5
2
2.5
3
Eleos (500MB DB) vanilla SGX (20MB DB)
Server threads
MemcachedSpeedup compared to vanilla SGX (500 MB)
No SGX Faults
No SGX Faults
22 May@Systor' 2017 Meni Orenbach, Technion 77
1 Thread 4 Threads0
0.5
1
1.5
2
2.5
3
Eleos (500MB DB) vanilla SGX (20MB DB)
Server threads
MemcachedSpeedup compared to vanilla SGX (500 MB)
Disclaimer: Eleos+Graphene is 3x slower than native
No SGX Faults
No SGX Faults
22 May@Systor' 2017 Meni Orenbach, Technion 78
Take aways
● Eleos eliminates enclave exits costs● Eleos available for Windows and Linux
– Makes memory demanding applications available on Windows today
● Eleos takes a modularize approach– Memory demanding app? Link to SUVM
– I/O intensive app? Link to RPC
– Maintaining small TCB
22 May@Systor' 2017 Meni Orenbach, Technion 79
Traditional SGX:Host-centric OS services
Enclave
Operating System
22 May@Systor' 2017 Meni Orenbach, Technion 80
Traditional SGX:Host-centric OS services
Enclave
Operating System
Getdata
22 May@Systor' 2017 Meni Orenbach, Technion 81
Traditional SGX:Host-centric OS services
Enclave
Operating System
Getdata
DataUnavailable
22 May@Systor' 2017 Meni Orenbach, Technion 82
Traditional SGX:Host-centric OS services
Enclave
Operating System
Fetch data
Getdata
DataUnavailable
22 May@Systor' 2017 Meni Orenbach, Technion 83
Traditional SGX:Host-centric OS services
Enclave
Operating System
Fetch data
Getdata
DataUnavailable
22 May@Systor' 2017 Meni Orenbach, Technion 84
Eleos Insight:Enclave-centric OS services
Enclave
Getdata
Fetch data
In-enclaveServices
22 May@Systor' 2017 Meni Orenbach, Technion 85
Take aways (2)
● Eleos adapts 'accelerator-centric management'– System calls: GPUfs [ASPLOS'13], GPUnet [OSDI'14]
– Virtual memory: ActivePointers [ISCA'16]
● We can do more!– Asynchronous DMA host copies
– Non-blocking enclave launches
More information at:
“SGX Enclaves as Accelerators" [Systex'16]
22 May@Systor' 2017 Meni Orenbach, Technion 86
Thank you
Code is available at:https://github.com/acsl-technion/eleos
22 May@Systor' 2017 Meni Orenbach, Technion 87
Backup slides