safebricks: shielding network functions in the cloud · nf tls ipsec tls safebricks also supports...

SafeBricks: Shielding Network Functions in the Cloud

Rishabh Poddar, Chang Lan, Raluca Ada Popa, Sylvia Ratnasamy

UC Berkeley

Network Functions (NFs) in the cloud

�2

DestinationEnterprise

Clients

Network Functions (NFs) in the cloud

�3

DestinationEnterprise

Clients

Network Functions (NFs) in the cloud

�4

NF providersNF providersNF providersNF providers

Enterprise

Clients

Destination

Problem: Security

�5

NF providersNF providersNF providersNF providers

Enterprise

Clients

Destination

Email

�6

NF providersNF providersNF providersNF providers

Enterprise

Clients

Destination

Email

Need to protect traffic from the cloud provider1

Hackers /curious employees

Problem: Security

Problem: Security

�7

NF providersNF providersNF providersNF providers

Enterprise

Clients

Destination

Email

Need to protect traffic from the NF providers

Exfiltration

2

Problem: Security

�8

NF providersNF providersNF providersNF providers

Enterprise

Clients

Destination

Email

Need to protect NF code and rulesets from client enterprise and cloud3

Cryptographic solutions do not suffice

�9

NF providersNF providersNF providersNF providers

Enterprise

Clients

Destination

Email

Cryptographic solutions do not suffice

�10

NF providersNF providersNF providersNF providers

Enterprise

Clients

Destination

Email

Standard encryption: e.g. end-to-end TLS1

Cryptographic solutions do not suffice

�11

NF providersNF providersNF providersNF providers

Enterprise

Clients

Destination

Email

Standard encryption: e.g. end-to-end TLS • Functionality: Doesn’t allow any computation on encrypted payload

?

1

Cryptographic solutions do not suffice

�12

NF providersNF providersNF providersNF providers

Enterprise

Clients

Destination

Email

Standard encryption: e.g. end-to-end TLS • Functionality: Doesn’t allow any computation on encrypted payload • Security: Unencrypted fields (e.g. IP headers) still leak information

?

1

Cryptographic solutions do not suffice

�13

Specialized encryption: e.g. BlindBox, Embark [Sherry et al. (SIGCOMM’15)]

[Lan et al. (NSDI’16)]

2

Cryptographic solutions do not suffice

�14

Specialized encryption: e.g. BlindBox, Embark • Too limited in functionality!

2

• Header-based comparisons

• Keyword matching

• Regular expressions

• Cross-flow analysis

• Statistical computations

How to achieve full functionality and

our security goals simultaneously?

�15

SafeBricks

�16

2

3

Protects traffic from the NF providers

Protects NF source code and rulesets from client enterprise and cloud

Protects traffic from the cloud provider1

SafeBricks

�17

2

3

Protects traffic from the NF providers

Protects traffic from the cloud provider1

Hardware enclaves + language-based isolation

Protects NF source code and rulesets from client enterprise and cloud

• Secure region of memory (enclaves) protected by hardware

Background: Hardware enclaves (e.g. Intel SGX)

�18

Operating System (untrusted)

Application (untrusted)

Enclave (trusted)

• Secure region of memory (enclaves) protected by hardware

�19

Operating System (untrusted)

Application (untrusted)

Enclave (trusted)

Secretdata

Trustedcode

Background: Hardware enclaves (e.g. Intel SGX)

• Secure region of memory (enclaves) protected by hardware

�20

Operating System (untrusted)

Application (untrusted)

Enclave (trusted)

Secretdata

Trustedcode

Client

• Remote attestation by clients

Background: Hardware enclaves (e.g. Intel SGX)

• Secure region of memory (enclaves) protected by hardware

�21

Operating System (untrusted)

Application (untrusted)

Enclave (trusted)

Secretdata

Trustedcode

Client

• Remote attestation by clients • Remotely verify enclave contents

Background: Hardware enclaves (e.g. Intel SGX)

• Secure region of memory (enclaves) protected by hardware

�22

Operating System (untrusted)

Application (untrusted)

Enclave (trusted)

Secretdata

Trustedcode

Client

• Remote attestation by clients • Remotely verify enclave contents • Establish a secure channel with enclave

Background: Hardware enclaves (e.g. Intel SGX)

Background: NetBricks

�23

[Panda et al. (OSDI’16)]

I/O interface

Poll for I/O

Programming abstractions

State abstractions

Scheduler

DPDK

NICs

NetBricks• Framework for developing arbitrary NFs

�24

[Panda et al. (OSDI’16)]

• Framework for developing arbitrary NFs • MapReduce like programming

abstractions (operators) for packet processing

I/O interface

Poll for I/O

Programming abstractions

State abstractions

Scheduler

DPDK

NICs

NetBricks

Background: NetBricks

�25

[Panda et al. (OSDI’16)]

• Framework for developing arbitrary NFs • MapReduce like programming

abstractions (operators) for packet processing

• NFs represented as a directed graph with operators as nodes

I/O interface

Poll for I/O

Programming abstractions

State abstractions

Scheduler

DPDK

NICs

NetBricks

Background: NetBricks

�26

• Written in Rust

• Fast, safe, zero-copy semantics

• Isolates NFs deployed in a chain while running them in the same address space

NICs

NetBricks

NF 1 NF 1

Background: NetBricks

SafeBricks

�27

Protects traffic from the NF providers

Protects traffic from the cloud provider

2

3

1

Protects NF source code and rulesets from client enterprise and cloud

SafeBricks

�28

Protects traffic from the NF providers

Protects traffic from the cloud provider

2

3

1

Protects NF source code and rulesets from client enterprise and cloud

Outsourcing NFs using hardware enclaves

�29 Enterprise

Clients

Gateway

Enclave

OS (untrusted)

Cloud (untrusted)

Destination

Outsourcing NFs using hardware enclaves

�30 Enterprise

Clients

Gateway

Enclave

OS (untrusted)

Cloud (untrusted)

Destination

NF

Outsourcing NFs using hardware enclaves

�31 Enterprise

Clients

Gateway

Enclave

OS (untrusted)

Cloud (untrusted)

Destination

NF

Outsourcing NFs using hardware enclaves

�32 Enterprise

Clients

Gateway

Enclave

OS (untrusted)

Cloud (untrusted)

Destination

NF

IPSec

IPSec

Outsourcing NFs using hardware enclaves

�33 Enterprise

Clients

Gateway

Enclave

OS (untrusted)

Cloud (untrusted)

Destination

NF

IPSec

IPSec

TLS

Interception proxy

Outsourcing NFs using hardware enclaves

�34 Enterprise

Clients

Gateway

Enclave

OS (untrusted)

Cloud (untrusted)

Destination

NF

TLS

IPSec

TLS

IPSec

Outsourcing NFs using hardware enclaves

�35 Enterprise

Clients

Gateway

Enclave

OS (untrusted)

Cloud (untrusted)

Destination

NF

TLSTLSTLSTLS

IPSecIPSec

TLS

IPSec

Outsourcing NFs using hardware enclaves

�36 Enterprise

Clients

Gateway

Enclave

OS (untrusted)

Cloud (untrusted)

Destination

NF

TLSTLSTLSTLS

IPSecIPSec

TLS

IPSec

Packet headers also encrypted

Outsourcing NFs using hardware enclaves

�37 Enterprise

Clients

Gateway

Enclave

OS (untrusted)

Cloud (untrusted)

Destination

NF

TLSTLSTLSTLS

IPSec

TLS

IPSec

TLS

IPSec

Outsourcing NFs using hardware enclaves

�38 Enterprise

Clients

Gateway

Enclave

OS (untrusted)

Cloud (untrusted)

Destination

NF

TLSTLSTLSTLS

IPSec

TLS

SafeBricks also supports “direct” delivery of traffic

IPSec

Outsourcing NFs using hardware enclaves

�39 Enterprise

Clients

Gateway

Enclave

OS (untrusted)

Cloud (untrusted)

Destination

NF

TLSTLSTLSTLS

IPSec

TLS

Can use general purpose frameworks,

e.g. Haven, Scone

IPSec

TLS

IPSec

Challenges

�40

Small trusted computing base (TCB) — enclave should contain minimal amount of code

1

Challenges

�41

High performance — Transitioning into / out of enclaves is expensive!

Small trusted computing base (TCB) — enclave should contain minimal amount of code

2

1

Challenges

�42

High performance — Transitioning into / out of enclaves is expensive!

Illegal enclave instructions — SGX does not support system calls or instructions that may lead to a VMEXIT

Small trusted computing base (TCB) — enclave should contain minimal amount of code

2

3

1

Challenges

�43

2

3

High performance — Transitioning into / out of enclaves is expensive!

Illegal enclave instructions — SGX does not support system calls or instructions that lead to a VMEXIT

Small trusted computing base (TCB) — enclave should contain minimal amount of code

1

�44

I/O interface

Poll for I/O

Programming abstractions

State abstractions

Scheduler

DPDK

NICs

NetBricks1

�45

I/O interface

Poll for I/O

Programming abstractions

State abstractions

Scheduler

DPDK

NICs

Enclave1

• Maximal TCB: NetBricks stack entirely within enclave

�46

I/O interface

Poll for I/O

Programming abstractions

State abstractions

Scheduler

DPDK

NICs

Enclave1

• Minimal TCB: Only security-critical components within enclave

• One enclave transition per node per packet batch

• Intermediate TCB • One enclave transition per

packet batch

�47

I/O interface

Poll for I/O

Programming abstractions

State abstractions

Scheduler

DPDK

NICs

Enclave1

�48

I/O interface

Poll for I/O

Programming abstractions

State abstractions

Scheduler

DPDK

NICs

Glue code (trusted)

Glue code (untrusted)

SafeBricksenclave(trusted)

SafeBrickshost

(untrusted)

1

• Partitioned NetBricks framework; glue code connects trusted and untrusted code

�49

I/O interface

Poll for I/O

Programming abstractions

State abstractions

Scheduler

DPDK

NICs

Glue code (trusted)

Glue code (untrusted)

SafeBricksenclave(trusted)

SafeBrickshost

(untrusted)

1

• Partitioned NetBricks framework; glue code connects trusted and untrusted code

Two new operators for packet transfer to/from enclave:

and toHosttoEnclave

Challenges

�50

2

3

High performance — Transitioning into / out of enclaves is expensive!

Small trusted computing base (TCB) — enclave should contain minimal amount of code

1

Illegal enclave instructions — SGX does not support system calls or instructions that lead to a VMEXIT

SafeBricks host

SafeBricks enclave

�51

NICs

2

toHosttoEnclave

NF

• One enclave transition per packet batch

SafeBricks host

SafeBricks enclave

�52 NICs

2

toHosttoEnclave

NF

• Shared queues in non-enclave heap

• Separate enclave and host threads

• Access queues without exiting enclave — zero enclave transitions

send recv

Enclave I/O

Host I/O

Challenges

�53

2

3

High performance — Transitioning into / out of enclaves is expensive!

Small trusted computing base (TCB) — enclave should contain minimal amount of code

1

Illegal enclave instructions — SGX does not support system calls or instructions that lead to a VMEXIT

�54

3

Observation: NFs in general do not require support for system calls / instructions that lead to VMEXITs

�55

3

Observation: NFs in general do not require support for system calls / instructions that lead to VMEXITs, except:

• Logging

• Timestamps (using rdtsc)

�56

Observation: NFs in general do not require support for system calls / instructions that lead to VMEXITs, except:

• Logging

• Timestamps (using rdtsc)

3

SafeBricks designs custom solutions for these operations without enclave transitions

�57

2

3

1

SafeBricks

Protects traffic from the NF providers

Protects traffic from the cloud provider

Protects NF source code and rulesets from client enterprise and cloud

Problem: Malicious NFs within enclaves

�58

Malicious NFs inside the enclave can exfiltrate or tamper with packets!!

Problem: Malicious NFs within enclaves

�59

Observation: NFs typically need access only to specific packet fields

• E.g. Firewall needs read-only access to TCP/IP headers

• E.g. NAT needs both read-write access to headers but not to packet payload

Malicious NFs inside the enclave can exfiltrate or tamper with packets!!

Problem: Malicious NFs within enclaves

�60

Observation: NFs typically need access only to specific packet fields

• E.g. Firewall needs read-only access to TCP/IP headers

• E.g. NAT needs both read-write access to headers but not to packet payload

Malicious NFs inside the enclave can exfiltrate or tamper with packets!!

IP addresses; TCP ports; HTTP

payload

Problem: Malicious NFs within enclaves

�61

Observation: NFs typically need access only to specific packet fields

• E.g. Firewall needs read-only access to TCP/IP headers

• E.g. NAT needs both read-write access to headers but not to packet payload

Malicious NFs inside the enclave can exfiltrate or tamper with packets!!

SafeBricks enforces least privilege across NFs within the enclave

toEnclave toHost

Host

�62

Run NFs within the same enclave

Least privilege enforcement

Firewall NAT

�63

Run NFs within the same enclaveFirewall

toEnclave

NAT

toHost

Host

Least privilege enforcement

�64

Run NFs within the same enclave

• Stitch NFs together interspersed with an operator ( wList ) that embeds a vector of permissions in packets — two bits per packet field

Firewall

toEnclave

NAT

toHost

Host

wList wList

wList

Least privilege enforcement

�65

Enforce permissions by mediating access to

packets using Rust’s ownership model

SafeBricks Controller

Least privilege enforcement

�66

Enforce permissions by mediating access to

packets using Rust’s ownership model

• Controller module holds ownership of packet buffers

SafeBricks Controller

Packet buffer

Least privilege enforcement

�67

Enforce permissions by mediating access to

packets using Rust’s ownership model

• Controller module holds ownership of packet buffers

• NFs borrow references to packet fields from the Controller, which checks permissions vector in packet

SafeBricks Controller

NAT Firewall

Packet buffer

Least privilege enforcement

�68

Enforce permissions by mediating access to

packets using Rust’s ownership model

• Controller module holds ownership of packet buffers

• NFs borrow references to packet fields from the Controller, which checks permissions vector in packet

SafeBricks Controller

Packet buffer

NAT Firewall

Returns an immutable reference for read-only access, and a mutable reference for

write access

Least privilege enforcement

Assumption: Trusted compilation of NFs

�69

Least privilege guarantees only hold if NFs are built using a compiler that prohibits unsafe operations!

Assumption: Trusted compilation of NFs

�70

Least privilege guarantees only hold if NFs are built using a compiler that prohibits unsafe operations!

E.g. Check array bounds, no pointer arithmetic, no

unsafe type casts

Assumption: Trusted compilation of NFs

�71

Least privilege guarantees only hold if NFs are built using a compiler that prohibits unsafe operations!

• Possible solution: Client obtains NF source codes from providers and assembles them locally

NF providersNF providersNF providersNF providers

Client Enterprise

NF code + rulesets

Assumption: Trusted compilation of NFs

�72

Least privilege guarantees only hold if NFs are built using a compiler that prohibits unsafe operations!

• Possible solution: Client obtains NF source codes from providers and assembles them locally

• Problem: This violates the confidentiality of NF source code!

NF providersNF providersNF providersNF providers

Enterprise

NF code + rulesets

�73

2

3

1

SafeBricks

Protects traffic from the NF providers

Protects traffic from the cloud provider

Protects NF source code and rulesets from client enterprise and cloud

Assembling NFs

�74

• Key idea: Build NFs within a special “meta”-enclave in the cloud using an agreed upon compiler

Assembling NFs

�75

• Key idea: Build NFs within a special “meta”-enclave in the cloud using an agreed upon compiler

• Both client and NF providers can verify the agreed upon compiler using remote attestation

Assembling NFs

�76

Enterprise

NF providersNF providersNF providersNF providers

Loader Compiler

Assembly enclave

Assembling NFs

�77

Enterprise

NF providersNF providersNF providersNF providers

Loader Compiler

Assembly enclave

Remote attestation

Assembling NFs

�78

Enterprise

NF providersNF providersNF providersNF providers

Loader Compiler

Assembly enclave

Remote attestation

Assembling NFs

�79

Enterprise

NF providersNF providersNF providersNF providers

Loader Compiler

Assembly enclave

Remote attestation

Assembling NFs

�80

Enterprise

NF providersNF providersNF providersNF providers

Loader Compiler

Assembly enclave

NF code + rulesets

Assembling NFs

�81

Enterprise

NF providersNF providersNF providersNF providers

Loader Compiler

Assembly enclave

NF code + rulesets

Config

Assembling NFs

�82

Enterprise

NF providersNF providersNF providersNF providers

Loader Compiler

Assembly enclave

NF code + rulesets

Config

Placement of NFs, least privilege policies per NF

Assembling NFs

�83

Enterprise

NF providersNF providersNF providersNF providers

Loader Compiler

Assembly enclave

NF code + rulesets

Config

Assembling NFs

�84

Enterprise

NF providersNF providersNF providersNF providers

Loader Compiler

Assembly enclave

Deployment enclave

SafeBricks

�85

2

3

Protects traffic from the NF providers

Protects NF source code and rulesets from client enterprise and cloud

Protects traffic from the cloud provider1

Performance

�86

Throughput decline across NFs

�87~0–15% overhead across applications for different packet sizes

Thro

ughp

ut d

eclin

e

0%

10%

20%

30%

40%

Packet size (bytes)64 256 512 1024

Firewall NATLoad Balancer DPI

DPI performance with increasing no. of rules

�88 Overhead spikes when NF working set exceeds enclave memory

Thro

ughp

ut d

eclin

e

0%

25%

50%

75%

100%

Number of rules

0 500010000

1500020000

25000

94MB64MB8MB

DPI performance with increasing no. of rules

�89 Overhead spikes when NF working set exceeds enclave memory

Thro

ughp

ut d

eclin

e

0%

25%

50%

75%

100%

Number of rules

0 500010000

1500020000

25000

94MB64MB8MB

Not a fundamental limitation

SafeBricks uses a combination of hardware enclaves

and language-based isolation to:

• Protect client traffic from the cloud provider

• Enforce least privilege across NFs

• Protect the confidentiality of NF code and rulesets

�90

Summary

Modest overhead across a range of applications

rishabhp@eecs.berkeley.edu