titus aws vpc networking for containers

●○○

●○○○

1

●●

○○

●○

●○

2

●○○○○

●●

○■

3

●

○●

○○

■■

4

●○○

●○

■○○

5

●●●●●

○●

6

7

Titusmesos

executor

Titus network driver

Docker engine

EC2 Instance

In: Network params

Out: Network pod rootNew task

Container create/start--net=container:<pod id>

Task Status

create/start pod container

●○

●

○●

○○

●8

9

Create

NS Configurator

IP Allocator

NS AllocatorHttp

IP + params

NS ref

Configured network ns (pod root)

params

Container id

●●●●●

○●●

○10

EC2 Instance eth1

ENI1SecGrp=A

eth2

ENI2SecGrp=X

eth3

ENI3SecGrp=Y,Z

IP 2 (primary)IP 3

IP 6 (primary)IP 1 (primary)

IP 4IP 5

IP 7IP 8

12

●○

●●

○●●

○

13

●○

●●●

○

○●

14

●○

●○

●○

15

●○

●●

○●

○●

16

No IP, SecGrp A

Task 0

SecGrp Y,Z

Task 1 Task 2 Task 3

Titus EC2 Host VMeth1

ENI1SecGrp=A

eth2

ENI2SecGrp=X

eth3

ENI3SecGrp=Y,Z

IP 1IP 2

IP 3

pod root

veth<id>

app

SecGrp X

pod root

veth<id>

app

SecGrp X

pod root

veth<id>

appapp

veth<id>

Linux Policy BasedRouting + Traffic Control

TitusEC2

Metadata Proxy

169.254.169.254IPTables NAT (*)

* **

169.254.169.254Non-routable IP

*

●○○

●

18

●● <IP>/32

○ via eth0

●

●

19

● Container IP: 100.66.23.19● Container Device: vethA

● Eni IP: 100.66.30.31/20● Eni GW: 100.66.16.1● Eni Device: eth1● Routing tables:

○ tocontainer, fromcontainer

20

# ip addr show eth0

eth0: … mtu 1500 qdisc tbf state UP group default

inet 100.66.23.19/32 ...

# ip route show

default via 100.66.30.31 dev eth0

100.66.30.31 dev eth0 scope link

21

# ip route show | grep eth1

100.66.16.0/20 dev eth1 proto kernel scope link src 100.66.30.31

# ip rule show | grep 100.66.23.19

from all to 100.66.23.19 iif eth1 lookup tocontainer

from 100.66.23.19 iif vethA lookup fromcontainer

# ip route show table tocontainer | grep 100.66.23.19

100.66.23.19 dev vethA scope link

# ip route show table fromcontainer

default via 100.66.16.1 dev eth1

22

●●●●

23

●●

○●

○

●

24

●○○

○○○

●○

25

26

27

28

●●●

○●

○○

29