ois2019 ipv6 docker€¦ · aws: step by step 1. design an ipv6 address plan 2. create an elastic...

Nicolas Leiva (@nleiv4)Solutions Architect

Why & How

Running IPv6-enabled Containers in the cloud

April 30, 2019

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Agenda


Agenda

• Intro

• Container networking basics• How can IPv6 help

• What can we do today in the cloud with IPv6

• Conclusion & Links


Container Networking basics


Container Networking IPv4

• Linux Bridge

• Private IP address space

• Network Address Translation


Connecting a Container

• Container• namespace: Isolate system resources• cgroup: Limits the resource usage


Multiple Containers per node

• You can run as many as you want depending on the resources in the host• Specify resource constraints!

packet


Multiple Containers

• Pool of machines (cluster)

• Orchestrator • Service lifecycle• Monitoring• Handle failure scenarios

• Kubernetes, Docker Swarm, Cloud Foundry, Nomad, Mesos, Nomad…

packet


Kubernetes

• Most popular open-source container orchestration system

• What about networking?. A Big LAN?


Kubernetes Networking

• All containers can communicate with all other containers without NAT

• All nodes can communicate with all containers (and vice-versa) without NAT

• The IP that a container sees itself as is the same IP that others see it as


Where do you run a Kubernetes Cluster?

• On-prem, Cloud Provider, etc.

• One environment is not too hard to master, however in distributed architectures the interconnection or the network becomes crucial (a pain point).


Interconnecting two clusters

• Interconnect private IPv4 islands between two different Cloud Providers over the Internet

packet



• Interconnect private IPv4 islands between two different Cloud Providers over a private network

packet



• Consider failure scenarios, add redundancy, bandwidth requirements.


Interconnecting three clusters

• If we add another cluster


Interconnecting four clusters

• And another.. Hub & Spoke


Interconnecting four clusters

• Or Full mesh -> N*(N-1)/2 -> (4 * 3)/2 = 6 links


How can IPv6 help


IPv4 intro

• The Internet protocol (IP) was defined in 1981• Internet addresses -> fixed length of four octets (32 bits)• Fourth version of the protocol -> IP version 4 (IPv4)

• In 1992, it became evident that we would eventually run out of IPv4 addresses• 1994: Re-usable private IP addresses • Network Address Translation (NAT): Translate private to a—public—IP address that is globally

unique

203.0.113.1/24


IPv4 in numbers

• The total number of public IPv4 addresses is ~3.7 billion• That less than the world population (~7.7 billion)• ~3.2 billion people will be online by the end of 2019

• The price of each IPv4 address is exceeding $20 nowadays


IPv6 intro

• In 1995, a new version of the Internet Protocol came out (expanded addressing capabilities)• IPv6 increases the IP address size from 32 bits to 128 bits• Lots of available public IPv6 address

• The problem? IPv6 is not backwards compatible with IPv4. Really slow transition• Over 20 years now with a current adoption of ~28% (*)

2001:db8::f00d/24

(*) https://www.google.com/intl/en/ipv6/statistics.html


IPv6 in numbers

• The total number of global IPv6 addresses is 2^125 • 42,535,295,865,117,307,932,921,825,928,971,026,432

• We could assign a—public—IPv6 address to EVERY ATOM ON THE SURFACE OF THE EARTH, and still have enough addresses left to do another 100+ earths (*)

billion

(*) https://itknowledgeexchange.techtarget.com/whatis/ipv6-addresses-how-many-is-that-in-numbers/


Kubernetes multi-cluster Networking, IPv6

• If we had infinite global IP addresses, we could forget about VPN’s (you can still do it at host level)


Security concerns?

• NAT does not block packets. • It does however hide internal addressing at the cost of keeping the state of the translation

somewhere else.


IPv6 Subnetting

• A Cloud Provider will assign you a /56 IPv6 prefix per VPC• You can breakdown this into smaller prefixes• -> 256 /64 networks

2001:db8:f00d:aa00::/56

2001:db8:f00d:aa00::/64

2001:db8:f00d:aa01::/64

2001:db8:f00d:aaff:/64

2001:db8:f00d:aa02::/64

2001:db8:f00d:aa03::/64

…

1

2

3

4

…

256


Addressing schema in a VPC

2001:db8:f00d:aa00::/56

2001:db8:f00d:aa00::/64

2001:db8:f00d:aa01::/64

2001:db8:f00d:aa02:/64

2001:db8:f00d:aa03::/64

2001:db8:f00d:aa04::/64

2001:db8:f00d:aa05::/64


What can we do today in the cloud with IPv6


IPv6 infrastructure support

Assign one or more global IPv6 addresses to an instance

Apply IPv6 Security policies

However, NO IPv6 Subnet Routing to an Instance (subnetting)• Alternative?: AWS Elastic Network Interfaces• Not exactly what I want, but is as good as it gets


VM

AWS: Step by Step

1. Design an IPv6 address plan

2. Create an Elastic Network Interface (ENI)

3. Create an EC2 instances with an ENI attached to it

4. Re-configure IPv6 addressing on the instance

5. Check we can reach the Internet over IPv6

6. Upgrade OS packages

7. Install Docker

8. Configure Docker to allocate IPv6 addresses

9. Run a couple of Containers using only IPv6

10. Test connectivity between containers

ENI

2600:1f18:47b::8/126

Container Container


1. Design an IPv6 addressing plan

• 1 IPv6 address for the node

• 4 IPv6 addresses for container allocation2600:1f18:47b::1:1

2600:1f18:47b::82600:1f18:47b::92600:1f18:47b::a2600:1f18:47b::b

2600:1f18:47b::8/126



• An ENI represents a virtual network card2600:1f18:47b::1:1

2600:1f18:47b::82600:1f18:47b::92600:1f18:47b::a2600:1f18:47b::b

ENI



• An ENI represents a virtual network card

eni1=`aws ec2 create-network-interface \--subnet-id $subnetId \--description "My IPv6 ENI 1" \--groups $sgId \--ipv6-addresses \Ipv6Address=2600:1f18:47b::1:1 \Ipv6Address=2600:1f18:47b::8 \Ipv6Address=2600:1f18:47b::9 \Ipv6Address=2600:1f18:47b::a \Ipv6Address=2600:1f18:47b::b \--query 'NetworkInterface.NetworkInterfaceId' \--output text`


3. Create an EC2 instances with an ENI attached

• Addresses are automagically routed to your instance

• The number of IP addresses you can assign to an instance is restricted by its type (max 50)


3. Create an EC2 instances with an ENI attached

• Attach the ENI we previously created, whose ID was stored in $eni1. • We keep the instance ID we receive back from AWS in $vm1

vm1=`aws ec2 run-instances \--key-name $AWS_SSH_KEY \--image-id ami-0ac019f4fcb7cb7e6 \--instance-type r5d.large \--network-interfaces DeviceIndex=0,NetworkInterfaceId=$eni1 \--query 'Instances[0].InstanceId' \--output text`


4. Re-configure IPv6 addressing on the instance

• Use Netplan if using Ubuntu 18.04• Netplan is a YAML network configuration abstraction.

network:version: 2ethernets:

ens5:dhcp6: noaccept-ra: noaddresses:- 2600:1f18:47b::1:1/64gateway6: fe80::1066:30ff:feb8:c008

$ sudo netplan --debug apply

/etc/netplan/50-cloud-init.yaml


5. Check we can reach the Internet over IPv6

• Inside the container• ping6 2600 ::• ping6 ipv6-test.com -c 1• ...

Full Outputs

root@d7c9480161f9:/# ping6 ipv6-test.com -c 1

PING ipv6-test.com(agaric.t0x.net (2001:41d0:8:e8ad::1)) 56 data bytes

64 bytes from agaric.t0x.net (2001:41d0:8:e8ad::1): icmp_seq=1 ttl=46 time=78.7 ms

--- ipv6-test.com ping statistics ---

1 packets transmitted, 1 received, 0% packet loss, time 0ms

rtt min/avg/max/mdev = 78.788/78.788/78.788/0.000 ms


6. Upgrade OS packages

• IPv6-only environment• Default package repository is not IPv6 friendly• To update packages modify your sources.list file

$ sudo apt-get -o Acquire::ForceIPv6=true updateGet:1 http://archive.ubuntu.com/ubuntu bionic InRelease [242 kB]Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [83.2 kB]...Get:40 http://archive.ubuntu.com/ubuntu bionic-backports/universe Translation-en [1604 B]Fetched 28.4 MB in 5s (5363 kB/s)Reading package lists... Done

us-east-1.ec2.archive.ubuntu.com

us-east-1.ec2.archive.ubuntu.com


7. Install Docker

• IPv6-only environment

$ sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"$ sudo apt-get -o Acquire::ForceIPv6=true install –y docker-ce

download.docker.com


8. Configure Docker to allocate IPv6 addresses

• Configure IPv6 address allocation on Docker’s bridge (docker0)

{"ipv6": true,"fixed-cidr-v6": "2600:1f18:47b:ca03::8/126"

}

$ sudo systemctl restart docker

/etc/docker/daemon.json


registry-1.docker.io

9. Run a couple of Containers using only IPv6

• Making the instances IPv6-friendly• Container registry: Google Container Registry

$ docker run -it --rm gcr.io/gcp-runtimes/ubuntu_18_0_4:latest bashlatest: Pulling from gcp-runtimes/ubuntu_18_0_4deabf7bad5e7: Pull complete…Digest: sha256:af51882c2cb15cb3ed133ac62debb744057e02d6dee8db25a54caac158be2a3cStatus: Downloaded newer image for gcr.io/gcp-runtimes/ubuntu_18_0_4:latestroot@bf2f00033d64:/#

hub.docker.com

$ docker run ubuntu


10. Test connectivity between containers

• Ping6 between containers

Full Outputs


Conclusion & Links


Blog posts

• Kubernetes Networking: Behind the sceneshttps://itnext.io/kubernetes-networking-behind-the-scenes-39a1ab1792bb

• Kubernetes multi-cluster networking made simple https://itnext.io/kubernetes-multi-cluster-networking-made-simple-c8f26827813

• How to run IPv6-enabled Docker containers on AWS https://medium.freecodecamp.org/how-to-run-ipv6-enabled-docker-containers-on-aws-87e090ab0397

https://itnext.io/kubernetes-networking-behind-the-scenes-39a1ab1792bb

https://itnext.io/kubernetes-multi-cluster-networking-made-simple-c8f26827813

https://medium.freecodecamp.org/how-to-run-ipv6-enabled-docker-containers-on-aws-87e090ab0397


Docker caveats

• Still need to plug this to Kubernetes

• IPv6 is disabled on containers in some Docker versions

• “The subnet for Docker containers should at least have a size of /80, so that an IPv6 address can end with the container’s MAC address and you prevent NDP neighbor cache invalidation issues in the Docker layer”

• …

ois2019 ipv6 docker€¦ · aws: step by step 1. design an ipv6 address plan 2. create an elastic...

Documents