ois2019 ipv6 docker€¦ · aws: step by step 1. design an ipv6 address plan 2. create an elastic...

Nicolas Leiva (@nleiv4)Solutions Architect

Why & How

Running IPv6-enabled Containers in the cloud

April 30, 2019

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Agenda

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Agenda

• Intro

• Container networking basics• How can IPv6 help

• What can we do today in the cloud with IPv6

• Conclusion & Links

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Container Networking basics

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Container Networking IPv4

• Linux Bridge

• Private IP address space

• Network Address Translation

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Container Networking IPv4

• Linux Bridge

• Private IP address space

• Network Address Translation

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Container Networking IPv4

• Linux Bridge

• Private IP address space

• Network Address Translation

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Connecting a Container

• Container• namespace: Isolate system resources• cgroup: Limits the resource usage

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Multiple Containers per node

• You can run as many as you want depending on the resources in the host• Specify resource constraints!

packet

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Multiple Containers

• Pool of machines (cluster)

• Orchestrator • Service lifecycle• Monitoring• Handle failure scenarios

• Kubernetes, Docker Swarm, Cloud Foundry, Nomad, Mesos, Nomad…

packet

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Kubernetes

• Most popular open-source container orchestration system

• What about networking?. A Big LAN?

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Kubernetes Networking

• All containers can communicate with all other containers without NAT

• All nodes can communicate with all containers (and vice-versa) without NAT

• The IP that a container sees itself as is the same IP that others see it as

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Where do you run a Kubernetes Cluster?

• On-prem, Cloud Provider, etc.

• One environment is not too hard to master, however in distributed architectures the interconnection or the network becomes crucial (a pain point).

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Interconnecting two clusters

• Interconnect private IPv4 islands between two different Cloud Providers over the Internet

packet

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Interconnecting two clusters

• Interconnect private IPv4 islands between two different Cloud Providers over a private network

packet

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Interconnecting two clusters

• Consider failure scenarios, add redundancy, bandwidth requirements.

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Interconnecting three clusters

• If we add another cluster

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Interconnecting four clusters

• And another.. Hub & Spoke

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Interconnecting four clusters

• Or Full mesh -> N*(N-1)/2 -> (4 * 3)/2 = 6 links

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

How can IPv6 help

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

IPv4 intro

• The Internet protocol (IP) was defined in 1981• Internet addresses -> fixed length of four octets (32 bits)• Fourth version of the protocol -> IP version 4 (IPv4)

• In 1992, it became evident that we would eventually run out of IPv4 addresses• 1994: Re-usable private IP addresses • Network Address Translation (NAT): Translate private to a—public—IP address that is globally

unique

203.0.113.1/24

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

IPv4 in numbers

• The total number of public IPv4 addresses is ~3.7 billion• That less than the world population (~7.7 billion)• ~3.2 billion people will be online by the end of 2019

• The price of each IPv4 address is exceeding $20 nowadays

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

IPv6 intro

• In 1995, a new version of the Internet Protocol came out (expanded addressing capabilities)• IPv6 increases the IP address size from 32 bits to 128 bits• Lots of available public IPv6 address

• The problem? IPv6 is not backwards compatible with IPv4. Really slow transition• Over 20 years now with a current adoption of ~28% (*)

2001:db8::f00d/24

(*) https://www.google.com/intl/en/ipv6/statistics.html

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

IPv6 in numbers

• The total number of global IPv6 addresses is 2^125 • 42,535,295,865,117,307,932,921,825,928,971,026,432

• We could assign a—public—IPv6 address to EVERY ATOM ON THE SURFACE OF THE EARTH, and still have enough addresses left to do another 100+ earths (*)

billion

(*) https://itknowledgeexchange.techtarget.com/whatis/ipv6-addresses-how-many-is-that-in-numbers/

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Kubernetes multi-cluster Networking, IPv6

• If we had infinite global IP addresses, we could forget about VPN’s (you can still do it at host level)

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Security concerns?

• NAT does not block packets. • It does however hide internal addressing at the cost of keeping the state of the translation

somewhere else.

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

IPv6 Subnetting

• A Cloud Provider will assign you a /56 IPv6 prefix per VPC• You can breakdown this into smaller prefixes• -> 256 /64 networks

2001:db8:f00d:aa00::/56

2001:db8:f00d:aa00::/64

2001:db8:f00d:aa01::/64

2001:db8:f00d:aaff:/64

2001:db8:f00d:aa02::/64

2001:db8:f00d:aa03::/64

…

1

2

3

4

…

256

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Addressing schema in a VPC

2001:db8:f00d:aa00::/56

2001:db8:f00d:aa00::/64

2001:db8:f00d:aa01::/64

2001:db8:f00d:aa02:/64

2001:db8:f00d:aa03::/64

2001:db8:f00d:aa04::/64

2001:db8:f00d:aa05::/64

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

What can we do today in the cloud with IPv6

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

IPv6 infrastructure support

Assign one or more global IPv6 addresses to an instance

Apply IPv6 Security policies

However, NO IPv6 Subnet Routing to an Instance (subnetting)• Alternative?: AWS Elastic Network Interfaces• Not exactly what I want, but is as good as it gets

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

VM

AWS: Step by Step

1. Design an IPv6 address plan

2. Create an Elastic Network Interface (ENI)

3. Create an EC2 instances with an ENI attached to it

4. Re-configure IPv6 addressing on the instance

5. Check we can reach the Internet over IPv6

6. Upgrade OS packages

7. Install Docker

8. Configure Docker to allocate IPv6 addresses

9. Run a couple of Containers using only IPv6

10. Test connectivity between containers

ENI

2600:1f18:47b::8/126

Container Container

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

1. Design an IPv6 addressing plan

• 1 IPv6 address for the node

• 4 IPv6 addresses for container allocation2600:1f18:47b::1:1

2600:1f18:47b::82600:1f18:47b::92600:1f18:47b::a2600:1f18:47b::b

2600:1f18:47b::8/126

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

2. Create an Elastic Network Interface (ENI)

• An ENI represents a virtual network card2600:1f18:47b::1:1

2600:1f18:47b::82600:1f18:47b::92600:1f18:47b::a2600:1f18:47b::b

ENI

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

2. Create an Elastic Network Interface (ENI)

• An ENI represents a virtual network card

eni1=`aws ec2 create-network-interface \--subnet-id $subnetId \--description "My IPv6 ENI 1" \--groups $sgId \--ipv6-addresses \Ipv6Address=2600:1f18:47b::1:1 \Ipv6Address=2600:1f18:47b::8 \Ipv6Address=2600:1f18:47b::9 \Ipv6Address=2600:1f18:47b::a \Ipv6Address=2600:1f18:47b::b \--query 'NetworkInterface.NetworkInterfaceId' \--output text`

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

3. Create an EC2 instances with an ENI attached

• Addresses are automagically routed to your instance

• The number of IP addresses you can assign to an instance is restricted by its type (max 50)

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

3. Create an EC2 instances with an ENI attached

• Attach the ENI we previously created, whose ID was stored in $eni1. • We keep the instance ID we receive back from AWS in $vm1

vm1=`aws ec2 run-instances \--key-name $AWS_SSH_KEY \--image-id ami-0ac019f4fcb7cb7e6 \--instance-type r5d.large \--network-interfaces DeviceIndex=0,NetworkInterfaceId=$eni1 \--query 'Instances[0].InstanceId' \--output text`

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

4. Re-configure IPv6 addressing on the instance

• Use Netplan if using Ubuntu 18.04• Netplan is a YAML network configuration abstraction.

network:version: 2ethernets:

ens5:dhcp6: noaccept-ra: noaddresses:- 2600:1f18:47b::1:1/64gateway6: fe80::1066:30ff:feb8:c008

$ sudo netplan --debug apply

/etc/netplan/50-cloud-init.yaml

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

5. Check we can reach the Internet over IPv6

• Inside the container• ping6 2600 ::• ping6 ipv6-test.com -c 1• ...

Full Outputs

root@d7c9480161f9:/# ping6 ipv6-test.com -c 1

PING ipv6-test.com(agaric.t0x.net (2001:41d0:8:e8ad::1)) 56 data bytes

64 bytes from agaric.t0x.net (2001:41d0:8:e8ad::1): icmp_seq=1 ttl=46 time=78.7 ms

--- ipv6-test.com ping statistics ---

1 packets transmitted, 1 received, 0% packet loss, time 0ms

rtt min/avg/max/mdev = 78.788/78.788/78.788/0.000 ms

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

6. Upgrade OS packages

• IPv6-only environment• Default package repository is not IPv6 friendly• To update packages modify your sources.list file

$ sudo apt-get -o Acquire::ForceIPv6=true updateGet:1 http://archive.ubuntu.com/ubuntu bionic InRelease [242 kB]Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [83.2 kB]...Get:40 http://archive.ubuntu.com/ubuntu bionic-backports/universe Translation-en [1604 B]Fetched 28.4 MB in 5s (5363 kB/s)Reading package lists... Done

us-east-1.ec2.archive.ubuntu.com

us-east-1.ec2.archive.ubuntu.com

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

7. Install Docker

• IPv6-only environment

$ sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"$ sudo apt-get -o Acquire::ForceIPv6=true install –y docker-ce

download.docker.com

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

8. Configure Docker to allocate IPv6 addresses

• Configure IPv6 address allocation on Docker’s bridge (docker0)

{"ipv6": true,"fixed-cidr-v6": "2600:1f18:47b:ca03::8/126"

}

$ sudo systemctl restart docker

/etc/docker/daemon.json

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

registry-1.docker.io

9. Run a couple of Containers using only IPv6

• Making the instances IPv6-friendly• Container registry: Google Container Registry

$ docker run -it --rm gcr.io/gcp-runtimes/ubuntu_18_0_4:latest bashlatest: Pulling from gcp-runtimes/ubuntu_18_0_4deabf7bad5e7: Pull complete…Digest: sha256:af51882c2cb15cb3ed133ac62debb744057e02d6dee8db25a54caac158be2a3cStatus: Downloaded newer image for gcr.io/gcp-runtimes/ubuntu_18_0_4:latestroot@bf2f00033d64:/#

hub.docker.com

$ docker run ubuntu

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

10. Test connectivity between containers

• Ping6 between containers

Full Outputs

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Conclusion & Links

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Blog posts

• Kubernetes Networking: Behind the sceneshttps://itnext.io/kubernetes-networking-behind-the-scenes-39a1ab1792bb

• Kubernetes multi-cluster networking made simple https://itnext.io/kubernetes-multi-cluster-networking-made-simple-c8f26827813

• How to run IPv6-enabled Docker containers on AWS https://medium.freecodecamp.org/how-to-run-ipv6-enabled-docker-containers-on-aws-87e090ab0397

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Docker caveats

• Still need to plug this to Kubernetes

• IPv6 is disabled on containers in some Docker versions

• “The subnet for Docker containers should at least have a size of /80, so that an IPv6 address can end with the container’s MAC address and you prevent NDP neighbor cache invalidation issues in the Docker layer”

• …