docker networking tutorial 101
DESCRIPTION
Introduction to Docker Networking options. We give in-depth description of the different options with single host examples. See our other presentations for multi-host, IPv6, and CoreOS Flannel descriptions.TRANSCRIPT
Key Takeaways
1. Docker networking is in early stage and diverse
2. Applications must choose what networking is right for their needs. It is possible to use same principles as VMs
3. Open vSwitch brings powerful networking capabilities
4. LorisPack is an easy way to add pod-level isolation for Docker containers
5. User space vs Kernel space packet processing is an important design choice
Copyright Reserved
• Over the past few years, LXC came up as an alternative to VM for running workload on hosts
• Each container is a clone of the host OS
• Docker brought Linux containers to prominence
‒ Tracks application configuration and possibly archives to DockerHub
Linux Containers
3
Container 1
App X
Container 2 Container 3
Host OS
Guest root
App Y
Guest root
App Z
Guest root
Copyright Reserved
Docker
• Excellent way to track application dependencies and configuration in a portable format.
• For instance, the Dockerfileon the right can be used to spawn a container with nginxLB and accessed at a host port
$ docker build XYZ
$ docker images
$ docker run --name=nginx1 -d –i nginx
4
# Pull base image.
FROM dockerfile/ubuntu
# Install Nginx.
RUN \
add-apt-repository -y ppa:nginx/stable && \
apt-get update && \
apt-get install -y nginx && \
rm -rf /var/lib/apt/lists/* && \
echo "\ndaemon off;" >> /etc/nginx/nginx.conf && \
chown -R www-data:www-data /var/lib/nginx
# Define mountable directories.
VOLUME ["/etc/nginx/sites-enabled", "/etc/nginx/certs", "/etc/nginx/conf.d", "/var/log/nginx"]
# Define working directory.
WORKDIR /etc/nginx
# Define default command.
CMD ["nginx"]
# Expose ports.
EXPOSE 80
EXPOSE 443
Copyright Reserved
Setup: 3 containers on 1 host
Spawn 1 nginx and 2 bash containers in detached mode
$ docker run –i –d --name=nginx1 nginx
$ docker run -i -d --name=c1 base /bin/bash
$ docker run -i -d --name=c2 base /bin/bash
List running containers Linux bridge ports
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4846cacc4a1e nginx:latest "nginx -g" 4 days ago Up 4 days 443/tcp,
80/tcp nginx1
0ad1a5eeff69 base:latest "/bin/bash" 4 days ago Up 4 days c1
a51357948bfd base:latest "/bin/bash" 4 days ago Up 4 days c2
5
Copyright Reserved
High-level Concepts
Namespace Containerized networking at the process level managed at /proc
Linux Bridge L2/MAC learning switch built into the Kernel to use for forwarding
OpenvSwitch Advanced bridge that is programmable and supports tunneling
NAT Network address translators are intermediate entities that translate IP address + Ports (Types: SNAT, DNAT)
iptables Policy engine in kernel that is used for managing packet forwarding, firewall, NAT features
Unix domain sockets
File descriptor based communication that is restricted to a single host. Works like a FIFO pipe.
User-space vs Kernel-space
Application domain that regulates access to resources and performance possible. • Container applications run in user-space• Typically network forwarding runs in kernel space
Options for Docker Networking
7
Copyright Reserved
"NetworkSettings": {"Bridge": "docker0","Gateway": "172.17.42.1","IPAddress": "172.17.0.19","IPPrefixLen": 16,"MacAddress": "02:42:ac:11:00:0f","PortMapping": null,"Ports": {
"443/tcp": [{
"HostIp": "0.0.0.0","HostPort": "49157"
}],"80/tcp": [
{"HostIp": "0.0.0.0","HostPort": "49158"
}]
}
Networking Still in Early Stages
• Today Docker usage is predominantly within a single laptop or host.
• The default network assigned is a port on Linux bridge docker0
• The network on right is allocated to the nginx container we spawned.
8
Copyright Reserved
Many ways to network in Docker
• Many of these are similar to what we can do with VM(except the Unix-domain socket method of direct access)
9
Host
Container
CContainer D Container E Container FContainer A Container B
Direct
Host
network
Unix-domain
sockets and
other IPC
Docker0
Linux bridge
Docker proxy
(using iptables)
Open vSwitch
Port
mapping
Copyright Reserved
Option 1: Docker0 bridge
• Default network automatically created when no additional options “--net“ or “-P” are specified
• Each container is addressed by a static IP address assigned by Docker
• Similar to what we have as default with KVM or VirtualBox
• Host can reach container with IP on the bridge
• But, outside traffic cannot reach the container
10
Host
Nginx1172.17.0.18
C1172.17.0.19
C2172.17.0.20
172.17.42.1
Docker0 bridge
eth0 eth0 eth0
veth002aa7a veth6df8377 veth7b0e4c6
eth0192.168.50.16
Copyright Reserved
Option 1: Docker0 bridge
Check Linux bridge ports and NAT rules under the hood
# iptables –L –t nat -n
...
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
MASQUERADE all -- 172.17.0.0/16 anywhere
# sudo brctl show
bridge name bridge id STP enabled interfaces
docker0 8000.56847afe9799 no veth002aa7a
veth6df8377
veth7b0e4c6
# docker inspect --format='{{.NetworkSettings}}' nginx1
(See for yourself)
11
Copyright Reserved
Option 2: Port mapping
• Provide access to the container from outside byallocating a DNAT port in the range 49153-65535
• Still uses Linux bridge docker0, butadds iptables rules for the DNAT
• In our example, nginx2 container is reachable byaccessing 192.168.50.16:49155
# docker run -P -d -i --name=nginx2 -t nginx
# iptables –L –t nat -n
...
Chain DOCKER (2 references)
target prot opt source destination
DNAT tcp -- anywhere anywhere tcp dpt:49155 to:172.17.0.19:80
DNAT tcp -- anywhere anywhere tcp dpt:49156 to:172.17.0.19:443
...
12
Host
nginx2 c1
172.17.42.1
Docker0 bridge
eth0 eth0
veth79ed06d veth6df8377
eth0192.168.50.16
Copyright Reserved
For the new nginx2 container, we show network settings below# docker inspect nginx2
"NetworkSettings": {"Bridge": "docker0","Gateway": "172.17.42.1","IPAddress": "172.17.0.19","IPPrefixLen": 16,"MacAddress": "02:42:ac:11:00:0f","PortMapping": null,"Ports": {
"443/tcp": [{
"HostIp": "0.0.0.0","HostPort": "49157"
}],"80/tcp": [
{"HostIp": "0.0.0.0","HostPort": "49158"
}]}}
Option 2: Port mapping
Advanced:‒ It is possible to restrict the port
mapping to listen on specific host IP address and/or a specific host port number
‒ Use -p option as follows
# docker run \–p host_IP:host_port:container_port\–d –i –t nginx
13
Copyright Reserved
Option 3: Host Option 4: Container
14
Give full access of the host network to the container using --net=host option# docker run --net=host
--name=c3 -i –d –t base /bin/bash
Check network within container using ifconfig command through exec# docker exec c3 ifconfig eth0
eth0 Link encap:EthernetHWaddr 52:54:00:0d:3c:9finet addr:192.168.50.16Bcast:192.168.50.255
Host can talk to container using lo (localhost) interface
Containers can listen on privileged ports (i.e., port numbers < 1024) of host
Give full access to network of a container XX to the new container YY using --net=container:XX option# docker run --net=container:nginx1
--name=c4 -i –d –t base /bin/bash
Check network within container using ifconfig command through exec# docker exec c4 ifconfig eth0
eth0 Link encap:EthernetHWaddr 02:42:ac:11:00:12inet addr:172.17.0.18Bcast:0.0.0.0
Container XX can talk to container YY using lo (localhost) interface
Copyright Reserved
Option 5: Open vSwitch (OVS)
• Similar to Linux bridge, but different technology
‒ Today, this is not the default with Docker
‒ Allows programming with OVSDB and OpenFlow protocols
• Why? OpenvSwitch has many useful features!
‒ VxLAN, GRE, VLAN based encapsulation and L2 forwarding
‒ Encapsulation allows containers to pick any MAC/IP they want
‒ Also possible to do L3 routing, ARP proxy etc, load-balancing
‒ Access control, traffic rate limiting and prioritization
‒ 10G/s or more packet processing throughput possible
‒ 1) kernel, or 2) userspace, with optionally DPDK acceleration
15
Copyright Reserved
Option 5: Open vSwitch (OVS)
Usage steps
1. Create OVS bridge and assign a gateway interface
2. Use --net=none when spawning container
3. Manually connect containers to the OVS bridge using veth pair
16
# ovs-vsctl add-br br0
# ovs-vsctl add-port br0 ovsbr0p1 -- set interface ovsbr0p1 type=internal
# ifconfig ovsbr0p1 192.168.50.1 netmask 255.255.255.0 up
# iptables -t nat -A POSTROUTING -s 192.168.50.0/24 -j MASQUERADE
# docker run --name=c1 --net=none -d -i -t base /bin/bash
Docker working towards allowing plugin
mechanism for integrating automation agents
Copyright Reserved
Option 5: Open vSwitch (OVS)
• Extract process ID for this container, and create network namespace for it
• Create veth device pair and move one to the processes’ network namespace
• Rename that device to eth0 and assign IP address + route to it
• Add pair interface to the OVS bridge
17
# pid=`docker inspect --format '{{ .State.Pid }}' $1`
# ln -s /proc/$pid/ns/net /var/run/netns/c1
# ip netns exec c1 ip link set dev peertapc1 name eth0
# ip netns exec c1 ip link set eth0 up
# ip netns exec c1 ip addr add 172.27.0.2/24 dev eth0
# ip netns exec c1 ip route add default via 172.27.0.1
# ip link add tapc1 type veth peer name peertapc1
# ip link set peertapc1 netns c1
# ifconfig tapc1 up
# ovs-vsctl add-port br0 tapc1
Copyright Reserved
Option 6: Unix-domain socket or pipe
• All inter-process communication mechanisms can be used with containers, especially Unix domain sockets or pipes. For instance,
• Apps on c1 and c2 can communicate over following Unix socket address
18
# docker run --name=c1 -v /var/run/foo:/var/run/foo -d -i -t base /bin/bash
# docker run --name=c2 -v /var/run/foo:/var/run/foo -d -i -t base /bin/bash
bind(socket_fd, (struct sockaddr *) &address,sizeof(struct sockaddr_un));
listen(socket_fd, 5);
while((connection_fd = accept(socket_fd,(struct sockaddr *) &address, &address_length)) > -1)
nbytes = read(connection_fd, buffer, 256);
struct sockaddr_un address;
address.sun_family = AF_UNIX;
snprintf(address.sun_path, UNIX_PATH_MAX, "/var/run/foo/bar");
connect(socket_fd, (struct sockaddr *) &address,sizeof(struct sockaddr_un));
write(socket_fd, buffer, nbytes);
C1 : Server.c C2 : Client.c
Other Parameters
19
Copyright Reserved
Other Networking Parameters
These are option assigned to the Docker daemon during startup
• Change default Linux bridge using --bridge
‒ IP address needs to be set for the bridge using --bip
• Change range of fixed IPs assigned to the container‒ Typical default is the same range as that assigned to bridge with --bip
• Change default MTU of the containers using --mtu option‒ The actual MTU will be the least of the uplink interface, and this mtu
• Set DNS server and domain using --dns, --dns-search options
• --ip-forward enables proxy forwarding at the bridge
• --iptables enable using iptables for access control
20
Securing containers using iptables
… default DROP policy
21
Copyright Reserved
Access Control using icc and links
Tighter access control on communication between containers
1. Start docker daemon with --icc=false option. This inserts a default DROP rule in iptables thereby preventing any spawned containers from communicating with each other.
22
$ sudo iptables –L –n...Chain FORWARD (policy ACCEPT)target prot opt source destinationACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHEDACCEPT all -- 0.0.0.0/0 0.0.0.0/0DROP all -- 0.0.0.0/0 0.0.0.0/0
Copyright Reserved
Access Control using icc and links
Tighter access control on communication between containers
2. Connect two containers using the --link option in docker run
‒ Some environment variables (e.g., NGINX1_PORT) are set in target container and entry in the /etc/hosts file for easy access to source
23
nginx1c1
Docker0 bridge
c2
X
$ docker run --name=nginx1 -P -d -i -t nginx$ docker run --name=c1 -d -i -t --link nginx1:nginx1 base /bin/bash$ docker run --name=c2 -d -i -t base /bin/bash
$ docker exec c1 cat /etc/hosts127.0.0.1 localhost::1 localhost ip6-localhost ip6-loopback172.17.0.10 nginx1
$ docker exec c1 printenv
(See for yourself)
Copyright Reserved
Access Control using icc and links
• The --link option allows access to the exposed ports (80, 443) of a source container (nginx1) from a target container (c1)
‒ Implemented using ACCEPT rules in iptables
24
$ sudo iptables –L -n...Chain FORWARD (policy ACCEPT)target prot opt source destinationACCEPT tcp -- 172.17.0.8 172.17.0.9 tcp spt:80ACCEPT tcp -- 172.17.0.9 172.17.0.8 tcp dpt:80ACCEPT tcp -- 172.17.0.8 172.17.0.9 tcp spt:443ACCEPT tcp -- 172.17.0.9 172.17.0.8 tcp dpt:443ACCEPT tcp -- 0.0.0.0/0 172.17.0.3 tcp dpt:443ACCEPT tcp -- 0.0.0.0/0 172.17.0.3 tcp dpt:80...
Insert by
link option
Exposed ports
in Dockerfile
Thank you.
https://github.com/sdnhub/lorispack© 2015 Copyright Reserved