Managing ejabberd platform with

Docker containers

What is Docker• Linux tool, written in Go language

• Started in 2010 as a concept at dotCloud SARL

• Published 2y ago by creator Solomon Hykes with Open Source license (Apache 2.0)

• Handled by Docker Inc

• Partnership with Red Hat, IBM, Google, Amazon...

• Automates the deployment of applications as standalone software containers

High level view• Own process space

• Own network interface

• Include all binaries/libs required by your software

• Works the same on every Linux host

• Just need Linux Kernel on host machine

Low level view• Like chroot++ or improved BSD jails, without need

of custom configuration and deployment scripts

• Shares Kernel with Host and other containers, but in isolated environment

• No need for device emulation (neither HVM nor PV)

• Adds API to existing Linux Kernel features: LXC, cgroups, namespaces

Container vs VM

• Virtual Machine: emulates hardware, runs the Operating System and applications

• Docker: system process with isolated view of the namespace and resources, runs libraries and applications. No emulation layer. Accumulates AUFS mounts for container filesystem.

Container vs VM

Why it matters• Containers start fast, are smaller than VM

• Build once run everywhere, matches cloud needs

• Containers are easy to replicate (docker hub)

• No runtime resources overhead, no emulation

• Allow low level packaging optimisation/setup

Drawbacks

• Works only on Linux (for now?)

• Requires Linux 3.8+

• Kernels older than 3.10 lack some of the features required to run some containers

ejabberd requirements

• Ejabberd beam files

• Erlang dependencies beam and NIF

• Configuration and runtime files

• Erlang VM and system library dependencies: libpthread, libm, libssl, libc...

Installer vs Container• Erlang is already a VM after all ! Installed may be sufficient ?

• Installer must include everything (code+database) and can not handle upgrade without external and custom scripts

• Installer is standalone but uses hosts's libc, which may differ from one host to another

• No clean way to use custom configuration and database with Installer/rpm/deb

• Containers gives deeper application integration, makes upgrade process simpler (more control on global environment), and ensure same environment deployed everywhere. (no hazardous dependencies issues)

Installations issues• Heterogeneous setup: installed from sources, from official binary

installer, from deb/rpm/pkgsrc/brew/etc... => impossible to reproduce clean, secure and optimised setup everywhere, with same version of erlang for a given ejabberd release

• Way too much combination, from system libs to erlang version. Many possible configuration, many patches on packaged Erlang and/or system libraries, etc...

• No hands on host's network settings and tunning

• Binary installer gives more control on software dependencies, but still depends on host's libc compatibility and does not ensure user uses bundled libraries.

Simple container• From Debian Jessie

• Download ProcessOne binary installer

• Install using unattended mode

• Apply own configuration

=> Build new container for every release

=> Depends on ProcessOne binary packaging

ejabberd container• By Rafael Römhild (rroemhild/ejabberd)

• Complete and customisable container

• Available for all tags since 14.12 up to 15.10

• Based on Debian Jessie

• Uses debian's Erlang/OTP, build ejabberd from sources.

EnvironmentEJABBERD_HTTPS true EJABBERD_STARTTLS true EJABBERD_S2S_SSL true EJABBERD_PROTOCOL_OPTIONS_TLSV1 false EJABBERD_LOGLEVEL 4 EJABBERD_MUC_CREATE_ADMIN_ONLY EJABBERD_REGISTER_ADMIN_ONLY EJABBERD_MOD_ADMIN_EXTRA true EJABBERD_MOD_MUC_ADMIN false EJABBERD_REGISTER_TRUSTED_NETWORK_ONLY EJABBERD_SKIP_MODULES_UPDATE false EJABBERD_AUTH_METHOD internal EJABBERD_ADMINS EJABBERD_USERS ERLANG_NODE ejabberd@localhost ERLANG_COOKIE ERLANG_OPTIONS -noshell XMPP_DOMAIN TZ Europe/Paris

Configuration• Few environment variables allows to change simple

configuration settings, but default configuration remains very generic, too generic...

• Better create your own container with your own configuration and certificates: FROM rroemhild/ejabberd:15.10ADD ./ejabberd.yml /etc/ejabberd/ejabberd.ymlADD ./ejabberdctl.cfg /etc/ejabberd/ejabberdctl.cfgADD ./example.com.pem /opt/ejabberd/ssl/example.com.pem

$ docker build -t p1/ejabberd:15.10 .

Persistent storage• Using a data container, hum... really ?

docker create --name ejabberd-data rroemhild/ejabberd-datadocker run -d \ --name "ejabberd" \ --volumes-from ejabberd-data \ .... \ rroemhild/ejabberd:15.10

• Using mounts from host filesystem docker run -d \ --name "ejabberd" \ -v $PWD/database:/opt/ejabberd/database \ -v $PWD/logs:/var/logs/ejabberd \ .... \ rroemhild/ejabberd:15.10

Using custom container• Simplify both configuration and customisations

• Allows better integration with your environment

• Allows implicit linking with your database container

• Very simple to write and maintain

• Build "ready to use" containers for your use

Deploy/share containers

• Need a docker hub account (public/private)

• docker push me/ejabberd:15.10

• docker pull me/ejabberd:15.10

• docker run me/ejabberd:15.10

Docker hub

• Public repository to share your app

• Private repository to share container with dev team

• Private repository to easy deploy app in-house

• https://docs.docker.com/docker-hub/

Start ejabberddocker run -i -t -P \ --name "15.09" \ -p 5222:5222 \ -p 5269:5269 \ -p 5280:5280 \ -h 'example.net' \ -e "XMPP_DOMAIN=example.net" \ -e "ERLANG_NODE=node1" \ -e "ERLANG_COOKIE=demo" \ -e "EJABBERD_ADMINS=admin@example.net" \ -e "EJABBERD_USERS=admin@example.net:password1234" \ -e "TZ=Europe/Paris" \ -v $PWD/database:/opt/ejabberd/database \ -v $PWD/logs/node1:/var/log/ejabberd \ p1/ejabberd:15.09 live

docker start 15.09 docker exec 15.09 ejabberdctl status docker stop 15.09

Simple upgrade• Simple switch from 15.09 to 15.10 when using

same data container or mounting point, and also using same configuration.

• Allow easy upgrade/rollback but stops service if serving with a single node. docker start 15.09docker exec 15.09 ejabberdctl statusdocker stop 15.09docker start 15.10docker exec 15.10 ejabberdctl status

DEMO

So what ?• Using our custom container, upgrading from 15.09 to

15.10

• Using host mounts for data and preconfigured ejabberd

• a simple start/stop of the right container is enough

=> NO EXTRA CHANGE REQUIRED !

• improvement: N+1 / N-1 automatic upgrade/migration code needed ejabberd side for full coverage.

Smooth upgrade• Using haproxy as front-end, two ejabberd containers running on

same host, mapping them to their own ports and own erlang node name to play with cluster

current container A: -p 15222:5222

next container B: -p 25222:5222

• Haproxy redirects 5222 to 15222, then start container B, then add B in the cluster, then haproxy redirects 5222 to 25222, after a while remove A from cluster and stop A.

• - : Requires lots of automated stuff, complex, not very clean

• + : Allows using only one host without disconnecting all your users

Cluster upgrade• Run ejabberd container on several hosts

• Get them as one ejabberd cluster

• Apply simple upgrade node after node

• Disconnected users automatically reconnect to another running node

• Load can be redirected at balancer level to reduce impact of disconnections (takes more time, depends on service use)

Cluster issue• Ejabberd nodes needs direct network link at each

other, being part of same network

• Containers runs on different hosts

• Docker links are only bound to local host

• Containers are isolated

• Need to resolve remote hosts: multi-host networking using overlay network driver is docker solution

Multi host networking• Needs Linux 3.16+

• Access to key-value store (Consul, Etcd or ZooKeeper)

• A configured Engine daemon on each host in the cluster

• Not straight forward solution $ docker network create -d overlay --subnet=192.168.0.0/16 --subnet=192.170.0.0/16 --gateway=192.168.0.100 --gateway=192.170.0.100 --ip-range=192.168.1.0/24 --aux-address a=192.168.1.5 --aux-address b=192.168.1.6 --aux-address a=192.170.1.5 --aux-address b=192.170.1.6 my-multihost-network

Simpler multi host• Docker container can access host interface being

bridged by its own interface

• Need private internal network (ip alias can work)

• Containers can ping by internal network IP

• Custom container needs /etc/hosts and inetrc with static IP for all your nodes

• But needs update /etc/hosts when adding nodes. So this is OK until our cluster changes over time (no easy scale)

Other DNS solutions• dnsdocker using docker compose

• etcd, consul or zookeeper for use with overlay

• other container management systems like Rancher

• possibles improvement with new Networking in Docker 1.9 (relay on an official image to provide clean DNS resolution across containers ?)

=> need improvements to support dynamic cluster

ejabberd container improvements (WIP)

• Allow easy custom configuration

• Native support of multi-host clustering and transient joincluster/leavecluster

• Consistency with standard installation

• Custom minimalist Erlang/OTP+Elixir with hipe support

• Smaller container (based on Alpine instead of Debian, from 245Mb to 35Mb !)

Questions

Christophe Romain <cromain@process-one.net>