Who are we?Jim Vanns

Aaron CareyProduction Engineers at ILM London

What we’re covering● Deploying a Mesos cluster in the cloud

using Ansible● Running services on Mesos using Ansible● Differences between AWS/GCE modules● Tips we’ve learnt along the way

Step 1:

Ansible Dockerfile (example follows)Fix the ansible versionInstall a custom dynamic inventory file (or

two)Consider your credentials and securityInstall (bake-in a ‘git archive’) your

playbooks

Concise Ansible DockerfileFROM ubuntu:14.04RUN apt-get update && apt-get -y install python python-pip python-devRUN pip install -U ansible==1.9.4 boto apache-libcloud httplib2ADD ansible.cfg /etc/ansible/ansible.cfgADD ec2.py /etc/ansible/ec2.pyADD ec2.ini /etc/ansible/ec2.iniADD gce.py /etc/ansible/gce.pyADD gce.ini /etc/ansible/gce.iniWORKDIR /srv/ansibleADD ansible .

Step 2: Deploy cloud hosts

RequirementsCloud Agnostic (Work with both AWS and GCE)Split provisioning and bootstrapping into rolesNeed a way to determine which are new hostsNeed a way to group hosts during and after

provisioningTagging!

Cloud Start Role# Launch the primary (master/leader) nameserver- name: Launch a bootstrap consul server hosts: localhost connection: local gather_facts: False vars: cloud_provider: ec2 count: 3 service_name: Consul launch_group: tag_consul_server_{{ consul_domain }} tags: consul: server_{{ consul_domain }} roles: - cloud_start

Cloud Start Role---- include: "{{ cloud_provider }}.yml"

Cloud Start Role- name: Launch instance ec2: key_name: "{{ key_name }}" instance_type: "{{ instance_type }}" image: "{{ image }}" wait: yes vpc_subnet_id: "{{ vpc_subnet_id }}" group_id: "{{ security_group_id }}" region: "{{ region }}" instance_tags: ansible: base count: "{{ count }}" assign_public_ip: "{{ public_ip }}" register: ec2

- name: Add instances to launched group add_host: hostname={{ item.private_ip }} groupname=launched_cloud_default with_items: ec2.instances

- name: Add instances to parameter group add_host: hostname={{ item.private_ip }} groupname={{ launch_group }} with_items: ec2.instances

Cloud Start Role- name: Tag instances ec2_tag: resource={{item.id}} state=present region={{region}} with_items: ec2.instances args: tags: "{{ tags }}"

- name: Wait for sshd to come up wait_for: host={{ item.private_ip }} port=22 state=started delay=60 timeout=300 connect_timeout=2 with_items: ec2.instances

Taking it further...Parallelise the provisioningTake advantage of host groups where you canUse ansible environment variables to target hosts

and groups

AWS vs GCEAWS - Tags are key-value pairs

GCE - Tags are string labels, Metadata are key-value pairs

We used a custom (slightly modified) GCE inventorytags = node.extra['tags']for t in tags: tag = 'tag_%s' % t if groups.has_key(tag): groups[tag].append(name) else: groups[tag] = [name]

if 'items' in node.extra['metadata']: for item in node.extra['metadata']['items']: tag = 'tag_%s_%s' % (item['key'], item['value']) if groups.has_key(tag): groups[tag].append(name) else: groups[tag] = [name]

Step 3: Mesos

Our Mesos ArchitectureZookeeper*Mesos Master*Mesos Agents*ConsulMarathon (The scheduler’s scheduler)Mesos-consulEverything else

Deploying Zookeeper- name: Set zookeeper ID facts gather_facts: True hosts: tag_zookeeper_server_{{ consul_domain }} user: ilm-user tasks: - set_fact: zkid={{ item.0 | int + 1 }} when: hostvars[item.1]['ansible_hostname'] == ansible_hostname with_indexed_items: groups['tag_zookeeper_server_{{ consul_domain }}']

- name: Apply zookeeper role gather_facts: True hosts: tag_zookeeper_server_{{ consul_domain }} sudo: True user: ilm-user roles: - zookeeper

Zookeeper Role- name: Register zookeeper name with consul uri: > url=http://127.0.0.1:8500/v1/agent/service/register HEADER_Content-Type=application/json method=PUT body_format=json body='{ "Name": "zookeeper", "Tags": [ "zookeeper", "{{ zkid }}" ], "Port": 2181 }'

- name: Register individual zookeeper node with consul uri: > url=http://127.0.0.1:8500/v1/agent/service/register method=PUT body_format=json HEADER_Content-Type=application/json body='{ "Name": "zookeeper-{{ zkid }}", "Tags": [ "zookeeper", "{{ zkid }}" ], "Port": 2181 }'

Zookeeper Role- name: Run zookeeper container docker: name: zookeeper image: "mesoscloud/zookeeper:3.4.6-ubuntu-14.04" state: started net: host restart_policy: always volumes: - /mnt/data/log:/tmp/zookeeper env: MYID: "{{ zkid }}" SERVERS: "zookeeper-1,zookeeper-2,zookeeper-3"

Mesos Containers- name: Run mesos-master container docker: name: mesos-master image: "mesosphere/mesos-master:{{ img_version }}" state: started net: host restart_policy: always volumes: - /mnt/data/log:/var/log env: MESOS_ZK: "zk://zookeeper:2181/mesos" MESOS_CLUSTER: "{{ cluster_name }}" MESOS_QUORUM: "1" MESOS_LOG_DIR: "/var/log/mesos" MESOS_WORK_DIR: "/var/lib/mesos" MESOS_HOSTNAME: "mesos-master"

Submit tasks to Marathon- name: Launch docker-registry hosts: "tag_build_docker_{{ consul_domain }}" gather_facts: False tasks: - name: Submit docker-registry job to marathon uri: > url=http://marathon:8080/v2/apps HEADER_Content-Type=application/json method=POST status_code=200,201,409 body_format=json

body='{ "args": [ ], "container": { "type": "DOCKER", "docker": { "network": "HOST", "image": "registry:2.2", "forcePullImage": true, "parameters": [ { "key": "env", "value": "REGISTRY_STORAGE=s3" } ] } }, "id": "docker-registry", "instances": 1, "cpus": 2, "mem": 4096 }'

Done