Nuvola: a tale of migration to AWS

Ansible + AWS: victory is mine!

Who am I?

Matteo Moretti

Who I am?

CTO @

website: madisoft.it

tech blog: labs.madisoft.it

It’s a story

●It’s our story

●It’s about a migration

●We did it!

●We’ve learnt a lot

●We want to share it with you

Nuvola

●~ 2M users

●~ 1000 databases

●~ 350GB of mysql data

●~ 25M of media files

●~ 4.50TB of media files

●~ 60 servers

Why a migration?

No automation, flexibility and autoscaling

Need of change

●Flexibility

●Horizontal scaling

●Infrastructure as code

●Multiple environments

●All services by one provider

●Cost optimization

pixabay.com

Change is coming

isn’t it?

pixabay.com

Obstacles

●It’s a distributed app

●Learn an entire new ecosystem

●Causing no troubles for users

●Few weeks of time

Tools and solutions

AWS

●100% automation

●Tons of services

●Very well integrated with Ansible

●IaaS services

●Autoscaling / Reserved instances / Spot instances

AWS

Autoscaling + reserverd instances + spot instances

Optimizing services while reducing costs

AWS

Ansible

●IT automation tool

●Easy lo learn

●No coding skills. It uses YAML

●No agents on target machines

●Ready-made AWS modules

●Can be easily idempotent

Ansible & AWS

AWS Azure Cloudstack Digital Ocean Google

86 18 31 5 10

Openstack Ovh Rackspace Softlayer Vmware

52 1 26 1 26

http://docs.ansible.com/ansible/list_of_cloud_modules.html

Ansible & AWS

ec2 - Create, terminate, start or stop an instance in ec2http://docs.ansible.com/ansible/ec2_module.html

ec2_asg - Create or delete AWS Autoscaling Groupshttp://docs.ansible.com/ansible/ec2_asg_module.html

ec2_elb_lb - Creates or destroys Amazon ELB.http://docs.ansible.com/ansible/ec2_elb_lb_module.html

ec2_snapshot - creates a snapshot from an existing volumehttp://docs.ansible.com/ansible/ec2_snapshot_module.html

ec2_tag - create and remove tag(s) to ec2 resources.http://docs.ansible.com/ansible/ec2_tag_module.html

s3 - manage objects in S3.http://docs.ansible.com/ansible/s3_module.html

route53 - add or delete entries in Amazons Route53 DNS servicehttp://docs.ansible.com/ansible/route53_module.html

Nuvola

Multiple environments

● ./infrastructure_nuvola_env_aws.sh --env prod● ./infrastructure_nuvola_env_aws.sh --env dev● ./infrastructure_nuvola_env_aws.sh --env lavorazione

Infrastructure

./infrastructure_nuvola_env_aws.sh --env prod

#!/bin/bash

. libs/limit_option_parser.sh

….

ansible-playbook --vault-password-file secrets/infrastructure_nuvola_env.secret \

ansible/infrastucture_nuvola_env.yml \ -e"$EXTRA_OPTIONS"….

Infrastructure playbook

infrastructure_nuvola_env.yml

tasks: - include: .../infrastructure_nuvola_vpc.yml tags: vpc

- include: .../infrastructure_nuvola_ec2.yml tags: ec2

- include: .../infrastructure_nuvola_elb.yml tags: elb

- include: .../infrastructure_nuvola_destroy.yml when: destroy == "true" and nuvola_env != "prod"

VPC

tasks/infrastructure_nuvola_vpc.yml

- name: INFRASTRUCTURE NUVOLA VPC | setting up vpc ec2_vpc:state: presentcidr_block: 10.0.0.0/16resource_tags: {

Name: "nuvola_{{ nuvola_env }}_vpc", nuvola_env: '{{ nuvola_env }}', nuvola_role: "vpc", billing: "{{ billing_tag_value }}"

}az: eu-west-1ainternet_gateway: True

register: vpc

VPC

infrastructure_nuvola_vpc.yml

- name: INFRASTRUCTURE NUVOLA VPC | vpc peering route ec2_vpc_route_table:

vpc_id: "{{ vpc['vpc']['id'] }}"tags:

Name: "nuvola_{{ nuvola_env }}_to_nuvola_default"subnets:

- "10.0.{{ ec2_vpc_subnet }}.0/24"routes:

- dest: 0.0.0.0/0 gateway_id: "{{ vpc.igw_id }}"

http://docs.ansible.com/ansible/ec2_vpc_route_table_module.html

ELB

infrastructure_nuvola_elb.yml- name: INFRASTRUCTURE NUVOLA ELB | Setup ELB ec2_elb_lb: state: present name: 'nuvola-{{ nuvola_env }}-elb' listeners: - protocol: http load_balancer_port: 80 instance_port: 80 - protocol: https load_balancer_port: 443 instance_protocol: http instance_port: 80 ssl_certificate_id: '{{ output.stdout }}'

http://docs.ansible.com/ansible/ec2_elb_lb_module.html

EC2 backend

infrastructure_nuvola_ec2.yml- name: INFRASTRUCTURE NUVOLA EC2 | Init backend instances ec2: key_name: '{{ ec2_key_name }}' instance_type: '{{ backend_instance_type }}' instance_tags: nuvola_type: "{{ nuvola_env }}_backend" nuvola_env: '{{ nuvola_env }}' nuvola_role: "backend" billing: "{{ billing_tag_value }}" image: "{{ ec2_ami_id }}" zone: "{{ ec2_zone }}" wait: yes wait_timeout: 600 group: "nuvola_{{ nuvola_env }}_backend_sg"

http://docs.ansible.com/ansible/ec2_module.html

EC2 backend

infrastructure_nuvola_ec2.ymlcount_tag: nuvola_type: "{{ nuvola_env }}_backend" exact_count: '{{ nuvola_backend_ec2_instances }}' vpc_subnet_id: "{{ vpc['subnets'][0]['id'] }}" assign_public_ip: yes termination_protection: "{{ delete_lock }}" volumes: - device_name: /dev/sda1 volume_type: gp2 volume_size: "{{ ec2_volume_size_backend }}" delete_on_termination: true instance_profile_name: "{{ ec2_instance_role }}" register: ec2_backend

http://docs.ansible.com/ansible/ec2_module.html

Provisioning

./provision_nuvola_backend.sh --limit "tag_nuvola_type_${ENV}_backend"

./provision_nuvola_dbserver.sh --limit "tag_nuvola_type_${ENV}_database"

./provision_nuvola_routine.sh --limit "tag_nuvola_type_${ENV}_routine"

if [ "$ENV" != "prod" ]; then./nuvola-init-not-prod-env.sh --env ${ENV}

./deploy_nuvola.sh --limit "tag_nuvola_type_${ENV}_backend" --env ${ENV}

fi

Provisioning PHP7

roles/php7/tasks/php7_prod.yml

- name: PHP7 ALL | install php packages apt: pkg={{ item }} state=latest update_cache=yes with_items: '{{ php7_packages }}'

- name: PHP7 ALL | Set php.ini CLI template:

src=roles/php7/templates/nuvola/php.ini.cli.j2 dest=/etc/php/7.0/cli/php.ini

- name: PHP7 ALL | Set php.ini php-fpm template:

src=roles/php7/templates/nuvola/php.ini.web.j2 dest=/etc/php/7.0/fpm/php.ini

Multiple env: how do I find it?

Route 53

● Public DNS○ nuvola-prod-backend-3.ops.madisoft.it○ nuvola-prod-database-24.ops.madisoft.it○ nuvola-dev-database-34.ops.madisoft.it○ nuvola-issue8978-database-34.ops.madisoft.it

● Private DNS○ local-prod-backend-0.ops.madisoft.it○ local-prod-cache-sessioni-0.ops.madisoft.it○ local-dev-database-14.ops.madisoft.it○ local-issue8978-backend-0.ops.madisoft.it

pixabay.com

DNS

infrastructure_nuvola_ec2.yml

- name: INFRASTRUCTURE NUVOLA EC2 | Assign backend dns route53: command: create zone: "{{ domain_tld }}" record: "nuvola-{{ nuvola_env }}-backend-{{ item.0 }}.{{ domain_tld }}" type: A value: '{{ item.1.public_ip }}' overwrite: yes ttl: "{{ ttl_expire }}" with_indexed_items: '{{ ec2_backend.instances }}'

nuvola-prod-backend-3.ops.madisoft.ithttp://docs.ansible.com/ansible/route53_module.html

DNS: local

infrastructure_nuvola_ec2.yml

- name: INFRASTRUCTURE NUVOLA EC2 | Assign database local dns route53: command: create zone: "{{ domain_tld }}" record: "local-{{ nuvola_env }}-database-{{ item.0 }}.{{ domain_tld }}" type: A value: '{{ item.1.private_ip }}' overwrite: yes with_indexed_items: '{{ ec2_database.instances }}'

local-dev-database-14.ops.madisoft.ithttp://docs.ansible.com/ansible/route53_module.html

DNS: local

infrastructure_nuvola_ec2.yml

- name: INFRASTRUCTURE NUVOLA EC2 | Assign database local dns route53: command: create zone: "{{ domain_tld }}" record: "local-{{ nuvola_env }}-database-{{ item.0 }}.{{ domain_tld }}" type: A value: '{{ item.1.private_ip }}' overwrite: yes with_indexed_items: '{{ ec2_database.instances }}'

local-dev-database-14.ops.madisoft.ithttp://docs.ansible.com/ansible/route53_module.html

Ready to move?

Warm up

Moving:

- static files from a shared NAS to S3- external standalone services to ec2- Jenkins CI to AWS- ELK stack to AWS- (and testing) Nuvola stage environment

Switch of

- Stop current app

- Create prod env infrastructure

- App deployment

- Copy db data

Infrastructure

./infrastructure_nuvola_env_aws.sh --env prod

#!/bin/bash

. libs/limit_option_parser.sh

….

ansible-playbook --vault-password-file secrets/infrastructure_nuvola_env.secret \

ansible/infrastucture_nuvola_env.yml \ -e"$EXTRA_OPTIONS"….

Migration with sharding

Db data migration

Many small databases on diferent machines

Use of parallelization

Mysql_migrate_dbserver.sh

….ansible-playbook -l $LIMIT \ ansible/mysql_migrate_dbserver.yml -e "nuvola_env=$ENV" \ --vault-password-file ./secrets/provision_nuvola_dbserver.secret….

App deploy

deploy_nuvola.sh

ansible-playbook

ansible/deploy_nuvola.yml

--extra-vars="nuvola_env=$ENV"

Switch of

Total time: ~ 50m

Achievement

Amazing migration!

WE ARE HIRING!

(wanna join? ask us at the end of the talk or visit our website)

@mat_teo8matteo.moretti@madisoft.it