Continuous Delivery of Windows microservices in the cloud

@owainperry

http://www.slideshare.net/perryofpeek/continuous-delivery-of-windows-micro-services-in-the-cloud

What?

The journey of the last 18 months, Justgiving has taken to ship more in less time

How we have implemented supporting infrastructure around micro services

Where we went wrong

Raise Money for Charities

Turnover ~400 Million a year

Process UK gift aid

Text Giving

Crowd funding

~1000 nodes

Stephen Sutton

Monolith legacy

High developer pain

Branching

2 week release cycles

Long build times

Long test times

Massive git repository

Legacy source code patterns (bin files)

etc...

Micro services

JustGiving micro service?

It's a small service! Usually a component makes up part of a business domain

Has one purpose

Is simple to reason about

Reality - various levels of granularity

Where we went next

New stuff:built using micro services

Hosted in the cloud

New pipelines for deployment

Monolith stuff:Chop out choice areas, bite size chunks at a time

Build a simpler pipeline

Simpler branching strategy

Infrastructure principles

Very Available

Repeatable through automation

Development define and build their infrastructure

Keep environments isolated

Be pragmatic.

Train track automation

Development - self service infrastructureFast to spin up infrastructure

Less Ops bottleneck

Reduce the need for AWS keys

Consistency benefits: Service architecture

Names and name conventions

Tags

Security groups

Scripting

Types of service

1. Windows service Either:Self hosted nancyfx web serverApplication

Health endpoint

Message based serviceHealth endpoint

2. IIS hosted applicationHealth endpoint

Core tools

AWS

Artifactory package repository

Chef open source v11

GO CD

Team City

Chocolatey

Auto scale group

Auto scale group

Micro service hosting architecture

Varnish is the load balancer

Elastic Load BalancerElastic Load BalancerVarnishNode 1Node 2Node 1Node 2

CD of micro services

Each micro service has its own Git repository

GO Pipeline

Team City build

Chef cookbook

Tests

AWS Infrastructure

Operations run book

Health check endpoint - /status/health

Blue / Green Deployment

Steps for a new micro service

Think of a name

Create a git repository

Open up micro service template in visual studioRename a few things

Teamcity build (copy n paste)

Go pipeline (copy n paste)

Define Infrastructure via json

The pipeline

Teamcity

GitChef(cookbook)

Go CDArtifactory(choco pkg)

DevStagingProdSandbox

Versions

Application chocolatey package

Cookbook

AWS infrastructure

Treat AWS as another resource

Heavy Weight Resource Providers for AWS

Light weight resource Providers to abstract the HWRP

json data_bags based infrastructure

Infrastructure builder node executes chef client for infrastructure

Most of our infrastructure is built this way, not just business application services

Chef resources

Elastic Load Balancer

Auto scale groups

Security groups

IAM roles

Launch configurations

DNS

Subnets

Ec2 instances

Route tables

HWRP example

aws_subnet 'subtest2' do accesskey accesskey secretkey secretkey region eu-west-1 vpc DEV-VPC azone 'eu-west-1a' cidr '10.20.30.0/24' routetable 'RTB-STG-PRIVATE' tags [ { "a" => "1"},{"c" => "2" }] action :createend

LWRP data bag - json

{ "id": "dev", "type": "asg-lcf", "region": "eu-west-1", "availability_zones": ["eu-west-1a" ], "vpc": "DEV-VPC", "routetable": "RTB-DEV-PRIVATE", "ami": "JG-UBUNTU12-04-V5", "instance_type": "m3.medium", "security_groups": ["********" ], "iam_role": "instance_default", "launch_configuration_name": "LCF-DEV-SENSU", "aws_autoscale_group_name": "ASG-DEV-SENSU", "health_check_type": "EC2", "health_check_grace_period": 300, "asg_min_size": 3, "asg_max_size": 3, "asg_desired_capacity": 3, "asg_default_cooldown": 200, "subnets": [ { "name": "SUB-DEV-LOGSTASH-ZA",

Micro service GO pipeline

Micro service cookbook

Chef is used to configure the node

Chef is used to deploy the application

deployment - is just version constraint in a chef environment

Deployment

Set the version constraint for the inactive environment

Chef search all nodes

For each node{ remote in an kick off chef client scheduled task}

Switch active environment

Poll for nodes to be on correct version.

Post deployment tests

Consistency in architecture

Name conventions

Build scripts

Packages

Supports Monitoring

Refactoring

Comprehension

Bootup

Chef is prebaked into AMI (faster)

Instance userdata script contains the bootup script (inc runlist)

Auto scale groups used to instantiate new instances

Automated change

Change log automated into Jira for releasesStaging and production

Change log automated for cookbook releases

Platform to include more over time.

Automated change log

Information

Pipelines for chef cookbooks

Chocolatey

Rebuild all packages for internal hosting. Don't want a dependency on 3rd party download

Control versions of software in use

Common pattern for git and builds Take ~10 mins to repackge

( This is a bit of a chore )

What we didn't get right...

Zombies.

Chocolatey (exe / bat)

AWS reboot

Cost control

Abstraction of infrastructure json

Chef search to configure load balancer

Not automating our base AMI's from the outset

Future

Rolling instance recycling daily / weekly?

Bake images in production

Ohai for sensu subscriptions

Service discovery rather than chef search

Application secret security

Code security scanning in the pipelines

Infrastructure penetration testing

Infrastructure security checks

Packer to build base ami's

So.

Micro services has helped productivity

AWS automation in hands of developers

Simple json defined infrastructure

Chef is core at the centre

We have made mistakes

But it's been a worthwhile journey so far.

Questions?