microservice scars - alt.net 2hr

Microservice Scars:Lessons we learnt

Abhaya Chauhan@AbhayaChauhan

www.abhayachauhan.comJoshua Toth@TothJoshuaJ

Question Policy

Let’s go on a Journey

Context

PageUp

Context

Talent Management

SaaS

Context

100% Cloud based

AWS + Azure

Context

19 years

Context

25% growth year on year

Context

PageUp = Market Leaders

Context

The Monolith

Context

Fragility

The Monolith

Context

Agility

The Monolith

Context

Complexity

The Monolith

Context

Innovation

The Monolith

Context

Deployment

The Monolith

Context

Production Hardened

The Monolith

Context

#Scale

Context

Large Projects

Context

Big Rewrite

Context

If you can’t build a monolith, what makes you think

microservices are the answer?

Simon Brown

Context

First rule of Distributed Computing

Context

Don’t do it.

Context

Why Microservices?

Context

Reduce time to market

Context

Reduce blast radius

Context

EnableEvolutionary Architecture

Context

Sprint Zero

Decisions

Sprint Zero

The stack

Sprint Zero

Sprint Zero

Sprint Zero

Frontend

Sprint Zero

Backend

Frontend

Sprint Zero

Backend

DB

Frontend

Sprint Zero

Elastic Beanstalk

S3

.Net

Sprint Zero

Self hosted

Sprint Zero

Angular

Sprint Zero

DynamoDB

Sprint Zero

Pact

Sprint Zero

Infancy

Sprint Zero

Limitations

Sprint Zero

Investment

Sprint Zero

Deployment

Sprint Zero

Continuous

Sprint Zero

Phabricator

Sprint Zero

Phabro slides had to be removed.

Please contact us for any queries

Sprint Zero

Pipeline

Sprint Zero

Sprint Zero

Pipeline

Sprint Zero

Pipeline

Sprint Zero

F# Make (Fake)

Sprint Zero

Monolith Scripts

Sprint Zero

Sharing != caring

Sprint Zero

Clear separation

Sprint Zero

Development Practices

Sprint Zero

TDD

Sprint Zero

BDD

Sprint Zero

Sprint Zero

Elastic Beanstalk

Full stack

Elastic Beanstalk

Visualization

Elastic Beanstalk

Monitoring

Elastic Beanstalk

Monitoring

Elastic Beanstalk

Monitoring

Elastic Beanstalk

Monitoring

Elastic Beanstalk

Elastic Beanstalk

Elastic Beanstalk

Elastic Beanstalk

Rebuilds

Elastic Beanstalk

No Downtime

Elastic Beanstalk

The Hack

Elastic Beanstalk

Elastic Beanstalk

Elastic Beanstalk

Elastic Beanstalk

Speed

Elastic Beanstalk

Care

Elastic Beanstalk

Synchronization

Elastic Beanstalk

50 shades of “fuck we can’t roll this back”

Grey

Elastic Beanstalk

Scaling

Elastic Beanstalk

Authentication

IdentityServer3(Locksmith)

OAuth2

Immediate return

OAuth2

Questions

Messaging

Messaging

Messaging

Messaging

Request Response Pain

Messaging

Resource intensive

Messaging

Fault Prone

Messaging

1 to 1 communication

Messaging

Event Driven Architecture

Messaging

Webhooks

Messaging

Messaging

Messaging

:`(

Messaging

Queuing (SQS)

Messaging

Resource Intensive

Fault Prone

1 - many Communication

Messaging

Resource Intensive

Fault Prone

1 - many Communication

Messaging

Resource Intensive

Fault Prone

1 - many Communication

Messaging

Resource Intensive

Fault Prone

1 - many Communication

Messaging

Simple Notification Service (SNS)

Messaging

Push messages

Messaging

Mobile Devices

Messaging

Mobile DevicesSMS

Messaging

Mobile DevicesSMS

Emails

Messaging

Mobile DevicesSMS

EmailsHTTP Endpoints

Messaging

Mobile DevicesSMS

EmailsHTTP Endpoints

Lambda

Messaging

Mobile DevicesSMS

EmailsHTTP Endpoints

LambdaQueues

Messaging

Messaging

Messaging

Messaging

Messaging

SNS + SQS = Amazing Service Bus

Resource Intensive

Fault Prone

1 - many Communication

Messaging

Circuit Breaker

Messaging

Messaging

CloudWatch

Messaging

Historical Events

Messaging

A store for events

Messaging

DynamoDb

Fully Managed NoSQL DB

DynamoDb

Fast and Predictable

DynamoDb

Distributed

DynamoDb

Key-ValueDocument

DynamoDb

Hash (Range)

Partition 1 Partition 2 Partition 3 Partition N

Read & Write capacity equally distributed

Max 400kb

DynamoDb

1 Read unit = 4KB/s1 Write unit = 1KB/s

DynamoDb

CQRS

DynamoDb

DynamoDb

Oh shit!

DynamoDb

Write is 20x more expensive

DynamoDb

DynamoDb

{"Key":"Mauritania","Value":"42803"},{"Key":"Mauritius","Value":"42804"},{"Key":"Mexico","Value":"42805"},{"Key":"Micronesia","Value":"42806"},{"Key":"Monaco","Value":"42807"},{"Key":"Mongolia","Value":"42808"},{"Key":"Montenegro","Value":"42809"},{"Key":"Montserrat","Value":"42810"},{"Key":"Morocco","Value":"42811"},{"Key":"Mozambique","Value":"42812"},{"Key":"Namibia","Value":"42813"},{"Key":"Nauru","Value":"42814"},{"Key":"Nepal","Value":"42815"},{"Key":"Netherlands","Value":"42816"},{"Key":"Netherlands Antilles","Value":"42817"},{"Key":"New Caledonia","Value":"42818"},{"Key":"New Zealand","Value":"42819"},{"Key":"Nicaragua","Value":"42820"},{"Key":"Niger","Value":"42821"},{"Key":"Nigeria","Value":"42822"},{"Key":"Niue","Value":"42823"},{"Key":"Northern Mariana Islands","Value":"42824"},{"Key":"Norway","Value":"42825"},{"Key":"Oman","Value":"42826"},{"Key":"Pakistan","Value":"42827"},{"Key":"Palau, Republic Of","Value":"42828"},{"Key":"Palestine","Value":"42829"},{"Key":"Panama","Value":"42830"},{"Key":"Papua New Guinea","Value":"42831"},{"Key":"Paraguay","Value":"42832"},{"Key":"Peru","Value":"42833"},{"Key":"Philippines","Value":"42834"},{"Key":"Poland","Value":"42835"},{"Key":"Portugal","Value":"42836"},{"Key":"Puerto Rico","Value":"42837"},{"Key":"Qatar","Value":"42838"},{"Key":"Republic Of Korea","Value":"42839"},{"Key":"Republic Of Moldova","Value":"42840"},{"Key":"Republic Of Yemen","Value":"42841"},{"Key":"Reunion","Value":"42842"},{"Key":"Romania","Value":"42843"},{"Key":"Russian Federation","Value":"42844"},{"Key":"Rwanda","Value":"42845"},{"Key":"Saint Kitts And Nevis","Value":"42846"},{"Key":"Saint Lucia","Value":"42847"},{"Key":"Sao Tome & Principe","Value":"42848"},{"Key":"Saudi Arabia","Value":"42849"},{"Key":"Senegal","Value":"42850"},{"Key":"Serbia","Value":"42851"},{"Key":"Serbia & Monten","Value":"42852"},{"Key":"Seychelles","Value":"42853"},{"Key":"Sierra Leone","Value":"42854"},{"Key":"Singapore","Value":"42855"},{"Key":"Slovakia","Value":"42856"},{"Key":"Slovenia","Value":"42857"},{"Key":"Solomon Islands","Value":"42858"},{"Key":"Somalia","Value":"42859"},{"Key":"South Africa","Value":"42860"},{"Key":"South Sudan","Value":"42861"},{"Key":"Spain","Value":"42862"},{"Key":"Sri Lanka","Value":"42863"},{"Key":"St. Vincent And The Grena","Value":"42864"},{"Key":"Sudan","Value":"42865"},{"Key":"Surinam","Value":"42866"},{"Key":"Swaziland","Value":"42867"},{"Key":"Sweden","Value":"42868"},{"Key":"Switzerland","Value":"42869"},{"Key":"Syrian Arab Republic","Value":"42870"},{"Key":"Tajikistan","Value":"42871"},{"Key":"Tanzania, United Rep. Of","Value":"42872"},{"Key":"Tfyr Of Macedonia","Value":"42873"},{"Key":"Thailand","Value":"42874"},{"Key":"Timor-Leste","Value":"42875"},{"Key":"Togo","Value":"42876"},{"Key":"Tokelau Islands","Value":"42877"},{"Key":"Tonga","Value":"42878"},{"Key":"Trinidad And Tobago","Value":"42879"},{"Key":"Tunisia","Value":"42880"},{"Key":"Turkey","Value":"42881"},{"Key":"Turkmenistan","Value":"42882"},{"Key":"Turks And Caicos Island","Value":"42883"},{"Key":"Mauritania","Value":"42803"},{"Key":"Mauritius","Value":"42804"},{"Key":"Mexico","Value":"42805"},{"Key":"Micronesia","Value":"42806"},{"Key":"Monaco","Value":"42807"},{"Key":"Mongolia","Value":"42808"},{"Key":"Montenegro","Value":"42809"},{"Key":"Montserrat","Value":"42810"},{"Key":"Morocco","Value":"42811"},{"Key":"Mozambique","Value":"42812"},{"Key":"Namibia","Value":"42813"},{"Key":"Nauru","Value":"42814"},{"Key":"Nepal","Value":"42815"},{"Key":"Netherlands","Value":"42816"},{"Key":"Netherlands Antilles","Value":"42817"},{"Key":"New Caledonia","Value":"42818"},{"Key":"New Zealand","Value":"42819"},{"Key":"Nicaragua","Value":"42820"},{"Key":"Niger","Value":"42821"},{"Key":"Nigeria","Value":"42822"},{"Key":"Niue","Value":"42823"},{"Key":"Northern Mariana Islands","Value":"42824"},{"Key":"Norway","Value":"42825"},{"Key":"Oman","Value":"42826"},{"Key":"Pakistan","Value":"42827"},{"Key":"Palau, Republic Of","Value":"42828"},{"Key":"Palestine","Value":"42829"},{"Key":"Panama","Value":"42830"},{"Key":"Papua New Guinea","Value":"42831"},{"Key":"Paraguay","Value":"42832"},{"Key":"Peru","Value":"42833"},{"Key":"Philippines","Value":"42834"},{"Key":"Poland","Value":"42835"},{"Key":"Portugal","Value":"42836"},{"Key":"Puerto Rico","Value":"42837"},{"Key":"Qatar","Value":"42838"},{"Key":"Republic Of Korea","Value":"42839"},{"Key":"Republic Of Moldova","Value":"42840"},{"Key":"Republic Of Yemen","Value":"42841"},{"Key":"Reunion","Value":"42842"},{"Key":"Romania","Value":"42843"},{"Key":"Russian Federation","Value":"42844"},{"Key":"Rwanda","Value":"42845"},{"Key":"Saint Kitts And Nevis","Value":"42846"},{"Key":"Saint Lucia","Value":"42847"},{"Key":"Sao Tome & Principe","Value":"42848"},{"Key":"Saudi Arabia","Value":"42849"},{"Key":"Senegal","Value":"42850"},{"Key":"Serbia","Value":"42851"},{"Key":"Serbia & Monten","Value":"42852"},{"Key":"Seychelles","Value":"42853"},{"Key":"Sierra Leone","Value":"42854"},{"Key":"Singapore","Value":"42855"},{"Key":"Slovakia","Value":"42856"},{"Key":"Slovenia","Value":"42857"}

DynamoDb

{"Key":"Mauritania","Value":"42803"},{"Key":"Mauritius","Value":"42804"},{"Key":"Mexico","Value":"42805"},{"Key":"Micronesia","Value":"42806"},{"Key":"Monaco","Value":"42807"},{"Key":"Mongolia","Value":"42808"},{"Key":"Montenegro","Value":"42809"},{"Key":"Montserrat","Value":"42810"},{"Key":"Morocco","Value":"42811"},{"Key":"Mozambique","Value":"42812"},{"Key":"Namibia","Value":"42813"},{"Key":"Nauru","Value":"42814"},{"Key":"Nepal","Value":"42815"},{"Key":"Netherlands","Value":"42816"},{"Key":"Netherlands Antilles","Value":"42817"},{"Key":"New Caledonia","Value":"42818"},{"Key":"New Zealand","Value":"42819"},{"Key":"Nicaragua","Value":"42820"},{"Key":"Niger","Value":"42821"},{"Key":"Nigeria","Value":"42822"},{"Key":"Niue","Value":"42823"},{"Key":"Northern Mariana Islands","Value":"42824"},{"Key":"Norway","Value":"42825"},{"Key":"Oman","Value":"42826"},{"Key":"Pakistan","Value":"42827"},{"Key":"Palau, Republic Of","Value":"42828"},{"Key":"Palestine","Value":"42829"},{"Key":"Panama","Value":"42830"},{"Key":"Papua New Guinea","Value":"42831"},{"Key":"Paraguay","Value":"42832"},{"Key":"Peru","Value":"42833"},{"Key":"Philippines","Value":"42834"},{"Key":"Poland","Value":"42835"},{"Key":"Portugal","Value":"42836"},{"Key":"Puerto Rico","Value":"42837"},{"Key":"Qatar","Value":"42838"},{"Key":"Republic Of Korea","Value":"42839"},{"Key":"Republic Of Moldova","Value":"42840"},{"Key":"Republic Of Yemen","Value":"42841"},{"Key":"Reunion","Value":"42842"},{"Key":"Romania","Value":"42843"},{"Key":"Russian Federation","Value":"42844"},{"Key":"Rwanda","Value":"42845"},{"Key":"Saint Kitts And Nevis","Value":"42846"},{"Key":"Saint Lucia","Value":"42847"},{"Key":"Sao Tome & Principe","Value":"42848"},{"Key":"Saudi Arabia","Value":"42849"},{"Key":"Senegal","Value":"42850"},{"Key":"Serbia","Value":"42851"},{"Key":"Serbia & Monten","Value":"42852"},{"Key":"Seychelles","Value":"42853"},{"Key":"Sierra Leone","Value":"42854"},{"Key":"Singapore","Value":"42855"},{"Key":"Slovakia","Value":"42856"},{"Key":"Slovenia","Value":"42857"},{"Key":"Solomon Islands","Value":"42858"},{"Key":"Somalia","Value":"42859"},{"Key":"South Africa","Value":"42860"},{"Key":"South

{"Key":"Papua New Guinea","Value":"42831"},{"Key":"Paraguay","Value":"42832"},{"Key":"Peru","Value":"42833"},{"Key":"Philippines","Value":"42834"},{"Key":"Poland","Value":"42835"},{"Key":"Portugal","Value":"42836"},{"Key":"Puerto Rico","Value":"42837"},{"Key":"Qatar","Value":"42838"},{"Key":"Republic Of Korea","Value":"42839"},{"Key":"Republic Of Moldova","Value":"42840"}

Static (Read Optimised) Volatile (Write Optimised)

Oh shit #2

DynamoDb

DynamoDb

Recovery Point Objective = 30

minutes

DynamoDb

Recovery Time Objective = 1 day

DynamoDb

Distributed Triggers

DynamoDb

DynamoDb

DynamoDb

DynamoDb

DynamoDb

S3 Versioned Buckets

DynamoDb

DynamoDb

DynamoDb

DynamoDb

DynamoDb Version Control

DynamoDb

Risk Profile

Low

High

Team

Skillset

Frontend

First Microservice

Quality

NoSql

Infrastructure

Security / DR

Estimation

Risk Profile at startup

PM 101 -Risk Management

Risk Profile: Team

1. Team availability2. Experience level of the Team3. Domain knowledge of the Team4. Technical skillset of the Team5. Team flow, size and track record

Risk Profile: Product

1. Product Complexity2. Product Criticality3. Product Stability4. Problem Clarity5. User perspection / expectations

Risk vs Returns

Risk vs Business Value

Retrospective

Monolith Gateway

Stakeholders

More than you think

Stakeholders

Metrics

Stakeholders

Transparency

Stakeholders

Support Balance

Business value

Team goals

Bake some goods

Time to iterate

Don’t underestimate

users

Pair pair pair

Questions

Thank you

https://github.com/PageUpPeopleOrg

https://en.wikipedia.org/wiki/Test-driven_developmenthttps://en.wikipedia.org/wiki/Behavior-driven_developmenthttps://angularjs.org/http://fsharp.github.io/FAKE/http://owin.org/https://aws.amazon.com/documentation/https://twitter.com/simonbrownhttp://phabricator.org/https://github.com/realestate-com-au/pacthttps://github.com/IdentityServer/IdentityServer3http://martinfowler.com/bliki/CQRS.html