Highway to heaven

Building microservices in the cloud Frankfurter Entwicklertag 2016

Agenda

Meet AutoScout24

Shifting gears

How we build our services

How we organize ourselves

@cdeger & @wolfwolf

AutoScout24 - 30.000ft

C C

Baseline AutoScout24 IT

Highly optimized, but of last decade

IT platform supported growth for >6 years

Microsoft oriented stack

Enterprise IT setup - MTBF over MTTR

Proven agile and lean principles

C

C C

New CEO

C C

Do you attract talent?

C C

C C

Project Tatsu

W W

Monolith to Microservices

Data center to AWS

.NET / Windows to JVM / Linux

Devs + Ops to DevOps culture

Involve product people

Five challenges

Technical transformation

W W

Change the wheels while driving

W W

Strategic Goals Goals of the business side

Architectural Principles High-Level Principles

Design and Delivery Principles Tactical measures

Reduce Time to Market Speed, Fast Feedback

Cost Efficiency Collect metrics to allow decisions cost vs. value.

Support Data-Driven Decisions Listen to users and validate hypothesis.

Provide as many relevant metrics & data as possible.

You build it, you run it The team is responsible for shaping, building, running and

maintaining its products. Fast feedback from live and customers

helps us to continuously improve.

Organized around Business Capabilities Build teams around products not projects. Follow the domain

and respect bounded contexts. Inverse Conway Maneuver.

Shared Nothing By default avoid sharing and tight coupling, except for the big

things in common. Don’t create the next monolith.

Macro and Micro Architecture Clear separation. Autonomous micro services within the rules

and constraints of the macro architecture.

AWS First Favor AWS platform service over managed service, over self-

hosted OSS, over self-rolled solutions.

Data-Driven/ Metric-Driven Collect metrics from processes and applications. Analyze, alert

and act on them.

Eliminate Accidental Complexity Strive to keep it simple. Focus on essential complexity. You build

one, you delete one.

Autonomous Teams Make fast local decisions. Be responsible. Know your

boundaries. Share findings.

Infrastructure As Code Automate everything: Reproducible, traceable and tested.

Immutable servers over snowflake servers.

Collaboration Culture Engineers from all backgrounds work together in collaborative

teams as engineers and share responsibilities. No silos.

Be Bold Go into production early. Value monitoring over tests. Recover

and learn. Optimize for MTTR not MTBF.

Security, Compliance and Data Privacy Security must be included from the beginning and everybody’s

concern. Keep data-privacy in mind.

Another goal Work in progress...

Containment and Boundaries Align blast radius and vendor lock-in with the boundaries of the

organization or business capabilities.

C C

Principles

Organized around business capabilities

You build it, you run it

Be bold

Macro and micro architecture

Shared nothing

Shared infrastructure

Shared nothing

Availability over shared nothing

Convenience offerings

No side effects

Fast local decisions over committee

Respect family ties W W

How (not) to share

Use over re-use

Re-use only after hardening

Copy n’paste, OSS, library

Pull instead of push

EventPublisher24

W W

How many environments?

Which versions on staging?

Prod differs anyway: Load, data, patterns

V2 V3

V6 V5

V4

V7

V5

V8

Engineer CI Dev Staging

V1

V4

Prod

C C

Dev and prod is enough

Consumer driven contracts

Shadow traffic

Semantic monitoring

Smoke tests or canary releases

V2 V3

V6 V5

V4

V7

V5

V8

Engineer CI Dev Prod

C C

Event Sourcing and data pumps

One way data highway

Event Sourcing - store history of all changes

W W

SQS + S3

Kinesis + S3

Kinesis + DynamoDB

SQS + DynamoDB

Proxy + DynamoDB

DynamoDB

Evolution

W W

Frontend integration

Shared nothing

Autonomous teams

One domain

High optimisation

C C

Frontend integration

C C

PageSpeed Module

css (page+fragment)

js (page+fragment)

ngx_pagespeed

css (page)

js (page)

css (fragment)

js (fragment)

C C

How to build autonomous teams

Do not fall back into old behaviours

Beware of Mandelbrot teams

Pager duty so that you run it

Fix broken windows

Part-time ops not working

Not all T-shapes are the same

Wolf

W W

Infrastructure guild

Agree on things to do

Share learnings

Delegate implementation to teams

Empty backlog should be normal

Infrastructure product teams needed?

W W

cdeger@autoscout24.com @cdeger

wschlegel@thoughtworks.com @wolfwolf

Attributions

Blue sky, white-gray clouds by nature protector Natubico, www.vivism.info [CC BY-SA 3.0]

http://commons.wikimedia.org/wiki/File%3ABlue_sky%2C_white-gray_clouds.JPG

A Danish Perspective by NASA [Public domain] http://commons.wikimedia.org/wiki/File%3AA_Danish_Perspective.jpg

http://commons.wikimedia.org/wiki/File%3ANASAComputerRoom7090.NARA.jpg

GREG

EINRAD

Amazon16 by Neil Palmer/CIAT [CC BY-SA 2.0] https://www.flickr.com/photos/ciat/5641594952

BERGSTEIGER

Barber in Cameroon by James Emery from Douglasville, United States (Daddy Joe_1355) [CC BY 2.0]

http://commons.wikimedia.org/wiki/File%3ABarber_in_Cameroon.jpg

Wide objectives by Kivela (Own work) [Public domain]

href="http://commons.wikimedia.org/wiki/File%3AWide_objectives.jpg

Transformer Fire Barrier by GerryS1 (Own work) [CC BY-SA 3.0 or GFDL]

http://commons.wikimedia.org/wiki/File%3ATransformer_Fire_Barrier.jpg

Attributions (cont)

Alonso Renault Pitstop Chinese GP 2008 by Bert van Dijk (Pitstop F1 ING Renault) [CC BY-SA 2.0]

http://commons.wikimedia.org/wiki/File%3AAlonso_Renault_Pitstop_Chinese_GP_2008.jpg

Principle of Panchasheel by Prakash Adhikary (Own work) [CC BY 3.0]

http://commons.wikimedia.org/wiki/File%3APrinciple_of_Panchasheel.JPG

Traffic Jam by Doo Ho Kim [CC BY-SA 2.0] https://www.flickr.com/photos/titicat/3049591547

Pellets by The original uploader was Richard Mayer at German Wikipedia [GFDL or CC-BY-SA-3.0]

http://commons.wikimedia.org/wiki/File%3APellets.jpg

Pipes and Valves by Uwe Hermann [CC BY-SA 2.0] https://www.flickr.com/photos/73628542@N00/6272975359

Size variation in Coccinella undecimpunctata (2127991716) by Gilles San Martin from Namur, Belgium [CC BY-SA 2.0]

http://commons.wikimedia.org/wiki/File%3ASize_variation_in_Coccinella_undecimpunctata_(2127991716).jpg

Mille crêpe by Laitr Keiows (Own work) [CC BY-SA 3.0 or GFDL]

http://commons.wikimedia.org/wiki/File%3AMille_cr%C3%AApe.jpg

Country Energy power line replacement 01 by Bidgee (Own work) [CC BY-SA 3.0]

http://commons.wikimedia.org/wiki/File%3ACountry_Energy_power_line_replacement_01.jpg

Puzzling by Bernd Gessler (Own work) [CC BY-SA 3.0] https://commons.wikimedia.org/wiki/File%3APuzzling.JPG

Attributions (cont)

Sharing Sucks (4536747557) by eyeliam from Portland, United States [CC BY 2.0]

http://commons.wikimedia.org/wiki/File%3ASharing_Sucks_(4536747557).jpg

7Line 9184 (8263568241) by Metropolitan Transportation Authority of the State of New York (7Line_9184 Uploaded by

tm) [CC BY 2.0] http://commons.wikimedia.org/wiki/File%3A7Line_9184_(8263568241).jpg

England rugby team 1905 by Russell & Sons (The Graphic) [Public domain or Public domain]

http://commons.wikimedia.org/wiki/File%3AEngland_rugby_team_1905.jpg

Wandergeselle by Sigismund von Dobschütz [CC BY-SA 3.0]

http://commons.wikimedia.org/wiki/File%3AWandergeselle_02.JPG

Faber-Rechenschieber 5304 by User:Karl Gruber (Own work) [CC BY-SA 4.0]

http://commons.wikimedia.org/wiki/File%3AFaber-Rechenschieber_5304.JPG

Wheel clamps Texas by Richard Anderson from Denton, United States (Boots.) [CC BY-SA 2.0]

http://commons.wikimedia.org/wiki/File%3AWheel_clamps_Texas.jpg

GuadalupeNOLA15Oct07Thanks by Infrogmation of New Orleans (Photo by Infrogmation) [GFDL or CC BY-SA 3.0]

http://commons.wikimedia.org/wiki/File%3AGuadalupeNOLA15Oct07Thanks.jpg

AtariBasic by Calin99 (Own work) [GPL] http://commons.wikimedia.org/wiki/File%3AAtariBasic.png Spare wheel by Brian Snelson [CC BY 2.0] https://commons.wikimedia.org/wiki/File:Spare_wheel_-_Flickr_-_exfordy.jpg

Backup

Unified

Logs

C C

Two stacks

Cash stack meets shiny new stack

One company

Lights on in cash stack

Feature freeze

Where to build new features?

Ease of integration helps business people C C

From documents to events

Refactoring toward deeper insight

From CRUD to sync

Offline first

Writes are expensive

USD 20 over USD 500

C C

How many layers to estimate a price

Evolving architecture

R backend and Play application

Single Play application

Play backend and Play web server

Long feedback cycles

Frequency of change

W

W W

Watchlist evolution

DynamoDB as JSON document store

Scheduled user journey test

Capacity exceeded

Requests throttled

Latency spikes for real users

C

C C

Focus sliders

Product over platform

time

W W

IT

business

Focus sliders

Product over platform

Delivery over learning

time

W W

IT

business