BUILDING A MODERNMICROSERVICES

ARCHITECTURE @

QCON LONDON - 2015YONI (JONATHAN) GOLDBERG

- GiltDirect, Sale Personalization, Loyalty, SEO, Post-purchase, Login/Registration

- MIT CS BS/Meng | Google | IBM | IDF

- Israel | Brooklyn | Coffee | JS/Node | Arduino | Running | Kite Surfing | Poker

THREE TAKE AWAYSFROM GILT'S STORY

Which problems will be solved?Which challenges will you face?Is it the right choice for you?

Flash Sales Business Founded in 2007

Top 50 Internet-Retailer~150 Engineers

WHAT IS GILT?

ANOTHER WAY TO LOOK ATGILT

THE CLASSIC STARTUPSTORY

THE EARLY DAYS2007 - Ruby on Rails the hottest new thing

The goal was to get to market fast

WE WERE ABLE TO HANDLE OURTRAFFIC PRETTY WELL

UNTIL LOUBOUTIN CAME TOGILT

TECHNOLOGY PAIN POINTS -2009

Spike required to launch 1,000s of ruby processesPostgres was overloadedRouting traffic between ruby processes sucked

DEV PAIN POINTS1000 Models/Controllers, 200K LOC, 100s of jobsLots of contributors + no ownershipDifficult deployments with long integration cyclesHard to identify root causes

WE NEEDED TO SOLVETHE PROBLEM FAST

THREE THINGS HAPPENEDStarted the transition to the JVMM(a/i)cro-Service Era StartedDedicated data stores

WHY JVM?Widely adoptedStableBetter support for concurrencyBetter GC vs MRI

FIRST 10 SERVICES

WE SOLVED 90% OF OUR SCALINGPROBLEMS

BUT NOT THE DEVELOPERS PAINPOINTS

SOLVED PAIN POINTSSpike required to launch 1,000s of ruby processesPostgres was overloadedRouting traffic between ruby processes sucked

STILL OPEN PAIN POINTSNew services became semi-monolithic1000 Models/Controllers, 200K LOC, 100s of jobsLots of contributors + no ownershipDifficult deployments with long integration cycles

WHY WE DOUBLED DOWN ON MICRO-SERVICES

Empower teams and ownershipSmaller scopeSimpler and Easier deployments and rollbacks

WE BEGAN THE TRANSITION TO SCALA AND PLAY

LOSA - LOTS OF SMALL (WEB) APPS [SAME AS MICRO-SERVICES BUT FOR WEB-APPS]

AS OF LAST WEEK WE HAVE AROUND 300 SERVICES IN PROD

WHY THE INCREASE?

APP BOOTSTRAP

rake bootstrap:admin-web # Bootstrap a admin-web servicerake bootstrap:client-server-core # Bootstrap a client-server-core servicerake bootstrap:play # Bootstrap a play servicerake bootstrap:play-ui-build # Bootstrap a play-ui-build servicerake bootstrap:sbt-library # Bootstrap a sbt-library servicerake bootstrap:schema # Bootstrap a schema service

DEMO

HOW TO DEFINE AMICROSERVICE?

FUNCTIONALITY SCOPENUMBER OF DEVS INVOLVED

"A SERVICE-ORIENTED ARCHITECTURECOMPOSED OF LOOSELY COUPLEDELEMENTS THAT HAVE BOUNDED

CONTEXTS"

- ADRIAN COCKCROFT

CURRENT CHALLENGESDeployments and Testing (Functional/Integration)Dev/Integration EnvironmentsService DiscoverabilityWho owns this service!?Monitoring

MICRO SERVICEDEPLOYMENT V3

CHALLENGES THAT WERE SOLVED IN V2:

Frustrating to deploy semi-manually (Capistrano)Scary to deploy other teams servicesHard to execute functional tests between services

SBTMotivation: Scala adoptionComplex Scala syntaxCool features: ~test, shell, consoleHard to debug

GILT-SBT-BUILDSimple config for all the servicesPulls many plugins: [nexus, testing, RPMs, run scripts, Monitoring, SemVer, ...]Custom commands (e.g 'sbt release')

ION-CANNON + SBTRun tests on dedicated EnvDark canary releasesEasy rollbacksIntegrated health checks

DATA CENTERLIMITATIONS

&& MORE TECHNOLOGY

OWNERSHIP

IMMUTABLEINFRASTRUCTURE

V3Per department AWS account and budgetLonger deploys [New instances / CNAME swaps]Each team has AWS expertise

API LOVE

WELL DEFINED REST APIS SOLVE DISCOVERABILITY, DOCUMENTATION

AND INNER ADOPTION

APIS @ GILTand

Describe the API in simple jsonAuto generates versioned docs, routes and clientsPer team - API design committee

www.apidoc.me Swagger.io

DEPENDENCY FUN [DEMO]

"MID-TIER MICRO-SERVICE"

BIGGEST PERFORMANCE CHALLENGENETWORK IO

ON DEV/INTEGRATION ENVIRONMENTSThe hardware is not strong enoughNo one wants to compile 20 servicesService Dependencies

EACH TEAM HAS A STAGINGENV

SERVICE_PORTS=[4001, #listing-service8235, #svc-user-set9420, #svc-free-fall7895, #svc-Loyalty8155, #web-loyalty9410, #web inventory status7898, #admin-loyalty7899, #notification7102, #rouge9530, #svc-component6802, #svc-waitlist-submit4066, #svc-action-sale....

PORT_FORWARD_ARGS=SERVICE_PORTS.map { |port|['-L', "#{port}:localhost:#{port}"]}

STAGING DIFFICULTIES:Hard to keep all the services up to dateMaxed our staging env capacitiesShared data across legacy apps

WITH OUR MIGRATION TO AWS [TEAM RESPONSIBILITY]A pool of integration instances and dbsDevelop with prod instances [Not a fan]

ON OWNERSHIP

"code stays much longer than people" - SB

CODE OWNERSHIP

CURRENT APPROACHCode Review!Code Review!Code Review!Team owns services, not individual developersOwnership transfer

DATA OWNERSHIP

WE TRANSITIONED TOMICRO-DBS

MICROSERVICES NEEDS THEIR OWNMONGODB | POSTGRES | RDS |

VOLDEMORT

MANAGE MICRO-RELATIONALDBS

SCHEMA EVOLUTIONMANAGER

https://github.com/gilt/schema-evolution-manager

PRINCIPLES OF SCHEMA EVOLUTION

MANAGERIndependent from the service codeManages the schema evolutions in a Git repoSchema changes are deployed as tar fliesNo rollbacksSchema changes are required to be incremental

ON MONITORING

THE TOOLS WE USE

GRAPHITE / OPENTSDB

BUT IT WASN'T ENOUGH

CAVE

HTTP://CAVELLC.GITHUB.IO

CONTINUOUS | AUDIT | VAULT |ENTERPRISE

ALERT EXAMPLES [SCALA]orders [shipTo: US].sum.5m < 1000response-time [svc: svc-team, env: prod].p999 > 1000

SUMMARY

WHAT'S NEXT ?BUILD YOUR NEXT FEATURE

IN A NEW SERVICE

@yoni_goldberg jgoldberg@gilt.com

QUESTION TIME

www.yonigoldberg.com