serverless

journey of

shop.LEGO.com

sheen brisalsLEGO

talk focus➢ reasons for the change

➢ high level architecture

➢ core principles

➢ overall process

➢ tools & techniques ➢ best practices

➢ development examples ➢ lessons

➢ what next?

why change?

BF-CM 2017

decouple

post BF-CM 2017

BF-CM 2018

operational reasons

• aging e-commerce platform

• release nightmare

• delayed features and fixes

• monolith

• tightly coupled dependencies

• limited scalability

• on premisis

business needs

• market expansion

• consumer experience

• scalable platform

• fast turnaround

• automation

• modernise

• data driven

Events

PAB

Site map

Pricing

Products

Stores

Promotions

Customer

Wish list

Payments

Prices

VIP

Discounts

Orders

Address

BAP

Cart

principles

➢ cloud & AWS

➢ serverless

➢ managed services

➢ API based

➢ decouple dependencies

architecture

set piece architecture development

A Set Piece Architecture requires planning and execution, but

can be rehearsed in advance

Tax

Import/Export

Data Stream

process

two phase development

Nov ‘18 May ‘19July ‘18

• ImpEx

• Product Description

• Product Contents

• Sales Tax

• Data Layer

• Checkout

• Payments

• Orders

• Customer

• VIP

• Promotions

agile and scrum

• practical agile if not text book style

daily standup

retro

planning

grooming

review• JIRA

• 2 week sprints

focus squads

services

checkout content

infra

management

• 4 focus areas

• change and adapt

• squad autonomy

• business analyst

• product owner

• delivery manager

• director

devops and automation

• not just programming

• embrace AWS & cloud

• shared responsibility with infra

• build, deploy & run culture

dev, qa, prod accounts

• single sign on

• ease of use

• cost control

• access separation

• fits well with CI/CD

design - review - decide

design

review

agree

develop

• decide quickly

• bold approach

• open for ideas

• change as needed

solution detailing

• detail the design

• do PoC if needed

• input from engineers

• change as needed

implement with tests• started off with units tests

• shared with QA

• integration tests now mandatory

• crucial for CI/CD pipeline

CI - CD• PR merge triggers CI

• builds tests and deploys to QA

Submit PR Code AnalysisPush to GIT

QA Tests

Unit Tests

IntegrationQA

• production deploy is separate pipeline at the moment

tools

techniques

javascript

• existing skills

• javascript vs Golang

• popularity

• framework support

const handler = NewRelic.monitorLambda(async (event, context, callback) =>

{

log.initDefaultParams(context);

let response;

try {

const country = get(event, 'queryStringParameters.country');

validateRequest(country);

const client = creatorWithToken(event.headers.Authorization);

let cart;

cart = await getCurrentCartForUserByCountry(client, country);

if (cart) {

cart = await recalculateCart(client, cart);

} else {

cart = await createMyCart(client, country, CartType.STANDARD);

}

response = createResponse(200, cart);

} catch (e) {

response = handleError(e);

}

callback(null, response);

});

serverless framework

• quick to get going

• easy of use

• popularity

• good plugins

• CloudFormation support

GitHub & mono repo

• independent service

deploy pipeline

• package per service

CircleCI

• not perfect but improving

• no auto deploy to production yet

best practices

cost awareness

requests cost + compute cost

requests cost + data transfer + cache

read (consistent/eventual) + write

(normal/transactional) + storage +others

built-in & managed services

use lambda triggers – S3, SQS, SNS,…

avoid lambda if not needed for integration

use TTL to expire unwanted data

use life cycle policies effectively

security - secure it

granular level policies

functions:

vip-crt-status-check:

name: ${self:service}-crt-status-check-${self:provider.stage}

handler: functions/statusCheck/statusCheck.handler

memorySize: 256

timeout: 6

environment:

….

iamRoleStatements:

- Effect: Allow

Action:

- ssm:GetParameters

Resource:

-…. :_common.PARAMETER_NAME_CRT_API_KEY}

- …. :_common.PARAMETER_NAME_CRT_TEST_USER_ID}

- …. :parameter/vip/crt/status/*

- Effect: Allow

Action:

- ssm:PutParameter

Resource:

-…. :_common.PARAMETER_NAME_CRT_API_STATUS}

- …. :_common.PARAMETER_NAME_CRT_POS_STATUS}

security - secure it

granular level policies api keys, lambda auth, usage plan

creds in Parameter Store secrets manager

S3 signed url item attribute level permission

log, trace & monitor

use CloudWatch logs add traceability in logs

enable CloudTrail

setup alerts and alarms monitor everything

elasticsearch

engineering

language choice code quality

develop & optimise be open for ideas

communicate collaborate co-locate

start-up mentality start small, scale fast

set piece development

examples

product - impex

product - impex

Product Feeds Commerce PlatformFeed Messages

product - impex

Product Feeds New PlatformTransform Feeds

impex – design considerations

• feeds in JSON format

• failure re-try process

• decoupled processing

• failure notification

• immediate handling of feeds

• independent feed flow

impex

impex

impex

data layer aka event streaming

event stream

events – design considerations

• low cost

• support different event types

• ability to add new consumers and producers

• near realtime processing

under a minute

• highly available

• event data in JSON

events

events{

"commerce_event":["service-data-layer-recsengine-STAGE","service-data-layer-newrelic-STAGE","service-data-layer-optimizely-STAGE"

],"page_event":[

"service-data-layer-recsengine-STAGE","service-data-layer-optimizely-STAGE"

],"recs_event":[

"service-data-layer-recsengine-STAGE"]

}

event stream

lessons - team

laydown the principles

• crucial to move

forward and

stick together

as a team

sensible solution

• start simple and

evolve

• listen, learn,

improve

be cost aware

• serverless is not free

• educate the team

• optimise

proof of concept (PoC)

• builds confidence

• hands-on learning

• stakeholder showcase

act fast

• agree & act

• just do it!

• small steps but faster

lessons - aws

monitor & optimise lambdas

Duration: 54908.63 ms Billed Duration: 55000 ms

Memory Size: 2048 MB Max Memory Used: 265 MB

don’t keep (pay for) unwanted data• S3 lifecycle policies • CloudWatch - retire logs

• use DynamoDB TTL • SQS / DLQ storage retension

S3 - have a structure• use folder structure • S3 not search friendly

Firehose – mind the buffer• use the choice wisely

• buffer data size can

vary

• design the app/lambda

accordingly

watch the DLQs

• prioritise failures

• selective & tiered monitoring

• implement re-play loop

• adjust DLQ delay time

what next?

➢ strengthen security

➢ lambda layers

➢ key/secrets rotation

➢ API caching

➢ unified CI/CD pipeline

➢ monitor ext services

➢ handling versions

➢ improve DLQ monitoring

➢ X-Ray

➢ staging env

➢ new projects

go

build

serverless

thank

you!

@sheenbrisals