cloudsolutionday 2016: docker & faas at getvero.com

TU HOANGSoftware Engineer / @rebyn / [email protected]

Docker @ VeroData Migration with Docker @ Vero

CLOUD FUNCTIONS @ VERO

@getveroAutomate customer interactions based on what your customers do, ie: fire welcome coupon campaign when● User signs up; and● Has at least an

outstanding item in cart for a week; and

● Has never received a coupon before.

+250 large consumer businesses use Vero daily:

#CONTAINERS- Opinionated: EC2-Spot,

ECS, ELB, Logentries- 20+ kinds of workers- Memory spec ranging

from 302MB to 2048MB memory

- Anytime: 148 containers (across 80 instances)

- Autoscaled: +350

- Peak: 16 million emails/day

- Weekly volume: 32.2M 75.2M

- Track 20.000 46.296 customer actions/min (+1B +2B/mth)

- Software engineering: 6- DevOps-y engineers: 3

Scale @getvero (with changes since Nov 2015)

Cloud Functions- Stateless compute containers- Ephemeral- Event-triggered- (AND/OR) Significantly depend on third-party

services

Cloud Function 1/7: autoscaling

Cloud Function 2/7: deliverability

Cloud Function 3/7: newsletter diagnosis

Cloud Function 3/7: deployment

@dustay ecs:deploy_by_image mailship:master

1. Check latest SHA’s pending status2. Check ongoing deployments (if any)

2a. YES: cancel | set a reminder2b. NO: DEPLOY AWAY


@dustay stats:low --threshold=50 | stats:refetch $newsletter_id

Triggershttps://bot.getvero.com:1234 /v1/triggers/UUID

HTTP

Chat commands

- Amazon S3

- Amazon DynamoDB

- Amazon Kinesis Streams

- Amazon Simple Notification Service

- Amazon CloudWatch Logs

- Scheduled Events (powered by Amazon CloudWatch Events)

- Amazon API Gateway

- Other Event Sources: Invoking a Lambda Function On Demand

Trigger Options

- Cloud agnostic (vanilla Docker)

- Navigate our setup of security firewalls (though not much desired since most services are public)

- Language agnostic (unified input/output)

Why we rolled out our own

cloud function service

- Of course, it’s always better if someone else manages it

- Startup latency = used to be an issue (then we learnt how to autoscale and/or prewarm infra)

- Horizontal scaling = rough around edges (also applies to our services (micro/nano), not just functions)

- Testing was a pain

Why we ate up along the

way

- Development friction = extremely minimal now

- Services are being created with confidence and scalability in mind despite small team

- 100% automation = within reach now

- Testing = microservice testing. Should bring about no changes.

- Matured autoscaling agent + spot fleet management = #BIGWIN for both functions and services

- See 75% cost reduction upon spot adoption

#WINS

- Stack decomposing = trend forward. Start first with parts of a monolith that interact with external services significantly.

- We control scaling-up. This is equally crucial.

#WINS

- Functions are being gradually treated as first class citizens

- We usually sit down and approach an issue re: can we narrow this business down to nano services / functions

Misc

- Credentials should be loaded upon instance provisioning or container provisioning, not hardcoded into scripts

- Vendor lockin = a concern, though not pressing. IAM- and security group-based services = kept to minimum and has failover.

- Instrumentation = crucial.

- Request/response mapping templates = must. Shared library.

Gotchas

Infrastructure Setup

Development Productivity

To quote Amazon:Focus on the code that mattersInnovate rapidlyReduce time to market/product

Continuous scaling

DEPLOYMENT(I run out of time)

https://www.getvero.com/careers(Senior Backend Engineers / Site Reliability Engineers

much needed!)

Thanks for tuning in.

cloudsolutionday 2016: docker & faas at getvero.com

Technology