Backend & Cloud devskickoff meetup2015-03-19

TRAFI

J U R G I S @ T R A F I .C O M

Plan1. About this meetup – why and what

2. TRAFI case◦ Architecture overview

◦ Azure vs. AWS

◦ Queues and data streams (AWS Kinesis)

◦ Google BigQuery

Why meetup?Community is good!

◦ Share knowledge

◦ Similar problems – someone else maybe solved it

◦ Different approaches – widens your view

◦ It’s fun

What is “backend developer”?BEFORE NOW

ASP.NET

SQL

Mobile

API API

Load balancer

Queues

Cache

Logs

Metrics SQL NoSQL

Web JS

Worker

IAPI

Lots to learnCan build a lot very easily… if you know how

Challenges◦ Mobile, cloud, service-oriented architecture

◦ New tools every day (libraries, frameworks, cloud services)

◦ Agile, cross-functional teams

◦ No longer “C# devs”, “Java devs” and “SQL devs” – need to understand bigger picture

Meetup focusArchitecture & patterns

◦ How to build cloud (distributed) solutions

Tools & technologies

◦ Know what’s out there (right tool for the right job)

Possible topicsArchitecture & patterns

◦ Service-oriented, queues, micro services◦ Async, reactive programming◦ Functional programming◦ Domain driven design, hexagonal architecture

Languages◦ C#, Scala, Go, Java8, C++11, Ruby, Node.js, …

NoSQL◦ MongoDB, Redis, Firebase

Big data◦ Hadoop, Spark, BigQuery◦ Machine learning

Cloud services◦ Amazon Web Services (38 services!)◦ Microsoft Azure◦ Google Cloud

Anything you’ve found out, you wished someone told you!

TRAFIUse case

◦ Android, iOS, Web apps

◦ REST API (~100 reqs/sec)

◦ Memory & Compute intensive

◦ External data integrations

◦ Background jobs

Code◦ C#, ASP.NET MVC

◦ Some Java

◦ Some Python

TRAFI architectureMobile

.NET API .NET API

Elastic Beanstalk(load balancer)

Data streaming(Kinesis)

ElastiCache

Logentries(logs)

Librato(metrics)

MS SQL BigQuery

Azure Website

Background jobs(Win VM)

Internal Java API(Linux VM)

Infrastructure◦ AWS Elastic Beanstalk

◦ AWS EC2 instances (c3.4xlarge)

◦ Azure websites

◦ Azure worker roles

Storage◦ MS SQL server

◦ AWS Kinesis

◦ AWS ElastiCache

◦ Google BigQuery

Metrics & Logs◦ Logentries

◦ Librato

Lesson #1: Azure vs AWSMigrated .NET API

◦ Azure: Web roles + Azure SQL

◦ AWS: Elastic Beanstalk + RDS SQL

Advantages◦ 2x faster CPU performance!

◦ Faster deployment (also “.NET-friendly”)

◦ More flexible load balancer, auto scaling

◦ Better metrics and logs

◦ SQL – faster, easier to manage, better metrics

Disadvantages◦ No “staged rollout”, only swap

Lesson #2: use messagingUse case: streaming data from external sources

Don’t reinvent the wheel – it’s not as easy as it looks!

External integration

API 1

API 2GPS data DB

Background

External integration

API 1

API 2GPS data

Background

Stream

Lesson #2: use messagingQueues

◦ AWS SQS

◦ Azure storage queues

◦ RabbitMQ

Data streams (publish/subscribe)◦ AWS Kinesis

◦ Azure Service bus

◦ Kafka

Queue

Worker 1

Worker 2B

A

B

Stream

Listener 1

Listener 2B

A

B

A

A

B

A

Lesson #3: storing logsUse case: store insert-only data for later analysis

◦ Web server request logs

◦ App event logs

◦ GPS logs…

Storing logs attempt 1Traditional SQL

◦ Good for small amount of data (<10 GB)

◦ Limited insert capacity

◦ Slow queries

◦ Expensive ($1.00 /GB/month)

Storing logs attempt 2Azure table storage (NoSQL)

◦ Cheap ($0.07 /GB/month)

◦ Unlimited data (partitioning)

◦ Only simple queries, slow

Storing logs attempt 3Google BigQuery (columnar store)

◦ Very Cheap ($0.02 /GB/month)*

◦ Unlimited data (partitioning)

◦ Fast advanced SQL queries!

◦ Structured data

Open questionsWebsites

◦ Pretty nice in Azure – easy deployment, cheap

Background jobs◦ No “worker roles” in AWS

◦ Running as “Windows Services” in custom EC2 VM

◦ Docker?

Thank you

Questions?

Your experience?

What’s next?

Beer time…