Go && Python Event driven ML

Lalafo case

How it all started

Part 1. The Phantom Menace

StackOverflow: Ready to rescue

200 RPM

$

Feedback

$

Feedback

Processes Monitoring

RAM - expiration?

Relational DB?

Any issues there?1. Request monitoring (?)2. Hard to reason GPU usage (varies 3 to 7GB)3. ~5Gb RAM out of the box (scale?)4. Redis is in-memory storage (temporary)5. Threading != Parallel

http://www.nooooooooooooooo.com/

What if we split API and ML

Part 2. A new hope

Requirements

Lalafo case

1. Single image/request processing time: < 2 seconds2. Visibility3. Persistence4. Scalability 5. Make it extendable for new features

a. Price predictionb. Similarity searchc. Segmentation

6. SDK friendly (well documented, tested etc)

$

Feedback

Processes Monitoring

RAM - expiration?

Relational DB?

After 2 months

API Listeners

Feedback

$

Requirements

Lalafo case

1. Single image/request processing time: < 2 seconds2. Visibility - Decouple request and prediction3. Persistence 4. Scalability 5. Make it extendable for new features

a. Price predictionb. Similarity searchc. Segmentation

6. SDK friendly (well documented, tested etc)

2 seconds per request

Part 3. The Empire Strikes Back

4 seconds per request

API Listeners

1 second0.01 sec

segmentio/kafka-go -> shopify/sarama

API Listeners

0.02 sec0.01 sec

3 seconds per request

API Listeners

0.02 sec

0.3 second

0.3 second

Synchronous request

How large is every image

2.4 seconds per request

API Listeners

0.02 sec

Synchronous request

2.4 seconds per request

API Listeners

0.02 sec

Synchronous request

2 seconds per request

API Listeners

0.02 sec

Synchronous request

Add ThreadPool

2 seconds per request

Success? Not yet

3 seconds per request

API Listeners

0.02 sec

3 seconds per request

API Listeners

0.02 sec

1 second and growing

3 seconds per request

2 seconds per request

¯\_(ツ)_/¯

2 seconds per request

API Listeners

0.02 sec

200 RPM -> 1200 RPM

2 seconds per request

API Listeners

0.02 sec

ScaleScale

Master -> Slave -> balancer

Topics -> Partitions

< 1 second per request

Success? Not yet

Out of 8Gb

How to PyTorch in production

7.6 Gb -> 600 Mb

1 second per request

Success? Not yet

Back to 2, 5, 10 seconds per request

Troubleshooting

API Listeners

0.02 sec

? ?

Troubleshooting

API Listeners

0.02 sec

?

Troubleshooting

API Listeners

0.02 sec

?175k records per request

Troubleshooting

API Listeners

0.02 sec?

Troubleshooting

API

SELECT * FROM ads LIMIT 1

25 seconds -> 0.02 seconds

0.02 seconds -> 0.002 second

Success? Not yet

Faust 1.4.6: No latest offset

https://github.com/robinhood/faust

3K images per minute 0_o

Success? Not yet

Issues to solve1. Occasional spikes in performance (GC, network latency)2. Message broker (Kafka rebalancing, offset etc)3. How to handle DB migrations4. Something we are not aware of yet

Lessons learnt1. CPU bound tasks != IO bound (¯\_(ツ)_/¯)2. High coupling - low cohesion3. You need to know how to cook MongoDB4. Go is not that obvious and library reach as Python5. Simple != Easier6. Concurrency != Parallelism (obviously)

< 1 second per request

+ Live statistics from PostgreSQL

API Listeners

Thank you everyone

Success?