call me never (full version)

Call Me Never

Building apps that don't wake you up

David Dossot

@ddossot

5 million visits a day...

... And counting.

Aiming for five nines...

... And 100x traffic

99.9962( °□°╯ ）╯︵ ┻━┻

ReadinessDegradation

RecoveryStrengthening

Load test

Readiness

Quick an dirty

Apache Bench

Richer scenarios

Apache jMeterGatling

Profiling

Readiness

Readiness

Replay access logs

Readiness

Welcome to the Internet

GET /admin.php

GET /catalog/compare/?action=DELETE_FROM_COMPARE_RESULT&IBLOCK_ID=3

Host: -- DROP DATABASE users;

Degradation

Decouple with queues and topics

Degradation

Delay, don't fall down

Degradation

Short circuit blocking-IOs

cache.get(key)

Degradation

Beware of benign code

Recovery

Self heal from transient failures

Recovery

Maintain stable state

Recovery

Replay stable state

Strengthening

Healthy health checks

Strengthening

Code defensively and log abundantly

try {

// risky stuff

} catch {

// useful log

}

Strengthening

Failure is opportunity for more tests

Thank you!

Q & A

call me never (full version)

Software