loki: an opensource zipkin/prometheus mashup written in go
TRANSCRIPT
![Page 1: Loki: An Opensource Zipkin/Prometheus Mashup written in Go](https://reader031.vdocument.in/reader031/viewer/2022021922/58e4a71e1a28abf5428b7389/html5/thumbnails/1.jpg)
Loki: a Zipkin/Prometheus Mashup@tom_wilkie, CNCFCon Berlin April 2017
+ =
![Page 2: Loki: An Opensource Zipkin/Prometheus Mashup written in Go](https://reader031.vdocument.in/reader031/viewer/2022021922/58e4a71e1a28abf5428b7389/html5/thumbnails/2.jpg)
![Page 3: Loki: An Opensource Zipkin/Prometheus Mashup written in Go](https://reader031.vdocument.in/reader031/viewer/2022021922/58e4a71e1a28abf5428b7389/html5/thumbnails/3.jpg)
Why did I write my own tracer?
![Page 4: Loki: An Opensource Zipkin/Prometheus Mashup written in Go](https://reader031.vdocument.in/reader031/viewer/2022021922/58e4a71e1a28abf5428b7389/html5/thumbnails/4.jpg)
Debugging a latency performance issue
with Cortex…
Distributor
Ingester Ingester…
![Page 5: Loki: An Opensource Zipkin/Prometheus Mashup written in Go](https://reader031.vdocument.in/reader031/viewer/2022021922/58e4a71e1a28abf5428b7389/html5/thumbnails/5.jpg)
Well, thats my rationalisation…
![Page 6: Loki: An Opensource Zipkin/Prometheus Mashup written in Go](https://reader031.vdocument.in/reader031/viewer/2022021922/58e4a71e1a28abf5428b7389/html5/thumbnails/6.jpg)
In reality, this is attempt #2
• Prototype “Weave Tracer” circa 2015
• Concept didn’t require application instrumentation
• Used ptrace to intercept syscalls and infer application behaviour
• Kinda worked, for a very limited definition of “worked”
![Page 7: Loki: An Opensource Zipkin/Prometheus Mashup written in Go](https://reader031.vdocument.in/reader031/viewer/2022021922/58e4a71e1a28abf5428b7389/html5/thumbnails/7.jpg)
Prometheus = Greek god. Loki = Norse equivalent?
![Page 8: Loki: An Opensource Zipkin/Prometheus Mashup written in Go](https://reader031.vdocument.in/reader031/viewer/2022021922/58e4a71e1a28abf5428b7389/html5/thumbnails/8.jpg)
So what makes Loki different?
![Page 9: Loki: An Opensource Zipkin/Prometheus Mashup written in Go](https://reader031.vdocument.in/reader031/viewer/2022021922/58e4a71e1a28abf5428b7389/html5/thumbnails/9.jpg)
Prometheus is to Graphite as
Loki is to Zipkin
![Page 10: Loki: An Opensource Zipkin/Prometheus Mashup written in Go](https://reader031.vdocument.in/reader031/viewer/2022021922/58e4a71e1a28abf5428b7389/html5/thumbnails/10.jpg)
Push vs Pull
https://prometheus.io/docs/introduction/faq/#why-do-you-pull-rather-than-push?
https://prometheus.io/blog/2016/07/23/pull-does-not-scale-or-does-it/
![Page 11: Loki: An Opensource Zipkin/Prometheus Mashup written in Go](https://reader031.vdocument.in/reader031/viewer/2022021922/58e4a71e1a28abf5428b7389/html5/thumbnails/11.jpg)
• Jobs must know where monitoring is • Can overwhelm graphite with too many
samples
Graphite Prometheus
scraping
your jobs
Prometheus
pushing
your jobs
Graphite
• Tell Prometheus where jobs are (via service discovery)
• Prometheus can back off when overwhelmed • Prometheus knows the identity of each job
![Page 12: Loki: An Opensource Zipkin/Prometheus Mashup written in Go](https://reader031.vdocument.in/reader031/viewer/2022021922/58e4a71e1a28abf5428b7389/html5/thumbnails/12.jpg)
Zipkin Loki
scraping
your jobs
Loki
pushing
your jobs
Zipkin
![Page 13: Loki: An Opensource Zipkin/Prometheus Mashup written in Go](https://reader031.vdocument.in/reader031/viewer/2022021922/58e4a71e1a28abf5428b7389/html5/thumbnails/13.jpg)
http://job/traces
your job
Loki client library
spans
scraping
Loki
• Client library keep pending spans in an in-memory ring buffer.
• /traces HTTP handler grabs all the in-memory spans and serialises them using Thrift.
• Spans will be dropped if not collected frequently enough.
![Page 14: Loki: An Opensource Zipkin/Prometheus Mashup written in Go](https://reader031.vdocument.in/reader031/viewer/2022021922/58e4a71e1a28abf5428b7389/html5/thumbnails/14.jpg)
• Retrieval library ‘knows’ identity of scraped endpoints, adds that to received spans
• … jobs don’t need to know their own identity
• … can be consistent with identity used in Prometheus
• Naive in memory storage implementation
• … makes queries slow, as its just a loop.
• Zipkin-compatible API endpoints
• UI _is_ the Zipkin UI
LokiPrometheus retrieval library
In memory storage
Zipkin API
Zipkin UI
![Page 15: Loki: An Opensource Zipkin/Prometheus Mashup written in Go](https://reader031.vdocument.in/reader031/viewer/2022021922/58e4a71e1a28abf5428b7389/html5/thumbnails/15.jpg)
Its all open source:
https://github.com/weaveworks-experiments/loki
…and it’s written in go
![Page 16: Loki: An Opensource Zipkin/Prometheus Mashup written in Go](https://reader031.vdocument.in/reader031/viewer/2022021922/58e4a71e1a28abf5428b7389/html5/thumbnails/16.jpg)
This all sounds great! Where’s the catch?
![Page 17: Loki: An Opensource Zipkin/Prometheus Mashup written in Go](https://reader031.vdocument.in/reader031/viewer/2022021922/58e4a71e1a28abf5428b7389/html5/thumbnails/17.jpg)
❌ Client library doesn’t actually support multiple scrapers (yet)
❌ Loki query performance sucks (for now)
❌ Loki single-process architecture limits scalability
❌ Can dropped spans, gets worse through jitter
![Page 18: Loki: An Opensource Zipkin/Prometheus Mashup written in Go](https://reader031.vdocument.in/reader031/viewer/2022021922/58e4a71e1a28abf5428b7389/html5/thumbnails/18.jpg)
… that Cortex performance issue
![Page 19: Loki: An Opensource Zipkin/Prometheus Mashup written in Go](https://reader031.vdocument.in/reader031/viewer/2022021922/58e4a71e1a28abf5428b7389/html5/thumbnails/19.jpg)
Debugging a latency performance issue
with Cortex…
Distributor
Ingester Ingester…
![Page 20: Loki: An Opensource Zipkin/Prometheus Mashup written in Go](https://reader031.vdocument.in/reader031/viewer/2022021922/58e4a71e1a28abf5428b7389/html5/thumbnails/20.jpg)
![Page 21: Loki: An Opensource Zipkin/Prometheus Mashup written in Go](https://reader031.vdocument.in/reader031/viewer/2022021922/58e4a71e1a28abf5428b7389/html5/thumbnails/21.jpg)
It was garbage collection…
![Page 22: Loki: An Opensource Zipkin/Prometheus Mashup written in Go](https://reader031.vdocument.in/reader031/viewer/2022021922/58e4a71e1a28abf5428b7389/html5/thumbnails/22.jpg)
100ms ➡ 25ms
![Page 23: Loki: An Opensource Zipkin/Prometheus Mashup written in Go](https://reader031.vdocument.in/reader031/viewer/2022021922/58e4a71e1a28abf5428b7389/html5/thumbnails/23.jpg)
Demo
![Page 24: Loki: An Opensource Zipkin/Prometheus Mashup written in Go](https://reader031.vdocument.in/reader031/viewer/2022021922/58e4a71e1a28abf5428b7389/html5/thumbnails/24.jpg)
Client Library
• Make is support multiple scrapers
• Move away from thrift to protos
• More languages
• Useful HTML /traces
Loki Server
• Local storage with BoltDB
• Make queries faster
• Make it distributed, use cloud storage
TODO
![Page 25: Loki: An Opensource Zipkin/Prometheus Mashup written in Go](https://reader031.vdocument.in/reader031/viewer/2022021922/58e4a71e1a28abf5428b7389/html5/thumbnails/25.jpg)
Why did I write my own tracer?
![Page 26: Loki: An Opensource Zipkin/Prometheus Mashup written in Go](https://reader031.vdocument.in/reader031/viewer/2022021922/58e4a71e1a28abf5428b7389/html5/thumbnails/26.jpg)
Because with OpenTracing, I can.
![Page 27: Loki: An Opensource Zipkin/Prometheus Mashup written in Go](https://reader031.vdocument.in/reader031/viewer/2022021922/58e4a71e1a28abf5428b7389/html5/thumbnails/27.jpg)
Thank you!Questions?
![Page 28: Loki: An Opensource Zipkin/Prometheus Mashup written in Go](https://reader031.vdocument.in/reader031/viewer/2022021922/58e4a71e1a28abf5428b7389/html5/thumbnails/28.jpg)