building a culture of observability at stripe
TRANSCRIPT
Building a Culture of Observability at Stripe
Maaaaaaaybe?
Cory “gphat” Watson
• Joined Stripe in August, 2015
• Previously at Keen IO and Twitter
• Generalist
Starting Point
• Stripe had some visibility, but not enough.
• No clear ownership, broken windows.
• Lack of confidence, vision for future.
• Very reactive.
This isn’t about a specific technology. This is about people.
Did it work?
You’re here because you know this is important.
How can we get others to agree and work toward it?
Stripe Org Facts
• ~450 employees, 100% growth in last year
• ~2 dozen teams
• ~200 services
• Thousands of hosts (AWS)
• Ruby, JVM, lots of OSS stuff
• Team: 3 + intern (starting Q2)
Where to begin?
Start Over, Kinda
• Spend time with the tools
• Improve if possible
• Replace if not
• Leverage past knowledge
Empathy and Respect
• People not generally evil, but they are busy!
• Stressed, doing best with what they have
• Being a hater is lazy
• Help people be great at their jobs
Replaced Existing System
• Maybe a bad call, technically better
• Overcoming momentum is hard, adds work
• Declaring bankruptcy
• Saved us ops headaches
• Still going
Tip: Nemawashi
• Start small, you’re a great guinea pig
• Quietly lay a foundation and gather feedback
• Ask how you can improve, follow up!
• Engage discontent! Usually fine. Sometimes you need whisky.
Identify Power Users
• Find interested parties
• Talk to them, give them what they need
• Empower them to help others
• Watch them grow!
Value
• What are you improving?
• How can you measure it?
• Is this the best way?
What is Observability?
Why do we want it?
In control theory, observability is a measure for how well internal states of a system can
be inferred by knowledge of its external outputs.
Systems output work.
If the internal state goes bad, the work goes bad.
We need to add sensors!
Make This Great
ProgrammerReference
System
Sensor(s)
Work
Flat Org Work Ethic
• Probably the biggest challenge, getting started
• So, ya know, get started
• Be willing to do the work, shave the preposterous line of yaks
• Stigmergy
• Strike when good opportunities arise (incidents, etc)
Advertise
• Don’t be afraid!
• Promote team accomplishments.
• Moreso, promote the accomplishment of others.
• Humbly ask to help, then learn.
• We send monthly “State of” addresses…
Make It Easy & Good
• Harder than it sounds (email!)
• Make it easy/automatic to do things right and hard to do wrong.
• Quality is important.
Automated Monitors
• Baseline monitoring
• Common problems, common solutions
• Users have no state, are surprised
• People care when you show them failure and how to fix it.
Automatic Ticket CreationAnd Resolution!
Investigation DashboardSuch Helpful!
Getting FeedbackHow we improve.
Teach the Basics
• Company curriculum: Teach ‘em early!
• Measuring work metrics
• Metrics types
• Schemas (dotted, tags, etc)
• Rates, histograms
• Visualizations
Ownership
• Poor story for this
• Org was ready for this, management was on board.
• Evolving, tools are lacking.
Did it work?
Yes, but not done.
• Some teams? Hell yes. Strong champions, huge improvement.
• Some other teams, kinda the same.
• Some other other teams, what is Observability and why do I care? Rare!
Usage?
• 200+ dashboards created, 339 in old (over 2 years)
• 200+ monitors created, dozens in old (nobody trusted, was unreliable!)
• ~3000 distinct metrics (can’t compare, tags now!)
• All positive feedback from automation. (Avg 4.5, 2.5% response)
Tools?
• Dozens of OSS PRs, OSS *StatsD library (Scala), internal libraries (we own)
• Vast improvement over old pipeline, no loss
• New styles, better naming, more consistency
• Being tied to a commercial product cuts both ways
Adjustments?
• Embracing other tools (log analysis, error catching)
• Beginning to work on strategic things (global timers, histograms and sets)
• Need to improve metrics on our own work (we got by easy for a while)
• Monitoring is hard, need to fix.
Summary
• Start small
• Seek feedback
• Think on your value
• Measure effectiveness
• Enjoy!
Thanks
Team @antifuchs and @shu, all of Stripe
onemogin.com
@gphat
github.com/gphat
Questions?
@gphatInfo
Slides
Feedback
Talk
Help me improve.