mean time to sleep: quantifying the on-call experience

@lozzd • @ryan_frantz

Mean Time to SleepQuantifying the on-call experience

Laurie Denness@lozzd

Ryan Frantz@ryan_frantz

Who is in an on-call rotation?

Who is on call right now?

Who feels like on-call sucks?

Welcome. How is on call?

Let’s help our people sleep

Make on-call more bearable

Incremental Changes

Email toAcknowledge

Email to Acknowledge• Replying “ack” with some context makes it appear in

IRC too

Email Only Alerts• Do you care if RAID becomes degraded in the middle of

the night?

• Do you care if one of your web/hadoop/X boxes dies in the middle of the night?

the night?

• Do you care if one of your web/hadoop/X boxes dies in the middle of the night?

• Can it wait until the morning?

Added Context• Previous service state

• Duration in that state

• Previous service state

• Alert recipients

• Notes

• Link to Runbook

• Notes

• Link to Runbook

• Notes

• Link to runbook

Alert Storms• Reduce noise when 200 things go wrong by aggregating

• Trigger alert percentage of pool over threshold

Low friction downtime• IRC commands to downtime hosts/sets of hosts

Downtime Reminders• Help prevent false notifications

Event Handlers• Teach Nagios to augment the team

• Restarting services (nscd)

• Re-running jobs (transient errors)

• Duplicate crons (Chef)

Incremental Improvements?• Maybe

• More ideas; hoped they’d stick

• We didn’t know because we didn’t measure

Measure Everything• “You can’t manage what you can’t measure.”

- Deming (not really)

Measure Everything• “You can’t manage what you can’t measure.”

- Deming (not really)

• But, we weren’t measuring anything

What should we measure?

What should we measure?• Volume of alerts (total, by severity)

• Alert categorization (actionable vs not)

• Alert times: Off-hours?

• Noisy hosts/services

Opsweekly

@lozzd • @ryan_frantz We have data.

Aggregate alerts1. Look at reports

2. Wow, look at all those alerts for the same thing

3. Aggregate alerts

4.Profit

Parent relationships• Prevent alerts due to upstream issues (downed switch)

• Standard Nagios feature

• Computers can do this for us!

Parent relationships• signalvnoise.com

• LLDP on host shows switch info

• Put switch info into Chef using ohai

• Create Nagios host configs based on data

Service Dependencies• Hundreds of Graphite-sourced checks

• Created new template that sets a servicegroup that depends on the Graphite service.

Keep on analyzing• It’s okay to just identify and delete alerts that don’t

mean anything!

Keep on analyzing• It’s okay to just identify and delete alerts that don’t

mean anything!

• Or move them to email only

More Quantification!

Reviewing the Year• Use reports

• Use search

• Identify noisiest alerts

Reviewing the YearYEARLY REPORT SCREENSHOTS

• Great time to look at this data and make improvements

Nagios Hack Day/Week

• Great time to look at this data and make improvements

• If Disk Space is the worst. Can we rethink that?

Nagios Hack Day/Week

Outsource Your Alerts• Etsy’s Search Team has on-call rotation

• A whole subset of alerts that don’t go to Ops

• More teams starting this but Search Team is at 100%

Sleep Tracking

“Track your life!” - @ph

Did it work?

Did it work?• Yes.

• Signal to noise ratio is much better

• Okay, so it’s a little more complicated than that

• Adding alerts all the time means new “annoying” things

• Keep monitoring

What’s next?

• We focus on people’s sleep

The Effect of Sleep

• But not the effect on the person when they come to work the next day

The Effect of Sleep

• How do we measure the impact of sleep loss/deprivation?

The Effect of Sleep

• How do we measure the impact of sleep loss/deprivation?

The Effect of Sleep

• Subjective: Pittsburgh Sleepiness Scale

• Objective: Psychomotor vigilance task (PVT) to measure alertness

Beyond Opsweekly• Employee wellness program

• Security have started using past sleep data to check for weird logins to systems

More context: nagios-herald

More reports• We have a bunch of data, we can build better reports,

drill down to analyze alerting trends

More reports• We have a bunch of data, we can build better reports,

drill down to analyze alerting trends

• Can we attribute particular actions to reduced noise volume?

• Aggregate alerts

• Non-downtimed alerts

Thanks

Etsy Ops Team

SewMona

Open Source/Links• http://ryanfrantz.com/mtts

• https://github.com/etsy/opsweekly

• https://github.com/etsy/nagios-herald

• https://github.com/jonlives/jawboneup_to_graphite

• http://codeascraft.com

Questions?

mean time to sleep: quantifying the on-call experience

Engineering

a guide to quantifying c-3 risk - society of actuaries...a...

quantifying mean inner potential of zno nanowires by...

quantifying thefuture

quantifying sleep architecture dynamics and individual...

quantifying the contribution of a subpopulation to...

sympathetic skin response in obstructive sleep apnea...

sleep apnea: test and but does that mean? a pcp’s · pdf...

christmas carols from poland polish trad., transl. douglas...

quantifying innovation

impact of sleep deficiency 2020 -...

quantifying and measuring morphological...

quantifying sensitivity

new relic infrastructure intro: increasing mean time between...

impact of prevagen on sleep quality - truth in...

quantifying mangrove

quantifying the ecological benefits of lakeshore restoration...

ontario public service guidance document for quantifying...

quantifying biology

my experiences quantifying my sleep states

better sleep can mean better grades at college