01.07.15

Rearview – time series based monitoring

What's Rearview?

• “Passive” monitoring framework

• Polls time series database (TSD) at scheduled intervals

• Runs Ruby scripts against data

• Can send alerts to:– Email

– VictorOps

– PagerDuty (cough)

● Semi-secure

Graphite

● TSD (think RRD tool for storage combined with Ganglia for visualization

● Rich HTTP API for querying/transforming metrics

● Incumbant (at LivingSocial)

● Does not scale well

● Wildcard queries hurt

● Hard to install

Yeah, but why?

● “I want New Relic for business metrics” - Chad Fowler (VPE LivingSocial)

● “Measure all the things!”

● Started with cron-based scripts

● Lots of code duplication

● Had to edit Chef/Nagios to deploy

● Painful

● Enable ease of deployment

● Better dashboards

System Overview

1) Rearview runs monitor at specified time

2) Fetches metric from Graphite

3) Runs Ruby script against data (optional)

4) Sends alert to alerting service or email (optional)

Monitors

● Ruby function called with results of TSDB query

● Some helper methods and implicit vars in scope

● Raise exception with error message to indicate failure state

● Runs in exernal, forked Ruby process in SAFE mode. Semi-secure.

● Scripts have 5s to complete (configurable) or are killed

Live demo! Lookout ...

Related Software

● Riemann (riemann.io)

● Stream receiver

● Clojure monitors

● Great documentation

● More efficient (no polling)

● Very flexible

Related Software (cont...)

● Grapfana (grafana.org)

● More configurable

● Supports

– Graphite

– InfluxDB

– OpenTSB

Future directions

● Handle recovery notifications

● Support InfluxDB

– Modern, well supported

– Scalable

● Support streaming data

● Port to Rust, Nim?

● Name change. Ideas?

Resources

● Rearview - https://github.com/fooblahblah/rearview

● Graphite - http://graphite.readthedocs.org/en/latest/

● Grafana - http://grafana.org/

● InfluxDB - http://influxdb.com/

● Riemann - http://riemann.io/