© The OpenNMS Group, Inc.

SCALING YOUR TIME SERIES DATA WITH NEWTS

by Jesse White

OPENNMS• World's First• Enterprise Grade• Network management• Application platform• Open Source

© The OpenNMS Group, Inc.

TIME SERIES

© The OpenNMS Group, Inc.

Source: https://en.wikipedia.org/wiki/RRDtool

RRDTOOL

STRATEGIES• rrdtool via JNI (1.0 - 2002)• JRobin (1.1.4 – 2004)• jrrd2 (17.0.0 - 2015)• Newts (17.0.0 - 2015)

© The OpenNMS Group, Inc.

Source: https://commons.wikimedia.org/wiki/File:Hard_disk_Western_Digital_WD740_1_(dark1).jpg

© The OpenNMS Group, Inc.

THE MODEL

Resources are used to group related metrics.

Metrics are used to identify specific values.

Samples represent the value of a resources metric at some instance in time.

© The OpenNMS Group, Inc.

Resource ID: 1:opennms-jvm:org_opennms_newts_name_repository_insert_timeMetrics: NewtsInsert50, NewtsInsert75, NewtsInsert95, NewtsInsert98, …

SAMPLES

© The OpenNMS Group, Inc.

LATE AGGREGATION

The target range is split into intervals, where we take weighted averages of the corresponding samples.

The heartbeat defines up to how long a sample is considered to be valid.

The result of the weighted averages produces evenly spaced results, called the primary data points.

We then apply a consolidation function i.e. MIN, MAX, AVG, the primary data points to achieve the desired resolution.

TERMS

SEARCH

env:uat AND (dc:dc1 OR dc:dc2)

NEWTS + OPENNMS

GRAPHING

SCALE UP AND OUT• 1015867.89 samples/s• Cassandra:

– 5 * i2.4xlarge• Load:

– 3 * m4.4xlarge

OPERATIONAL ISSUES• Estimating disk space usage• Cluster management

– Repairs• Data mangling

– Deleting a point– Renaming a series

NEWTS.IO