rdo hangout on gnocchi

13
Rebasing Ceilometer storage on Gnocchi OpenStack Telemetry 1

Upload: eoghan-glynn

Post on 28-Jul-2015

392 views

Category:

Software


4 download

TRANSCRIPT

Page 1: RDO hangout on gnocchi

Rebasing Ceilometer storage on GnocchiOpenStack Telemetry

1

Page 2: RDO hangout on gnocchi

2

What’s the problem we’re trying to solve here?

● Flexible but heavyweight Ceilometer samples model with free-form metadata

● This legacy issue has led to many of the problems impacting Ceilometer adoption:● massive storage footprint● suboptimal data ingestion● non-scaling query API

● Also gives us a weakly-typed API representation

Page 3: RDO hangout on gnocchi

● Track strongly-typed resource attributes● Rely on events to reconstruct resource state timeline● Eagerly pre-aggregate metric data● Support restricted cross-metric aggregation

3

Key approaches taken by Gnocchi

Page 4: RDO hangout on gnocchi

4

Compare and contrast ...

“classic” Ceilometer Gnocchi

Heavy-weight samples with embedded metadata

Light-weight time-series shorn of metadata

Global data expiry policy set across the board

Per time-series configurable retention policies

Page 5: RDO hangout on gnocchi

5

Compare and contrast ...

“classic” Ceilometer Gnocchi

On-demand aggregation Eager pre-aggregation

Intertwined storage of resources and samples

Separated storage and data models for resources & time-series data

Page 6: RDO hangout on gnocchi

● Resource = cloud resource (instance, volume, etc.)

● Metric = anything you’d like to collect data about

● identified by UUID, or by name combined with resource ID

● Measure = (timestamp, value) time-series datapoint

6

Gnocchi basics

Page 7: RDO hangout on gnocchi

● Archive policy = data storage policy defined by admin● 1 second resolution over a day, 1 hour resolution over a year, or

even both● Consists of granularity (in seconds) and retention time-span

● Aggregation = function used to roll up data● Retention = do not store fine grained data forever,

instead store aggregated data according to the per-metric archive policies

Gnocchi basics

7

Page 8: RDO hangout on gnocchi

8

Gnocchi aggregation mechanism

Time

1s

1m

1h

Now

Data to be calculated in run-time

Most recent and small-granularity aggregated

data

Original non-aggregated measures to calculate aggregations from

Additional data - 1 h

Aggregated data

Page 9: RDO hangout on gnocchi

● Capturing measurements for different metrics is the main concept of Gnocchi

● Although, metrics have no actual use without some-resource association

● Resources have strongly-typed attributes● Metric association is by name (e.g. “cpu_util”)● Metrics for loosely associated resources can be cross-

aggregated

9

Gnocchi Indexer concept

Page 10: RDO hangout on gnocchi

● Gnocchi indexer is responsible for indexing entities, resources, and linking them together

● Resources and their attributes are well-defined, typed, and indexed● The generic type can be used if the resource type is

unknown to Gnocchi

10

Gnocchi Indexer concept

Page 11: RDO hangout on gnocchi

● Alarming to drive Heat autoscaling, based on aggregating samples across all instances with matching metadata● use cross-metric aggregation based on strongly-typed

resource attributes, as opposed to free-from metadata● Reconstructing the resource state timeline, from

per-sample resource metadata● use queries over relatively infrequent events capturing

state transitions11

Covering existing ceilometer use-cases

Page 12: RDO hangout on gnocchi

● Existing specialized metrics-oriented DBs can be leveraged by Gnocchi’s pluggable driver model● actively working on drivers for InfluxDB and OpenTSDB

● Gnocchi itself provides a canonical storage driver based on Pandas and Swift

● In the specialized TSBD use-case, Gnocchi manages the resource-metric association & abstract archive policy concepts

12

No, we’re not re-inventing TSDB here

Page 13: RDO hangout on gnocchi