Overview of OpenStack Ceilometer plans for Gnocchi integration

1. Rebasing Ceilometer storage on Gnocchi
OpenStack Telemetry
1

2. 2
What’s the problem we’re trying to solve here?
● Flexible but heavyweight Ceilometer samples model
with free-form metadata
● This legacy issue has led to many of the problems
impacting Ceilometer adoption:
● massive storage footprint
● suboptimal data ingestion
● non-scaling query API
● Also gives us a weakly-typed API representation

3. ● Track strongly-typed resource attributes
● Rely on events to reconstruct resource state timeline
● Eagerly pre-aggregate metric data
● Support restricted cross-metric aggregation
3
Key approaches taken by Gnocchi

4. 4
Compare and contrast ...
“classic” Ceilometer Gnocchi
Heavy-weight samples with
embedded metadata
Light-weight time-series
shorn of metadata
Global data expiry policy
set across the board
Per time-series
configurable retention
policies

5. 5
Compare and contrast ...
“classic” Ceilometer Gnocchi
On-demand aggregation Eager pre-aggregation
Intertwined storage of
resources and samples
Separated storage and
data models for resources
& time-series data

6. ● Resource = cloud resource (instance, volume, etc.)
● Metric = anything you’d like to collect data about
● identified by UUID, or by name combined with resource ID
● Measure = (timestamp, value) time-series datapoint
6
Gnocchi basics

7. ● Archive policy = data storage policy defined by admin
● 1 second resolution over a day, 1 hour resolution over a year, or
even both
● Consists of granularity (in seconds) and retention time-span
● Aggregation = function used to roll up data
● Retention = do not store fine grained data forever,
instead store aggregated data according to the per-
metric archive policies
Gnocchi basics
7

8. 8
Gnocchi aggregation mechanism
Time
1s
1m
1h
Now
Data to be calculated in
run-time
Most recent and small-
granularity aggregated
data
Original non-aggregated measures to
calculate aggregations from
Additional
data - 1 h
Aggregated
data

9. ● Capturing measurements for different metrics is the
main concept of Gnocchi
● Although, metrics have no actual use without some-
resource association
● Resources have strongly-typed attributes
● Metric association is by name (e.g. “cpu_util”)
● Metrics for loosely associated resources can be cross-
aggregated
9
Gnocchi Indexer concept

10. ● Gnocchi indexer is responsible for indexing
entities, resources, and linking them together
● Resources and their attributes are well-defined,
typed, and indexed
● The generic type can be used if the resource type is
unknown to Gnocchi
10
Gnocchi Indexer concept

11. ● Alarming to drive Heat autoscaling, based on
aggregating samples across all instances with
matching metadata
● use cross-metric aggregation based on strongly-typed
resource attributes, as opposed to free-from metadata
● Reconstructing the resource state timeline, from
per-sample resource metadata
● use queries over relatively infrequent events capturing
state transitions
11
Covering existing ceilometer use-cases

12. ● Existing specialized metrics-oriented DBs can be
leveraged by Gnocchi’s pluggable driver model
● actively working on drivers for InfluxDB and OpenTSDB
● Gnocchi itself provides a canonical storage driver
based on Pandas and Swift
● In the specialized TSBD use-case, Gnocchi manages
the resource-metric association & abstract archive
policy concepts
12
No, we’re not re-inventing TSDB here