5. Copyright, The OpenNMS Group, Inc.
Samples
CREATE TABLE newts.samples (
context text,
partition int,
resource text,
collected_at timestamp,
metric_name text,
attributes map<text, text>,
value blob,
PRIMARY KEY ((context, partition, resource), collected_at, metric_name)
) WITH CLUSTERING ORDER BY (collected_at ASC, metric_name ASC);
6. Copyright, The OpenNMS Group, Inc.
Indexing
CREATE TABLE newts.terms (
context text,
field text,
value text,
resource text,
PRIMARY KEY ((context, field, value), resource)
) WITH CLUSTERING ORDER BY (resource ASC);
7. Copyright, The OpenNMS Group, Inc.
Searching
{
"resource": {
"id": "ike:vendingMachine",
"attributes": {
"slot015name": "Cheerwine"
}
},
"timestamp": 1412009776000,
"name": "slot015sold",
"type": "COUNTER",
"value": 8
}
Field Value
_all ike
_all vendingMachine
_all Cheerwine
slot015name Cheerwine
● Cheerwine
● slot015name:Cheerwine
● Cheerwine AND (MountainDew or “RC Cola”)
Search
Index
Composite key: ((distribute), unique)
Partition in seconds, higher => less larger reads, lower => more smaller reads
Value: 9 bytes, first byte is the type code (counter, gauge, absolute, derive) and next 8 bytes is the value (long or double in big-endian)
Drawbacks
Default field is ‘_all’
Caching
Cardinality
Cold-start
Split on the colon
Attributes are added to both the ‘_all’ field and their named field
Compound queries and handled client side (union, intersect)
Collectors build CollectionSets
CollectionSets -> (Threshd, Persistor)
Time Series Strategy
jrb, jrrd, jrrd2, tcp, queued
Parsing
Adds OpenNMS specific attributes (more next slide)
Add to ring buffer in batches
Batches used to limit size of Cassandra inserts
Ring Buffer has fixed size, must be a power of 2
Producer will wait if buffer is full
Consumer threads
Write to samples table
Merge results with cache
LRU cache
In JVM, or external cache such as Redis
Write to terms table
Resource ID different from the Newts Resource ID
Resources are organized in various folders on disk
Resources refer to a particular folder on disk
Resources have attributes, which refer to various files in the folder
Metrics are store in .rrd (storeByGroup)
Key-value pairs are stored in strings.properties
Other in .meta files
With .rrd we just look at the files on disk
Add attributes that allow us to query the available resources for a given type in a single query