Building an Event-oriented Data Platform with Kafka, Eric Sammer

1. building a system for machine and event-oriented data e. sammer | @esammer kafka summit 2016

3. © 2015 Rocana, Inc. All Rights Reserved. what we do 3 • we build a system for the operation of modern data centers • triage and diagnostics, exploration, trends, advanced analytics of complex systems • our data: logs, metrics, human activity, anything that occurs in the data center • “enterprise software” (i.e. we build for others.) • today: how we built what we built

4. © 2015 Rocana, Inc. All Rights Reserved. our typical customer use cases 4 • >100K events / sec (8.6B events / day), sub-second end to end latency, full fidelity retention, critical use cases • quality of service - “are credit card transactions happening fast enough?” • fraud detection - “detect, investigate, prosecute, and learn from fraud.” • forensic diagnostics - “what really caused the outage last friday?” • security - “who’s doing what, where, when, why, and how, and is that ok?” • user behavior - ”capture and correlate user behavior with system performance, then feed it to downstream systems in realtime.”

8. © 2015 Rocana, Inc. All Rights Reserved. guarantees 8 • no single point of failure exists • all components scale horizontally[1] • data retention and latency is a function of cost, not tech[1] • every event is delivered provided no more than N - 1 failures occur (where N is the kafka replication level) • all operations, including upgrade, are online[2] • every event is (or appears to be) delivered exactly once[3] [1] we’re positive there’s a limit, but thus far it has been cost. [2] from the user’s perspective, at a system level. [3] when queried via our UI. lots of details here.

10. © 2015 Rocana, Inc. All Rights Reserved. modeling our world 10 • everything is an event • each event contains a timestamp, type, location, host, service, body, and type- specific attributes (k/v pairs) • build specialized aggregates as necessary - just optimized views of the data

11. © 2015 Rocana, Inc. All Rights Reserved. event schema 11 { id: string, ts: long, event_type_id: int, location: string, host: string, service: string, body: [ null, string ], attributes: map<string> }

12. © 2015 Rocana, Inc. All Rights Reserved. event types 12 • some event types are standard – syslog, http, log4j, generic text record, … • users define custom event types • producers populate event type • transformations can turn an event of type A into B • event type metadata tells downstream systems how to interpret body and attributes

13. © 2015 Rocana, Inc. All Rights Reserved. ex: generic syslog event 13 event_type_id: 100, // rfc3164, rfc5424 (syslog) body: … // raw syslog message bytes attributes: { // extracted fields from body syslog_message: “DHCPACK from 10.10.0.1 (xid=0x45b63bdc)”, syslog_severity: “6”, // info severity syslog_facility: “3”, // daemon facility syslog_process: “dhclient”, syslog_pid: “668”, … }

14. © 2015 Rocana, Inc. All Rights Reserved. ex: generic http event 14 event_type_id: 102, // generic http event body: … // raw http log message bytes attributes: { http_req_method: “GET”, http_req_vhost: “w2a-demo-02”, http_req_path: “/api/v1/search?q=service%3Asshd&p=1&s=200”, http_req_query: “q=service%3Asshd&p=1&s=200”, http_resp_code: “200”, … }

17. © 2015 Rocana, Inc. All Rights Reserved. data processing 17 • each processing job gets a full stream of the fire hose, decides what it wants to consider or operate on • output of “non-terminal” jobs always just events • result: all processing jobs are composable • many jobs take user rules or configuration from our ui

18. © 2015 Rocana, Inc. All Rights Reserved. the jobs 18 • transformation engine: configuration-based data transformation • metric aggregation: olap cube construction of time series data (e.g. host 17 user cpu time) • model build/eval: train/evaluate various kinds of models (e.g. anomaly detection) • trigger engine: detect complex patterns in the stream, emit events on match (e.g. complex event processing, automated workflow, alerting) • action engine: perform some action upon receiving a specific event type (e.g. email notification, 3rd party api invocation) • storage: write all the things to hdfs

22. © 2015 Rocana, Inc. All Rights Reserved. aggregation 22 • mostly for time series metrics • two halves: on write and on query • data model: (dimensions) => (aggregates) • on write – reduce(a: A, b: A) => A over window – store “base” aggregates, all associative and commutative • on query – perform same aggregate or derivative aggregates – group by the same dimensions – we use SQL (Impala)

23. © 2015 Rocana, Inc. All Rights Reserved. aside: late arriving data (it’s a thing) 23 • never trust a (wall) clock • producer determines event time, rest of the system uses this always • data that shows up late always processed according to event time • apache beam describes these issues perfectly • this is real and you must deal with it

24. © 2015 Rocana, Inc. All Rights Reserved. ex: service event volume by host and minute 24 • dimensions: ts, window, location, host, service, metric • on write, aggregates: count, sum, min, max, last • epoch, 60000, us-west-2a, w2a-demo-1, sshd, event_volume => 17, 42, 1, 10, 8 • on query: – SELECT floor(ts / 60000) as bin, host, service, metric, sum(value_sum) FROM events WHERE ts BETWEEN x AND y AND metric = ”event_volume” GROUP BY bin, host, service, metric • if late arriving data existed in events, the same dimensions would repeat with a another set of aggregates and would be rolled up as a result of the group by • tl;dr: normal window aggregation operations

26. © 2015 Rocana, Inc. All Rights Reserved. extending the system 26 • custom producers • custom consumers • event types • parser / transformation plugins • custom metric definition and aggregate functions • custom processing jobs on landed data

27. © 2015 Rocana, Inc. All Rights Reserved. pain (aka: the struggle is real) 27 • lots of tradeoffs when picking a stream processing solution – samza: right features, but low level programming model, not supported by vendors. missing security features. – storm: too rigid, too slow. not supported by all Hadoop vendors. – flink: relatively new, fledgling community. growing. – spark streaming: tons of issues initially, but lots of community energy. improving. • stack complexity, (relative im)maturity • beam-style retractions required for correct, timely, efficient aggregates of complex metrics (non-assoc/commutative)

28. © 2015 Rocana, Inc. All Rights Reserved. if you’re going to try this… 28 • read all the literature on stream processing[1] • treat it like the distributed systems problem it is • understand, make, and make good on guarantees • find the right abstractions • never trust the hand waving or “hello worlds” • fully evaluate the projects/products in this space • understand it’s not just about search [1] wait, like all of it? yea, like all of it.

29. © 2015 Rocana, Inc. All Rights Reserved. things I didn’t talk about 29 • reprocessing data when bad code / transformations are detected • dealing with data quality issues (“the struggle is real” part 2) • the user interface and all the fancy analytics – data visualization and exploration – event search – anomalous trend and event detection – metric, source, and event correlation – motif finding – noise reduction and dithering • event delivery semantics (e.g. at least once, exactly once, etc.) • alerting, notification, and other subsystems

Building an Event-oriented Data Platform with Kafka, Eric Sammer

confluent

Building an Event-oriented Data Platform with Kafka, Eric Sammer