End-to-End “Exactly Once” with Heron & Pulsar by Ivan Kelly at Big Data Spain 2017
•
1 like•1,455 views
Heron is an open-source streaming engine, employed by Twitter, Microsoft and Google, to process billions of events every day. https://www.bigdataspain.org/2017/talk/end-to-end-exactly-once-with-heron-pulsar Big Data Spain 2017 November 16th - 17th Kinépolis Madrid
Recommended
Recommended
More Related Content
Similar to End-to-End “Exactly Once” with Heron & Pulsar by Ivan Kelly at Big Data Spain 2017
Similar to End-to-End “Exactly Once” with Heron & Pulsar by Ivan Kelly at Big Data Spain 2017 (20)
More from Big Data Spain
More from Big Data Spain (20)
Recently uploaded
Recently uploaded (20)
End-to-End “Exactly Once” with Heron & Pulsar by Ivan Kelly at Big Data Spain 2017
- 2. End-to-end Effectively Once with Heron and Pulsar Ivan Kelly @ivankelly
- 4. 3 common semantics in stream processing • At most once • At least once • Effectively once
- 5. Only interesting in the case of failures!
- 6. Example application State in each count node
- 7. At most once Drops tuples
- 8. At least once Retries tuples
- 9. ☒ A processing node will only ever process a tuple once ☑ Result of a processed tuple only perceived once Effectively once?
- 10. Optimistic & Pessimistic I/O requirements versus recovery latency Effectively once?
- 13. Effectively Once Support Optimistic Pessimistic Twitter Heron Apache Flink Google MillWheel Apache Kafka Apache Apex Twitter Heron (In-development)
- 14. There are more ways to fail
- 16. On failure, the messaging system must not: • Lose messages • Duplicate messages • Reorder messages*
- 17. Subscribers must be able to reattach where they left off Catching up should not adversely affect other operations
- 18. Once acked by messaging, data is persisted there But before this point, publishers must not: • Lose data • Duplicate data
- 19. Both PubSub and Queue semantics Multi-tenant & Cross-DC Apache Incubating project 2 years production usage at scale in Yahoo Apache Pulsar
- 21. No loss, No dupes, No reorder* = Total Order Atomic Broadcast (TOAB) = Consensus
- 22. KV/Filesystem API • No access to log Single replicated log • Can’t scale out
- 23. Horizontally scalable replicated logging Pulsar topic replicated log is sequence of bookkeeper ledgers (stored in ZK) Uses ZooKeeper to close ledgers Many different streams per storage node (bookies) Isolated read and write paths Apache BookKeeper
- 24. Reattach with Cursors Enhanced offsets Managed by Pulsar Stored in BookKeeper ledgers (no scale issue)
- 26. Idempotent publish Publisher communicates offset in source data to broker Deduplicate published messages will lower or equal offset Broker stores publisher offsets in a cursor
- 27. Pulling it all together Optimistic Effectively Once Pessimistic Effectively Once (Coming soon) Total Order Atomic Broadcast Scalable Storage for many streams Cursors Idempotent Publish