Resilient Event Driven Systems With Kafka
•
0 j'aime•217 vues
Talk at Craft conf 2018 on how to build resilient event driven systems with Kafka. Walks through an example of building notifications feature for chat app Stride.
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à Resilient Event Driven Systems With Kafka
Similaire à Resilient Event Driven Systems With Kafka (20)
Plus de Iccha Sethi
Plus de Iccha Sethi (8)
Dernier
Dernier (20)
Resilient Event Driven Systems With Kafka
- 1. ICCHA SETHI | PRINCIPAL DEVELOPER, ATLASSIAN | @ICCHASETHI From Requirements To Resilient Event Driven Systems With Kafka
- 3. It all begins with a feature spec… STORY OF NOTIFICATIONS @IcchaSethi
- 7. What did we know about the technical requirements of Notifications feature?
- 8. What did we know? Asynchronous
- 9. Scale What did we know? Asynchronous
- 10. Scale What did we know? Graceful degradationAsynchronous @IcchaSethi
- 11. Event-driven architecture, is a software architecture pattern promoting the production, detection, consumption of, and reaction to events. Wikipedia, 2018
- 12. “The event-driven architecture pattern is a popular distributed asynchronous architecture pattern used to produce highly scalable applications” Software Architecture Patterns, Mark Richards
- 13. Kafka 0 1 2 3 4 5 6 7 8 9 Partition 0 0 1 2 3 4 5 Partition 3 Topic Foo Producer Consumer 0 Consumer 1 Consumer Group B Consumer 0 Consumer 1 Consumer Group A Consumer Group B Consumer Group A
- 14. MESSAGE SERVICE
- 17. NOTIFICATIONS WORKER Send notification Dependency A Dependency B Dependency C
- 18. MICRO SERVICES MEET EVENT DRIVEN ARCHITECTURE
- 19. Direct Messages Worker Send Notification Worker Group Messages Worker Check User Online Worker Dependency A Dependency B Send notification
- 20. Direct Messages Worker Send Notification Worker Group Messages Worker Check User Online Worker Dependency A Dependency B Send notification
- 22. Direct Messages Worker Send Notification Worker Group Messages Worker Check User Online Worker Send notification Another Rule Worker
- 24. Build for extensibility and flexibility to account for evolving requirements! TAKEAWAY @IcchaSethi
- 25. Good job Team! We are done!
- 27. How do we want notifications to behave when things go wrong?
- 28. Check User Online Worker Presence Service
- 29. Low latency is of utmost importance Ordering of Events is of utmost important ORDERING LATENCY
- 34. Deadletter using Kafka 0 1 2 3 4 5 6 7 8 9 Topic Foo Consumer 0 Consumer 1 Consumer Group B
- 35. Deadletter using Kafka 0 1 2 3 4 5 6 7 8 9 Topic Foo Consumer 0 Consumer 1 Consumer Group B 0 1 2 3 4 5 Topic Foo.deadletter
- 36. Deadletter using Kafka 0 1 2 3 4 5 6 7 8 9 Topic Foo Consumer 0 Consumer 1 Consumer Group B 0 1 2 3 4 5 Topic Foo.deadletter
- 39. Use when order does not matter Do not deadletter un- recoverable errors Utilize metadata to track attempts Deadletter with Kafka
- 40. RESILIENCY REQUIREMENTS Ordering HTTP Retries Latency Deadletter queues Timeouts
- 41. RESILIENCY REQUIREMENTS Ordering HTTP Retries Latency Deadletter queues Timeouts Circuit breakers
- 42. RESILIENCY REQUIREMENTS Ordering HTTP Retries Latency Deadletter queues Timeouts Circuit breakers @IcchaSethi
- 43. Check User Online Worker Presence Service
- 44. SCENARIO ONE - WHAT TOOLS WOULD YOU USE? Order doesn’t matter! Send notifications ASAP!
- 45. SCENARIO TWO - WHAT TOOLS WOULD YOU USE? I guess upto 1 second delay is acceptable to preserve order. Else order doesn’t matter.
- 46. Always ASK what if things go wrong? TAKEAWAY @IcchaSethi
- 47. HOW DO YOU SCALE RESILIENCY?
- 48. HOW DO YOU SCALE RESILIENCY?
- 49. Make it low-cost to opt in for resiliency and have sensible default values TAKEAWAY
- 51. Why didn’t the customer get the notification?
- 52. For any product Observability is an unwritten requirement ME
- 53. Observability Stats Logs Tracing What should we measure? • Data for Current state • Data to predict future state
- 54. THE ANSWER LIES IN QUEUING THEORY Arrivals Queue Service
- 55. THE ANSWER LIES IN QUEUING THEORY Arrivals Queue Service Arrival Rate
- 56. THE ANSWER LIES IN QUEUING THEORY Arrivals Queue Service Arrival Rate Queue Length
- 57. THE ANSWER LIES IN QUEUING THEORY Arrivals Queue Service Arrival Rate Wait Time Residence Time Service Time Queue Length
- 58. THE ANSWER LIES IN QUEUING THEORY Arrivals Queue Service Arrival Rate Wait Time Residence Time Service Time Utilization = Arrival Rate * Service Time Queue Length
- 59. THE ANSWER LIES IN QUEUING THEORY Arrivals Queue Service Arrival Rate Wait Time Arrival Rate Service Time Utilization Queue Length
- 60. UTILIZATION = RATE OF ARRIVAL*SERVICE TIME Hockey Stick Residence Time vs Utilization
- 62. UTILIZATION = RATE OF ARRIVAL * SERVICE TIME Direct Messages Worker Consumer Lag Residence Time Service Time Broker
- 63. UTILIZATION = RATE OF ARRIVAL * SERVICE TIME Direct Messages Worker Broker Counter for messages produced
- 64. UTILIZATION = RATE OF ARRIVAL * SERVICE TIME
- 66. Observability Stats Logs Tracing What should we measure? • Data for Current state • Data to predict future state • When will HOT partitions in Kafka occur? • Will scaling my consumers help?
- 67. Observability is not only for measure of internal state of system but can also be used to predict its future state TAKEAWAY @IcchaSethi
- 68. Observability Stats Logs Tracing Logs • All producer/consumer success and failure handlers are wrapped with consistent log fields. • topic • partition • offset • Kafka timestamp
- 69. LOGS
- 70. Observability Stats Logs Tracing “… one of the deficiencies of the Event Notification pattern is the lack of visibility into the flow of the system, which makes it hard to debug.” - Martin Fowler
- 72. Observability Stats Logs Tracing Kafka Event Metadata • id • producer of the event • source • tags • timestamp • version
- 73. Observability Stats Logs Tracing "bitbucket_org-hipchat-schemas-platform- ZipkinTrace": { "properties": { "parent_id": { "type": "string" }, "sampled": { "type": "boolean" }, "span_id": { "type": "string" }, "trace_id": { "type": "string" } } },
- 80. Product Requirements • Build for evolving requirements - extensibility and flexibility • Always ask WHAT if things go wrong Resiliency • Resiliency affects product behavior • Make it a low cost option to adopt resiliency with sensible default values Observability Observability is not only the measure of the internal state but can also be used to predict future state Takeaways @IcchaSethi