Safeguarding - Protecting Your Kafka from Misbehaving Clients with Tom Scott
Safeguarding - Protecting Your Kafka from Misbehaving Clients with Tom Scott
Kafka is a VERY flexible and configurable system. That flexibility has undoubtedly been key to its adoption, but it's also a double-edged sword. How can you ensure applications use cluster resources effectively, knowing that much configuration can be performed client-side? Hidden away from the watchful eye of operators. An incorrectly chosen value hidden in a config file somewhere can have drastic effects on application and Kafka cluster performance. At Conduktor, we have observed many examples of such issues reported by our customers. Common issues include producers with inefficiently configured batch size and linger, consumer groups with more members than subscribed partitions, and topics that unintentionally mix data with schema and without. In many of these cases, the impact of the issue is invisible to the client application but its impact on Kafka cluster performance is significant. Join us for a detective hunt as we discuss tools and metrics used to detect misconfigurations in clients, how to address them once discovered, and ways in which to ensure that new occurrences are prevented from arising in the future.
Recommandé
Contenu connexe
Similaire à Safeguarding - Protecting Your Kafka from Misbehaving Clients with Tom Scott
Similaire à Safeguarding - Protecting Your Kafka from Misbehaving Clients with Tom Scott(20)
Plus de HostedbyConfluent
Plus de HostedbyConfluent(20)
Dernier
Dernier(20)
Safeguarding - Protecting Your Kafka from Misbehaving Clients with Tom Scott
- 1. Safeguarding - protecting your clusters from misbehaving clients. Tom Scott
- 2. ● Principal Engineer - Conduktor Gateway ● 6 years in Kafka across many roles ● 15+ yrs in all things data Who am I?
- 3. ● To minimise outages ● To respond gracefully to change ● To reduce friction between domains Result: Velocity! Start with Why
- 4. The practice of protecting your data and infrastructure from intentional and unintentional threats, both internal and external. What is Safeguarding? Data Governance “ Technical Best Practices Safeguarding
- 5. A collection of loose, negotiable contracts enforced by humans. Case in point: The “production readiness” check These suffer from: ● Relevance - scoped for multiple applications in multiple technologies ● Timeliness - Seen as a box check to be completed just before go live ● Perception - Rules to be bent/ignored, “what can we get away with?” Stage 1: safeguarding by habit
- 6. “Your application is behaving badly” Now what? ● Uncomfortable decisions ● Temporary solutions (that become permanent) ● Tech debt Stage 2: reactive automation
- 7. ● Enforced at the earliest point ● Applied in every environment ● Specific and detailed information on breaches Examples include: ● Schema validation ● Partition count limits ● Automatic encryption Stage 3: proactive automation
- 8. How? (but actually where) A Client Plugin A Broker Plugin A Proxy
- 9. Application Proxy Application Encryption Interceptor Schema Validation Interceptor Ring fenced Conduktor Gateway - a Kafka Proxy ● Interceptors - functions that perform transformation/filtering/projection on Kafka requests/responses ● Gateway embeds one or more interceptors ● To a client: Gateway == Kafka Brokers
- 10. ● Proactive safeguarding: ○ Prevents outages ○ Resets perception ● Enforced inter-domain contracts help: ○ React gracefully to change ○ Reach broader technical goals ○ Reduce friction ● Proactive safeguarding does not: ○ Define governance/best practice policies ○ Fully replace reactive solutions In summary
- 11. Q&A