Presentation by Youssef Francis, Jun He, Airbnb

1. Every Message Counts
Youssef Francis / Jun He / Kafka Summit 2017
Kafka as a Foundation for
Highly Reliable Logging at
Airbnb

2. Overview
Data we collect at Airbnb
• Product Events: Click streams, metrics, experiment
reporting.
• Database Exports: Streaming and batch replicas of
production databases.
• Service Events: Services uses these to communicate with
each other. Events act as a log of mutations.
• Derived Data: Computed from a combination of streaming
and batch data.

3. A Simple Logging Pipeline

4. Schemas are a contract Events are delivered
reliably
Events are available
in real time
Producers only emit
well-formed events.
Errors detected in
development.
Predictable routing and
storage. Automatic
recovery from transient
failures.
Monitoring enables
immediately actionable
alerts. Events can be
inspected as they flow
through the pipeline.
Easy to find schemas
in use.
Ownership and
metadata provide
useful context.
Schemas and data
are discoverable
Tenets of Reliable Logging

5. Our Production Logging Pipeline

6. What is Jitney?
Standardized message bus at
Airbnb
Jitney is a unified solution for
propagating business context
with standardized events.

7. Jitney is an
ecosystem
• Central schema repository
• Client SDK
• Producer proxy
• Consumer agent
• Self-service portal

8. Jitney Schemas
Schemas
• Contract owned by the producer and shared by consumers.
• Rich metadata provided by the Jitney API.
Central Schema Repository
• Single source of truth.
• Schemas are easily discoverable.
• All schemas undergo code review and automated validation.
Why Thrift?
• Rich ecosystem; write once, use everywhere.
• Good performance in Java and Ruby.
• Efficient binary encoding.

9. Jitney client SDK
Jitney everywhere
• Standardized routing logic.
• Automatic topic creation.
• Cluster discovery.
• Seek and Replay API: fast bootstrap and recovery.
• Built-in metric reporting.
• Available on all platforms in use at Airbnb.

10. Jitney producer
proxy and
consumer agents
Powered by Kafka
Jitney producer agent
• Local and distributed proxy modes
• Open to the public internet
• Local rate limiting and event filtering
• Single and batch publishing modes
• Horizontally scalable
Jitney consumer agent
• Local Java agent to support non-Java consumers
• Delivery at least once guarantee

11. Self-service portal
• Real time message tailing
• Topic creation
• Offset management

12. We built our logging pipeline using Jitney

13. Logging Pipeline
Schema
Authoring
Schema
Deployment
Implementation
and Publishing
Processing and
Storage
Monitoring and
Auditing
Continuous Integration Workflow

14. Schema
Authoring
Engineers and
Data Scientists
define their
expectations in a
thrift schema,
which undergoes
code review.
Logging Pipeline
Continuous Integration Workflow

15. Schema
Authoring
Schema
Deployment
Once approved,
schema changes
are published
automatically in
new client
packages and
deployed to
consumers.
Logging Pipeline
Continuous Integration Workflow

16. Schema
Authoring
Schema
Deployment
Implementation
and Publishing
Engineers
implement logging
using client SDKs.
Schemas enforced
automatically.
Logging Pipeline
Continuous Integration Workflow

17. Schema
Authoring
Schema
Deployment
Implementation
and Publishing
Events are
consumed by
services and
stream processing
apps, and stored
in the warehouse.
Processing and
Storage
Logging Pipeline
Continuous Integration Workflow

18. Schema
Authoring
Schema
Deployment
Implementation
and Publishing
Processing and
Storage
Monitoring and
Auditing
Continuous loss
reporting and
anomaly detection.
Logging Pipeline
Continuous Integration Workflow

19. Some numbers
• 150 brokers across 5 clusters. 1 million messages per
second at peak.
• 10 billion logging events collected per day. Highly seasonal.
• 95% of logging events are available for querying in the
warehouse within 3 minutes.
• SLA guarantees no more than 0.01% loss. Less than
0.000001% end-to-end loss in practice.

20. Questions