Gobblin is used at LinkedIn to ingest data from 3 Kafka clusters containing over 2500 topics and 50 billion records totaling over 15TB daily into related systems like copycat. It supports at least once, at most once, and exactly once semantics for publish and checkpoint persistence. Gobblin uses normal and two-level bin packing for load balancing across mappers and supports both deduplicated and non-deduplicated compaction with options for handling late events. More information can be found on Gobblin's GitHub page and user forum.

1. ingestion from kafka using gobblin
Ziyang Liu

2. in production @linkedin
3 Kafka clusters
> 2500 topics
> 50 billion records
> 15TB
daily:

3. related systems
copycat

4. semantics
at least once
publish data before
persisting checkpoints
at most once
persist checkpoints
before publishing data
exactly once
publish data and persist
checkpoints atomically

5. load balancing
normal bin packing two level bin packing
- Different partitions
of a topic usually go
to different mappers.
- Less skew, more
small files & task
overhead.
- First group certain
partitions of a topic
together.
- More skew.

6. compaction
Dedup or non-dedup
Multiple levels of compaction
Multiple options for handling late events

7. to learn more
https://github.com/linkedin/gobblin
Kafka-HDFS Ingestion (with end-to-end examples)
Camus → Gobblin Migration
https://groups.google.com/forum/#!forum/gobblin-users
Countless active users and discussions
Post any question, get the answer usually within a day
Join us!
Data And Analytics (DNA)