31. Delivery Semantics (Produce)
● At least once
○ Retry if no ACK
● At most once
○ Never retry
● Exactly once
○ Idempotence (new in 0.11)
KafkaProducer
32. Delivery Semantics (Consume)
● At least once
○ Process messages, then commit offsets
● At most once
○ Commit offsets, then process messages
● Exactly once
○ At least once & idempotence
Kafka Consumer
33. Producer
val props = new Properties()
props.put("bootstrap.servers", "localhost:9092")
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer")
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer")
val producer = new KafkaProducer[String, String](props)
val f: java.util.concurrent.Future[RecordMetadata] =
producer.send(new ProducerRecord[String, String]("my-topic", "key", "value"))
f.get // sync
producer.close
34. Consumer
val props = new Properties()
props.put("bootstrap.servers", "localhost:9092")
props.put("group.id", "group1") // consumer group
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer")
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer")
val consumer = new KafkaConsumer[String, String](props)
consumer.subscribe(Arrays.asList("my-topic"))
while (true) {
val records = consumer.poll(1000) // timeout for long pull
for (record <- records.asScala)
println(record.partition, record.offset, record.key, record.value)
consumer.commitSync
}
40. Old / New Consumer API
● Old API
○ --zookeeper localhost:9092
○ Offsets stored in ZooKeeper
● New API
○ --bootstrap-server localhost:9092
○ Offsets stored in Kafka’s internal topic
41. About Partitions
● Support increase the number of partitions only
● Configs (broker & topic)
○ retention.ms
○ retention.bytes
○ segment.bytes