"During development and automated tests, it is common to create Kafka clusters from scratch and run workloads against those short-lived clusters. Starting a Kafka broker typically takes several seconds, and those seconds add up to precious time and resources. How about spinning up a Kafka broker in less than 0.2 seconds with less memory overhead? In this session, we will talk about kafka-native, which leverages GraalVM native image for compiling Kafka broker to native executable using Quarkus framework. After going through some implementation details, we will focus on how it can be used in a Docker container with Testcontainers to speed up integration testing of Kafka applications. We will finally discuss some current caveats and future opportunities of a native-compiled Kafka for cloud-native production clusters."

1. Running Kafka as
a Native Binary using GraalVM
Ozan Gunalp, Red Hat

2. Ozan Gunalp, Red Hat – Quarkus
@ozangunalp
Running Kafka as a
Native Binary using GraalVM

3. The Problem
Kafka is fast!
3
Can we make it faster?
Reducing broker startup time as much as possible
sub second startup time

4. Integration Tests with Kafka
KafkaContainer
- Confluent platform image, provided by
org.testcontainers:kafka
- Zookeeper & Kraft-mode
- Not compatible with random
bootstrap.servers port
StrimziKafkaContainer
- Strimzi image, provided by
io.strimzi:strimzi-test-container
- Zookeeper & Kraft-mode
- Can be used with random port
- Cluster support
- Toxiproxy support to simulate network
conditions
4
Both are slow to startup
Kraft with format-storage uses another Java process to create log-dir

5. Quarkus Dev Services
5

6. Quarkus
6
Frictionless development
Optimized for Distributed Architectures

7. Quarkus Developer Joy
7
Dev Mode with live reload, Dev services and Continuous testing
Unified Configuration
Customizable defaults

8. Quarkus Kube-Native
8
Low memory overhead + Fast startup time + Small container size
=
Increase the deployment density
Observability & Resilience
❤

9. Quarkus Build-Time
9

10. kafka-native
10
Kafka broker embedded in a Quarkus application
Default configuration for dev/test purposes
Completely configurable
Compiled to native binary using GraalVM

11. GraalVM Native Image
Run the same workload on smaller machines on the cloud
11
🚀 🔐 🗜
Smaller size images
– no JVM included
Minimal attack
surface
🍃
Smaller memory
footprint
Faster startup time
– No warmup

12. native-image compilation
Reachability analysis
Build-time Class Initialization
Heap snapshotting
Metadata for Runtime Reflection, Security Providers 12

13. Compiling Kafka to native
Netty GraalVM support inherited from Quarkus
Security Providers
Plain, Scram, OAuth, GSSAPI
Substitutions
No Random init on static fields, No thread initialization
Bytecode manipulation
No JMX Support – YammerMetrics, MBean
13

14. kafka-native Quarkus Dev Service
14
Startup time RSS at startup
quay.io/strimzi-test-container/test-
container
@0.103.0-kafka-3.3.2
~3s 370Mb
vectorized/redpanda
@v22.3.4
~200ms 135Mb
quay.io/ogunalp/kafka-native
@0.2.0-kafka-3.4.0
~120ms 45Mb
quarkus.kafka.devservice.provider=kafka-native

15. kafka-native-test-container
<!-- https://mvnrepository.com/artifact/com.ozangunalp/kafka-native-test-container -->
<dependency>
<groupId>com.ozangunalp</groupId>
<artifactId>kafka-native-test-container</artifactId>
<version>0.2.0</version>
</dependency>
15
try (var container = new KafkaNativeContainer()) {
container.start();
String bootstrapServers = container.getBootstrapServers();
// create topics
// produce
// consume
}

16. kafka-native-test-container
16
String clusterId = Uuid.randomUuid().toString();
String quorumVotes = "1@broker1:9094";
var controllerAndBroker = new KafkaNativeContainer();
var brokerOnly = new KafkaNativeContainer();
var common = Map.of(
"SERVER_CLUSTER_ID", clusterId,
"KAFKA_CONTROLLER_QUORUM_VOTERS", quorumVotes);
common.forEach(controllerAndBroker::addEnv);
common.forEach(brokerOnly::addEnv);
controllerAndBroker.withNetworkAliases(broker1);
controllerAndBroker.withNetwork(network);
controllerAndBroker.addEnv("SERVER_HOST", "broker1");
controllerAndBroker.addEnv("KAFKA_BROKER_ID", "1");
brokerOnly.withNetworkAliases(broker2);
brokerOnly.withNetwork(network);
brokerOnly.addEnv("SERVER_HOST", "broker2");
brokerOnly.addEnv("KAFKA_BROKER_ID", "2");
brokerOnly.addEnv("KAFKA_PROCESS_ROLES", "broker");
Startables.deepStart(controllerAndBroker, brokerOnly);
var bs = brokerOnly.getBootstrapServers();
// produce, consume

17. kafka-native on the Cloud
17
Azure Container Instances
Single-node 1vCPU - 1Gb
~$1.30 per day (us-central)
1rps, 100rps, 1000rps, 5000rps
This is not a benchmark

18. Caveats
Good for tests and dev
Not production-ready
Classes are linked statically: limited strategy classes
GC performance – only Serial GC in GraalVM CE for now
Lacks managed beans / JMX for observability
Experimental support coming in GraalVM 23
18

19. What’s Next?
19
Support Zookeeper (Thanks to @kwall)
Support multiple Kafka versions (Thanks to @franvila)
Multi-arch images (ARM support)
Move to Smallrye organization
Experimental JMX Support
https://github.com/ozangunalp/kafka-native

20. https://quarkus.io
@quarkusio
https://quarkusio.zulipchat.com
https://code.quarkus.io
$>

21. vs Redpanda
kafka-native IS Kafka upstream code
Redpanda is more than a Kafka-compatible broker
Schema registry, HTTP proxy and UI console
Redpanda behaves slightly differently
Redpanda transactions do not support complete Kafka transactions protocol (?)
21

22. Strategy config
principal.builder.class, default: DefaultKafkaPrincipalBuilder
replica.selector.class
sasl.client.callback.handler.class
sasl.login.callback.handler.class
sasl.login.class
sasl.server.callback.handler.class
security.providers
ssl.engine.factory.class
alter.config.policy.class.name
authorizer.class.name
client.quota.callback.class
create.cluster.link.policy.class.name
create.topic.policy.class.name
kafka.metrics.reporters
metric.reporters
22