Kafka Rest Proxy - Installation on Linux CentOS7 - producer, consumer example

2. Intro
Kafka Cluster
incoming message
- without kafka client library support
(e.g. Android or IOS app, IOT devices, …)
- send message using Rest APIs

3. Intro
source : https://docs.confluent.io/platform/current/kafka-rest/index.html#confluent-rest-apis

4. License
source : https://www.confluent.io/confluent-community-license-faq/

5. Installation
• wget https://packages.confluent.io/archive/7.3/confluent-community-7.3.0.tar.gz
• tar -zxvf confluent-community-7.3.0.tar.gz
[root@master ~]# cd confluent-7.3.0/
[root@master confluent-7.3.0]# vi etc/kafka-rest/kafka-rest.properties
#id=kafka-rest-test-server
#schema.registry.url=http://localhost:8081
zookeeper.connect=master:2181,slave1:2181,slave2:2181
bootstrap.servers=master:9092,slave1:9092,slave2:9092
By default the server starts bound to port 8082,
does not specify a unique instance ID (required to safely run multiple proxies concurrently)
configuration settings : https://docs.confluent.io/platform/current/kafka-rest/production-deployment/rest-proxy/config.html#configuration-options-for-https

6. Run & Test
• 1. Zookeeper Ensemble, Kafka Cluster Running
• 2. execute shell at kafka rest home directory
sh bin/kafka-rest-start etc/kafka-rest/kafka-rest.properties
default port : 8082

7. 1. List all the topics in a Kafka cluster
• curl -X GET "http://localhost:8082/topics/"

8. 2. Query Kafka Topic
• curl -X GET -H "Accept: application/vnd.kafka.v2+json"
"http://localhost:8082/topics/kopo-topic"

9. 3. Sending data to Kafka Topic
• curl -X POST -H "Content-Type: application/vnd.kafka.json.v2+json" --
data '{"records":[{"value":{"name": "testUser"}}]}'
"http://localhost:8082/topics/kopo-topic"
Producer
Consumer

10. 4. Prepare Consumer
• consumer id and group registration
• curl -X POST -H "Content-Type: application/vnd.kafka.v2+json" --data '{"name":
"my_consumer_instance", "format": "json", "auto.offset.reset": "earliest"}'
http://localhost:8082/consumers/my_consumer_group

11. 4-1. Mapping topic with consumer
• curl -X POST -H "Content-Type: application/vnd.kafka.v2+json" --data
'{"topics":["kopo-topic"]}'
http://localhost:8082/consumers/my_consumer_group/instances/my_consumer_instance
/subscription

12. 4-2. Receiving data
• curl -X GET -H "Accept: application/vnd.kafka.json.v2+json"
http://localhost:8082/consumers/my_consumer_group/instances/my_consumer_instance/records

13. etc.
• Idle consumer will be destroyed after 300 seconds by default
• Delete
curl -X DELETE -H "Content-Type: application/vnd.kafka.protobuf.v2+json" http://localhost:8082/consumers/[consumer group]/instances/[consumer
instance id]
source : https://docs.confluent.io/platform/current/kafka-rest/production-deployment/rest-proxy/config.html#configuration-options-for-https
for more examples : https://docs.confluent.io/platform/current/kafka-rest/quickstart.html#produce-and-consume-json-schema-messages

14. I Think,
• In environments where the Kafka-client is not available, Rest
Proxy is a good option to consider.
• Detailed options are not available compared to implementing
directly with the Kafka-client.
• The Rest Proxy shall be used after the performance test.
• I think it is more appropriate to apply a reverse proxy using
nginx, etc., and link it with a back-end application using the
kafka-client.