apidays LIVE India 2021 - Connecting 1.3 billion digital innovators
May 20, 2021
REST the Events - REST APIs for Event-Driven Architecture
Mark Teehan, Principal Solution Engineer at Confluent APAC
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
apidays LIVE India - REST the Events - REST APIs for Event-Driven Architecture by Mark Teehan, Confluent
1. REST the Events - REST APIs for Event-
Driven Architecture
Mark Teehan - Principal Engineer - Confluent Singapore
API Days – India
teehan@confluent.io
linkedin
2. REST the Events - REST APIs for Event-
Driven Architecture
All things apache kafka in asia pacific
Teehan@confluent.io
3. What is Kafka
A modern
distributed
platform for
data streams
Confluent helps to build
• Event-driven apps
• Data in motion
• Mainframe offload (core banking)
• Shipping logs, metrics, traces
6. Request-Response - http
Client sends request and waits for response
something that should happen
Image: The most useless Machine (imgur)
Low Latency
Point to point
Presumption of a response
Pre-defined API
7. Event Streaming - kafka
Messages received in a queue:
drop and process later
something that has happened
Image: reddit - oddly satisfying
Continuous Processing
Event-driven
No presumption of a response
General Purpose Events
8. Request-Response - http
Challenges
● Difficult to enforce standards across
services
● Does not scale if servers are synchronous
● Risky inter-service dependencies
● Services required to maintain state
● Complex management and version
compatibility—slows development
● Requires load balancing
9. Event Streaming - kafka
Challenges
● Complex Infrastructure
● Event thinking is hard
● Adopting the Kafka Way
11. So when do we want to
Use HTTP with Kafka?
● Management Plane
● Data Plane
● Other Reasons
12. ● Management Plane
● Data Plane
● Other Reasons
Cluster configuration: Management of Kafka topics, consumer groups, ACLs, etc.
CI/CD and DevOps integration:
Build delivery pipelines and to automate administration
HTTP APIs - Yes Python/Others - no
13. ● Management Plane
● Data Plane
● Other Reasons
Legacy apps: use of offbeat middleware
Usually good ETL options, but poor streaming options
HTTP/REST is sometimes the only protocol in common
Mobile apps: Natural fit for request response
Use of websockets and Server Sent Events
API Gateway: Most API management tools do not have native event streaming support
Common for integration with partner systems and service monetization
HTTP/REST is sometimes the only protocol in common
Other Cobol,Erlang, Kotlin, Node.js, PHP, Ruby, Rust; 0% ~ some% support
Languages:
14. ● Management Plane
● Data Plane
● Other Reasons
Familiarity: Quicker and easier than learning a new API
Technology
Lock In Avoid all Client dependencies
Security: Securing HTTP ports is easier (F5 proxies owned by Infosec)
Securing TCP is harder
Domain Use both:
Driven HTTP/REST for synchronous client/server
Design: Kafka protocol for decoupling
A service mesh to combine both is a common architecture.
15. So that was “When to use HTTP with Kafka”
Lets talk about three of the Kafka REST APIs
16. Three Kafka REST API :
Confluent REST Proxy
Confluent Broker REST
Confluent Cloud REST API
18. a RESTful interface to an Apache Kafka® cluster
making it easy to
● produce and consume messages
● view the metadata of the cluster
● perform administrative actions
Confluent REST Proxy
Confluent Broker REST
Confluent Cloud REST API
19. Management Plane (metadata)
● Brokers
● Topics
● Consumer Groups
● ACLS
Data Plane (data)
● Produce
● Consume
Low throughput
Limited API calls
Continuous consumption
Clustering for throughput
Confluent REST Proxy
Confluent Broker REST
Confluent Cloud REST API
21. Brokers: Describe, List, Configure
Topics: create, delete, describe, list, configure
Consumer Groups: delete, describe, list
ACLs: create, delete, describe, list
Partition Reassignments: list
Runs on the broker - not a separate service
Confluent REST Proxy
Confluent Broker REST
Confluent Cloud REST API