Manage Consistent Configurations Across Multiple Kafka Environments with Nagashree B & S Vinod Kumar
![6
V3.0
V2.0
V1.0
Release
API
DEV
[{
"topic_name": "Topic2",
"number_of_partitions": 10,
"deployment_model": "multi_region",
"max_read_throughput": 10000,
"max_write_throughput_bytes": 10000,
"max_parallel_client_connections": 50,
“scaling_
➢ "configurations": [
{
"retention.ms": 864000
},
{
"min.insync.replica": 2
}
],
➢ "producers": [
{
"dev_user_principal": "dev_user",
"prod_user_principal": "prod_user"
}
]
➢ "consumers": [
{
"dev_user_principal":"dev_consumer_1",
"prod_user_principal":"prod_consumer_1",
"group_id": "group1"
},
{
"dev_user_principal": "dev_consumer_2",
"prod_user_principal":"prod_conusmer_2",
"group_id": "group2"
}
]
}]
GIT
Release API
Application
Owner
Dev Kafka
Cluster
V1.0
V2.0
Release
Snapshots
DEV](https://image.slidesharecdn.com/lt1120230517ksl23fidelitynagashreeb-230613151541-67601242/75/manage-consistent-configurations-across-multiple-kafka-environments-with-nagashree-b-s-vinod-kumar-6-2048.jpg?cb=1686669686)
Manage Consistent Configurations Across Multiple Kafka Environments with Nagashree B & S Vinod Kumar
"Conventional Kafka developers are liable to perform monotonous functions like creating topics, managing topic configurations, granting access for producers/consumers, topic sizing, capacity planning and other day-2 operations. At Fidelity Investments we support hundreds of applications publishing and subscribing to tens of thousands of event streams across business units. Changes to applications are constant. Expecting developers to manually repeat commands across environments slows development and increases the chance for misconfigurations, ultimately degrading the quality of work. In this talk, we will discuss how Fidelity Investments modeled a unique API to seamlessly lift and shift application topics, ACLs, quotas, and every other entity from lower environment to higher environment clusters, thus resulting in the propagation of exact configurations across environments with zero human errors. We will also highlight the salient features of this in-house REST API in providing options to scale differently in higher environments. Flexibility of modifying users across environments while preserving complete Kafka cluster configuration is an added bonus. This entire process empowers Kafka applications teams to promote version-controlled releases across multiple environments with just a single click and a safety net to rollback to older configurations."
Recommandé
Contenu connexe
Similaire à Manage Consistent Configurations Across Multiple Kafka Environments with Nagashree B & S Vinod Kumar
Similaire à Manage Consistent Configurations Across Multiple Kafka Environments with Nagashree B & S Vinod Kumar(20)
Plus de HostedbyConfluent
Plus de HostedbyConfluent(20)
Dernier
Dernier(20)
Manage Consistent Configurations Across Multiple Kafka Environments with Nagashree B & S Vinod Kumar
- 1. Promote, Don't Repeat! How to manage consistent configurations across multiple Kafka environments Nagashree B S Vinod Kumar Event Streaming Platform Team Fidelity Investments
- 2. The Curse of Scale: Repetition & Inconsistency Dine in LIVE Party Order Choose Bun Type Select Patty Cheese? Veggies of choice Pack for Delivery Ready!! Customize your Burger
- 3. Create Topic Change Topic Config Grant Access Schemas? MVP DEV The Curse of Scale: Repetition & Inconsistency QA PROD retention = 3 days retention = 2 days ACLs P C C Topic1 P0 P1 P2 Topic1 P0 P1 P2 P3 retention = 3 days Inconsistency Inconsistency ACLs Topic1 P0 P1 P2 ACLs P C
- 4. API Management Plane 4 Cluster Topology Topic1 P1 P0 Topic Management • Create Topic • Modify Topic Configurations • Scale Topic • Delete Topic Access Data Plane • Add Producer • Add Consumer • Remove Producer • Remove Consumer Schema Management • Register Subject • Add Schemas • Remove Schemas • Change Compatibility Persist state ➢ Multi-cluster model ➢ Active-Active replication across Regions ➢ Separation of Data Plane and Management Plane API Management Plane Topic1 P1 P0
- 5. 5 DEV QA/PERF PROD API Management Plane 5 Topic Management • Create Topic • Modify Topic Configurations • Scale Topic • Delete Topic Access Data Plane • Add Producer • Add Consumer • Remove Producer • Remove Consumer Schema Management • Register Subject • Add Schemas • Remove Schemas • Change Compatibility Persist state 5 API Management Plane ➢ Multi-cluster model ➢ Active-Active replication across Regions ➢ Separation of Data Plane and Management Plane New Creations & Updates Release & Promote
- 6. 6 V3.0 V2.0 V1.0 Release API DEV [{ "topic_name": "Topic2", "number_of_partitions": 10, "deployment_model": "multi_region", "max_read_throughput": 10000, "max_write_throughput_bytes": 10000, "max_parallel_client_connections": 50, “scaling_ ➢ "configurations": [ { "retention.ms": 864000 }, { "min.insync.replica": 2 } ], ➢ "producers": [ { "dev_user_principal": "dev_user", "prod_user_principal": "prod_user" } ] ➢ "consumers": [ { "dev_user_principal":"dev_consumer_1", "prod_user_principal":"prod_consumer_1", "group_id": "group1" }, { "dev_user_principal": "dev_consumer_2", "prod_user_principal":"prod_conusmer_2", "group_id": "group2" } ] }] GIT Release API Application Owner Dev Kafka Cluster V1.0 V2.0 Release Snapshots DEV
- 7. 7 PROD QA DEV Promote API QA DEV PROD Customize configs using scaling factors Customize configs using scaling factors QA is the New PROD JSON changes to be added is displayed as response Apply latest release changes to the higher env cluster Promote API Existing Resource Configs in QA Cluster ~ SRE Return JSON changes True False Release type == dryrun Incoming change Retention to 3 days JSON V2.0 QA V1.0 JSON V1.0 JSON V1.0 JSON V2.0 JSON V2.0 JSON V3.0 JSON V3.0 JSON V3.0 JSON V3.0 JSON
- 8. Thank You! Kafka Summit London 2023 Nagashree B /bnagashree S Vinod Kumar /s-vinod-kumar F I N D Y O U R F I D E L I T Y