Migrating applications to serverless Apache Kafka + KSQL

•Télécharger en tant que PPTX, PDF•

2 j'aime•150 vues

The document describes the steps to migrate a legacy movie ratings application to a serverless architecture using Apache Kafka and KSQL. The legacy application uses a monolithic, database-centric design to calculate average movie ratings. The migration plan involves: 1) Using Kafka Connect to extract rating data to Kafka topics, 2) Setting up a Confluent Cloud Kafka cluster, 3) Replicating data to the cloud cluster, 4) Processing data with KSQL queries, 5) Building microservices powered by KSQL output, and 6) Decommissioning the monolith once migration is complete.

Technologie

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Migrating Applications to Serverless
Apache Kafka and KSQL
Tim Berglund
S e s s i o n I D
Sr. Director, Developer Relations
Confluent

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

Our Movie Rating Service, Legacy Edition
• Users rate movies, ratings go into Kafka
• Monolithic, database-centric application calculates averages
• Serves them to users through a web UI and API
Movie
Ratings
Users
Movies
Top Rated Movies
My Favorite Movie
Moviegoers

Kafka served well
• Decouples event input from processing
• Easily understood abstraction for event processing
• Not exactly a pleasure to operate, but we can’t complain
Movie
Ratings
Users
Movies
Top Rated Movies
My Favorite Movie
Moviegoers

Our Monolith’s Problems
• It will do a bad job managing complexity as our service grows
• The Kafka Consumer code is bespoke
• It is a textbook pre-cloud architecture
• We cannot trivially scale to larger message volumes
Movie
Ratings
Users
Movies
Top Rated Movies
My Favorite Movie
Moviegoers

Our Refactoring Plan
• Capture Users and Movies as Kafka topics
• Migrate all topics to Confluent Cloud using Confluent Replicator
• Refactor monolith to microservices
• Keep web UI nearly untouched
• Never touch the on-prem system until the migration is complete
Movie
Ratings
Users
Movies
Top Rated Movies
My Favorite Movie
Moviegoers

Step One: Fewer Databases
• Use Kafka Connect to extract Users and Movies tables to Kafka topics
Movie
Ratings
Moviegoers
Movies
Users
Kafka
Connect

Step Two: Spin up a Confluent Cloud cluster
• We want to get out of the business of managing Kafka ourselves
Movie
Ratings
Movies
Users

Step Three: Deploy Confluent Replicator
• Use Kafka Connect to extract Users and Movies tables to Kafka topics
Movie
Ratings
Movies
Users
Movie
Ratings
Movies
Users
Replicator
Replicator
Replicator

Step Four: Convert to KSQL
• Bespoke Consumer code implements non-differentiated functionality
Movie
Ratings
Movies
Users
CREATE TABLE movie_ratings AS
SELECT title,
SUM(rating)/COUNT(rating) AS avg_rating,
COUNT(rating) AS num_ratings
FROM ratings
LEFT OUTER JOIN movies
ON ratings.movie_id = movies.movie_id
GROUP BY title;

Step Four: Convert to KSQL
• The rating averaging query
Movie
Ratings
Movies
Users
Rated
Movies
KSQL
magic goes here

Step Four: Convert to KSQL
• The user favorite query
Movie
Ratings
Movies
Users
Rated
Movies
KSQL
magic goes here
more
KSQL
magic goes here
User
Favorites

Step Five: Extract the rating average service
• Now serve rating averages from KSQL output
• Monolith no longer serves these results
Rating Averages
Rated
Movies

Step Six: Extract the user favorite service
• Now serve rating averages from KSQL output
• Monolith no longer serves these results
User Favorites User
Favorites

Step Seven: Stand down the monolith
• Now serve rating averages from KSQL output
• Monolith no longer serves these results
User Favorites
Rated Movies
User
Favorites
Movie
Ratings
Moviegoers
Rated
Movies
so much
KSQL
magic
Movies
Users

Step Eight: Stand down Replicator
• All data is in Confluent Cloud now
• For hybrid on-prem/cloud deployment

Thank you!
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Tim Berglund
@tlberglund
https://kafka-tutorials.confluent.io
https://slackpass.io/confluentcommunity

Contenu connexe

Tendances

Input features are the building blocks for machine learning models. You cannot have a great model without great features. By building on top of Apache Pulsar's infinite retention of events, we built infrastructure to serve features in production and to generate training datasets. It allowed our machine learning teams to change, test, and deploy personalization features at an extraordinary rate to 10s of millions of end-users. This talk will discuss: - What event-sourcing is and why it's so powerful for machine learning infrastructure. - How we built the StreamSQL feature store on top of Pulsar, Flink, and Cassandra. - How a feature store accelerates ML development.

StreamSQL Feature Store (Apache Pulsar Summit)

Simba Khadder

At Hootsuite, we've been transitioning from a single monolithic PHP application to a set of scalable Scala-based microservices. To avoid excessive coupling between services, we've implemented an event system using Apache Kafka that allows events to be reliably produced + consumed asynchronously from services as well as data stores. In this presentation, I talk about: - Why we chose Kafka - How we set up our Kafka clusters to be scalable, highly available, and multi-data-center aware. - How we produce + consume events - How we ensure that events can be understood by all parts of our system (Some that are implemented in other programming languages like PHP and Python) and how we handle evolving event payload data.

Building an Event Bus at Scale

jimriecken

Whilst Kafka has the ability to encrypt data in transit, it does not have the functionality out of the box to encrypt data at rest. This places the responsibility of encryption of data placed on message queues on developers. Implementing cryptography correctly in our applications is challenging and time consuming. In this demo-driven talk, I will show you how you can use HashiCorp Vault’s API to implement a simple workflow that offsets the complexity of cryptography to Vault. In just a few lines of code, I will demonstrate how message producers will be able to encrypt its data, whilst message consumers can decrypt message payloads with minimal development effort. I will also show how to troubleshoot common errors from the API. By the end of this talk, you will learn how to implement symmetric and asymmetric encryption of your application data before placing it on Kafka message queues. You will also learn how to implement this workflow using Format Preserving Encryption (FPE).

Encrypting Kafka messages at rest to secure applications | Robert Barnes, Has...

HostedbyConfluent

Building a derived data store using Kafka

Venu Ryali

Introduction to Kafka

Akash Vacher

Kafka aws

Ariel Moskovich

Kafka meetup seattle 2019 mirus reliable, high performance replication for ap...

Nitin Kumar

How to Lock Down Apache Kafka and Keep Your Streams Safe

confluent

Fundamentals and Architecture of Apache Kafka

Angelo Cesaro

Confluent building a real-time streaming platform using kafka streams and k...

Thomas Alex

Apache kafka

Long Nguyen

Even though Kafka is scalable by design, proper handling of over one petabyte of data a day requires much more than Kafka’s scalability. Several challenges present themselves in a data centric business at this scale. These challenges include capacity planning, provisioning, message auditing, monitoring and alerting, rebalancing workloads with changes in traffic patterns, data lineage, handling service degradation and system outages, optimizing cost, upgrades, etc. In this talk we describe how at Pinterest we tackle some of these challenges and share some of the key lessons that we learned in the process. Specifically we will share how we: * Automate Kafka cluster maintenance * Manage over 150K partitions * Manage upgrade lifecycle * Track / troubleshoot thousands of data pipelines

Organic Growth and A Good Night Sleep: Effective Kafka Operations at Pinteres...

confluent

What happened when our biggest and most important Kafka cluster went rogue all of a sudden, and while trying to recover it, a single, crucial misconfiguration made things even worse? At a company like Taboola, where service availability and latency are our top priority, this was a disaster. With 300K messages/sec and 250TB of messages produced each day to our on-premise Kafka clusters, and mirrored to our central Kafka cluster, we always try to ensure Kafka behaves well under high loads of traffic and unexpected cluster failures. So when our main Kafka cluster went crazy we had a serious issue on our hands. This session is the story of how we learned the hard way about mitigating cluster failures with the proper configurations in place.

Oops! I started a broker | Yinon Kahta, Taboola

HostedbyConfluent

___________________________________________ Meetup#7 | Session 2 | 21/03/2018 | Taboola _____________________________________________ In this talk, we will present our multi-DC Kafka architecture, and discuss how we tackle sending and handling 10B+ messages per day, with maximum availability and no tolerance for data loss. Our architecture includes technologies such as Cassandra, Spark, HDFS, and Vertica - with Kafka as the backbone that feeds them all.

Distributed Kafka Architecture Taboola Scale

Apache Kafka TLV

Building a company-wide data pipeline on Apache Kafka - engineering for 150 b...

LINE Corporation

Have you ever migrated Kafka clusters from one data center to another being completely transparent to client applications? At PayPal, as part of a massive datacenter migration initiative, Kafka team successfully moved all PayPal Kafka traffic across data centers. This initiative involved migrating 20+ Kafka clusters (1000+ broker and zookeeper nodes), as well as 60+ mirrormaker groups which seamlessly handle Kafka traffic volumes as high as 1 trillion messages per day. Throughout the course of this migration, applications required no modification, encountered 0% service outage, 0% message loss and duplicated messages. The whole migration process was fully transparent to Kafka applications. In this session, you will learn the strategies, techniques and tools the PayPal Kafka team has utilized for managing the migration process. You will also learn the lessons and pitfalls they experienced during this exercise, as well as the secret sauce of making the migration successful.

How did we move the mountain? - Migrating 1 trillion+ messages per day across...

HostedbyConfluent

Introduction to apache kafka

Dimitris Kontokostas

As architectures have become more complex, no single protocol or service can fulfill every use case. As a result, teams have turned to leveraging a multitude of services, like Kafka, REST, GraphQL, gRPC, SOAP and MQTT depending on the use case. However, this introduces a multitude of problems. The design experience is inconsistent, as each service has its own specification or schema, from OpenAPI to Avro, Protobuf to AsyncAPI. Each API or service requires different utilities, products or SDKs to simply send and receive data. Creating tests or mocks requires imperfect and fragile scripts. Join this talk to learn how SmartBear is building the world's first universal and protocol-agnostic API platform and the lessons learned along the way.

Becoming Protocol-Agnostic with Kafka, REST, GraphQL & gRPC | Tyler Mills, Sm...

HostedbyConfluent

Capture the Streams of Database Changes

confluent

Organizations have a need to protect Personally Identifiable Information (PII). As Event Streaming Architecture (ESA) becomes ubiquitous in the enterprise, the prevalence of PII within data streams will only increase. Data architects must be cognizant of how their data pipelines can allow for potential leaks. In highly distributed systems, zero-trust networking has become an industry best practice. We can do the same with Kafka by introducing message-level security. A DevSecOps Engineer with some Kafka experience can leverage Kafka Streams to protect PII by enforcing role-based access control using Open Policy Agent. Rather than implementing a REST API to handle message-level security, Kafka Streams can filter, or even transform outgoing messages in order to redact PII data while leveraging the native capabilities of Kafka. In our proposed presentation, we will provide a live demonstration that consists of two consumers subscribing to the same Kafka topic, but receiving different messages based on the rules specified in Open Policy Agent. At the conclusion of the presentation, we will provide attendees with a GitHub repository, so that they can enjoy a sandbox environment for hands-on experimentation with message-level security.

Securing the Message Bus with Kafka Streams | Paul Otto and Ryan Salcido, Raf...

HostedbyConfluent

Tendances (20)

StreamSQL Feature Store (Apache Pulsar Summit)

Building an Event Bus at Scale

Encrypting Kafka messages at rest to secure applications | Robert Barnes, Has...

Building a derived data store using Kafka

Introduction to Kafka

Kafka aws

Kafka meetup seattle 2019 mirus reliable, high performance replication for ap...

How to Lock Down Apache Kafka and Keep Your Streams Safe

Fundamentals and Architecture of Apache Kafka

Confluent building a real-time streaming platform using kafka streams and k...

Apache kafka

Organic Growth and A Good Night Sleep: Effective Kafka Operations at Pinteres...

Oops! I started a broker | Yinon Kahta, Taboola

Distributed Kafka Architecture Taboola Scale

Building a company-wide data pipeline on Apache Kafka - engineering for 150 b...

How did we move the mountain? - Migrating 1 trillion+ messages per day across...

Introduction to apache kafka

Becoming Protocol-Agnostic with Kafka, REST, GraphQL & gRPC | Tyler Mills, Sm...

Capture the Streams of Database Changes

Securing the Message Bus with Kafka Streams | Paul Otto and Ryan Salcido, Raf...

Similaire à Migrating applications to serverless Apache Kafka + KSQL

Going Cloud Native with IBM Cloud and NetflixOSS for Dev@Pulse

aspyker

Immutable infrastructure tsap_v2

Volodymyr Tsap

Serverless architectures let you build and deploy applications and services with infrastructure resources that require zero administration. In the past, you had to provision and scale servers to run your application code, install and operate distributed databases, and build and run custom software to handle API requests. Now, AWS provides a stack of scalable, fully-managed services that eliminates these operational complexities. In this session, you learn about the concepts and benefits of serverless architectures and the basics of the serverless stack AWS provides (e.g., AWS Lambda and Amazon API Gateway). We discuss use cases such as data processing, website backends, serverless applications and "operational glue". After that, you get practical tips and tricks, best practices, and architecture patterns that you can take back and implement immediately.

AWS re:Invent 2016: Getting Started with Serverless Architectures (CMP211)

Amazon Web Services

Gcp intro-20160721

Haeseung Lee

In this workshop, we build out an end-to-end Amazon AppStream 2.0 environment for your organization. We create a master image containing desktop application and configure a streaming fleet and streaming stack. We walk through network configuration options, and we show you how to connect to resources in your VPC. Finally, we show you how to create streaming URLs that users need to access their applications. To complete this workshop, you must bring your laptop, have an individual AWS account that has already been provisioned, and have working knowledge of AWS concepts. Also, it is beneficial to attend the session, "Securely Deliver Desktop Applications with Amazon AppStream 2.0.”

Build an AppStream 2.0 Environment to Deliver Desktop Applications to Any Com...

Amazon Web Services

Active web page chapter for reading purpose

SambalSwetank

Yelowsoft delivers super app to bbr one of the biggest ride hailing companies...

Yelowsoft

Yelowsoft delivers super app to bbr one of the biggest ride hailing companies...

Yelowsoft

Oscon2014 Netflix API - Top 10 Lessons Learned

Sangeeta Narayanan

Can We Make Maps from Videos? ~From AI Algorithm to Engineering for Continuou...

DeNA

Scalability and flexibility can make or break your online video success. When putting an online video plan in place for your organization, you want to be certain that you will be able to accommodate adoption and usage, increasing content quantity and quality, and any new features you may need. In Kaltura Inspire this webinar we discussed the fundamentals of API-driven online video solutions and how you can leverage this to gain scalability, flexibility, and control of end-user experience. How to avoid potentially slow, risky, and error prone manual deployments by using the Kaltura API to fully automate your cluster deployments and upgrades. See how you could plan and execute a video platform that better fits your needs and have more influence on how your users experience online video.

Kaltura Inspire Webinar: API Driven Video Platform - The Key to Scalability a...

Zohar Babin

About captionator english 20150622

Will Kim

Canada DevOps Summit 2020 Presentation Nov_03_2020

Varun Manik

Spring Cloud Pipelines is an opinionated framework that automates the creation of structured continuous deployment pipelines. In this presentation we’ll go through the contents of the Spring Cloud Pipelines project. We’ll start a new project for which we’ll have a deployment pipeline set up in no time. We’ll deploy to Cloud Foundry and check if our application is backwards compatible so that we can roll it back on production.

Continuous Deployment of your Application @SpringOne

ciberkleid

This session provides an overview of the AWS Digital Media Partner ecosystem and explains how APN partners, including ISVs, SIs and solution providers, leverage the AWS Cloud to enable media workloads to be executed in the cloud. Included is an overview of the AWS media workflow categories and a discussion of the alignment of these partners to these categories as well as to the sub-vertical industries that comprise Media and Entertainment.

[AWS LA Media & Entertainment Event 2015]: M&E Ecosystem Update Q4 2015

Amazon Web Services

Geeeez, after demanding you unit test, system test, black box test, white box test, test-test-test everything, your manager is now demanding you load test your brand spanking new Oracle web application. How on earth can you do this? This technical presentation will explain the concepts behind preparing for load testing, the Http protocol's request/response model, and live demonstrations using Oracle's Http Analyzer and Apache's JMeter to stress test your Oracle web application. The presentation is suitable for anybody, be it DBAs or developers, who are concerned about the performance of any web based application, possibly an Apex or JDeveloper or 3rd party web application. Knowledge of Apex or JDeveloper is not mandatory for this presentation and they will not be covered in any depth.

Take a load off! Load testing your Oracle APEX or JDeveloper web applications

Sage Computing Services

Top 10 Lessons Learned from the Netflix API - OSCON 2014

Daniel Jacobson

Continuous Deployment of your Application - SpringOne Tour Dallas

VMware Tanzu

IBM Connect 2016: Speaker Session with Teresa Deane, Senior Developer, BCC

BCC - Solutions for IBM Collaboration Software

Presented by Yaron Inger, CTO @Lightricks in the TLV iOS Developers Meetup 23/12/15 http://www.meetup.com/Tel-Aviv-iOS-Developers-Meetup Enlight is an iOS app designed to be a one-stop-shop for all your photo editing needs, that's been recently selected by Apple as the *2015 App of the Year* in Canada, UK, Germany, China and more, and as runner-up in the US. In this talk I will reveal some of the concepts and infrastructures that enabled us to create this extremely complex app with a relatively small team of designers and developers.

Yaron Inger - Enlight - Inside the app of the year

tlv-ios-dev

Similaire à Migrating applications to serverless Apache Kafka + KSQL (20)

Going Cloud Native with IBM Cloud and NetflixOSS for Dev@Pulse

Immutable infrastructure tsap_v2

AWS re:Invent 2016: Getting Started with Serverless Architectures (CMP211)

Gcp intro-20160721

Build an AppStream 2.0 Environment to Deliver Desktop Applications to Any Com...

Active web page chapter for reading purpose

Yelowsoft delivers super app to bbr one of the biggest ride hailing companies...

Oscon2014 Netflix API - Top 10 Lessons Learned

Can We Make Maps from Videos? ~From AI Algorithm to Engineering for Continuou...

Kaltura Inspire Webinar: API Driven Video Platform - The Key to Scalability a...

About captionator english 20150622

Canada DevOps Summit 2020 Presentation Nov_03_2020

Continuous Deployment of your Application @SpringOne

[AWS LA Media & Entertainment Event 2015]: M&E Ecosystem Update Q4 2015

Take a load off! Load testing your Oracle APEX or JDeveloper web applications

Top 10 Lessons Learned from the Netflix API - OSCON 2014

Continuous Deployment of your Application - SpringOne Tour Dallas

IBM Connect 2016: Speaker Session with Teresa Deane, Senior Developer, BCC

Yaron Inger - Enlight - Inside the app of the year

Plus de confluent

Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...

confluent

Santander Stream Processing with Apache Flink

confluent

Unlocking the Power of IoT: A comprehensive approach to real-time insights

confluent

El Stream processing es un requisito previo de la pila de data streaming, que impulsa aplicaciones y pipelines en tiempo real. Permite una mayor portabilidad de datos, una utilización optimizada de recursos y una mejor experiencia del cliente al procesar flujos de datos en tiempo real. En nuestro taller práctico híbrido, aprenderás cómo filtrar, unir y enriquecer fácilmente datos en tiempo real dentro de Confluent Cloud utilizando nuestro servicio Flink sin servidor.

Workshop híbrido: Stream Processing con Flink

confluent

Our talk will explore the transformative impact of integrating Confluent, HiveMQ, and SparkPlug in Industry 4.0, emphasizing the creation of a Unified Namespace. In addition to the creation of a Unified Namespace, our webinar will also delve into Stream Governance and Scaling, highlighting how these aspects are crucial for managing complex data flows and ensuring robust, scalable IIoT-Platforms. You will learn how to ensure data accuracy and reliability, expand your data processing capabilities, and optimize your data management processes. Don't miss out on this opportunity to learn from industry experts and take your business to the next level.

Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...

confluent

La arquitectura impulsada por eventos (EDA) será el corazón del ecosistema de MAPFRE. Para seguir siendo competitivas, las empresas de hoy dependen cada vez más del análisis de datos en tiempo real, lo que les permite obtener información y tiempos de respuesta más rápidos. Los negocios con datos en tiempo real consisten en tomar conciencia de la situación, detectar y responder a lo que está sucediendo en el mundo ahora.

AWS Immersion Day Mapfre - Confluent

confluent

Eventos y Microservicios - Santander TechTalk

confluent

Q&A with Confluent Experts: Navigating Networking in Confluent Cloud

confluent

Citi TechTalk Session 2: Kafka Deep Dive

confluent

Traditional data pipelines often face scalability issues and challenges related to cost, their monolithic design, and reliance on batch data processing. They also typically operate under the premise that all data needs to be stored in a single centralized data source before it's put to practical use. Confluent Cloud on Amazon Web Services (AWS) provides a fully managed cloud-native platform that helps you simplify the way you build real-time data flows using streaming data pipelines and Apache Kafka.

Build real-time streaming data pipelines to AWS with Confluent

confluent

Q&A with Confluent Professional Services: Confluent Service Mesh

confluent

Citi Tech Talk: Event Driven Kafka Microservices

confluent

An in depth look at how Confluent is being used in the financial services industry. Gain an understanding of how organisations are utilising data in motion to solve common problems and gain benefits from their real time data capabilities. It will look more deeply into some specific use cases and show how Confluent technology is used to manage costs and mitigate risks. This session is aimed at Solutions Architects, Sales Engineers and Pre Sales, and also the more technically minded business aligned people. Whilst this is not a deeply technical session, a level of knowledge around Kafka would be helpful.

Confluent & GSI Webinars series - Session 3

confluent

Transforming applications built with traditional messaging solutions such as TIBCO, MQ and Solace to be scalable, reliable and ready for the move to cloud How can applications built with traditional messaging technologies like TIBCO, Solace and IBM MQ be modernised and be made cloud ready? What are the advantages to Event Streaming approaches to pub/sub vs traditional message queues? What are the strengeths and weaknesses of both approaches, and what use cases and requirements are actually a better fit for messaging than Kafka?

Citi Tech Talk: Messaging Modernization

confluent

Citi Tech Talk: Data Governance for streaming and real time data

confluent

Confluent & GSI Webinars series: Session 2

confluent

Vous apprendrez également à : • Créer plus rapidement des produits et fonctionnalités à l’aide d’une suite complète de connecteurs et d’outils de gestion des flux, et à connecter vos environnements à des pipelines de données • Protéger vos données et charges de travail les plus critiques grâce à des garanties intégrées en matière de sécurité, de gouvernance et de résilience • Déployer Kafka à grande échelle en quelques minutes tout en réduisant les coûts et la charge opérationnelle associés

Data In Motion Paris 2023

confluent

Confluent Partner Tech Talk with Synthesis

confluent

The Future of Application Development - API Days - Melbourne 2023

confluent

The Playful Bond Between REST And Data Streams

confluent

Plus de confluent (20)

Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...

Santander Stream Processing with Apache Flink

Unlocking the Power of IoT: A comprehensive approach to real-time insights

Workshop híbrido: Stream Processing con Flink

Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...

AWS Immersion Day Mapfre - Confluent

Eventos y Microservicios - Santander TechTalk

Q&A with Confluent Experts: Navigating Networking in Confluent Cloud

Citi TechTalk Session 2: Kafka Deep Dive

Build real-time streaming data pipelines to AWS with Confluent

Q&A with Confluent Professional Services: Confluent Service Mesh

Citi Tech Talk: Event Driven Kafka Microservices

Confluent & GSI Webinars series - Session 3

Citi Tech Talk: Messaging Modernization

Citi Tech Talk: Data Governance for streaming and real time data

Confluent & GSI Webinars series: Session 2

Data In Motion Paris 2023

Confluent Partner Tech Talk with Synthesis

The Future of Application Development - API Days - Melbourne 2023

The Playful Bond Between REST And Data Streams

Dernier

Strategies for Landing an Oracle DBA Job as a Fresher

Remote DBA Services

Automating Google Workspace (GWS) & more with Apps Script

wesley chun

Three things you will take away from the session: • How to run an effective tenant-to-tenant migration • Best practices for before, during, and after migration • Tips for using migration as a springboard to prepare for Copilot in Microsoft 365 Main ideas: Migration Overview: The presentation covers the current reality of cross-tenant migrations, the triggers, phases, best practices, and benefits of a successful tenant migration Considerations: When considering a migration, it is important to consider the migration scope, performance, customization, flexibility, user-friendly interface, automation, monitoring, support, training, scalability, data integrity, data security, cost, and licensing structure Next Wave: The next wave of change includes the launch of Copilot, which requires businesses to be prepared for upcoming changes related to Copilot and the cloud, and to consolidate data and tighten governance ShareGate: ShareGate can help with pre-migration analysis, configurable migration tool, and automated, end-user driven collaborative governance

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

sammart93

Effective data discovery is crucial for maintaining compliance and mitigating risks in today's rapidly evolving privacy landscape. However, traditional manual approaches often struggle to keep pace with the growing volume and complexity of data. Join us for an insightful webinar where industry leaders from TrustArc and Privya will share their expertise on leveraging AI-powered solutions to revolutionize data discovery. You'll learn how to: - Effortlessly maintain a comprehensive, up-to-date data inventory - Harness code scanning insights to gain complete visibility into data flows leveraging the advantages of code scanning over DB scanning - Simplify compliance by leveraging Privya's integration with TrustArc - Implement proven strategies to mitigate third-party risks Our panel of experts will discuss real-world case studies and share practical strategies for overcoming common data discovery challenges. They'll also explore the latest trends and innovations in AI-driven data management, and how these technologies can help organizations stay ahead of the curve in an ever-changing privacy landscape.

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

TrustArc

Axa Assurance Maroc - Insurer Innovation Award 2024

The Digital Insurer

Imagine a world where information flows as swiftly as thought itself, making decision-making as fluid as the data driving it. Every moment is critical, and the right tools can significantly boost your organization’s performance. The power of real-time data automation through FME can turn this vision into reality. Aimed at professionals eager to leverage real-time data for enhanced decision-making and efficiency, this webinar will cover the essentials of real-time data and its significance. We’ll explore: FME’s role in real-time event processing, from data intake and analysis to transformation and reporting An overview of leveraging streams vs. automations FME’s impact across various industries highlighted by real-life case studies Live demonstrations on setting up FME workflows for real-time data Practical advice on getting started, best practices, and tips for effective implementation Join us to enhance your skills in real-time data automation with FME, and take your operational capabilities to the next level.

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

Safe Software

This presentations targets students or working professionals. You may know Google for search, YouTube, Android, Chrome, and Gmail, but did you know Google has many developer tools, platforms & APIs? This comprehensive yet still high-level overview outlines the most impactful tools for where to run your code, store & analyze your data. It will also inspire you as to what's possible. This talk is 50 minutes in length.

Powerful Google developer tools for immediate impact! (2023-24 C)

wesley chun

Data Cloud, More than a CDP by Matt Robison

Anna Loughnan Colquhoun

Partners Life - Insurer Innovation Award 2024

The Digital Insurer

Tech Trends Report 2024 Future Today Institute.pdf

hans926745

Scaling API-first – The story of a global engineering organization

Radu Cotescu

🐬 The future of MySQL is Postgres 🐘

RTylerCroy

Handwritten Text Recognition for manuscripts and early printed texts

Maria Levchenko

Histor y of HAM Radio presentation slide

vu2urc

presentation ICT roal in 21st century education

jfdjdjcjdnsjd

Building Digital Trust in a Digital Economy Veronica Tan, Director - Cyber Security Agency of Singapore Apidays Singapore 2024: Connecting Customers, Business and Technology (April 17 & 18, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

apidays

Finology Group – Insurtech Innovation Award 2024

The Digital Insurer

Boost Fertility New Invention Ups Success Rates.pdf

sudhanshuwaghmare1

Join our latest Connector Corner webinar to discover how UiPath Integration Service revolutionizes API-centric automation in a 'Quote to Cash' process—and how that automation empowers businesses to accelerate revenue generation. A comprehensive demo will explore connecting systems, GenAI, and people, through powerful pre-built connectors designed to speed process cycle times. Speakers: James Dickson, Senior Software Engineer Charlie Greenberg, Host, Product Marketing Manager

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

DianaGray10

As privacy and data protection regulations evolve rapidly, organizations operating in multiple jurisdictions face mounting challenges to ensure compliance and safeguard customer data. With state-specific privacy laws coming up in multiple states this year, it is essential to understand what their unique data protection regulations will require clearly. How will data privacy evolve in the US in 2024? How to stay compliant? Our panellists will guide you through the intricacies of these states' specific data privacy laws, clarifying complex legal frameworks and compliance requirements. This webinar will review: - The essential aspects of each state's privacy landscape and the latest updates - Common compliance challenges faced by organizations operating in multiple states and best practices to achieve regulatory adherence - Valuable insights into potential changes to existing regulations and prepare your organization for the evolving landscape

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

TrustArc

Dernier (20)

Strategies for Landing an Oracle DBA Job as a Fresher

Automating Google Workspace (GWS) & more with Apps Script

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

Axa Assurance Maroc - Insurer Innovation Award 2024

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

Powerful Google developer tools for immediate impact! (2023-24 C)

Data Cloud, More than a CDP by Matt Robison

Partners Life - Insurer Innovation Award 2024

Tech Trends Report 2024 Future Today Institute.pdf

Scaling API-first – The story of a global engineering organization

🐬 The future of MySQL is Postgres 🐘

Handwritten Text Recognition for manuscripts and early printed texts

Histor y of HAM Radio presentation slide

presentation ICT roal in 21st century education

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

Finology Group – Insurtech Innovation Award 2024

Boost Fertility New Invention Ups Success Rates.pdf

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

Migrating applications to serverless Apache Kafka + KSQL

1. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Migrating Applications to Serverless Apache Kafka and KSQL Tim Berglund S e s s i o n I D Sr. Director, Developer Relations Confluent

3. Our Movie Rating Service, Legacy Edition • Users rate movies, ratings go into Kafka • Monolithic, database-centric application calculates averages • Serves them to users through a web UI and API Movie Ratings Users Movies Top Rated Movies My Favorite Movie Moviegoers

4. Kafka served well • Decouples event input from processing • Easily understood abstraction for event processing • Not exactly a pleasure to operate, but we can’t complain Movie Ratings Users Movies Top Rated Movies My Favorite Movie Moviegoers

5. Or can we?

6. Our Monolith’s Problems • It will do a bad job managing complexity as our service grows • The Kafka Consumer code is bespoke • It is a textbook pre-cloud architecture • We cannot trivially scale to larger message volumes Movie Ratings Users Movies Top Rated Movies My Favorite Movie Moviegoers

8. Our Refactoring Plan • Capture Users and Movies as Kafka topics • Migrate all topics to Confluent Cloud using Confluent Replicator • Refactor monolith to microservices • Keep web UI nearly untouched • Never touch the on-prem system until the migration is complete Movie Ratings Users Movies Top Rated Movies My Favorite Movie Moviegoers

9. Step One: Fewer Databases • Use Kafka Connect to extract Users and Movies tables to Kafka topics Movie Ratings Moviegoers Movies Users Kafka Connect

10. Step Two: Spin up a Confluent Cloud cluster • We want to get out of the business of managing Kafka ourselves Movie Ratings Movies Users

11. Step Three: Deploy Confluent Replicator • Use Kafka Connect to extract Users and Movies tables to Kafka topics Movie Ratings Movies Users Movie Ratings Movies Users Replicator Replicator Replicator

12. Step Four: Convert to KSQL • Bespoke Consumer code implements non-differentiated functionality Movie Ratings Movies Users CREATE TABLE movie_ratings AS SELECT title, SUM(rating)/COUNT(rating) AS avg_rating, COUNT(rating) AS num_ratings FROM ratings LEFT OUTER JOIN movies ON ratings.movie_id = movies.movie_id GROUP BY title;

13. Step Four: Convert to KSQL • The rating averaging query Movie Ratings Movies Users Rated Movies KSQL magic goes here

14. Step Four: Convert to KSQL • The user favorite query Movie Ratings Movies Users Rated Movies KSQL magic goes here more KSQL magic goes here User Favorites

15. Step Five: Extract the rating average service • Now serve rating averages from KSQL output • Monolith no longer serves these results Rating Averages Rated Movies

16. Step Six: Extract the user favorite service • Now serve rating averages from KSQL output • Monolith no longer serves these results User Favorites User Favorites

17. Step Seven: Stand down the monolith • Now serve rating averages from KSQL output • Monolith no longer serves these results User Favorites Rated Movies User Favorites Movie Ratings Moviegoers Rated Movies so much KSQL magic Movies Users

18. Step Eight: Stand down Replicator • All data is in Confluent Cloud now • For hybrid on-prem/cloud deployment

Migrating applications to serverless Apache Kafka + KSQL

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Migrating applications to serverless Apache Kafka + KSQL

Similaire à Migrating applications to serverless Apache Kafka + KSQL (20)

Plus de confluent

Plus de confluent (20)

Dernier

Dernier (20)

Migrating applications to serverless Apache Kafka + KSQL