SlideShare a Scribd company logo

Sweet Streams (Are made of this)

•Download as PPTX, PDF•
•
Corneil du Plessis
Corneil du Plessis

Spring Cloud Data Flow - a Gentle introductions

Software
1 of 37
Version 1.0 October 2022 Sweet Streams (Are made of this) Spring Cloud Data Flow - A gentle overview Copyright © 2022 VMware, Inc. or its affiliates.
Cover w/ Image Agenda ● What is Spring Cloud Data Flow? ● Spring Batch ● Spring Task ● Spring Cloud Functions ● Spring Cloud Stream Applications ● Interview with a User ● Q+A
What is Spring Cloud Data Flow? ● Microservice based Enterprise Application Integration. ○ Batch Jobs ○ Tasks ○ Streams
What is Spring Cloud Data Flow? ● Orchestration and Deployment ○ Tanzu Application Service / Cloud Foundry ○ Kubernetes ○ Local
What is Spring Cloud Data Flow? ● Observability ○ Micrometer ○ Prometheus ○ WaveFront ○ InfluxDB ○ More…
Stream DSL file | splitter | csv-to-json: transform > :input-stream http > :input-stream :input-stream > filter | aggregator > :output-stream :output-stream > jdbc :output-stream > json-to-ws-notification: transform | websocket Legend Stream App Topic App Label
Stream DSL
Stream DSL
Stream Definitions
Stream Definitions
Stream Definitions
Batch/Task Workflows ● DSL can define complex Batch and or Task application topologies in addition to a single Batch or Task application ● Each box is a Spring Batch or Task application. ● Application Flow can split and join, based on optional conditional expressions
How did we build Spring Cloud Data Flow? Spring Cloud Data Flow Spring Boot Spring Cloud Spring Batch Spring Integration Spring AMQP Spring Kafka Spring Security Spring Data
Spring Cloud Data Flow Architecture ● Data Flow UI, REST Client or Shell ● Data Flow Server ● Skipper Server ● Messaging Middleware
Observability Streams
Observability Tasks
Pre-packaged Stream Applications aggregator analytics bridge cassandra cdc-debezium dataflow-tasklauncher elasticsearch file filter ftp groovy header-enricher http http-request image- recognition jdbc jms load-generator log mail mongodb mqtt object-detection pgcopy rabbit redis router rsocket s3 script semantic-segmentation sftp splitter syslog tcp throughput time transform twitter-message twitter-search twitter-stream twitter-trend twitter-update wavefront websocket zeromq
Pre-packaged Stream Applications aggregator dataflow-tasklauncher filter http jdbc splitter transform your-application
Introducing: A user.
Data ingest Data comes in via file or web service Once in the DB the batch process can consume Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum
Batch process
Batch process
Technology
Technology
Design ● How do you eat an elephant? ● Enable more real-time processing ● Improve the feedback loop for failures ● Use Java ● Functional programming paradigm ● How do you eat an elephant?
Design Spring Cloud Data Flow
Stream processing System is built to be event driven Files are broken into individual events File and web service events are processed by streaming applications All streaming apps are custom. (Didn’t use any off the shelf applications) Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum
Stream processing Failures can be handled on individual events Streaming apps are designed to be idempotent Load can be spread out much better Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Error handler
Batch processing System is built to be event driven Files are broken into individual events Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Spring Batch
Idempotent processing ● At some point you must write to the DB (sink) ● Processes are designed to be idempotent ● The sink that persists is designed to cater for items being replayed
Developer experience ● Everything is a Spring Boot application (Our Devs love this) ● Changes can be tested locally ● Custom templates ● Dependency management (Solution BOM) ● Easy to understand applications that are small and do one thing ● Unit and integration testing is easy ● Containers Paving the road to production
Deploying Data Flow
Deployment Data Flow deployment ● Deployed to Kubernetes (using Bitnami Helm Charts) ● Message broker is Apache Kafka ● Customized to use non-OSS database ● JDK17 Stream and Batch applications ● GitOps ○ Applications are defined in git ○ Stream definition (DSL) and deployment info is in git ○ Deployments via Dataflow REST API
Management and Observability Monitoring ● Application metrics (uptime, resource utilization) ● Monitor topic lag ● Logs exported via sidecar application Scaling ● Make sure JVM properties are set (Heap memory, CPU count, GC, etc) ● Single instance of most apps ● No dynamic scaling yet
Questions?
© 2020 Spring. A VMware-backed project. Resources ● dataflow.spring.io ● github.com/spring-cloud/spring-cloud-dataflow ● dataflow.spring.io/docs/applications/pre-packaged/ ● via.vmw.com/sweet-streams
Thank you Contact me at pcorneil@vmware.com @corneil © 2020 Spring. A VMware-backed project.

Recommended

Dataflow.pptx
Dataflow.pptxDataflow.pptx
Dataflow.pptxSadeka Islam
 
Building cloud native data microservice
Building cloud native data microserviceBuilding cloud native data microservice
Building cloud native data microserviceNilanjan Roy
 
How Docker Accelerates Continuous Development at ironSource: Containers #101 ...
How Docker Accelerates Continuous Development at ironSource: Containers #101 ...How Docker Accelerates Continuous Development at ironSource: Containers #101 ...
How Docker Accelerates Continuous Development at ironSource: Containers #101 ...Brittany Ingram
 
AWS Lambda and Serverless framework: lessons learned while building a serverl...
AWS Lambda and Serverless framework: lessons learned while building a serverl...AWS Lambda and Serverless framework: lessons learned while building a serverl...
AWS Lambda and Serverless framework: lessons learned while building a serverl...Luciano Mammino
 
Apache Airflow in the Cloud: Programmatically orchestrating workloads with Py...
Apache Airflow in the Cloud: Programmatically orchestrating workloads with Py...Apache Airflow in the Cloud: Programmatically orchestrating workloads with Py...
Apache Airflow in the Cloud: Programmatically orchestrating workloads with Py...Kaxil Naik
 
My past-3 yeas-developer-journey-at-linkedin-by-iantsai
My past-3 yeas-developer-journey-at-linkedin-by-iantsaiMy past-3 yeas-developer-journey-at-linkedin-by-iantsai
My past-3 yeas-developer-journey-at-linkedin-by-iantsaiKim Kao
 
Delivering Cloud Native Batch Solutions - Dodd Pfeffer
Delivering Cloud Native Batch Solutions - Dodd PfefferDelivering Cloud Native Batch Solutions - Dodd Pfeffer
Delivering Cloud Native Batch Solutions - Dodd PfefferVMware Tanzu
 
Leapfrog into Serverless - a Deloitte-Amtrak Case Study | Serverless Confere...
Leapfrog into Serverless - a Deloitte-Amtrak Case Study | Serverless Confere...Leapfrog into Serverless - a Deloitte-Amtrak Case Study | Serverless Confere...
Leapfrog into Serverless - a Deloitte-Amtrak Case Study | Serverless Confere...Gary Arora
 

Recommended

Dataflow.pptx
Dataflow.pptxDataflow.pptx
Dataflow.pptxSadeka Islam
 
Building cloud native data microservice
Building cloud native data microserviceBuilding cloud native data microservice
Building cloud native data microserviceNilanjan Roy
 
How Docker Accelerates Continuous Development at ironSource: Containers #101 ...
How Docker Accelerates Continuous Development at ironSource: Containers #101 ...How Docker Accelerates Continuous Development at ironSource: Containers #101 ...
How Docker Accelerates Continuous Development at ironSource: Containers #101 ...Brittany Ingram
 
AWS Lambda and Serverless framework: lessons learned while building a serverl...
AWS Lambda and Serverless framework: lessons learned while building a serverl...AWS Lambda and Serverless framework: lessons learned while building a serverl...
AWS Lambda and Serverless framework: lessons learned while building a serverl...Luciano Mammino
 
Apache Airflow in the Cloud: Programmatically orchestrating workloads with Py...
Apache Airflow in the Cloud: Programmatically orchestrating workloads with Py...Apache Airflow in the Cloud: Programmatically orchestrating workloads with Py...
Apache Airflow in the Cloud: Programmatically orchestrating workloads with Py...Kaxil Naik
 
My past-3 yeas-developer-journey-at-linkedin-by-iantsai
My past-3 yeas-developer-journey-at-linkedin-by-iantsaiMy past-3 yeas-developer-journey-at-linkedin-by-iantsai
My past-3 yeas-developer-journey-at-linkedin-by-iantsaiKim Kao
 
Delivering Cloud Native Batch Solutions - Dodd Pfeffer
Delivering Cloud Native Batch Solutions - Dodd PfefferDelivering Cloud Native Batch Solutions - Dodd Pfeffer
Delivering Cloud Native Batch Solutions - Dodd PfefferVMware Tanzu
 
Leapfrog into Serverless - a Deloitte-Amtrak Case Study | Serverless Confere...
Leapfrog into Serverless - a Deloitte-Amtrak Case Study | Serverless Confere...Leapfrog into Serverless - a Deloitte-Amtrak Case Study | Serverless Confere...
Leapfrog into Serverless - a Deloitte-Amtrak Case Study | Serverless Confere...Gary Arora
 
Building Analytic Apps for SaaS: “Analytics as a Service”
Building Analytic Apps for SaaS: “Analytics as a Service”Building Analytic Apps for SaaS: “Analytics as a Service”
Building Analytic Apps for SaaS: “Analytics as a Service”Amazon Web Services
 
Game Analytics at London Apache Druid Meetup
Game Analytics at London Apache Druid MeetupGame Analytics at London Apache Druid Meetup
Game Analytics at London Apache Druid MeetupJelena Zanko
 
Story of migrating event pipeline from batch to streaming
Story of migrating event pipeline from batch to streamingStory of migrating event pipeline from batch to streaming
Story of migrating event pipeline from batch to streaminglohitvijayarenu
 
Web app job and functions - TUGAIT 2017
Web app job and functions - TUGAIT 2017Web app job and functions - TUGAIT 2017
Web app job and functions - TUGAIT 2017Steef-Jan Wiggers
 
Ultimate Guide to Microservice Architecture on Kubernetes
Ultimate Guide to Microservice Architecture on KubernetesUltimate Guide to Microservice Architecture on Kubernetes
Ultimate Guide to Microservice Architecture on Kuberneteskloia
 
Discover MongoDB Atlas and MongoDB Stitch - DEM02-S - Mexico City AWS Summit
Discover MongoDB Atlas and MongoDB Stitch - DEM02-S - Mexico City AWS SummitDiscover MongoDB Atlas and MongoDB Stitch - DEM02-S - Mexico City AWS Summit
Discover MongoDB Atlas and MongoDB Stitch - DEM02-S - Mexico City AWS SummitAmazon Web Services
 
Cloud-Native Patterns for Data-Intensive Applications
Cloud-Native Patterns for Data-Intensive ApplicationsCloud-Native Patterns for Data-Intensive Applications
Cloud-Native Patterns for Data-Intensive ApplicationsVMware Tanzu
 
Stream Processing – Concepts and Frameworks
Stream Processing – Concepts and FrameworksStream Processing – Concepts and Frameworks
Stream Processing – Concepts and FrameworksGuido Schmutz
 
Evolution of Real-time User Engagement Event Consumption at Pinterest
Evolution of Real-time User Engagement Event Consumption at PinterestEvolution of Real-time User Engagement Event Consumption at Pinterest
Evolution of Real-time User Engagement Event Consumption at PinterestHostedbyConfluent
 
[WSO2Con EU 2018] The Rise of Streaming SQL
[WSO2Con EU 2018] The Rise of Streaming SQL[WSO2Con EU 2018] The Rise of Streaming SQL
[WSO2Con EU 2018] The Rise of Streaming SQLWSO2
 
TenMax Data Pipeline Experience Sharing
TenMax Data Pipeline Experience SharingTenMax Data Pipeline Experience Sharing
TenMax Data Pipeline Experience SharingChen-en Lu
 
Deep dive into serverless on Google Cloud
Deep dive into serverless on Google CloudDeep dive into serverless on Google Cloud
Deep dive into serverless on Google CloudBret McGowen - NYC Google Developer Advocate
 
Supporting Digital Media Workflows in the Cloud with Perforce Helix
Supporting Digital Media Workflows in the Cloud with Perforce HelixSupporting Digital Media Workflows in the Cloud with Perforce Helix
Supporting Digital Media Workflows in the Cloud with Perforce HelixPerforce
 
MongoDB World 2018: Ch-Ch-Ch-Ch-Changes: Taking Your Stitch Application to th...
MongoDB World 2018: Ch-Ch-Ch-Ch-Changes: Taking Your Stitch Application to th...MongoDB World 2018: Ch-Ch-Ch-Ch-Changes: Taking Your Stitch Application to th...
MongoDB World 2018: Ch-Ch-Ch-Ch-Changes: Taking Your Stitch Application to th...MongoDB
 
6. DISZ - Webalkalmazások skálázhatósága a Google Cloud Platformon
6. DISZ - Webalkalmazások skálázhatósága a Google Cloud Platformon6. DISZ - Webalkalmazások skálázhatósága a Google Cloud Platformon
6. DISZ - Webalkalmazások skálázhatósága a Google Cloud PlatformonMárton Kodok
 
Cloud Experience: Data-driven Applications Made Simple and Fast
Cloud Experience: Data-driven Applications Made Simple and FastCloud Experience: Data-driven Applications Made Simple and Fast
Cloud Experience: Data-driven Applications Made Simple and FastDatabricks
 
Dynomite @ RedisConf 2017
Dynomite @ RedisConf 2017Dynomite @ RedisConf 2017
Dynomite @ RedisConf 2017Ioannis Papapanagiotou
 
Serverlessusecase workshop feb3_v2
Serverlessusecase workshop feb3_v2Serverlessusecase workshop feb3_v2
Serverlessusecase workshop feb3_v2kartraj
 
Challenges In Modern Application
Challenges In Modern ApplicationChallenges In Modern Application
Challenges In Modern ApplicationRahul Kumar Gupta
 
Webinar september 2013
Webinar september 2013Webinar september 2013
Webinar september 2013Marc Gille
 
Cloud Native Applications for Cloud Foundry using Spring Cloud : A Workshop
Cloud Native Applications for Cloud Foundry using Spring Cloud : A WorkshopCloud Native Applications for Cloud Foundry using Spring Cloud : A Workshop
Cloud Native Applications for Cloud Foundry using Spring Cloud : A WorkshopCorneil du Plessis
 
QueryDSL - Lightning Talk
QueryDSL - Lightning TalkQueryDSL - Lightning Talk
QueryDSL - Lightning TalkCorneil du Plessis
 

More Related Content

Similar to Sweet Streams (Are made of this)

Building Analytic Apps for SaaS: “Analytics as a Service”
Building Analytic Apps for SaaS: “Analytics as a Service”Building Analytic Apps for SaaS: “Analytics as a Service”
Building Analytic Apps for SaaS: “Analytics as a Service”Amazon Web Services
 
Game Analytics at London Apache Druid Meetup
Game Analytics at London Apache Druid MeetupGame Analytics at London Apache Druid Meetup
Game Analytics at London Apache Druid MeetupJelena Zanko
 
Story of migrating event pipeline from batch to streaming
Story of migrating event pipeline from batch to streamingStory of migrating event pipeline from batch to streaming
Story of migrating event pipeline from batch to streaminglohitvijayarenu
 
Web app job and functions - TUGAIT 2017
Web app job and functions - TUGAIT 2017Web app job and functions - TUGAIT 2017
Web app job and functions - TUGAIT 2017Steef-Jan Wiggers
 
Ultimate Guide to Microservice Architecture on Kubernetes
Ultimate Guide to Microservice Architecture on KubernetesUltimate Guide to Microservice Architecture on Kubernetes
Ultimate Guide to Microservice Architecture on Kuberneteskloia
 
Discover MongoDB Atlas and MongoDB Stitch - DEM02-S - Mexico City AWS Summit
Discover MongoDB Atlas and MongoDB Stitch - DEM02-S - Mexico City AWS SummitDiscover MongoDB Atlas and MongoDB Stitch - DEM02-S - Mexico City AWS Summit
Discover MongoDB Atlas and MongoDB Stitch - DEM02-S - Mexico City AWS SummitAmazon Web Services
 
Cloud-Native Patterns for Data-Intensive Applications
Cloud-Native Patterns for Data-Intensive ApplicationsCloud-Native Patterns for Data-Intensive Applications
Cloud-Native Patterns for Data-Intensive ApplicationsVMware Tanzu
 
Stream Processing – Concepts and Frameworks
Stream Processing – Concepts and FrameworksStream Processing – Concepts and Frameworks
Stream Processing – Concepts and FrameworksGuido Schmutz
 
Evolution of Real-time User Engagement Event Consumption at Pinterest
Evolution of Real-time User Engagement Event Consumption at PinterestEvolution of Real-time User Engagement Event Consumption at Pinterest
Evolution of Real-time User Engagement Event Consumption at PinterestHostedbyConfluent
 
[WSO2Con EU 2018] The Rise of Streaming SQL
[WSO2Con EU 2018] The Rise of Streaming SQL[WSO2Con EU 2018] The Rise of Streaming SQL
[WSO2Con EU 2018] The Rise of Streaming SQLWSO2
 
TenMax Data Pipeline Experience Sharing
TenMax Data Pipeline Experience SharingTenMax Data Pipeline Experience Sharing
TenMax Data Pipeline Experience SharingChen-en Lu
 
Supporting Digital Media Workflows in the Cloud with Perforce Helix
Supporting Digital Media Workflows in the Cloud with Perforce HelixSupporting Digital Media Workflows in the Cloud with Perforce Helix
Supporting Digital Media Workflows in the Cloud with Perforce HelixPerforce
 
MongoDB World 2018: Ch-Ch-Ch-Ch-Changes: Taking Your Stitch Application to th...
MongoDB World 2018: Ch-Ch-Ch-Ch-Changes: Taking Your Stitch Application to th...MongoDB World 2018: Ch-Ch-Ch-Ch-Changes: Taking Your Stitch Application to th...
MongoDB World 2018: Ch-Ch-Ch-Ch-Changes: Taking Your Stitch Application to th...MongoDB
 
6. DISZ - Webalkalmazások skálázhatósága a Google Cloud Platformon
6. DISZ - Webalkalmazások skálázhatósága a Google Cloud Platformon6. DISZ - Webalkalmazások skálázhatósága a Google Cloud Platformon
6. DISZ - Webalkalmazások skálázhatósága a Google Cloud PlatformonMárton Kodok
 
Cloud Experience: Data-driven Applications Made Simple and Fast
Cloud Experience: Data-driven Applications Made Simple and FastCloud Experience: Data-driven Applications Made Simple and Fast
Cloud Experience: Data-driven Applications Made Simple and FastDatabricks
 
Serverlessusecase workshop feb3_v2
Serverlessusecase workshop feb3_v2Serverlessusecase workshop feb3_v2
Serverlessusecase workshop feb3_v2kartraj
 
Challenges In Modern Application
Challenges In Modern ApplicationChallenges In Modern Application
Challenges In Modern ApplicationRahul Kumar Gupta
 
Webinar september 2013
Webinar september 2013Webinar september 2013
Webinar september 2013Marc Gille
 

Similar to Sweet Streams (Are made of this) (20)

Building Analytic Apps for SaaS: “Analytics as a Service”
Building Analytic Apps for SaaS: “Analytics as a Service”Building Analytic Apps for SaaS: “Analytics as a Service”
Building Analytic Apps for SaaS: “Analytics as a Service”
 
Game Analytics at London Apache Druid Meetup
Game Analytics at London Apache Druid MeetupGame Analytics at London Apache Druid Meetup
Game Analytics at London Apache Druid Meetup
 
Story of migrating event pipeline from batch to streaming
Story of migrating event pipeline from batch to streamingStory of migrating event pipeline from batch to streaming
Story of migrating event pipeline from batch to streaming
 
Web app job and functions - TUGAIT 2017
Web app job and functions - TUGAIT 2017Web app job and functions - TUGAIT 2017
Web app job and functions - TUGAIT 2017
 
Ultimate Guide to Microservice Architecture on Kubernetes
Ultimate Guide to Microservice Architecture on KubernetesUltimate Guide to Microservice Architecture on Kubernetes
Ultimate Guide to Microservice Architecture on Kubernetes
 
Discover MongoDB Atlas and MongoDB Stitch - DEM02-S - Mexico City AWS Summit
Discover MongoDB Atlas and MongoDB Stitch - DEM02-S - Mexico City AWS SummitDiscover MongoDB Atlas and MongoDB Stitch - DEM02-S - Mexico City AWS Summit
Discover MongoDB Atlas and MongoDB Stitch - DEM02-S - Mexico City AWS Summit
 
Cloud-Native Patterns for Data-Intensive Applications
Cloud-Native Patterns for Data-Intensive ApplicationsCloud-Native Patterns for Data-Intensive Applications
Cloud-Native Patterns for Data-Intensive Applications
 
Stream Processing – Concepts and Frameworks
Stream Processing – Concepts and FrameworksStream Processing – Concepts and Frameworks
Stream Processing – Concepts and Frameworks
 
Evolution of Real-time User Engagement Event Consumption at Pinterest
Evolution of Real-time User Engagement Event Consumption at PinterestEvolution of Real-time User Engagement Event Consumption at Pinterest
Evolution of Real-time User Engagement Event Consumption at Pinterest
 
[WSO2Con EU 2018] The Rise of Streaming SQL
[WSO2Con EU 2018] The Rise of Streaming SQL[WSO2Con EU 2018] The Rise of Streaming SQL
[WSO2Con EU 2018] The Rise of Streaming SQL
 
TenMax Data Pipeline Experience Sharing
TenMax Data Pipeline Experience SharingTenMax Data Pipeline Experience Sharing
TenMax Data Pipeline Experience Sharing
 
Deep dive into serverless on Google Cloud
Deep dive into serverless on Google CloudDeep dive into serverless on Google Cloud
Deep dive into serverless on Google Cloud
 
Supporting Digital Media Workflows in the Cloud with Perforce Helix
Supporting Digital Media Workflows in the Cloud with Perforce HelixSupporting Digital Media Workflows in the Cloud with Perforce Helix
Supporting Digital Media Workflows in the Cloud with Perforce Helix
 
MongoDB World 2018: Ch-Ch-Ch-Ch-Changes: Taking Your Stitch Application to th...
MongoDB World 2018: Ch-Ch-Ch-Ch-Changes: Taking Your Stitch Application to th...MongoDB World 2018: Ch-Ch-Ch-Ch-Changes: Taking Your Stitch Application to th...
MongoDB World 2018: Ch-Ch-Ch-Ch-Changes: Taking Your Stitch Application to th...
 
6. DISZ - Webalkalmazások skálázhatósága a Google Cloud Platformon
6. DISZ - Webalkalmazások skálázhatósága a Google Cloud Platformon6. DISZ - Webalkalmazások skálázhatósága a Google Cloud Platformon
6. DISZ - Webalkalmazások skálázhatósága a Google Cloud Platformon
 
Cloud Experience: Data-driven Applications Made Simple and Fast
Cloud Experience: Data-driven Applications Made Simple and FastCloud Experience: Data-driven Applications Made Simple and Fast
Cloud Experience: Data-driven Applications Made Simple and Fast
 
Dynomite @ RedisConf 2017
Dynomite @ RedisConf 2017Dynomite @ RedisConf 2017
Dynomite @ RedisConf 2017
 
Serverlessusecase workshop feb3_v2
Serverlessusecase workshop feb3_v2Serverlessusecase workshop feb3_v2
Serverlessusecase workshop feb3_v2
 
Challenges In Modern Application
Challenges In Modern ApplicationChallenges In Modern Application
Challenges In Modern Application
 
Webinar september 2013
Webinar september 2013Webinar september 2013
Webinar september 2013
 

More from Corneil du Plessis

Cloud Native Applications for Cloud Foundry using Spring Cloud : A Workshop
Cloud Native Applications for Cloud Foundry using Spring Cloud : A WorkshopCloud Native Applications for Cloud Foundry using Spring Cloud : A Workshop
Cloud Native Applications for Cloud Foundry using Spring Cloud : A WorkshopCorneil du Plessis
 
Enhancements in Java 9 Streams
Enhancements in Java 9 StreamsEnhancements in Java 9 Streams
Enhancements in Java 9 StreamsCorneil du Plessis
 
Performance Comparison JVM Languages
Performance Comparison JVM LanguagesPerformance Comparison JVM Languages
Performance Comparison JVM LanguagesCorneil du Plessis
 
Microservices Patterns and Anti-Patterns
Microservices Patterns and Anti-PatternsMicroservices Patterns and Anti-Patterns
Microservices Patterns and Anti-PatternsCorneil du Plessis
 
Consume Spring Data Rest with Angularjs
Consume Spring Data Rest with AngularjsConsume Spring Data Rest with Angularjs
Consume Spring Data Rest with AngularjsCorneil du Plessis
 
Gradle: The Build System you have been waiting for!
Gradle: The Build System you have been waiting for!Gradle: The Build System you have been waiting for!
Gradle: The Build System you have been waiting for!Corneil du Plessis
 
Polyglot persistence with Spring Data
Polyglot persistence with Spring DataPolyglot persistence with Spring Data
Polyglot persistence with Spring DataCorneil du Plessis
 
Gradle: The Build system you have been waiting for
Gradle: The Build system you have been waiting forGradle: The Build system you have been waiting for
Gradle: The Build system you have been waiting forCorneil du Plessis
 
Dependency Injection in Spring in 10min
Dependency Injection in Spring in 10minDependency Injection in Spring in 10min
Dependency Injection in Spring in 10minCorneil du Plessis
 

More from Corneil du Plessis (15)

Cloud Native Applications for Cloud Foundry using Spring Cloud : A Workshop
Cloud Native Applications for Cloud Foundry using Spring Cloud : A WorkshopCloud Native Applications for Cloud Foundry using Spring Cloud : A Workshop
Cloud Native Applications for Cloud Foundry using Spring Cloud : A Workshop
 
QueryDSL - Lightning Talk
QueryDSL - Lightning TalkQueryDSL - Lightning Talk
QueryDSL - Lightning Talk
 
Enhancements in Java 9 Streams
Enhancements in Java 9 StreamsEnhancements in Java 9 Streams
Enhancements in Java 9 Streams
 
Reactive Spring 5
Reactive Spring 5Reactive Spring 5
Reactive Spring 5
 
Empathic API-Design
Empathic API-DesignEmpathic API-Design
Empathic API-Design
 
Performance Comparison JVM Languages
Performance Comparison JVM LanguagesPerformance Comparison JVM Languages
Performance Comparison JVM Languages
 
Microservices Patterns and Anti-Patterns
Microservices Patterns and Anti-PatternsMicroservices Patterns and Anti-Patterns
Microservices Patterns and Anti-Patterns
 
Consume Spring Data Rest with Angularjs
Consume Spring Data Rest with AngularjsConsume Spring Data Rest with Angularjs
Consume Spring Data Rest with Angularjs
 
The Evolution of Java
The Evolution of JavaThe Evolution of Java
The Evolution of Java
 
Gradle: The Build System you have been waiting for!
Gradle: The Build System you have been waiting for!Gradle: The Build System you have been waiting for!
Gradle: The Build System you have been waiting for!
 
Polyglot persistence with Spring Data
Polyglot persistence with Spring DataPolyglot persistence with Spring Data
Polyglot persistence with Spring Data
 
Data repositories
Data repositoriesData repositories
Data repositories
 
Gradle: The Build system you have been waiting for
Gradle: The Build system you have been waiting forGradle: The Build system you have been waiting for
Gradle: The Build system you have been waiting for
 
Dependency Injection in Spring in 10min
Dependency Injection in Spring in 10minDependency Injection in Spring in 10min
Dependency Injection in Spring in 10min
 
Spring Data in 10 minutes
Spring Data in 10 minutesSpring Data in 10 minutes
Spring Data in 10 minutes
 

Recently uploaded

BusinessGPT - Security and Governance for Generative AI
BusinessGPT - Security and Governance for Generative AIBusinessGPT - Security and Governance for Generative AI
BusinessGPT - Security and Governance for Generative AIAGATSoftware
 
Navigation in flutter – how to add stack, tab, and drawer navigators to your ...
Navigation in flutter – how to add stack, tab, and drawer navigators to your ...Navigation in flutter – how to add stack, tab, and drawer navigators to your ...
Navigation in flutter – how to add stack, tab, and drawer navigators to your ...Flutter Agency
 
GraphSummit Milan & Stockholm - Neo4j: The Art of the Possible with Graph
GraphSummit Milan & Stockholm - Neo4j: The Art of the Possible with GraphGraphSummit Milan & Stockholm - Neo4j: The Art of the Possible with Graph
GraphSummit Milan & Stockholm - Neo4j: The Art of the Possible with GraphNeo4j
 
What is a Recruitment Management Software?
What is a Recruitment Management Software?What is a Recruitment Management Software?
What is a Recruitment Management Software?NYGGS Automation Suite
 
OpenChain Webinar: AboutCode and Beyond - End-to-End SCA
OpenChain Webinar: AboutCode and Beyond - End-to-End SCAOpenChain Webinar: AboutCode and Beyond - End-to-End SCA
OpenChain Webinar: AboutCode and Beyond - End-to-End SCAShane Coughlan
 
Jax, FL Admin Community Group 05.14.2024 Combined Deck
Jax, FL Admin Community Group 05.14.2024 Combined DeckJax, FL Admin Community Group 05.14.2024 Combined Deck
Jax, FL Admin Community Group 05.14.2024 Combined DeckMarc Lester
 
Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...
Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...
Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...Abortion Clinic
 
The Evolution of Web App Testing_ An Ultimate Guide to Future Trends.pdf
The Evolution of Web App Testing_ An Ultimate Guide to Future Trends.pdfThe Evolution of Web App Testing_ An Ultimate Guide to Future Trends.pdf
The Evolution of Web App Testing_ An Ultimate Guide to Future Trends.pdfkalichargn70th171
 
Encryption Recap: A Refresher on Key Concepts
Encryption Recap: A Refresher on Key ConceptsEncryption Recap: A Refresher on Key Concepts
Encryption Recap: A Refresher on Key Conceptsthomashtkim
 
The Strategic Impact of Buying vs Building in Test Automation
The Strategic Impact of Buying vs Building in Test AutomationThe Strategic Impact of Buying vs Building in Test Automation
The Strategic Impact of Buying vs Building in Test AutomationElement34
 
Effective Strategies for Wix's Scaling challenges - GeeCon
Effective Strategies for Wix's Scaling challenges - GeeConEffective Strategies for Wix's Scaling challenges - GeeCon
Effective Strategies for Wix's Scaling challenges - GeeConNatan Silnitsky
 
Lessons Learned from Building a Serverless Notifications System.pdf
Lessons Learned from Building a Serverless Notifications System.pdfLessons Learned from Building a Serverless Notifications System.pdf
Lessons Learned from Building a Serverless Notifications System.pdfSrushith Repakula
 
Automate your OpenSIPS config tests - OpenSIPS Summit 2024
Automate your OpenSIPS config tests - OpenSIPS Summit 2024Automate your OpenSIPS config tests - OpenSIPS Summit 2024
Automate your OpenSIPS config tests - OpenSIPS Summit 2024Andreas Granig
 
Community is Just as Important as Code by Andrea Goulet
Community is Just as Important as Code by Andrea GouletCommunity is Just as Important as Code by Andrea Goulet
Community is Just as Important as Code by Andrea GouletAndrea Goulet
 
Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...
Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...
Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...Andrea Goulet
 
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...Lisi Hocke
 
Optimizing Operations by Aligning Resources with Strategic Objectives Using O...
Optimizing Operations by Aligning Resources with Strategic Objectives Using O...Optimizing Operations by Aligning Resources with Strategic Objectives Using O...
Optimizing Operations by Aligning Resources with Strategic Objectives Using O...OnePlan Solutions
 
Alluxio Monthly Webinar | Simplify Data Access for AI in Multi-Cloud
Alluxio Monthly Webinar | Simplify Data Access for AI in Multi-CloudAlluxio Monthly Webinar | Simplify Data Access for AI in Multi-Cloud
Alluxio Monthly Webinar | Simplify Data Access for AI in Multi-CloudAlluxio, Inc.
 
Auto Affiliate AI Earns First Commission in 3 Hours..pdf
Auto Affiliate AI Earns First Commission in 3 Hours..pdfAuto Affiliate AI Earns First Commission in 3 Hours..pdf
Auto Affiliate AI Earns First Commission in 3 Hours..pdfSelfMade bd
 

Recently uploaded (20)

BusinessGPT - Security and Governance for Generative AI
BusinessGPT - Security and Governance for Generative AIBusinessGPT - Security and Governance for Generative AI
BusinessGPT - Security and Governance for Generative AI
 
Navigation in flutter – how to add stack, tab, and drawer navigators to your ...
Navigation in flutter – how to add stack, tab, and drawer navigators to your ...Navigation in flutter – how to add stack, tab, and drawer navigators to your ...
Navigation in flutter – how to add stack, tab, and drawer navigators to your ...
 
GraphSummit Milan & Stockholm - Neo4j: The Art of the Possible with Graph
GraphSummit Milan & Stockholm - Neo4j: The Art of the Possible with GraphGraphSummit Milan & Stockholm - Neo4j: The Art of the Possible with Graph
GraphSummit Milan & Stockholm - Neo4j: The Art of the Possible with Graph
 
What is a Recruitment Management Software?
What is a Recruitment Management Software?What is a Recruitment Management Software?
What is a Recruitment Management Software?
 
OpenChain Webinar: AboutCode and Beyond - End-to-End SCA
OpenChain Webinar: AboutCode and Beyond - End-to-End SCAOpenChain Webinar: AboutCode and Beyond - End-to-End SCA
OpenChain Webinar: AboutCode and Beyond - End-to-End SCA
 
Jax, FL Admin Community Group 05.14.2024 Combined Deck
Jax, FL Admin Community Group 05.14.2024 Combined DeckJax, FL Admin Community Group 05.14.2024 Combined Deck
Jax, FL Admin Community Group 05.14.2024 Combined Deck
 
Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...
Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...
Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...
 
The Evolution of Web App Testing_ An Ultimate Guide to Future Trends.pdf
The Evolution of Web App Testing_ An Ultimate Guide to Future Trends.pdfThe Evolution of Web App Testing_ An Ultimate Guide to Future Trends.pdf
The Evolution of Web App Testing_ An Ultimate Guide to Future Trends.pdf
 
Encryption Recap: A Refresher on Key Concepts
Encryption Recap: A Refresher on Key ConceptsEncryption Recap: A Refresher on Key Concepts
Encryption Recap: A Refresher on Key Concepts
 
The Strategic Impact of Buying vs Building in Test Automation
The Strategic Impact of Buying vs Building in Test AutomationThe Strategic Impact of Buying vs Building in Test Automation
The Strategic Impact of Buying vs Building in Test Automation
 
Effective Strategies for Wix's Scaling challenges - GeeCon
Effective Strategies for Wix's Scaling challenges - GeeConEffective Strategies for Wix's Scaling challenges - GeeCon
Effective Strategies for Wix's Scaling challenges - GeeCon
 
Lessons Learned from Building a Serverless Notifications System.pdf
Lessons Learned from Building a Serverless Notifications System.pdfLessons Learned from Building a Serverless Notifications System.pdf
Lessons Learned from Building a Serverless Notifications System.pdf
 
Automate your OpenSIPS config tests - OpenSIPS Summit 2024
Automate your OpenSIPS config tests - OpenSIPS Summit 2024Automate your OpenSIPS config tests - OpenSIPS Summit 2024
Automate your OpenSIPS config tests - OpenSIPS Summit 2024
 
Community is Just as Important as Code by Andrea Goulet
Community is Just as Important as Code by Andrea GouletCommunity is Just as Important as Code by Andrea Goulet
Community is Just as Important as Code by Andrea Goulet
 
Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...
Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...
Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...
 
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
 
Optimizing Operations by Aligning Resources with Strategic Objectives Using O...
Optimizing Operations by Aligning Resources with Strategic Objectives Using O...Optimizing Operations by Aligning Resources with Strategic Objectives Using O...
Optimizing Operations by Aligning Resources with Strategic Objectives Using O...
 
Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...
Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...
Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...
 
Alluxio Monthly Webinar | Simplify Data Access for AI in Multi-Cloud
Alluxio Monthly Webinar | Simplify Data Access for AI in Multi-CloudAlluxio Monthly Webinar | Simplify Data Access for AI in Multi-Cloud
Alluxio Monthly Webinar | Simplify Data Access for AI in Multi-Cloud
 
Auto Affiliate AI Earns First Commission in 3 Hours..pdf
Auto Affiliate AI Earns First Commission in 3 Hours..pdfAuto Affiliate AI Earns First Commission in 3 Hours..pdf
Auto Affiliate AI Earns First Commission in 3 Hours..pdf
 

Sweet Streams (Are made of this)

  • 1. Version 1.0 October 2022 Sweet Streams (Are made of this) Spring Cloud Data Flow - A gentle overview Copyright © 2022 VMware, Inc. or its affiliates.
  • 2. Cover w/ Image Agenda ● What is Spring Cloud Data Flow? ● Spring Batch ● Spring Task ● Spring Cloud Functions ● Spring Cloud Stream Applications ● Interview with a User ● Q+A
  • 3. What is Spring Cloud Data Flow? ● Microservice based Enterprise Application Integration. ○ Batch Jobs ○ Tasks ○ Streams
  • 4. What is Spring Cloud Data Flow? ● Orchestration and Deployment ○ Tanzu Application Service / Cloud Foundry ○ Kubernetes ○ Local
  • 5. What is Spring Cloud Data Flow? ● Observability ○ Micrometer ○ Prometheus ○ WaveFront ○ InfluxDB ○ More…
  • 6. Stream DSL file | splitter | csv-to-json: transform > :input-stream http > :input-stream :input-stream > filter | aggregator > :output-stream :output-stream > jdbc :output-stream > json-to-ws-notification: transform | websocket Legend Stream App Topic App Label
  • 12. Batch/Task Workflows ● DSL can define complex Batch and or Task application topologies in addition to a single Batch or Task application ● Each box is a Spring Batch or Task application. ● Application Flow can split and join, based on optional conditional expressions
  • 13. How did we build Spring Cloud Data Flow? Spring Cloud Data Flow Spring Boot Spring Cloud Spring Batch Spring Integration Spring AMQP Spring Kafka Spring Security Spring Data
  • 14. Spring Cloud Data Flow Architecture ● Data Flow UI, REST Client or Shell ● Data Flow Server ● Skipper Server ● Messaging Middleware
  • 17. Pre-packaged Stream Applications aggregator analytics bridge cassandra cdc-debezium dataflow-tasklauncher elasticsearch file filter ftp groovy header-enricher http http-request image- recognition jdbc jms load-generator log mail mongodb mqtt object-detection pgcopy rabbit redis router rsocket s3 script semantic-segmentation sftp splitter syslog tcp throughput time transform twitter-message twitter-search twitter-stream twitter-trend twitter-update wavefront websocket zeromq
  • 18. Pre-packaged Stream Applications aggregator dataflow-tasklauncher filter http jdbc splitter transform your-application
  • 20. Data ingest Data comes in via file or web service Once in the DB the batch process can consume Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum
  • 25. Design ● How do you eat an elephant? ● Enable more real-time processing ● Improve the feedback loop for failures ● Use Java ● Functional programming paradigm ● How do you eat an elephant?
  • 27. Stream processing System is built to be event driven Files are broken into individual events File and web service events are processed by streaming applications All streaming apps are custom. (Didn’t use any off the shelf applications) Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum
  • 28. Stream processing Failures can be handled on individual events Streaming apps are designed to be idempotent Load can be spread out much better Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Error handler
  • 29. Batch processing System is built to be event driven Files are broken into individual events Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Spring Batch
  • 30. Idempotent processing ● At some point you must write to the DB (sink) ● Processes are designed to be idempotent ● The sink that persists is designed to cater for items being replayed
  • 31. Developer experience ● Everything is a Spring Boot application (Our Devs love this) ● Changes can be tested locally ● Custom templates ● Dependency management (Solution BOM) ● Easy to understand applications that are small and do one thing ● Unit and integration testing is easy ● Containers Paving the road to production
  • 33. Deployment Data Flow deployment ● Deployed to Kubernetes (using Bitnami Helm Charts) ● Message broker is Apache Kafka ● Customized to use non-OSS database ● JDK17 Stream and Batch applications ● GitOps ○ Applications are defined in git ○ Stream definition (DSL) and deployment info is in git ○ Deployments via Dataflow REST API
  • 34. Management and Observability Monitoring ● Application metrics (uptime, resource utilization) ● Monitor topic lag ● Logs exported via sidecar application Scaling ● Make sure JVM properties are set (Heap memory, CPU count, GC, etc) ● Single instance of most apps ● No dynamic scaling yet
  • 36. © 2020 Spring. A VMware-backed project. Resources ● dataflow.spring.io ● github.com/spring-cloud/spring-cloud-dataflow ● dataflow.spring.io/docs/applications/pre-packaged/ ● via.vmw.com/sweet-streams
  • 37. Thank you Contact me at pcorneil@vmware.com @corneil © 2020 Spring. A VMware-backed project.

Editor's Notes

  1. Have you heard of Enterprise Integration Patterns? SCDF provides for the deployment and orchestration of batches, tasks and streams. These are deployed as microservice based Spring Boot Applications into Cloud Foundry / Tanzu Application Service, Kubernetes or as processes on a single machine.
  2. These are deployed as microservice based Spring Boot Applications into Cloud Foundry / Tanzu Application Service, Kubernetes or as processes on a single machine. The platform manages the availability and scalability
  3. Micrometer should be thought of as SLF4J for metrics.
  4. Spring Cloud Data Flow use Spring Boot, Spring Security, Spring Data and Spring Cloud Deployer modules.
  5. I would like to introduce you to one of the rarest sightings in Open source. Vincent Chegwidden works at Discovery using Spring Cloud Data Flow.
  6. Q: How did you get to use Spring Cloud Data Flow?
  7. Q: So I suppose you use or create Spring Cloud Stream applications…
  8. Q: Do you use any Spring Batch functionality?
  9. Q: Do any of the processors have side-effects?
  10. Q: What does the development cycle look like for your custom stream apps?
  11. Q: How do you deploy and manage SCDF and your solution?
  12. Future: Boot 3 support for Stream apps, Task and Batch apps and observability improvements. Data Flow interacting with multiple deployment environments. Environments may be connected…. Polyglot apps, Web Assembly, IoT, Edge.