SlideShare a Scribd company logo

Sweet Streams (Are made of this)

•Download as PPTX, PDF•
•
Corneil du Plessis
Corneil du Plessis

Spring Cloud Data Flow - a Gentle introductions

Software
1 of 37
Version 1.0 October 2022 Sweet Streams (Are made of this) Spring Cloud Data Flow - A gentle overview Copyright © 2022 VMware, Inc. or its affiliates.
Cover w/ Image Agenda ● What is Spring Cloud Data Flow? ● Spring Batch ● Spring Task ● Spring Cloud Functions ● Spring Cloud Stream Applications ● Interview with a User ● Q+A
What is Spring Cloud Data Flow? ● Microservice based Enterprise Application Integration. ○ Batch Jobs ○ Tasks ○ Streams
What is Spring Cloud Data Flow? ● Orchestration and Deployment ○ Tanzu Application Service / Cloud Foundry ○ Kubernetes ○ Local
What is Spring Cloud Data Flow? ● Observability ○ Micrometer ○ Prometheus ○ WaveFront ○ InfluxDB ○ More…
Stream DSL file | splitter | csv-to-json: transform > :input-stream http > :input-stream :input-stream > filter | aggregator > :output-stream :output-stream > jdbc :output-stream > json-to-ws-notification: transform | websocket Legend Stream App Topic App Label
Stream DSL
Stream DSL
Stream Definitions
Stream Definitions
Stream Definitions
Batch/Task Workflows ● DSL can define complex Batch and or Task application topologies in addition to a single Batch or Task application ● Each box is a Spring Batch or Task application. ● Application Flow can split and join, based on optional conditional expressions
How did we build Spring Cloud Data Flow? Spring Cloud Data Flow Spring Boot Spring Cloud Spring Batch Spring Integration Spring AMQP Spring Kafka Spring Security Spring Data
Spring Cloud Data Flow Architecture ● Data Flow UI, REST Client or Shell ● Data Flow Server ● Skipper Server ● Messaging Middleware
Observability Streams
Observability Tasks
Pre-packaged Stream Applications aggregator analytics bridge cassandra cdc-debezium dataflow-tasklauncher elasticsearch file filter ftp groovy header-enricher http http-request image- recognition jdbc jms load-generator log mail mongodb mqtt object-detection pgcopy rabbit redis router rsocket s3 script semantic-segmentation sftp splitter syslog tcp throughput time transform twitter-message twitter-search twitter-stream twitter-trend twitter-update wavefront websocket zeromq
Pre-packaged Stream Applications aggregator dataflow-tasklauncher filter http jdbc splitter transform your-application
Introducing: A user.
Data ingest Data comes in via file or web service Once in the DB the batch process can consume Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum
Batch process
Batch process
Technology
Technology
Design ● How do you eat an elephant? ● Enable more real-time processing ● Improve the feedback loop for failures ● Use Java ● Functional programming paradigm ● How do you eat an elephant?
Design Spring Cloud Data Flow
Stream processing System is built to be event driven Files are broken into individual events File and web service events are processed by streaming applications All streaming apps are custom. (Didn’t use any off the shelf applications) Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum
Stream processing Failures can be handled on individual events Streaming apps are designed to be idempotent Load can be spread out much better Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Error handler
Batch processing System is built to be event driven Files are broken into individual events Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Spring Batch
Idempotent processing ● At some point you must write to the DB (sink) ● Processes are designed to be idempotent ● The sink that persists is designed to cater for items being replayed
Developer experience ● Everything is a Spring Boot application (Our Devs love this) ● Changes can be tested locally ● Custom templates ● Dependency management (Solution BOM) ● Easy to understand applications that are small and do one thing ● Unit and integration testing is easy ● Containers Paving the road to production
Deploying Data Flow
Deployment Data Flow deployment ● Deployed to Kubernetes (using Bitnami Helm Charts) ● Message broker is Apache Kafka ● Customized to use non-OSS database ● JDK17 Stream and Batch applications ● GitOps ○ Applications are defined in git ○ Stream definition (DSL) and deployment info is in git ○ Deployments via Dataflow REST API
Management and Observability Monitoring ● Application metrics (uptime, resource utilization) ● Monitor topic lag ● Logs exported via sidecar application Scaling ● Make sure JVM properties are set (Heap memory, CPU count, GC, etc) ● Single instance of most apps ● No dynamic scaling yet
Questions?
© 2020 Spring. A VMware-backed project. Resources ● dataflow.spring.io ● github.com/spring-cloud/spring-cloud-dataflow ● dataflow.spring.io/docs/applications/pre-packaged/ ● via.vmw.com/sweet-streams
Thank you Contact me at pcorneil@vmware.com @corneil © 2020 Spring. A VMware-backed project.

Recommended

Dataflow.pptx
Dataflow.pptxDataflow.pptx
Dataflow.pptxSadeka Islam
 
Building cloud native data microservice
Building cloud native data microserviceBuilding cloud native data microservice
Building cloud native data microserviceNilanjan Roy
 
How Docker Accelerates Continuous Development at ironSource: Containers #101 ...
How Docker Accelerates Continuous Development at ironSource: Containers #101 ...How Docker Accelerates Continuous Development at ironSource: Containers #101 ...
How Docker Accelerates Continuous Development at ironSource: Containers #101 ...Brittany Ingram
 
AWS Lambda and Serverless framework: lessons learned while building a serverl...
AWS Lambda and Serverless framework: lessons learned while building a serverl...AWS Lambda and Serverless framework: lessons learned while building a serverl...
AWS Lambda and Serverless framework: lessons learned while building a serverl...Luciano Mammino
 
Apache Airflow in the Cloud: Programmatically orchestrating workloads with Py...
Apache Airflow in the Cloud: Programmatically orchestrating workloads with Py...Apache Airflow in the Cloud: Programmatically orchestrating workloads with Py...
Apache Airflow in the Cloud: Programmatically orchestrating workloads with Py...Kaxil Naik
 
My past-3 yeas-developer-journey-at-linkedin-by-iantsai
My past-3 yeas-developer-journey-at-linkedin-by-iantsaiMy past-3 yeas-developer-journey-at-linkedin-by-iantsai
My past-3 yeas-developer-journey-at-linkedin-by-iantsaiKim Kao
 
Delivering Cloud Native Batch Solutions - Dodd Pfeffer
Delivering Cloud Native Batch Solutions - Dodd PfefferDelivering Cloud Native Batch Solutions - Dodd Pfeffer
Delivering Cloud Native Batch Solutions - Dodd PfefferVMware Tanzu
 
Leapfrog into Serverless - a Deloitte-Amtrak Case Study | Serverless Confere...
Leapfrog into Serverless - a Deloitte-Amtrak Case Study | Serverless Confere...Leapfrog into Serverless - a Deloitte-Amtrak Case Study | Serverless Confere...
Leapfrog into Serverless - a Deloitte-Amtrak Case Study | Serverless Confere...Gary Arora
 

Recommended

Dataflow.pptx
Dataflow.pptxDataflow.pptx
Dataflow.pptxSadeka Islam
 
Building cloud native data microservice
Building cloud native data microserviceBuilding cloud native data microservice
Building cloud native data microserviceNilanjan Roy
 
How Docker Accelerates Continuous Development at ironSource: Containers #101 ...
How Docker Accelerates Continuous Development at ironSource: Containers #101 ...How Docker Accelerates Continuous Development at ironSource: Containers #101 ...
How Docker Accelerates Continuous Development at ironSource: Containers #101 ...Brittany Ingram
 
AWS Lambda and Serverless framework: lessons learned while building a serverl...
AWS Lambda and Serverless framework: lessons learned while building a serverl...AWS Lambda and Serverless framework: lessons learned while building a serverl...
AWS Lambda and Serverless framework: lessons learned while building a serverl...Luciano Mammino
 
Apache Airflow in the Cloud: Programmatically orchestrating workloads with Py...
Apache Airflow in the Cloud: Programmatically orchestrating workloads with Py...Apache Airflow in the Cloud: Programmatically orchestrating workloads with Py...
Apache Airflow in the Cloud: Programmatically orchestrating workloads with Py...Kaxil Naik
 
My past-3 yeas-developer-journey-at-linkedin-by-iantsai
My past-3 yeas-developer-journey-at-linkedin-by-iantsaiMy past-3 yeas-developer-journey-at-linkedin-by-iantsai
My past-3 yeas-developer-journey-at-linkedin-by-iantsaiKim Kao
 
Delivering Cloud Native Batch Solutions - Dodd Pfeffer
Delivering Cloud Native Batch Solutions - Dodd PfefferDelivering Cloud Native Batch Solutions - Dodd Pfeffer
Delivering Cloud Native Batch Solutions - Dodd PfefferVMware Tanzu
 
Leapfrog into Serverless - a Deloitte-Amtrak Case Study | Serverless Confere...
Leapfrog into Serverless - a Deloitte-Amtrak Case Study | Serverless Confere...Leapfrog into Serverless - a Deloitte-Amtrak Case Study | Serverless Confere...
Leapfrog into Serverless - a Deloitte-Amtrak Case Study | Serverless Confere...Gary Arora
 
Building Analytic Apps for SaaS: “Analytics as a Service”
Building Analytic Apps for SaaS: “Analytics as a Service”Building Analytic Apps for SaaS: “Analytics as a Service”
Building Analytic Apps for SaaS: “Analytics as a Service”Amazon Web Services
 
Game Analytics at London Apache Druid Meetup
Game Analytics at London Apache Druid MeetupGame Analytics at London Apache Druid Meetup
Game Analytics at London Apache Druid MeetupJelena Zanko
 
Story of migrating event pipeline from batch to streaming
Story of migrating event pipeline from batch to streamingStory of migrating event pipeline from batch to streaming
Story of migrating event pipeline from batch to streaminglohitvijayarenu
 
Web app job and functions - TUGAIT 2017
Web app job and functions - TUGAIT 2017Web app job and functions - TUGAIT 2017
Web app job and functions - TUGAIT 2017Steef-Jan Wiggers
 
Ultimate Guide to Microservice Architecture on Kubernetes
Ultimate Guide to Microservice Architecture on KubernetesUltimate Guide to Microservice Architecture on Kubernetes
Ultimate Guide to Microservice Architecture on Kuberneteskloia
 
Discover MongoDB Atlas and MongoDB Stitch - DEM02-S - Mexico City AWS Summit
Discover MongoDB Atlas and MongoDB Stitch - DEM02-S - Mexico City AWS SummitDiscover MongoDB Atlas and MongoDB Stitch - DEM02-S - Mexico City AWS Summit
Discover MongoDB Atlas and MongoDB Stitch - DEM02-S - Mexico City AWS SummitAmazon Web Services
 
Cloud-Native Patterns for Data-Intensive Applications
Cloud-Native Patterns for Data-Intensive ApplicationsCloud-Native Patterns for Data-Intensive Applications
Cloud-Native Patterns for Data-Intensive ApplicationsVMware Tanzu
 
Stream Processing – Concepts and Frameworks
Stream Processing – Concepts and FrameworksStream Processing – Concepts and Frameworks
Stream Processing – Concepts and FrameworksGuido Schmutz
 
Evolution of Real-time User Engagement Event Consumption at Pinterest
Evolution of Real-time User Engagement Event Consumption at PinterestEvolution of Real-time User Engagement Event Consumption at Pinterest
Evolution of Real-time User Engagement Event Consumption at PinterestHostedbyConfluent
 
[WSO2Con EU 2018] The Rise of Streaming SQL
[WSO2Con EU 2018] The Rise of Streaming SQL[WSO2Con EU 2018] The Rise of Streaming SQL
[WSO2Con EU 2018] The Rise of Streaming SQLWSO2
 
TenMax Data Pipeline Experience Sharing
TenMax Data Pipeline Experience SharingTenMax Data Pipeline Experience Sharing
TenMax Data Pipeline Experience SharingChen-en Lu
 
Deep dive into serverless on Google Cloud
Deep dive into serverless on Google CloudDeep dive into serverless on Google Cloud
Deep dive into serverless on Google CloudBret McGowen - NYC Google Developer Advocate
 
Supporting Digital Media Workflows in the Cloud with Perforce Helix
Supporting Digital Media Workflows in the Cloud with Perforce HelixSupporting Digital Media Workflows in the Cloud with Perforce Helix
Supporting Digital Media Workflows in the Cloud with Perforce HelixPerforce
 
MongoDB World 2018: Ch-Ch-Ch-Ch-Changes: Taking Your Stitch Application to th...
MongoDB World 2018: Ch-Ch-Ch-Ch-Changes: Taking Your Stitch Application to th...MongoDB World 2018: Ch-Ch-Ch-Ch-Changes: Taking Your Stitch Application to th...
MongoDB World 2018: Ch-Ch-Ch-Ch-Changes: Taking Your Stitch Application to th...MongoDB
 
6. DISZ - Webalkalmazások skálázhatósága a Google Cloud Platformon
6. DISZ - Webalkalmazások skálázhatósága a Google Cloud Platformon6. DISZ - Webalkalmazások skálázhatósága a Google Cloud Platformon
6. DISZ - Webalkalmazások skálázhatósága a Google Cloud PlatformonMárton Kodok
 
Cloud Experience: Data-driven Applications Made Simple and Fast
Cloud Experience: Data-driven Applications Made Simple and FastCloud Experience: Data-driven Applications Made Simple and Fast
Cloud Experience: Data-driven Applications Made Simple and FastDatabricks
 
Dynomite @ RedisConf 2017
Dynomite @ RedisConf 2017Dynomite @ RedisConf 2017
Dynomite @ RedisConf 2017Ioannis Papapanagiotou
 
Serverlessusecase workshop feb3_v2
Serverlessusecase workshop feb3_v2Serverlessusecase workshop feb3_v2
Serverlessusecase workshop feb3_v2kartraj
 
Challenges In Modern Application
Challenges In Modern ApplicationChallenges In Modern Application
Challenges In Modern ApplicationRahul Kumar Gupta
 
Webinar september 2013
Webinar september 2013Webinar september 2013
Webinar september 2013Marc Gille
 
Cloud Native Applications for Cloud Foundry using Spring Cloud : A Workshop
Cloud Native Applications for Cloud Foundry using Spring Cloud : A WorkshopCloud Native Applications for Cloud Foundry using Spring Cloud : A Workshop
Cloud Native Applications for Cloud Foundry using Spring Cloud : A WorkshopCorneil du Plessis
 
QueryDSL - Lightning Talk
QueryDSL - Lightning TalkQueryDSL - Lightning Talk
QueryDSL - Lightning TalkCorneil du Plessis
 

More Related Content

Similar to Sweet Streams (Are made of this)

Building Analytic Apps for SaaS: “Analytics as a Service”
Building Analytic Apps for SaaS: “Analytics as a Service”Building Analytic Apps for SaaS: “Analytics as a Service”
Building Analytic Apps for SaaS: “Analytics as a Service”Amazon Web Services
 
Game Analytics at London Apache Druid Meetup
Game Analytics at London Apache Druid MeetupGame Analytics at London Apache Druid Meetup
Game Analytics at London Apache Druid MeetupJelena Zanko
 
Story of migrating event pipeline from batch to streaming
Story of migrating event pipeline from batch to streamingStory of migrating event pipeline from batch to streaming
Story of migrating event pipeline from batch to streaminglohitvijayarenu
 
Web app job and functions - TUGAIT 2017
Web app job and functions - TUGAIT 2017Web app job and functions - TUGAIT 2017
Web app job and functions - TUGAIT 2017Steef-Jan Wiggers
 
Ultimate Guide to Microservice Architecture on Kubernetes
Ultimate Guide to Microservice Architecture on KubernetesUltimate Guide to Microservice Architecture on Kubernetes
Ultimate Guide to Microservice Architecture on Kuberneteskloia
 
Discover MongoDB Atlas and MongoDB Stitch - DEM02-S - Mexico City AWS Summit
Discover MongoDB Atlas and MongoDB Stitch - DEM02-S - Mexico City AWS SummitDiscover MongoDB Atlas and MongoDB Stitch - DEM02-S - Mexico City AWS Summit
Discover MongoDB Atlas and MongoDB Stitch - DEM02-S - Mexico City AWS SummitAmazon Web Services
 
Cloud-Native Patterns for Data-Intensive Applications
Cloud-Native Patterns for Data-Intensive ApplicationsCloud-Native Patterns for Data-Intensive Applications
Cloud-Native Patterns for Data-Intensive ApplicationsVMware Tanzu
 
Stream Processing – Concepts and Frameworks
Stream Processing – Concepts and FrameworksStream Processing – Concepts and Frameworks
Stream Processing – Concepts and FrameworksGuido Schmutz
 
Evolution of Real-time User Engagement Event Consumption at Pinterest
Evolution of Real-time User Engagement Event Consumption at PinterestEvolution of Real-time User Engagement Event Consumption at Pinterest
Evolution of Real-time User Engagement Event Consumption at PinterestHostedbyConfluent
 
[WSO2Con EU 2018] The Rise of Streaming SQL
[WSO2Con EU 2018] The Rise of Streaming SQL[WSO2Con EU 2018] The Rise of Streaming SQL
[WSO2Con EU 2018] The Rise of Streaming SQLWSO2
 
TenMax Data Pipeline Experience Sharing
TenMax Data Pipeline Experience SharingTenMax Data Pipeline Experience Sharing
TenMax Data Pipeline Experience SharingChen-en Lu
 
Supporting Digital Media Workflows in the Cloud with Perforce Helix
Supporting Digital Media Workflows in the Cloud with Perforce HelixSupporting Digital Media Workflows in the Cloud with Perforce Helix
Supporting Digital Media Workflows in the Cloud with Perforce HelixPerforce
 
MongoDB World 2018: Ch-Ch-Ch-Ch-Changes: Taking Your Stitch Application to th...
MongoDB World 2018: Ch-Ch-Ch-Ch-Changes: Taking Your Stitch Application to th...MongoDB World 2018: Ch-Ch-Ch-Ch-Changes: Taking Your Stitch Application to th...
MongoDB World 2018: Ch-Ch-Ch-Ch-Changes: Taking Your Stitch Application to th...MongoDB
 
6. DISZ - Webalkalmazások skálázhatósága a Google Cloud Platformon
6. DISZ - Webalkalmazások skálázhatósága a Google Cloud Platformon6. DISZ - Webalkalmazások skálázhatósága a Google Cloud Platformon
6. DISZ - Webalkalmazások skálázhatósága a Google Cloud PlatformonMárton Kodok
 
Cloud Experience: Data-driven Applications Made Simple and Fast
Cloud Experience: Data-driven Applications Made Simple and FastCloud Experience: Data-driven Applications Made Simple and Fast
Cloud Experience: Data-driven Applications Made Simple and FastDatabricks
 
Serverlessusecase workshop feb3_v2
Serverlessusecase workshop feb3_v2Serverlessusecase workshop feb3_v2
Serverlessusecase workshop feb3_v2kartraj
 
Challenges In Modern Application
Challenges In Modern ApplicationChallenges In Modern Application
Challenges In Modern ApplicationRahul Kumar Gupta
 
Webinar september 2013
Webinar september 2013Webinar september 2013
Webinar september 2013Marc Gille
 

Similar to Sweet Streams (Are made of this) (20)

Building Analytic Apps for SaaS: “Analytics as a Service”
Building Analytic Apps for SaaS: “Analytics as a Service”Building Analytic Apps for SaaS: “Analytics as a Service”
Building Analytic Apps for SaaS: “Analytics as a Service”
 
Game Analytics at London Apache Druid Meetup
Game Analytics at London Apache Druid MeetupGame Analytics at London Apache Druid Meetup
Game Analytics at London Apache Druid Meetup
 
Story of migrating event pipeline from batch to streaming
Story of migrating event pipeline from batch to streamingStory of migrating event pipeline from batch to streaming
Story of migrating event pipeline from batch to streaming
 
Web app job and functions - TUGAIT 2017
Web app job and functions - TUGAIT 2017Web app job and functions - TUGAIT 2017
Web app job and functions - TUGAIT 2017
 
Ultimate Guide to Microservice Architecture on Kubernetes
Ultimate Guide to Microservice Architecture on KubernetesUltimate Guide to Microservice Architecture on Kubernetes
Ultimate Guide to Microservice Architecture on Kubernetes
 
Discover MongoDB Atlas and MongoDB Stitch - DEM02-S - Mexico City AWS Summit
Discover MongoDB Atlas and MongoDB Stitch - DEM02-S - Mexico City AWS SummitDiscover MongoDB Atlas and MongoDB Stitch - DEM02-S - Mexico City AWS Summit
Discover MongoDB Atlas and MongoDB Stitch - DEM02-S - Mexico City AWS Summit
 
Cloud-Native Patterns for Data-Intensive Applications
Cloud-Native Patterns for Data-Intensive ApplicationsCloud-Native Patterns for Data-Intensive Applications
Cloud-Native Patterns for Data-Intensive Applications
 
Stream Processing – Concepts and Frameworks
Stream Processing – Concepts and FrameworksStream Processing – Concepts and Frameworks
Stream Processing – Concepts and Frameworks
 
Evolution of Real-time User Engagement Event Consumption at Pinterest
Evolution of Real-time User Engagement Event Consumption at PinterestEvolution of Real-time User Engagement Event Consumption at Pinterest
Evolution of Real-time User Engagement Event Consumption at Pinterest
 
[WSO2Con EU 2018] The Rise of Streaming SQL
[WSO2Con EU 2018] The Rise of Streaming SQL[WSO2Con EU 2018] The Rise of Streaming SQL
[WSO2Con EU 2018] The Rise of Streaming SQL
 
TenMax Data Pipeline Experience Sharing
TenMax Data Pipeline Experience SharingTenMax Data Pipeline Experience Sharing
TenMax Data Pipeline Experience Sharing
 
Deep dive into serverless on Google Cloud
Deep dive into serverless on Google CloudDeep dive into serverless on Google Cloud
Deep dive into serverless on Google Cloud
 
Supporting Digital Media Workflows in the Cloud with Perforce Helix
Supporting Digital Media Workflows in the Cloud with Perforce HelixSupporting Digital Media Workflows in the Cloud with Perforce Helix
Supporting Digital Media Workflows in the Cloud with Perforce Helix
 
MongoDB World 2018: Ch-Ch-Ch-Ch-Changes: Taking Your Stitch Application to th...
MongoDB World 2018: Ch-Ch-Ch-Ch-Changes: Taking Your Stitch Application to th...MongoDB World 2018: Ch-Ch-Ch-Ch-Changes: Taking Your Stitch Application to th...
MongoDB World 2018: Ch-Ch-Ch-Ch-Changes: Taking Your Stitch Application to th...
 
6. DISZ - Webalkalmazások skálázhatósága a Google Cloud Platformon
6. DISZ - Webalkalmazások skálázhatósága a Google Cloud Platformon6. DISZ - Webalkalmazások skálázhatósága a Google Cloud Platformon
6. DISZ - Webalkalmazások skálázhatósága a Google Cloud Platformon
 
Cloud Experience: Data-driven Applications Made Simple and Fast
Cloud Experience: Data-driven Applications Made Simple and FastCloud Experience: Data-driven Applications Made Simple and Fast
Cloud Experience: Data-driven Applications Made Simple and Fast
 
Dynomite @ RedisConf 2017
Dynomite @ RedisConf 2017Dynomite @ RedisConf 2017
Dynomite @ RedisConf 2017
 
Serverlessusecase workshop feb3_v2
Serverlessusecase workshop feb3_v2Serverlessusecase workshop feb3_v2
Serverlessusecase workshop feb3_v2
 
Challenges In Modern Application
Challenges In Modern ApplicationChallenges In Modern Application
Challenges In Modern Application
 
Webinar september 2013
Webinar september 2013Webinar september 2013
Webinar september 2013
 

More from Corneil du Plessis

Cloud Native Applications for Cloud Foundry using Spring Cloud : A Workshop
Cloud Native Applications for Cloud Foundry using Spring Cloud : A WorkshopCloud Native Applications for Cloud Foundry using Spring Cloud : A Workshop
Cloud Native Applications for Cloud Foundry using Spring Cloud : A WorkshopCorneil du Plessis
 
Enhancements in Java 9 Streams
Enhancements in Java 9 StreamsEnhancements in Java 9 Streams
Enhancements in Java 9 StreamsCorneil du Plessis
 
Performance Comparison JVM Languages
Performance Comparison JVM LanguagesPerformance Comparison JVM Languages
Performance Comparison JVM LanguagesCorneil du Plessis
 
Microservices Patterns and Anti-Patterns
Microservices Patterns and Anti-PatternsMicroservices Patterns and Anti-Patterns
Microservices Patterns and Anti-PatternsCorneil du Plessis
 
Consume Spring Data Rest with Angularjs
Consume Spring Data Rest with AngularjsConsume Spring Data Rest with Angularjs
Consume Spring Data Rest with AngularjsCorneil du Plessis
 
Gradle: The Build System you have been waiting for!
Gradle: The Build System you have been waiting for!Gradle: The Build System you have been waiting for!
Gradle: The Build System you have been waiting for!Corneil du Plessis
 
Polyglot persistence with Spring Data
Polyglot persistence with Spring DataPolyglot persistence with Spring Data
Polyglot persistence with Spring DataCorneil du Plessis
 
Gradle: The Build system you have been waiting for
Gradle: The Build system you have been waiting forGradle: The Build system you have been waiting for
Gradle: The Build system you have been waiting forCorneil du Plessis
 
Dependency Injection in Spring in 10min
Dependency Injection in Spring in 10minDependency Injection in Spring in 10min
Dependency Injection in Spring in 10minCorneil du Plessis
 

More from Corneil du Plessis (15)

Cloud Native Applications for Cloud Foundry using Spring Cloud : A Workshop
Cloud Native Applications for Cloud Foundry using Spring Cloud : A WorkshopCloud Native Applications for Cloud Foundry using Spring Cloud : A Workshop
Cloud Native Applications for Cloud Foundry using Spring Cloud : A Workshop
 
QueryDSL - Lightning Talk
QueryDSL - Lightning TalkQueryDSL - Lightning Talk
QueryDSL - Lightning Talk
 
Enhancements in Java 9 Streams
Enhancements in Java 9 StreamsEnhancements in Java 9 Streams
Enhancements in Java 9 Streams
 
Reactive Spring 5
Reactive Spring 5Reactive Spring 5
Reactive Spring 5
 
Empathic API-Design
Empathic API-DesignEmpathic API-Design
Empathic API-Design
 
Performance Comparison JVM Languages
Performance Comparison JVM LanguagesPerformance Comparison JVM Languages
Performance Comparison JVM Languages
 
Microservices Patterns and Anti-Patterns
Microservices Patterns and Anti-PatternsMicroservices Patterns and Anti-Patterns
Microservices Patterns and Anti-Patterns
 
Consume Spring Data Rest with Angularjs
Consume Spring Data Rest with AngularjsConsume Spring Data Rest with Angularjs
Consume Spring Data Rest with Angularjs
 
The Evolution of Java
The Evolution of JavaThe Evolution of Java
The Evolution of Java
 
Gradle: The Build System you have been waiting for!
Gradle: The Build System you have been waiting for!Gradle: The Build System you have been waiting for!
Gradle: The Build System you have been waiting for!
 
Polyglot persistence with Spring Data
Polyglot persistence with Spring DataPolyglot persistence with Spring Data
Polyglot persistence with Spring Data
 
Data repositories
Data repositoriesData repositories
Data repositories
 
Gradle: The Build system you have been waiting for
Gradle: The Build system you have been waiting forGradle: The Build system you have been waiting for
Gradle: The Build system you have been waiting for
 
Dependency Injection in Spring in 10min
Dependency Injection in Spring in 10minDependency Injection in Spring in 10min
Dependency Injection in Spring in 10min
 
Spring Data in 10 minutes
Spring Data in 10 minutesSpring Data in 10 minutes
Spring Data in 10 minutes
 

Recently uploaded

INGKA DIGITAL: Linked Metadata by Design
INGKA DIGITAL: Linked Metadata by DesignINGKA DIGITAL: Linked Metadata by Design
INGKA DIGITAL: Linked Metadata by DesignNeo4j
 
The Impact of PLM Software on Fashion Production
The Impact of PLM Software on Fashion ProductionThe Impact of PLM Software on Fashion Production
The Impact of PLM Software on Fashion ProductionWave PLM
 
Microsoft 365 Copilot; An AI tool changing the world of work _PDF.pdf
Microsoft 365 Copilot; An AI tool changing the world of work _PDF.pdfMicrosoft 365 Copilot; An AI tool changing the world of work _PDF.pdf
Microsoft 365 Copilot; An AI tool changing the world of work _PDF.pdfQ-Advise
 
IT Software Development Resume, Vaibhav jha 2024
IT Software Development Resume, Vaibhav jha 2024IT Software Development Resume, Vaibhav jha 2024
IT Software Development Resume, Vaibhav jha 2024vaibhav130304
 
A Guideline to Zendesk to Re:amaze Data Migration
A Guideline to Zendesk to Re:amaze Data MigrationA Guideline to Zendesk to Re:amaze Data Migration
A Guideline to Zendesk to Re:amaze Data MigrationHelp Desk Migration
 
A Comprehensive Appium Guide for Hybrid App Automation Testing.pdf
A Comprehensive Appium Guide for Hybrid App Automation Testing.pdfA Comprehensive Appium Guide for Hybrid App Automation Testing.pdf
A Comprehensive Appium Guide for Hybrid App Automation Testing.pdfkalichargn70th171
 
AI/ML Infra Meetup | ML explainability in Michelangelo
AI/ML Infra Meetup | ML explainability in MichelangeloAI/ML Infra Meetup | ML explainability in Michelangelo
AI/ML Infra Meetup | ML explainability in MichelangeloAlluxio, Inc.
 
AI/ML Infra Meetup | Perspective on Deep Learning Framework
AI/ML Infra Meetup | Perspective on Deep Learning FrameworkAI/ML Infra Meetup | Perspective on Deep Learning Framework
AI/ML Infra Meetup | Perspective on Deep Learning FrameworkAlluxio, Inc.
 
KLARNA - Language Models and Knowledge Graphs: A Systems Approach
KLARNA - Language Models and Knowledge Graphs: A Systems ApproachKLARNA - Language Models and Knowledge Graphs: A Systems Approach
KLARNA - Language Models and Knowledge Graphs: A Systems ApproachNeo4j
 
Crafting the Perfect Measurement Sheet with PLM Integration
Crafting the Perfect Measurement Sheet with PLM IntegrationCrafting the Perfect Measurement Sheet with PLM Integration
Crafting the Perfect Measurement Sheet with PLM IntegrationWave PLM
 
Workforce Efficiency with Employee Time Tracking Software.pdf
Workforce Efficiency with Employee Time Tracking Software.pdfWorkforce Efficiency with Employee Time Tracking Software.pdf
Workforce Efficiency with Employee Time Tracking Software.pdfDeskTrack
 
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdf
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdfImplementing KPIs and Right Metrics for Agile Delivery Teams.pdf
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdfVictor Lopez
 
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...naitiksharma1124
 
APVP,apvp apvp High quality supplier safe spot transport, 98% purity
APVP,apvp apvp High quality supplier safe spot transport, 98% purityAPVP,apvp apvp High quality supplier safe spot transport, 98% purity
APVP,apvp apvp High quality supplier safe spot transport, 98% purityamy56318795
 
GraphSummit Stockholm - Neo4j - Knowledge Graphs and Product Updates
GraphSummit Stockholm - Neo4j - Knowledge Graphs and Product UpdatesGraphSummit Stockholm - Neo4j - Knowledge Graphs and Product Updates
GraphSummit Stockholm - Neo4j - Knowledge Graphs and Product UpdatesNeo4j
 
how-to-download-files-safely-from-the-internet.pdf
how-to-download-files-safely-from-the-internet.pdfhow-to-download-files-safely-from-the-internet.pdf
how-to-download-files-safely-from-the-internet.pdfMehmet Akar
 
SQL Injection Introduction and Prevention
SQL Injection Introduction and PreventionSQL Injection Introduction and Prevention
SQL Injection Introduction and PreventionMohammed Fazuluddin
 
Mastering Windows 7 A Comprehensive Guide for Power Users .pdf
Mastering Windows 7 A Comprehensive Guide for Power Users .pdfMastering Windows 7 A Comprehensive Guide for Power Users .pdf
Mastering Windows 7 A Comprehensive Guide for Power Users .pdfmbmh111980
 
Agnieszka Andrzejewska - BIM School Course in Kraków
Agnieszka Andrzejewska - BIM School Course in KrakówAgnieszka Andrzejewska - BIM School Course in Kraków
Agnieszka Andrzejewska - BIM School Course in Krakówbim.edu.pl
 
AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAG
AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAGAI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAG
AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAGAlluxio, Inc.
 

Recently uploaded (20)

INGKA DIGITAL: Linked Metadata by Design
INGKA DIGITAL: Linked Metadata by DesignINGKA DIGITAL: Linked Metadata by Design
INGKA DIGITAL: Linked Metadata by Design
 
The Impact of PLM Software on Fashion Production
The Impact of PLM Software on Fashion ProductionThe Impact of PLM Software on Fashion Production
The Impact of PLM Software on Fashion Production
 
Microsoft 365 Copilot; An AI tool changing the world of work _PDF.pdf
Microsoft 365 Copilot; An AI tool changing the world of work _PDF.pdfMicrosoft 365 Copilot; An AI tool changing the world of work _PDF.pdf
Microsoft 365 Copilot; An AI tool changing the world of work _PDF.pdf
 
IT Software Development Resume, Vaibhav jha 2024
IT Software Development Resume, Vaibhav jha 2024IT Software Development Resume, Vaibhav jha 2024
IT Software Development Resume, Vaibhav jha 2024
 
A Guideline to Zendesk to Re:amaze Data Migration
A Guideline to Zendesk to Re:amaze Data MigrationA Guideline to Zendesk to Re:amaze Data Migration
A Guideline to Zendesk to Re:amaze Data Migration
 
A Comprehensive Appium Guide for Hybrid App Automation Testing.pdf
A Comprehensive Appium Guide for Hybrid App Automation Testing.pdfA Comprehensive Appium Guide for Hybrid App Automation Testing.pdf
A Comprehensive Appium Guide for Hybrid App Automation Testing.pdf
 
AI/ML Infra Meetup | ML explainability in Michelangelo
AI/ML Infra Meetup | ML explainability in MichelangeloAI/ML Infra Meetup | ML explainability in Michelangelo
AI/ML Infra Meetup | ML explainability in Michelangelo
 
AI/ML Infra Meetup | Perspective on Deep Learning Framework
AI/ML Infra Meetup | Perspective on Deep Learning FrameworkAI/ML Infra Meetup | Perspective on Deep Learning Framework
AI/ML Infra Meetup | Perspective on Deep Learning Framework
 
KLARNA - Language Models and Knowledge Graphs: A Systems Approach
KLARNA - Language Models and Knowledge Graphs: A Systems ApproachKLARNA - Language Models and Knowledge Graphs: A Systems Approach
KLARNA - Language Models and Knowledge Graphs: A Systems Approach
 
Crafting the Perfect Measurement Sheet with PLM Integration
Crafting the Perfect Measurement Sheet with PLM IntegrationCrafting the Perfect Measurement Sheet with PLM Integration
Crafting the Perfect Measurement Sheet with PLM Integration
 
Workforce Efficiency with Employee Time Tracking Software.pdf
Workforce Efficiency with Employee Time Tracking Software.pdfWorkforce Efficiency with Employee Time Tracking Software.pdf
Workforce Efficiency with Employee Time Tracking Software.pdf
 
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdf
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdfImplementing KPIs and Right Metrics for Agile Delivery Teams.pdf
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdf
 
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...
 
APVP,apvp apvp High quality supplier safe spot transport, 98% purity
APVP,apvp apvp High quality supplier safe spot transport, 98% purityAPVP,apvp apvp High quality supplier safe spot transport, 98% purity
APVP,apvp apvp High quality supplier safe spot transport, 98% purity
 
GraphSummit Stockholm - Neo4j - Knowledge Graphs and Product Updates
GraphSummit Stockholm - Neo4j - Knowledge Graphs and Product UpdatesGraphSummit Stockholm - Neo4j - Knowledge Graphs and Product Updates
GraphSummit Stockholm - Neo4j - Knowledge Graphs and Product Updates
 
how-to-download-files-safely-from-the-internet.pdf
how-to-download-files-safely-from-the-internet.pdfhow-to-download-files-safely-from-the-internet.pdf
how-to-download-files-safely-from-the-internet.pdf
 
SQL Injection Introduction and Prevention
SQL Injection Introduction and PreventionSQL Injection Introduction and Prevention
SQL Injection Introduction and Prevention
 
Mastering Windows 7 A Comprehensive Guide for Power Users .pdf
Mastering Windows 7 A Comprehensive Guide for Power Users .pdfMastering Windows 7 A Comprehensive Guide for Power Users .pdf
Mastering Windows 7 A Comprehensive Guide for Power Users .pdf
 
Agnieszka Andrzejewska - BIM School Course in Kraków
Agnieszka Andrzejewska - BIM School Course in KrakówAgnieszka Andrzejewska - BIM School Course in Kraków
Agnieszka Andrzejewska - BIM School Course in Kraków
 
AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAG
AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAGAI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAG
AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAG
 

Sweet Streams (Are made of this)

  • 1. Version 1.0 October 2022 Sweet Streams (Are made of this) Spring Cloud Data Flow - A gentle overview Copyright © 2022 VMware, Inc. or its affiliates.
  • 2. Cover w/ Image Agenda ● What is Spring Cloud Data Flow? ● Spring Batch ● Spring Task ● Spring Cloud Functions ● Spring Cloud Stream Applications ● Interview with a User ● Q+A
  • 3. What is Spring Cloud Data Flow? ● Microservice based Enterprise Application Integration. ○ Batch Jobs ○ Tasks ○ Streams
  • 4. What is Spring Cloud Data Flow? ● Orchestration and Deployment ○ Tanzu Application Service / Cloud Foundry ○ Kubernetes ○ Local
  • 5. What is Spring Cloud Data Flow? ● Observability ○ Micrometer ○ Prometheus ○ WaveFront ○ InfluxDB ○ More…
  • 6. Stream DSL file | splitter | csv-to-json: transform > :input-stream http > :input-stream :input-stream > filter | aggregator > :output-stream :output-stream > jdbc :output-stream > json-to-ws-notification: transform | websocket Legend Stream App Topic App Label
  • 12. Batch/Task Workflows ● DSL can define complex Batch and or Task application topologies in addition to a single Batch or Task application ● Each box is a Spring Batch or Task application. ● Application Flow can split and join, based on optional conditional expressions
  • 13. How did we build Spring Cloud Data Flow? Spring Cloud Data Flow Spring Boot Spring Cloud Spring Batch Spring Integration Spring AMQP Spring Kafka Spring Security Spring Data
  • 14. Spring Cloud Data Flow Architecture ● Data Flow UI, REST Client or Shell ● Data Flow Server ● Skipper Server ● Messaging Middleware
  • 17. Pre-packaged Stream Applications aggregator analytics bridge cassandra cdc-debezium dataflow-tasklauncher elasticsearch file filter ftp groovy header-enricher http http-request image- recognition jdbc jms load-generator log mail mongodb mqtt object-detection pgcopy rabbit redis router rsocket s3 script semantic-segmentation sftp splitter syslog tcp throughput time transform twitter-message twitter-search twitter-stream twitter-trend twitter-update wavefront websocket zeromq
  • 18. Pre-packaged Stream Applications aggregator dataflow-tasklauncher filter http jdbc splitter transform your-application
  • 20. Data ingest Data comes in via file or web service Once in the DB the batch process can consume Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum
  • 25. Design ● How do you eat an elephant? ● Enable more real-time processing ● Improve the feedback loop for failures ● Use Java ● Functional programming paradigm ● How do you eat an elephant?
  • 27. Stream processing System is built to be event driven Files are broken into individual events File and web service events are processed by streaming applications All streaming apps are custom. (Didn’t use any off the shelf applications) Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum
  • 28. Stream processing Failures can be handled on individual events Streaming apps are designed to be idempotent Load can be spread out much better Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Error handler
  • 29. Batch processing System is built to be event driven Files are broken into individual events Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Spring Batch
  • 30. Idempotent processing ● At some point you must write to the DB (sink) ● Processes are designed to be idempotent ● The sink that persists is designed to cater for items being replayed
  • 31. Developer experience ● Everything is a Spring Boot application (Our Devs love this) ● Changes can be tested locally ● Custom templates ● Dependency management (Solution BOM) ● Easy to understand applications that are small and do one thing ● Unit and integration testing is easy ● Containers Paving the road to production
  • 33. Deployment Data Flow deployment ● Deployed to Kubernetes (using Bitnami Helm Charts) ● Message broker is Apache Kafka ● Customized to use non-OSS database ● JDK17 Stream and Batch applications ● GitOps ○ Applications are defined in git ○ Stream definition (DSL) and deployment info is in git ○ Deployments via Dataflow REST API
  • 34. Management and Observability Monitoring ● Application metrics (uptime, resource utilization) ● Monitor topic lag ● Logs exported via sidecar application Scaling ● Make sure JVM properties are set (Heap memory, CPU count, GC, etc) ● Single instance of most apps ● No dynamic scaling yet
  • 36. © 2020 Spring. A VMware-backed project. Resources ● dataflow.spring.io ● github.com/spring-cloud/spring-cloud-dataflow ● dataflow.spring.io/docs/applications/pre-packaged/ ● via.vmw.com/sweet-streams
  • 37. Thank you Contact me at pcorneil@vmware.com @corneil © 2020 Spring. A VMware-backed project.

Editor's Notes

  1. Have you heard of Enterprise Integration Patterns? SCDF provides for the deployment and orchestration of batches, tasks and streams. These are deployed as microservice based Spring Boot Applications into Cloud Foundry / Tanzu Application Service, Kubernetes or as processes on a single machine.
  2. These are deployed as microservice based Spring Boot Applications into Cloud Foundry / Tanzu Application Service, Kubernetes or as processes on a single machine. The platform manages the availability and scalability
  3. Micrometer should be thought of as SLF4J for metrics.
  4. Spring Cloud Data Flow use Spring Boot, Spring Security, Spring Data and Spring Cloud Deployer modules.
  5. I would like to introduce you to one of the rarest sightings in Open source. Vincent Chegwidden works at Discovery using Spring Cloud Data Flow.
  6. Q: How did you get to use Spring Cloud Data Flow?
  7. Q: So I suppose you use or create Spring Cloud Stream applications…
  8. Q: Do you use any Spring Batch functionality?
  9. Q: Do any of the processors have side-effects?
  10. Q: What does the development cycle look like for your custom stream apps?
  11. Q: How do you deploy and manage SCDF and your solution?
  12. Future: Boot 3 support for Stream apps, Task and Batch apps and observability improvements. Data Flow interacting with multiple deployment environments. Environments may be connected…. Polyglot apps, Web Assembly, IoT, Edge.