Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Delivering Cloud Native Batch Solutions - Dodd Pfeffer

128 vues

Publié le

Delivering Cloud Native Batch Solutions with Dodd Pfeffer at SpringOne Tour 2019

Publié dans : Logiciels
  • Soyez le premier à commenter

  • Soyez le premier à aimer ceci

Delivering Cloud Native Batch Solutions - Dodd Pfeffer

  1. 1. © Copyright 2019 Pivotal Software, Inc. All rights Reserved. Dodd Pfeffer Pivotal Platform Architect Cloud Native Batch
  2. 2. Use Cases 3
  3. 3. Modernize monolithic ETL workloads SQL scripts, stored procedures, and in-house bash scripts to cloud- native architecture. Small and incremental releases. Continuous delivery is the focus. 4
  4. 4. Scheduled batch-jobs Whether it is for predictive model training, massive file movement, or the classic data migration batch-jobs, they are typically schedule driven. 5
  5. 5. Dragonstone Finance ● Good DDD design approaches ● Internal ratings service ○ Influenced by a series of analyst evaluations provided via file integration from various sources ● Internal auditing function ○ Accepts trade data via file integration from various sources ○ Provides feedback to traders and trading system on healthiness of trades
  6. 6. What would a cloud native batch solution look like ● Batch apps run on ephemeral containers whose capacity is only consumed during batch execution ● Polling for new files delivered by data microservice pipelines via ingest streams ● Scheduler provided by the platform ● Batch app properties externally managed through config server ● Batch app external service dependency facilitated through service registry ● Cloud platform dynamically binds persistent volume-service to container during batch execution ● Database connection information dynamically injected into batch app environment ● Databases provisioned and managed by via cloud platform ● Batch app logs aggregated and stored on the platform, optional streamed to external log source ● Access to manage and monitor the batch applications consistent with other microservices on the platform
  7. 7. Dragonstone Finance ● Spring Batch ● Spring Cloud Task ● Spring Cloud Data Flow ● Spring Cloud Config Server ● Spring Cloud Service Registry ● Pivotal Application Services ● Volume Services for PCF ● MySql for PCF
  8. 8. File Ingest Reference Architecture https://content.pivotal.io/blog/need-24x7-etl-then-move-to-cloud-native-file-ingest-with-spring-cloud-data-flow
  9. 9. Spring Batch ● Lightweight framework ● Familiar spring interfaces ● Transaction management ● Chunk based processing ● Declarative I/O ● Start/Stop/Restart ● Retry/Skip https://docs.spring.io/spring-batch/4.1.x/reference/html/index-single.html
  10. 10. Spring Cloud Stream ● Stream is composed of long lived Boot Microservice apps ● Event driven based Microservices ● Loose coupling via pub/sub messaging ● Opinionated Middleware Primitives ● For integration and stream processing use-cases ● Each microservice app can be packaged as an uber-jar or as a Docker container ● Out-of-the-box apps: https://cloud.spring.io/spring-cloud-stream-app-starters/
  11. 11. Spring Cloud Task ● Tasks are short lived Boot Microservice apps ● System tracks invocations, exit-status ● Spring Batch Jobs are wrapped as Tasks ● Useful for ETL between databases and filesystems ● Each microservice app can be packaged as an uber-jar or as a Docker container ● Out-of-the-box apps: https://cloud.spring.io/spring-cloud-task-app-starters/
  12. 12. Spring Cloud Stream ● Spring Boot Application that stays on listening for a data source. ● Provides abstraction to a message broker implementation. Stream Versus Task Spring Cloud Task ● Spring Boot application that terminates its process when it has completed its task. ● Implements a CommandLineRunner bean ● May publish messages using Spring Cloud Stream. ● Maintains its own task repository to track task analytics
  13. 13. What is Spring Cloud Data Flow? Spring Cloud Data Flow is a Microservices toolkit for building data integration and real-time data processing pipelines. Pipelines consist of Spring Boot apps, using Spring Cloud Stream for events or Spring Cloud Task for batch processes. The Data Flow server provides interfaces to compose and deploy pipelines onto a modern platform like Cloud Foundry.
  14. 14. Cover w/ Image Spring Cloud Services for Pivotal Cloud Foundry ■ Pivotal has bundled a number of these OSS projects into Spring Cloud Services for PCF Service Registry Config Server Circuit Breaker Dashboard
  15. 15. Increase speed and deploy code to production thousands of times per month. Use PAS to run Java, .NET, and Node apps. Pivotal Application Service (PAS): A Runtime for Apps Best runtime for Spring and Spring Boot — Spring’s microservice patterns—and Spring Boot’s executable jars—are ready-made for PAS. Turnkey microservices operations and security — Spring Cloud Services brings microservices best practices to PAS. It includes Config Server, Service Registry, and Circuit Breaker Dashboard. A native Windows and .NET experience — Use PAS to run new apps built with .NET Core. Run your legacy .NET Framework apps on PAS too, using the .NET Hosted Web Core buildpack. Push applications to containers running on Windows Server 2016. Built for apps — PAS has everything to need to run apps. Buildpacks manage runtime dependencies; metrics, logging, and scaling are done for you. Multitenancy, and blue/green deployment patterns are built-in. Extend apps with a rich service catalog. Container-ready — PAS supports the OCI format for Docker images. Run platform-built and developer-built containers.
  16. 16. Let’s take a look
  17. 17. References ● 24x7 ETL / File Ingest Reference Architecture: https://content.pivotal.io/blog/need-24x7-etl-then- move-to-cloud-native-file-ingest-with-spring-cloud-data-flow ● Spring Batch Docs: https://docs.spring.io/spring-batch/4.1.x/reference/html/index-single.html ● Spring Cloud Data Flow Docs: https://docs.spring.io/spring-cloud- dataflow/docs/current/reference/htmlsingle/ ● Volume Services: https://docs.pivotal.io/pivotalcf/2-5/devguide/services/using-vol- services.html#create-nfs ● Demo Source Repo: https://github.com/doddatpivotal/dragonstone-finance
  18. 18. Register Today Early Bird ends 7/8 Use Discount Code S1P_Save200 October 7–10, 2019 Austin Convention Center and Save!
  19. 19. Transforming How The World Builds Software © Copyright 2017 Pivotal Software, Inc. All rights Reserved.Revised 1/25/19