Scheduling Apps in the Cloud with Glenn Renfro & Roy Clarkson at SpringOne Tour 2019

1. © Copyright 2018 Pivotal Software, Inc. All rights Reserved.
Roy Clarkson
@RoyClarkson
Glenn Renfro
@cppwfs
Scheduling Apps In The
Cloud

2. First Let’s Get This Out
Of Our System

3. Because Scheduling is
Exciting stuff...

4. But Seriously
What does Scheduling Look
Like In The ?

5. Cover w/ Image
Agenda
■ What does scheduling mean?
■ Ways to Schedule
■ Cloud support scheduling
■ Spring Cloud Task
■ Scheduling Simplified

6. Vocabulary… Woot! Woot!
Task - Is a short-lived microservice. Also known as a
ephemeral microservice.
It is also a 12 factor app: https://12factor.net/

7. ● Launching the task
● Tracking the task execution
● Obtaining its final state
● Managing failure conditions
What Does
Scheduling
Mean?

8. App that executes tasks based on
calendar events
Launching Tasks based on Event
Ways to Schedule

9. Cloud Scheduling Cloud Foundry - PCF Scheduler
Kubernetes Cron Jobs

10. Cloud Scheduling
Things that they both have in
common

11. Cloud Foundry - PCF Scheduler
Kubernetes - Cron Jobs
Cloud Scheduling ● Both CF and Kubernetes have a
concept of ephemeral apps.
○ Jobs
○ Tasks

12. Cloud Foundry - PCF Scheduler
Kubernetes - Cron Jobs
Cloud Scheduling
● A schedule name was applied to an
application name
● An application was specified (one
pushed the other pulled from a
repository)
● An ability to pass in command line
args and environment
variables(properties)
● and...

13. Cloud Foundry - PCF Scheduler
Kubernetes - Cron Jobs
Cloud Scheduling
My Gosh! It’s full of CRON!

14. Cloud Foundry - PCF Scheduler
Kubernetes - Cron Jobs
Cloud Scheduling
● Tasks can be scheduled via :
○ RESTFul API
○ Command Line Interface

15. Cloud Foundry - PCF Scheduler
Cloud Scheduling
● PCF Scheduler Does offer a UI
Suspicious Cat knew
there would be a
difference!

16. PCF - Scheduler (basic steps)
● Push app to Cloud Foundry
● Bind the services you need to the application
● Create your Job
● Create your schedule

17. Cloud Foundry - PCF Scheduler
cf push test-app -p test-app.jar
cf bind-service test-app mydatabase
cf create-job test-app my-schedule-job
"JAVA_OPTS="..."
cf schedule-job my-schedule-job "0 12 ? * *"

18. Kubernetes - CronJobs (basic steps)
● Push your docker image up to repository
● Create your specification
○ Create your .spec file to include
■ Schedule spec
■ Job Template Spec
○ -or-
○ Used kubectl run to create a schedule

19. Kubernetes - Cron Jobs via Yaml
kind: CronJob
metadata:
name: hello
spec:
schedule: "*/1 * * * *"
jobTemplate:
spec:
template:
spec:
containers:
- name: hello
image: springcloud/spring-cloud-scheduler-spi-test-app
args:
- spring.application.name=hello
restartPolicy: OnFailure
kubectl create -f ./cronjob.yaml

20. Kubernetes - Cron Jobs
kubectl run mysched --schedule="*/5 * * * *"
--restart=Never
--image=springcloud/spring-cloud-scheduler-spi-test-app

21. ● Launching the task
● Tracking the task execution
● Obtaining its final state
● Managing failure conditions
What Does
Scheduling
Mean?

22. ● Tracking the task execution
● Obtaining its final state
● Managing failure conditions
Spring Cloud Task
&
Spring Batch

23. So let’s simplify things a
little with
Spring Cloud Data Flow
23

24. A toolkit for building
data integration,
real-time streaming,
and batch data
processing pipelines.
WHAT IS SPRING CLOUD DATA FLOW?
24

25. Data pipelines consist of Spring Boot apps, using
Spring Cloud Stream for event-streaming or
Spring Cloud Task for batch processes.
Ready for Data Integration with >60
out-of-the-box streaming and batch Apps.
DSL, GUI, and REST-APIs to build and
orchestrate data pipelines onto platforms like
Kubernetes and Cloud Foundry.
Continuous delivery for streaming data pipelines
using Spring Cloud Skipper.
Cron-job scheduler for batch data pipelines using
Spring Cloud Scheduler.
A toolkit for building
data integration,
real-time streaming,
and batch data
processing pipelines.
WHAT IS SPRING CLOUD DATA FLOW?
25

26. Data pipelines consist of Spring Boot apps, using
Spring Cloud Stream for event-streaming or
Spring Cloud Task for batch processes.
Ready for Data Integration with >60
out-of-the-box streaming and batch Apps.
DSL, GUI, and REST-APIs to build and
orchestrate data pipelines onto platforms like
Kubernetes and Cloud Foundry.
Continuous delivery for streaming data pipelines
using Spring Cloud Skipper.
Cron-job scheduler for batch data pipelines using
Spring Cloud Scheduler.
A toolkit for building
data integration,
real-time streaming,
and batch data
processing pipelines.
WHAT IS SPRING CLOUD DATA FLOW?
26

27. Data pipelines consist of Spring Boot apps, using
Spring Cloud Stream for event-streaming or
Spring Cloud Task for batch processes.
Ready for Data Integration with >60
out-of-the-box streaming and batch Apps.
DSL, GUI, and REST-APIs to build and
orchestrate data pipelines onto platforms like
Kubernetes and Cloud Foundry.
Continuous delivery for streaming data pipelines
using Spring Cloud Skipper.
Cron-job scheduler for batch data pipelines using
Spring Cloud Scheduler.
A toolkit for building
data integration,
real-time streaming,
and batch data
processing pipelines.
WHAT IS SPRING CLOUD DATA FLOW?
27

28. Data pipelines consist of Spring Boot apps, using
Spring Cloud Stream for event-streaming or
Spring Cloud Task for batch processes.
Ready for Data Integration with >60
out-of-the-box streaming and batch Apps.
DSL, GUI, and REST-APIs to build and
orchestrate data pipelines onto platforms like
Kubernetes and Cloud Foundry.
Cron-job scheduler for batch data pipelines using
Spring Cloud Scheduler.
A toolkit for building
data integration,
real-time streaming,
and batch data
processing pipelines.
WHAT IS SPRING CLOUD DATA FLOW?
28

29. Demo
How to schedule a task in Spring Cloud Data Flow on
Kubernetes (PKS - Pivotal Container Service)

30. Local Version runs on your
machine.
Cloud Foundry Version can be
installed by hand or by the PCF
SCDF TIle
Kubernetes Version can be install
by hand or by our Helm Chart
Data Flow
Installation

31. SCDF
OSS or PCF Tile?

32. SCDF for PCF
Addition of the Spring Cloud Data Flow server to the
Marketplace as a managed service
Automatic integration with dependent PCF services:
- MySQL for PCF (or a user-provided relational
database service) for apps, pipelines, and task
history
- RabbitMQ for PCF (or a user-provided RabbitMQ
or Kafka service) for event messaging
- Redis for PCF (or a user-provided Redis service)
for capturing analytics data
Integration of Data Flow server with PCF’s UAA
security model

33. SCDF Tile on Pivotal Network

34. PCF Operations Manager Tiles

35. Versions SCDF for PCF tile v1.5.1
is packaged with
SCDF OSS v2.0.1

36. SCDF for PCF Architecture

37. Demo
How to schedule a task in Spring Cloud Data Flow for
Pivotal Cloud Foundry (PCF)
1. Provision new SCDF SI via GUI
2. Switch to previously installed SCDF service
3. Open management dashboard
4. Create a new task (timestamp)
5. Create a new schedule (0/5 0 ? * * *)
6. Create a second schedule (0/2 0 ? * * *)

38. CF CLI
$ cf create-service
p-dataflow standard df1 -c
'{"scheduler": {"name":
"scheduler-for-pcf",
"plan": "standard"}}'

39. SCDF CLI Plugin
$ cf install-plugin -r
CF-Community
"spring-cloud-dataflow-for
-pcf"

40. SCDF CLI Plugin
$ cf dataflow-shell
data-flow

41. What about the case of a task getting kicked
off by an event outside of a calendar based
event?
41
Seems Like We are Missing Something…

42. Task Launched via an event
handled in a stream.
Task Launching
By Event

43. Also, what if I have a scheduler already, can
I take advantage of that?
43
Seems Like We are Missing Something…

44. Questions?

45. Transforming How The World Builds Software
© Copyright 2018 Pivotal Software, Inc. All rights Reserved.

46. More Info
- https://cloud.spring.io/spring-cloud-dataflow/
- https://network.pivotal.io/products/p-dataflow
- https://docs.pivotal.io/scdf/1-3/
- https://spring.io/projects/spring-cloud-task
- https://spring.io/projects/spring-cloud-stream
- https://github.com/spring-cloud/spring-cloud-data
flow