Pivoting Spring XD to Spring Cloud Data Flow: A microservice based architecture for stream processing Microservice based architectures are not just for distributed web applications! They are also a powerful approach for creating distributed stream processing applications. Spring Cloud Data Flow enables you to create and orchestrate standalone executable applications that communicate over messaging middleware such as Kafka and RabbitMQ that when run together, form a distributed stream processing application. This allows you to scale, version and operationalize stream processing applications following microservice based patterns and practices on a variety of runtime platforms such as Cloud Foundry, Apache YARN and others. About Sabby Anandan Sabby Anandan is a Product Manager at Pivotal. Sabby is focused on building products that eliminate the barriers between application development, cloud, and big data.

1. Pivoting
Spring XD to Spring Cloud Data Flow
Sabby Anandan | Product Manager | @sabbyanandan

2. business premise
microservices
what it means
for data
tools / frameworks
what we learned
how it all comes
together

3. Sense of Urgency
business
results
business
strategy

4. Sense of Urgency
OR
Need for Momentum?

5. business
strategy
business
results

6. business
strategy
business
results
Continuous Delivery

7. Continuous Delivery

8. Deliver Continuously
1 2

9. frequent roll out updates, even multiple times a day
gives you more ways to maintain availability
reduce risks
allows you to make sound technology choices
gives your team more velocity

10. microservices architecture
business
strategy
business
results
devops culture
continuous delivery
Casey West
Principal Technologist, Cloud Foundry

11. “how do we get there?”

12. Continuously Delivered
Microservices

13. Microservices
architecture composed of loosely coupled
elements that have bounded contexts
Adrian Cockcroft
Battery Ventures
previously: Cloud Architect at Netflix

14. “what tools can I use?”

15. Spring

16. Spring Boot
AutoConfigure
Starters
CLI
Actuator
Dev-Tools
Health Checks
Security
Logging

17. BUILDING ADAPTIVE APPLICATIONS IS HARD
built on Cloud Foundry
code will be open sourced.
Demo: Spring Boot

18. Emerging Challenges
• separate deployable units (registry, discovery,
versioning)
• reliable operationalization (fault tolerance)
• too many moving parts (coordination, routing,
load balancing)

19. Spring

20. Spring Cloud

21. Spring Cloud
Hystrix
Eureka
Ribbon
Zuul
Feign
Service Discovery
Config Server
Control Bus
Zookeeper Distributed TracingConfig Server

22. Spring Cloud
@EnableConfigServer
@EnableEurekaServer
@EnableDiscoveryClient
@EnableCircuitBreaker
@EnableHystrix
@EnableAtlas
@EnableTurbine
@EnableZuulProxy

23. OK, I get it..
..what’s specical about
microservices?

24. Microservices
a single application as a suite of small services,
each running in its own process and
communicating with lightweight mechanisms, often
an HTTP resource API
James Lewis and Martin Fowler
ThoughtWorks

25. Microservices
a single application as a suite of small services,
each running in its own process and
communicating with lightweight
mechanisms, often an an
HTTP resource API

26. OK, I get it..
..what if I have a lot
of data?

27. every software company cares
about data

28. Data-driven Microservices
composition of small standalone executable
applications communicating over messaging
middleware
Mark Fisher and Mark Pollack
Spring/Pivotal

29. Data-driven Microservices
composition of small standalone executable
applications communicating over
messaging middleware
operates by consuming from and producing to
message queues and data stores

30. “i need real-time insights
from data”

31. Spring

32. Developer Toolkits
Spring Integration
enables lightweight
messaging within Spring-
based applications and
supports integration with
external systems Spring Cloud Stream
enables you to create
powerful distributed
streaming data applications
with a minimal amount of
coding

33. Spring Cloud Stream
@EnableBinding(Source.class)
@EnableRXJavaProcessor
@EnableBinding(Sink.class)
@EnableBinding(Processor.class)

34. http | transform | log

35. | = ?
http | transform | log

36. | = Binder

37. Binders

38. BUILDING ADAPTIVE APPLICATIONS IS HARD
built on Cloud Foundry
code will be open sourced.
Demo: Spring Cloud Stream

39. @EnableBinding(Source.class)
Discover
Output
Channel
Discover
Binder
Discover
Type
Converter
Bind
Producer
Auto
Configure
$ java -jar http-source.jar

40. $ java -jar transform-processor.jar
@EnableBinding(Processor.class)
Discover Input
and Output
Channels
Discover
Binder
Discover
Type
Converter
Bind
Producer and
Consumer
Auto
Configure

41. @EnableBinding(Sink.class)
Discover
Input
Channel
Discover
Binder
Discover
Type
Converter
Bind
Consumer
Auto
Configure
$ java -jar log-sink.jar

42. “i have large volumes of
asynchronous data
systems”

43. Spring

44. Reactive Streams
@EnableRXJavaProcessor

45. 10 11 12 13 14 15 16 17 18 19.. ..
10 11 12 13 14 15 16 17 18 19.. ..
i0
i1
12
17

46. “i need offline data analysis”

47. Spring

48. Batch-processing Microservices
Spring Batch
enables the development
of robust batch applications
vital for the daily operations of
enterprise systems
Spring Cloud Task
enables you to develop and
run short-lived executable
data applications locally or in
the cloud

49. @EnableTask
Discover
DataSource
Bind
DataSource
Auto
Configure
$ java -jar timestamp-task.jar

50. BUILDING ADAPTIVE APPLICATIONS IS HARD
built on Cloud Foundry
code will be open sourced.
Recap

51. I need real-time insights from data
I need offline data analysis
Spring Cloud Stream
Spring Cloud Task
Stream
Batch

52. Stream Batch
Data Architecture

53. Spring

54. Spring XD

55. Kodiak: Turnkey Data Integration Platform Sep 2012
Apr 2013
Aug 2013
Jan 2014
Apr 2014
Nov 2014
Dec 2014
Jan 2015
Feb 2015
Batch Jobs (Gaps: Oozie; Flume; Sqoop)
PMML (closed-loop-analytics; IoT), YARN
Kafka and Sqoop Support
Spring XD on CloudFoundry (beta)
Spring XD Launch
Builds on Spring Boot
Spark and Python Support
Spark Streaming, RxJava, Reactor Support
Apr 2015 BDS [HAWQ, GPDB, GemFire, PHD 3.0/HDP]
Flo UI, HDP/Ambari, Lattice Integration Jun 2015
Nov 2015 Flo Composed Jobs/Workflows, Perf. Improvements

56. Kodiak: Turnkey Data Integration Platform Sep 2012
Apr 2013
Aug 2013
Jan 2014
Apr 2014
Nov 2014
Dec 2014
Jan 2015
Feb 2015
Batch Jobs (Gaps: Oozie; Flume; Sqoop)
PMML (closed-loop-analytics; IoT), YARN
Kafka and Sqoop Support
Spring XD on CloudFoundry (beta)
Spring XD Launch
Builds on Spring Boot
Spark and Python Support
Spark Streaming, RxJava, Reactor Support
Apr 2015 BDS [HAWQ, GPDB, GemFire, PHD 3.0/HDP]
Flo UI, HDP/Ambari, Lattice Integration Jun 2015
Nov 2015 Flo Composed Jobs/Workflows, Perf. Improvements

57. Not Quiet There
• couldn’t scale without undeploying
running stream or batch pipelines
• couldn’t do rolling upgrades
• couldn’t do continuous deployments

58. longer running
services
short lived
processes
Data Microservices Architecture
Stream Batch

59. Spring XD

60. Spring Cloud Data Flow

61. Spring Cloud Data Flow is
a cloud native programming and operating model
for composable data microservices on a
structured platform

62. Spring Cloud Data Flow is
a cloud native programming and operating
model for composable data microservices on a
structured platform

63. Spring Cloud Data Flow is
a cloud native programming and operating
model for composable data microservices on a
structured platform
@EnableBinding(Source)
class Greeter {
@InboundChannelAdapter(Source.OUTPUT)
String greet() {
"hello world"
}
}
canary
deployments
continuous
delivery
monitoring
@EnableBinding(Source.class)
@EnableBinding(Processor.class)
@EnableBinding(Sink.class)

64. Spring Cloud Data Flow is
a cloud native programming and operating model
for composable data microservices on a
structured platform
http
transform
log
job foo
< bar || baz & jaz
> bye
Streams
Jobs
foo
bar jaz
baz
bye

65. Spring Cloud Data Flow is
a cloud native programming and operating model
for composable data microservices on a
structured platform
Private and Public PaaS
Polyglot
Multiple Platforms
Service Marketplace
Direct Access to Containers
Metrics, Logging, Monitoring
Dynamic Scaling and Resource
Allocation
Integrated Security and
Auditing

66. Spring Cloud Data Flow is
a cloud native programming and operating model
for composable data microservices on a
structured platform
Private and Public PaaS
Polyglot
Multiple Platforms
Service Marketplace
Direct Access to Containers
Metrics, Logging, Monitoring
Dynamic Scaling and Resource
Allocation
Integrated Security and
Auditing

67. Spring Cloud Data Flow is
a cloud native programming and operating model
for composable data microservices on a
structured platform
……
Polyglot
Multiple Platforms
……
Direct Access to Containers
Metrics, Logging, Monitoring
Dynamic Scaling and Resource
Allocation
Integrated Security and
Auditing
YARN

68. XD Modules
XD
Admin
XD Containers
ZooKeeper
ZooKeeper
Admin / Flo UI
Shell
CURL
Database

69. Admin
Admin / Flo UI
Shell
CURL
??X
YARN
Stream/Task Boot Apps

70. Orchestrate Composable
Data Microservices
HOST
Cloud Foundry YARN Local
Spring Cloud Data Flow
µPCF
Spring Cloud Stream/Task AppsSpring Cloud Stream Binders [Rabbit, Kafka, Redis]

71. BUILDING ADAPTIVE APPLICATIONS IS HARD
built on Cloud Foundry
code will be open sourced.
Demo: Spring Cloud Data Flow

72. cf push
• here is my source code
• run it on the cloud for me
• i do not care how

73. yarn push
• here is my source code
• run it on the hadoop for me
• i do not care how

74. “simple things should be simple,
complex things should be possible”
- alan kay

75. Resources
Spring Cloud Data Flow: http://cloud.spring.io/spring-
cloud-dataflow/
Spring Cloud Data Flow Samples: https://github.com/
spring-cloud/spring-cloud-dataflow-samples
Spring Cloud Stream: http://cloud.spring.io/spring-cloud-
stream/
Spring Cloud Task: http://cloud.spring.io/spring-cloud-task/