Lightbend Fast Data Platform - A Technical Overview
Dean Wampler, O’Reilly author and Big Data Strategist in the office of the CTO at Lightbend discusses practical tips for architecting stream-processing applications and explains how you can tame some of the complexity in moving from data at rest to data in motion.
7. • Tangled responsibilities, lead to
infrequent, “big-bang” deployments
• App lifetimes months to forever!
Monoliths
8.
9. • Each does one thing, so must be
message driven & asynchronous
• Updates easier, deployments frequent
• App lifetimes: minutes! to forever
Microservices
10. • But be careful:
• Message overhead is much better than
function calls!
Microservices
26. 1. An accelerated on ramp for building
streaming data systems, data
applications, and other microservices.
Value Three Ways
27. 2. Best practices guidance for solving
specific design problems:
- Sample apps
- Documentation
- Enablement services
Value Three Ways
28. 3. Machine learning-based monitoring
and management:
-Keep your systems resilient, scalable,
and responsive with minimal user
intervention.
Value Three Ways
29. Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:
On Premise or Cloud
FDP
Kafka
Connect
Storage
HDFS,
S3, …
SQL/
NoSQL
ES
Kafka
Streams
Cloud Hosted
FDP.ai
Cluster
Analysis
Machine
LearningMicroservices
ProducDon Suite
Machine Learning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Management…Consoles
30. Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:
On Premise or Cloud
FDP
Kafka
Connect
Storage
HDFS,
S3, …
SQL/
NoSQL
ES
Kafka
Streams
Cloud Hosted
FDP.ai
Cluster
Analysis
Machine
LearningMicroservices
ProducDon Suite
Machine Learning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Management…Consoles
31. Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:
On Premise or Cloud
FDP
Kafka
Connect
Storage
HDFS,
S3, …
SQL/
NoSQL
ES
Kafka
Streams
Cloud Hosted
FDP.ai
Cluster
Analysis
Machine
LearningMicroservices
ProducDon Suite
Machine Learning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Management…Consoles
32. Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:
On Premise or Cloud
FDP
Kafka
Connect
Storage
HDFS,
S3, …
SQL/
NoSQL
ES
Kafka
Streams
Cloud Hosted
FDP.ai
Cluster
Analysis
Machine
LearningMicroservices
ProducDon Suite
Machine Learning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Management…Consoles
33. Service 1
Log &
Other Files
Internet
Services
Service 2
Service 3
Services
Services
N * M links ConsumersProducers
34. Service 1
Log &
Other Files
Internet
Services
Service 2
Service 3
Services
Services
N + M links ConsumersProducers
35. Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:
On Premise or Cloud
FDP
Kafka
Connect
Storage
HDFS,
S3, …
SQL/
NoSQL
ES
Kafka
Streams
Cloud Hosted
FDP.ai
Cluster
Analysis
Machine
LearningMicroservices
ProducDon Suite
Machine Learning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Management…Consoles
36. Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:
On Premise or Cloud
FDP
Kafka
Connect
Storage
HDFS,
S3, …
SQL/
NoSQL
ES
Kafka
Streams
Cloud Hosted
FDP.ai
Cluster
Analysis
Machine
LearningMicroservices
ProducDon Suite
Machine Learning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Management…Consoles
37. Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:
On Premise or Cloud
FDP
Kafka
Connect
Storage
HDFS,
S3, …
SQL/
NoSQL
ES
Kafka
Streams
Cloud Hosted
FDP.ai
Cluster
Analysis
Machine
LearningMicroservices
ProducDon Suite
Machine Learning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Management…Consoles
38. •Low latency? How low?
•High volume? How high?
Streaming Tradeoffs (1/3)
38
39. •Which kinds of data processing &
analytics are required?
•How will this processing be done?
•Individual processing of events?
•Bulk processing of records?
Streaming Tradeoffs (2/3)
39
40. •Which tools and data sources/sinks
must interoperate with your
streaming tool?
Streaming Tradeoffs (3/3)
40
50. Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:
On Premise or Cloud
FDP
Kafka
Connect
Storage
HDFS,
S3, …
SQL/
NoSQL
ES
Kafka
Streams
Cloud Hosted
FDP.ai
Cluster
Analysis
Machine
LearningMicroservices
ProducDon Suite
Machine Learning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Management…Consoles
51. • Only metrics are uploaded, not sensitive
domain data.
• One less service for you to manage.
• …
Why Only Hosted Services?
52. • …
• We can rapidly evolve this service.
without impacting your environment.
• You benefit from aggregated knowledge
from all FDP clusters.
Why Only Hosted Services?
53. Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:
On Premise or Cloud
FDP
Kafka
Connect
Storage
HDFS,
S3, …
SQL/
NoSQL
ES
Kafka
Streams
Cloud Hosted
FDP.ai
Cluster
Analysis
Machine
LearningMicroservices
ProducDon Suite
Machine Learning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Management…Consoles
55. Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS, Marathon
On Premise or Cloud
FDP
Kafka
Connect
Storage
HDFS,
S3, …
SQL/
NoSQL
ES
Kafka
Streams
Cloud Hosted
Graviton
Cluster
Analysis
Machine
LearningMicroservices
ProducCon Suite
Machine Learning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Management…Consoles
For More Information:
lightbend.com/fast-data-platform