You should spend your time using the powerful Apache Flink ecosystem to get value from your data, not on your data processing infrastructure. Cloud environments can help you with this problem by providing managed services and infrastructure. Since Google Cloud Dataproc, Google's managed service to power the Apache big data ecosystem, runs Flink, you can easily combine the benefits of cloud with your Flink data pipelines. With new support for Flink and long-running streaming jobs, we will show you how you can set up a cluster and a streaming job in less than three minutes.

1. Make the cloud work for you
Easy, fast, and low-cost streaming with Apache
Flink on Google Cloud Platform
James Malone [ jamesmalone@google.com ]

2. Open source
powerful but complex

3. The Apache ecosystem

4. Typical OSS deployments

5. Scaling makes your life difficult
Scaling can take hours, days, or
weeks to perform which may
delay needed data processing
Capacityneeded
Time needed to obtain new capacity
Time

6. Scaling should be painless and fast
Things take seconds to minutes,
not hours or weeks.
Capacityneeded
Time

7. You have to babysit utilization
Clusterutilization
Time
Requires effort to pack clusters
so the it does not have periods of
inactivity and wasted resources
Cluster
Inactive

8. Only use clusters when you need them
Be an expert with your data, not
your infrastructure
Clusterutilization
cluster 1 cluster 2
Time
Clusterutilization
Time

9. You are not paying for what you use
Cost
Time
You are paying for more (spare)
capacity than you actually need to
process your data

10. Pay for exactly what you use
Cost
Time
You only pay for resources when
you need them

11. Open source
on Google Cloud

12. Google is passionate about open source
MapReduce
BigTable DremelColossus
FlumeMegastore
SpannerPubSub
Millwheel
Cloud
Dataflow
Cloud
Dataproc
Apache
Beam

13. What is Cloud Dataflow?
Intelligently scales to millions of QPS
Open source programming model
Unified batch and streaming processing
Fully managed, no-ops data processing

14. Cloud Dataflow
Compute and storage
Unbounded
Bounded
Resource management
Resource auto-scaler
Dynamic work
rebalancer
Work scheduler
Monitoring
Log collection
Graph optimization
Auto-healing
Intelligent watermarkingS
O
U
R
C
E
S
I
N
K

15. Cloud Dataproc offers a spectrum
Cloud Dataflow Cloud Dataproc

16. Proprietary + Confidential
Cloud Dataflow
Cloud Dataflow is a real-time data
processing service for batch and
stream data processing.
● Fully managed
● Unified programming model
● Integrated and open source
● Resource management
● Autoscaling
● Monitoring

17. What is Cloud Dataproc?
Google Cloud Dataproc is a fast, easy to use, low cost
and fully-managed service, powered by Google Cloud
Platform, that helps you take advantage of the Spark,
Flink, and Hadoop ecosystem.

18. Google Cloud Dataproc
Fast
Things take seconds
to minutes, not
hours or weeks
Easy
Be an expert with
your data, not your
data infrastructure
Cost-effective
Pay for exactly what
you use to process
your data, not more

19. Cloud Dataproc clusters
Create cluster
(0 seconds)
Configure cluster
(20 seconds)
Use cluster
(90 seconds)
Scale anytime

20. Cloud Dataproc offers a spectrum
BA
A
B
C
D
E
F

21. Connecting OSS to Cloud Platform
Input
Cloud
Dataproc
BigQuery
Compute
Engine
Cloud
Storage
Cloud
Bigtable
Output
Cloud
Dataproc
BigQuery
Compute
Engine
Cloud
Storage
Cloud
Bigtable

22. Cloud Dataproc - under the hood
Dataproc Cluster
Dataproc Agent
Master node(s)
Compute Engine
Managed instance Group
Persistent
Workers
Compute Engine
PVM Worker(s)
Compute Engine
HDFS
Persistent Disk
Cluster bucket
Cloud Storage
Networking
Cloud Network
Cloud
IAM

23. Cloud Dataproc - under the hood
Dataproc Cluster
Dataproc Agent
HDFS
Persistent Disk
Cluster bucket
Cloud Storage
Comput engine nodes
Dataproc Image OSS
Apache Flink
Apache Hadoop
Apache Hive
(many others)
Clients
Cloud Dataproc API
Clusters
Operations
Jobs

24. Cloud Dataproc - under the hood
Clients
Developers Console
Cloud SDK
SSH
Customer project(s)
Dataproc API Clusters
Cloud Dataproc

25. Cloud Storage performance (and improvement)

26. Cloud Dataproc demo

27. $12.85

28. Example new features
and their impact

29. Restartable jobs (beta)
● Any job submitted through the Cloud Dataproc Jobs API can now be set
to automatically restart on failure
● Very useful for both batch and streaming jobs. Jobs which checkpoint
can also be automatically restarted
● Specified with the switch --max_retries_per_hour when using the
Cloud SDK (gcloud) the max_failures_per_hour in the Jobs API

30. Clusters with GPUs (beta)
● Cloud Dataproc clusters support Compute Engine nodes with Nvidia
Tesla K80 GPUs attached to them
● We expect GPU support will continue to grow in the open source data
processing ecosystem throughout 2017
● Easily add GPUs to a Cloud Dataproc cluster with the switch
--master/worker_accelerator with the Cloud SDK (gcloud)

31. Single-node clusters (beta)
● Create a sandbox Cloud Dataproc “cluster” with only one node instead
of the typical three node design (1 master, 2 workers)
● Great for lightweight data science, small-scale testing, proof of concept
building, and education
● Use the --single-node argument in the Cloud SDK or select “Single
node” when creating a cluster in the Google Cloud Console

32. Regional endpoints & private IP clusters (beta)
● Cloud Dataproc supports a “global” endpoint and “regional endpoints” in
each Compute Engine region. This allows you to isolate Cloud Dataproc
interactions to one specific region
● Traditionally clusters have needed a public IP attached to them. Cloud
Dataproc now supports (easy to setup) “private IP only” clusters which
do not require a public IP address on Compute Engine nodes

33. Cloud platform
architecture concepts

34. Disaggregation of storage and compute
Analysis
Cloud Datalab
Development & Test
Data sinksProduction
Cloud
Dataproc
External applications
Storage
Cloud Storage
Application Logs
Storage
BigQuery
Clusters
Development
Cloud
Dataproc
Test
Cloud
Dataproc
Data sources
Storage
Cloud Bigtable
Storage
Cloud Storage
Storage
BigQuery
Storage
Cloud Bigtable
Data scienceCluster monitoring
Monitor
Stackdriver
Logs
Logging

35. Dataproc Cluster
Cost savings through preemptible VMs
Master node(s)
Compute Engine
Managed instance Group
Persistent
Workers
Compute Engine
PVM Worker(s)
Compute Engine
HDFS
Persistent Disk
Cluster bucket
Cloud Storage
Networking
Cloud Network
Cloud
IAM
Master node(s)
Compute Engine

36. Right-sizing your hardware with custom machine
types
Workload n1-standard-4 n1-highmem-4 n1-standard-8custom-6-30720

37. Split clusters and jobs
Customer’s on-premise datacenter
Flink/Hadoop cluster
Job 1
Job 2
Job 3
Job 4
Cluster 1
Cloud Dataproc
Cluster 2
Cloud Dataproc
Cluster 3
Cloud Dataproc
Job 1
Job 2
Job 3
Job 4

38. Separate development and production
Customer’s on-premise datacenter
Production
Cloud Dataproc
Development
Cloud Dataproc
Staging
Cloud Dataproc
Production cluster
Compute HDFS
Development cluster
Compute HDFS
Cluster bucket
Cloud Storage
Read/write
Read only

39. Semi-long-lived clusters - group and select by label
Ephemeral and semi-long-lived clusters
Clusters per Job
Cluster
Cloud Dataproc
Cluster
Cloud Dataproc
Cluster
Cloud Dataproc
Cloud Storage
Cluster
Cloud Dataproc
Cluster
Cloud Dataproc
Cluster
Cloud Dataproc
Edge Nodes
Compute Engine
"Canary"
v1.2
"Current"
v1.1
Clients Clients Clients
ClientsClients

40. Questions and
next steps

41. Getting started
Codelabs - codelabs.developers.google.com/codelabs/cloud-dataproc-starter/
Cloud Dataproc quickstarts - cloud.google.com/dataproc/docs/quickstarts
Cloud Dataproc tutorials - cloud.google.com/dataproc/docs/tutorials
Cloud Dataproc initialization actions - github.com/GoogleCloudPlatform/dataproc-initialization-actions

42. Getting help
Cloud Dataproc documentation - cloud.google.com/dataproc/docs
Cloud Dataproc release notes - cloud.google.com/dataproc/docs
Stack Overflow - google-cloud-dataproc
Cloud Dataproc email discussion - cloud-dataproc-discuss@googlegroups.com
Google Cloud Support - cloud.google.com/support

43. Thank you
https://goo.gl/Qyf5U7