This document discusses Google Cloud Platform's Internet of Things (IoT) architecture and services. It describes how IoT data can be captured using protocols and streaming into Google Cloud Pub/Sub. Machine learning algorithms can then detect patterns in real-time streams. Data is also archived in Cloud Storage. Google Cloud Dataflow is highlighted for processing both batch and stream workloads, with features like autoscaling, intuitive programming model, and unified processing of data.

1. IoT @ Google Scale
James Chittenden
Google Cloud Platform Solutions Engineer
jameschi@google.com

2. +James Chittenden
(Big Data Cloud Engineer)
jameschi@google.com

3. Big Data at Google
aka. Data at Google

4. Manage the Entire Lifecycle of Big Data
Cloud Logs
Google App
Engine
Google Analytics
Premium
Cloud Pub/Sub
BigQuery Storage
(tables)
Cloud Bigtable
(noSQL)
Cloud Storage
(files)
Cloud Dataflow
BigQuery Analytics
(SQL)
Capture Store Analyze
Batch
Real time analytics
and Alerts
Cloud DataStore
Process
Stream
Cloud Dataflow
Cloud
Monitoring

5. End to End View of the GCP IoT Architecture

6. Device to Device Protocols
● Device Discovery
● Device to Device authentication
● Device Configuration
● Protocol Routing

7. Machine Learning: Pattern Detection and Prediction
● Subscribers scan real time
streams and feed data into the
Machine Learning Recognition
algorithm
● Dataflow Orchestrates
streaming algorithms which
compare data streams against
Experience Database
● Correlators detect known
patterns and publish alerts
using Cloud Pub/Sub

8. Cloud Storage Archival and Retrieval
● Data is periodically unloaded
from Big Table and stored in
Cloud Storage for archival
● Data in Cloud Storage can be
quickly re-loaded in Big Table
should it need to be re-
processed.

9. Cloud Pub/Sub
Real-time and reliable messaging with Pub/Sub

10. Messaging is a shock-absorber
Throughput LatencyAvailability
Images by Connie
Zhou
• Buffer new requests
during outages
• Prevent overloads that
cause outages
• Redirect requests to
recover from outages
• Smooth out spikes in
new request rate
• Balance load across
multiple workers
• Balance arrival rate
with service rate
• Accept requests closer
to the network edge
• Optimize message
flow across regions
• Leverage shared
efforts to improve
protocols

11. Pub/Sub is a change-absorber
Sinks TransformsSources
Images by Connie
Zhou
• New data sources can
plug into old data
flows
• New data sources can
use new schemas
• Common security
policies for all sources
• Data can be sent to
new destinations
• Push and Pull delivery
are both available
• Spans organizational
boundaries
• Select subsets of
messages that matter
• Helps manage schema
and version changes
• Can merge streams
into new topics

12. Chat & Mobile
Every time your GMail box
pops up a new message,
it’s because of a push
notification to your
browser or mobile
device.
One of the most important
real-time information
streams in the company is
advertising revenue — we
use Pub/Sub to broadcast
budgets to our entire fleet
of search engines
Google Cloud Messaging
for Android delivers
billions of messages a
day, reliably and securely
for Google’s own mobile
apps and the entire
developer community
Updating search results as
you type is a feat of real-
time indexing that
depends on Pub/Sub to
update caches with
breaking news
Ads & Budgets Instant SearchPush Notifications
Pub/Sub at Google

13. HTTP Server
Subscriber
Pub/Sub System
Webhook
Delivery
Publisher
Topic
Subscription
HTTP Push
Delivery
Google
App Engine
Pull
Subscriber
Subscription Subscription
Google RPC
Delivery
Cloud
Dataflow
Subscription
On-Prem/Cloud Any Environment

14. Subscriber
Msg
Pub/Sub System
Subscriber
Msg
Pub/Sub System
Ack
RPC Send
RPC Return
Ack
Push Subscription Pull Subscription

15. “We don’t really run MapReduce at Google anymore”
- Urs Hoelzle
Google Dataflow

16. Google Technologies
SpannerDremelMapReduce
Big Table
MillWheel
2012 2014+2002 2004 2006 2008 2010
GFS
2013
More!
Flumejava
Colossus

17. Autoscaling mid-job
Fully managed - No-Ops
Intuitive Data Processing Framework
Batch and Stream Processing in one
Liquid sharding mid-job
1
2
3
4
5
Dataflow Goodies

18. Autoscaling mid-job
Fully managed - No-Ops
Intuitive Data Processing Framework
Batch and Stream Processing in one
Liquid sharding mid-job
1
2
3
4
5
Pipeline p = Pipeline.create();
p.begin()
.apply(TextIO.Read.from(“gs://…”))
.apply(ParDo.of(new ExtractTags())
.apply(Count.create())
.apply(ParDo.of(new ExpandPrefixes())
.apply(Top.largestPerKey(3))
.apply(TextIO.Write.to(“gs://…”));
p.run();
Dataflow Goodies

19. Autoscaling mid-job
Fully managed - No-Ops
Intuitive Data Processing Framework
Batch and Stream Processing in one
Liquid sharding mid-job
1
2
3
4
5
Deploy
Schedule & Monitor
Dataflow Goodies

20. Autoscaling mid-job
Fully managed - No-Ops
Intuitive Data Processing Framework
Batch and Stream Processing in one
Liquid sharding mid-job
1
2
3
4
5
800 RPS 1200 RPS 5000 RPS 50 RPS
Dataflow Goodies

21. Autoscaling mid-job
Fully managed - No-Ops
Intuitive Data Processing Framework
Batch and Stream Processing in one
Liquid sharding mid-job
1
2
3
4
5
Dataflow Goodies

22. Autoscaling mid-job
Fully managed - No-Ops
Intuitive Data Processing Framework
Batch and Stream Processing in one
Liquid sharding mid-job
1
2
3
4
5
Pipeline p = Pipeline.create();
p.begin()
.apply(TextIO.Read.from(“gs://…”))
.apply(ParDo.of(new ExtractTags())
.apply(Count.create())
.apply(ParDo.of(new ExpandPrefixes())
.apply(Top.largestPerKey(3))
.apply(TextIO.Write.to(“gs://…”));
p.run();
.apply(PubsubIO.Read.from(“input_topic”))
.apply(Window.<Integer>by(FixedWindows.of(5, MINUTES))
.apply(PubsubIO.Write.to(“output_topic”));
Dataflow Goodies

23. Unified Model

24. Unified Model

25. Pub/Sub + Dataflow + BigQuery Demo

26. Life of a Pipeline

28. Dataflow
Your Data
BigQuery
Fast ETL
Regex
JSON
UDFs
Spreadsheets
BI Tools
Coworkers
Applications + Reports
PubSub
Cloud Storage
BigTable
Enterprise Big Data Architecture on Google

29. Plus True Stream Processing
Plus Autoscaling and per-minute billing
All the benefits of Hadoop-on-Google
Plus a Fully-Managed Service
Plus New, Intuitive Framework
1
2
3
4
5
Why Dataflow?

30. Questions?