Talk on public cloud architectures, particularly for data pipelines. Includes Serverless and Server-based architectures built on AWS and GCP.

2. • 5 – 15x market share of all
others (US market)AWS Dominates
• AWS cloud bills are for ‘old
services’ (EC2, RDS)80% EC2++
• First-time cloud projects
outcome rate*50% FAIL

3. Definitions
Vendors
Patterns
Lessons
1
2
3
4

6. 10 Million Executions
= ~ $ 500 USD

7. Files
AWS
S3
RDBMS
AWS
RDS
VMs
AWS
EC2

11. Use image
Files ->
“BIG”
Application
PLUS OS
Coordinate
with Load
Balancers
Mature
Technology
Use
Dockerfiles ->
“small”
Application
Coordinate
with Container
Managers
New
Technology

13. Just write code –> “tiny”
Function / Method
Auto-scales
Very NewTechnology
Use Dockerfiles –> “small”
Application
Coordinate via Container Mgr.
NewTechnology

14. 100
10
1
VM Container Lambda
Drive to Lambda – Save Money
Cost

15. 5
50
95
Lambda Container VM
UseVMs – Keep Control
Control

18. “But why are
Compute AND File Storage
commodities
on Azure, AWS AND GCP?”

20. Compute
EC2
Containers
Lambda
Files
S3
Glacier
Data
RDS
DynamoDB
Other
Machine
Learning
Kinesis
Serverless

21. Alpha
Some parts
may work
Service may
be changed
Service may
be
discontinued
Beta
Many parts
should
work
Service may
be changed
Year One
Most parts
should
work
Can include
some
service
integrations
YearTwo
All parts
should
work
Patterns
and scripts
emerge
YearThree
Service is
stable
Tools and
partners
emerge

23. $$ $ $$$

24. Functions
• Logic Apps
• No Code
• Generates JSON
Which
Vendor?

27. Server-based Solutions

28. Server-based
Solutions
High Availability
Core Security

29. Server-based
Solutions
High Availability
Security

30. Server-based
Solutions
High
Availability
Security
Scalability
Cost Control

31. Google
Compute
Engine
Very fast to
start
…globally
Automatically
discounted
for sustained
use
Easier to size
via the ’slider’

37. Hadoop/Spark
Transform
Visualization Client
Data Lake
ANSI SQL
Exploratory
ANSI SQL
Warehouse

38. Kappa Architecture on the Cloud – Servers?

39. PySpark ETL
Glue
Visualization Client - QuickSight
Data Lake – S3
Explore SQL
Athena
MPP SQL
Spectrum

40. AWS

41. AWS

42. Beam ETL
Dataflow
Visualization Client - DataStudio
Data Lake – GCS
Explore SQL
BigQuery
MPP SQL
BigQuery

48. Servers / IaaS?
PaaS?
Serverless?
Integration
testing?
Orchestration?
Deployment?

49. “My”
Programming
Language ?
Debugging?
Unit testing?
Integration
testing?
Orchestration?
Deployment?

51. AWS X-Ray

56. Reduce
Attack
Surface
Test
external
connections
Minimal
permissions
Granular
policies
Unique
credentials

59. Service
Costs
Training
Costs
Tooling
Costs
Migration
Costs
Learning
Costs

60. ServiceType Servers (or Containers) Serverless
Compute EC2 Lambda
Files File Services on EC2 S3
SQL on Relational Data RDBMS on EC2 or RDS Athena / Redshift Spectrum
Data Pipeline Kafka cluster on EC2 Kinesis
Machine Learning EMR with Spark ML or
Hadoop on EC2
Machine Learning API
IoT MQTT Message Broker RabbitMQ on EC2 IoT Broker
NoSQL MongoDB… on EC2 DynamoDB