Sydney: Certus Data 2.0 Vault Meetup with Snowflake - Data Vault In The Cloud

1© 2018 Snowflake Computing Inc. All Rights Reserved.
Data Vault in the Cloud

• 640 employees
• 1500+ customers
• $923M in funding
US
2014
Founded 2012 EMEA
2016
APAC
2017
Snowflake

Complexity
Infrastructure
and pipelines
that are too hard
to manage
Limited
Scalability
Unable to
service the
needs of the
business
Diversity
Unable to easily
consolidate
siloed datasets
Inadequate
Elasticity
Struggling with
rigid, inflexible
architectures
Rigid
Cost
Forced to keep
the lights on 24/7
Key Challenges

All-new SQL data
warehouse
Fully relational, ANSI SQL,
ACID compliant
Built for the cloud
Instantly scale up and down
as needed
Delivered as a service
No infrastructure, knobs, or
tuning to worry about
What is Snowflake?

6© 2018 Snowflake Computing Inc. All Rights Reserved.© 2018 Snowflake Computing Inc. All Rights Reserved. 6
Traditional architectures
Shared-disk
Shared storage
Single cluster
Snowflake
Shared-nothing
Decentralized, local storage
Single cluster
Multi-cluster, shared data
Centralized, scale-out storage
Multiple, independent compute clusters
Snowflake is the first new database scalability architecture in
30 years!

Workload Isolation =
Increased Agility

© 2018 Snowflake Computing Inc. All Rights Reserved.
ETL/ELT
snowpipe
XS
S
M
M
L
Sales
Data
Science
Global Services
Transactional Control
Security
Query Planning & Optimisation
Logical Model
AWS QuickSight
S
AWS Glue

• Allows loading several
tables from one source
• In one statement
Multi-Table Insert
INSERT ALL
WHEN (SELECT COUNT(*)
FROM dv_hub_country hc
WHERE hc.hub_country_key= stg.hash_key) = 0
THEN
INTO dv.hub_country (hub_country_key, country_abbrv, hub_load_dts,
hub_rec_src)
VALUES (stg.hash_key, stg.country_abbrv, stg.load_dts, stg.rec_src)
WHEN (SELECT COUNT(*)
FROM dv.sat_countries sc
WHERE sc.hub_country_key = stg.hash_key AND
sc.hash_diff = MD5(country_name)) = 0
THEN
INTO dv.sat_countries (hub_country_key, sat_load_dts,
hash_diff, sat_rec_src, country_name)
VALUES (stg.hash_key, stg.load_dts,
stg.hash_diff, stg.rec_src, stg.country_name)
SELECT MD5(country_abbrv) AS hash_key, country_abbv, country_name,
MD5(country_name) AS hash_diff,
CURRENT_TIMESTAMP AS load_dts,
stage_rec_src AS rec_src
FROM stage.country stg;

Hashing
• Snowflake supports both MD5, SHA1 and SHA2 algorithms in standard, HEX, and
binary formats
• i.e. SHA2(), SHA2_HEX() or SHA2_BINARY()
SELECT SHA2_BINARY('Snowflake', 384);
736BD8A53845348830B1EE63A8CD3972F031F13B111F66FFDEC2271A7AE709662E503A0CA305
BD50DA8D1CED48CD45D9

Instant Elasticity

ETL/ELT
snowpipe
XS M
Sales
Data
Science
XLS
Global Services
Security
Logical Model
AWS QuickSight
AWS Glue

ETL/ELT
snowpipe
XS M
Sales
Data
Science
M…
XL
Multi-cluster
Global Services
Security
Logical Model
AWS QuickSight
AWS Glue

Data Vault Loading Pattern
- Major Synchronization Points
• All processes within sets done in
parallel
• Process sets ”wait” for previous set
completion
• Sets run as soon as data is ready
• No other waiting needed
• Load dependencies maximally
reduced
Stage Loads Vault Loads

Applying Virtual Warehouses to Data Vault
1. Use a Multi-Cluster
Warehouse for Source to
Stage loads
2. Use a separate Warehouse
for each DV entity type
• Zero contention
• Maximum parallelism
• Maximum flexibility -
each can be
independently sized

New Tricks

{
"city": {
"coord": {
"lat": 27.716667,
"lon": 85.316666
},
"country": "NP",
"findname": "KATHMANDU",
"id": 1283240,
"name": "Kathmandu",
"zoom": 7
},
"clouds": {
"all": 75
},
"main": {
"humidity": 94,
"pressure": 1012,
"temp": 294.15,
"temp_max": 294.15,
"temp_min": 294.15
},
"time": 1504476371,
"weather": [
{

ETL/ELT
snowpipe
XS M
Sales
Data
Science
M…
XL
Multi-cluster
Structured &
semi-structured
Global Services
Security
Logical Model
AWS QuickSight
AWS Glue

Query JSON with SQL
Apple 101.12 250 FIH-2316
Pear 56.22 202 IHO-6912
Orange 98.21 600 WHQ-6090
Structured data
(CSV)
Semi-structured data
(e.g. JSON, Avro, XML)
{ "firstName": "John",
"lastName": "Smith",
"height_cm": 167.64,
"address": {
"streetAddress": "21 2nd Street",
"city": "New York",
"state": "NY",
"postalCode": "10021-3100"
},
"phoneNumbers": [
{ "type": "home", "number": "212 555-1234" },
{ "type": "office", "number": "646 555-4567" }
]
}
select v:lastName::string as last_name
from json_demo;
All Your Data!

• Create Sats using a VARIANT
• Hub Key
• Load DTS
• Rec Source
• VARIANT
• Load JSON doc into VARIANT column
• Create a View on the Sat for Business Vault
• View presents only the JSON attributes needed for current use
case
Combine Data Vault & Schema-on-Read
Benefits
➢No change to Sat
definitions if structure
changes
➢Business Vault view is
unaffected
➢Change or add new BV
views over time as needed

ETL/ELT
Snowpipe
XS
S
M
L
Finance
Sales
Data
Science
M…
Test/Dev
Clone
Data protection
& time travel
XL
Multi-cluster
Structured &
semi-structured
Global Services
Security
Logical Model
AWS QuickSight
AWS Glue

Story Time

Founded in 2008, Aptus Health connects health and life sciences
companies with healthcare professionals, healthcare consumers,
and other members of the healthcare ecosystem.
The company creates multichannel marketing campaigns aimed at
physicians and healthcare consumers through the many digital
channels the company owns and operates.
Aptus Health

Where They Started
Mobile App
Campaign
Management
Sample Requests
Daily ETL
Apply Business Rules
Data Warehouse
- User Activity
- Campaign Activity
- Sample Request Activity
- User Data
- Mobile App Activity
Cube
Reports
Web & Mobile
Analytics
Extracts
Fill
Stage
Load F
D
D
D
D
D
D
D
D
Engagement
Platform

• A business-driven roadmap in collaboration with Engineering to drive priorities
• Central availability of data related to Healthcare Providers across all platforms
• Ability to efficiently answer business questions: new or previous
• New types of analysis should not require weeks of data preparation
• Do not lose data that could have future analytical value
• Ability to analyze historic events
• Clear documentation of all calculations/derivations of data for complete transparency in
meaning of information & to instill confidence
• Master Data Management: stop passing around spreadsheets
• Framework that will be extensible to both US and international data
• Timely data: minutes, not days
• Business reporting enabled with limited impact to data management projects
What did the business need?

Diversity
One place for
all your data
Scalability
Any scale of
data, users and
workloads
Flexible
Cost
Pay for what
you use, when
you use it
Simplicity
Minimal admin,
plug-and-play
solution
Elasticity
Size for what
you need
right now
Key Values

YOUR DATA, NO LIMITS
Thank You!

Sydney: Certus Data 2.0 Vault Meetup with Snowflake - Data Vault In The Cloud

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (19)

Similaire à Sydney: Certus Data 2.0 Vault Meetup with Snowflake - Data Vault In The Cloud

Similaire à Sydney: Certus Data 2.0 Vault Meetup with Snowflake - Data Vault In The Cloud (20)

Plus de Certus Solutions

Plus de Certus Solutions (20)

Dernier

Dernier (20)

Sydney: Certus Data 2.0 Vault Meetup with Snowflake - Data Vault In The Cloud