Introduction to Data Vault 2.0 on Snowflake

INTRODUCTION TO
DATA VAULT 2.0 ON
SNOWFLAKE
KENT GRAZIANO, CHIEF TECHNICAL EVANGELIST
KentGraziano

© 2018 Snowflake Computing Inc. All Rights Reserved. Snowflake Proprietary. Not for Redistribution.
© 2021 Snowflake Inc. All Rights Reserved
› Chief Technical Evangelist, Snowflake Inc
› Blogger: The Data Warrior
› Certified Data Vault Master and DV 2.0 Practitioner
› Oracle ACE Director (Alumni)
› OakTable Member
› Member – DAMA Houston & DAMA International
› Data Modeling, Data Architecture and Data Warehouse
Specialist
› 30+ years in IT
› 25+ years of Oracle-related work
› 20+ years of data warehousing experience
› Former-Member: Boulder BI Brain Trust
(http://www.boulderbibraintrust.org/)
› Author & Co-Author of a bunch of books
› Past-President of ODTUG and Rocky Mountain Oracle User
Group
MY BIO

AGENDA
• Why talk about Data Vault?
• What is a Data Vault?
• Data Vault 2.0 modeling basics
• Applying Snowflake Features
• Virtual Information Marts
• Reference Architecture
• Critical Success Factors
• Customer Examples
• References

WHY DATA VAULT?

SOLUTIONS
CUSTOMERS LOOK FOR
Future Proof
supporting complex scenarios
scalable
extendable / agile
Controlled
governed
transparent
Modern
attractive for talent
latest & greatest
Easy to engineer
pattern based
quick time to value
automated
Greenfield
Migration
Immediate
pain
Centralize
and
democratize
New Business
Opportunities
Analytics 2.0

CUSTOMERS ASKING
FOR DATA VAULT
We are new to DW.
What is the best
pattern to use for
my cloud data
project?
We did DW before.
Should we try
Data Vault? We did Data Vault
before. What are the
best practices on
Snowflake?
300+ DV customers

WHAT IS DATA VAULT?

ORIGINAL DATA VAULT DEFINITION
The Data Vault is a detail oriented, historical tracking and uniquely linked
set of normalized tables that support one or more functional areas of
business.
It is a hybrid approach encompassing the best of breed between 3rd normal
form (3NF) and star schema. The design is flexible, scalable, consistent
and adaptable to the needs of the enterprise.
Architected specifically to meet the needs of today’s
enterprise data warehouses
Dan Linstedt: Defining the Data Vault

© 2020 Empowered Holdings & Dan Linstedt. All Rights Reserved
WHAT IS DATA VAULT 2.0?
What is Data Vault?
A System of Business
Intelligence containing the
necessary components needed to
accomplish enterprise vision in
Data Warehousing and Information
Delivery.
Foundational Pillars of Data Vault 2.0
Far more than just a
data modeling technique,
it is a complete
methodology.

• What are our other Enterprise Data
Warehouse Options?
• Third-Normal Form (3NF): Complex
primary keys (PK’s) with cascading
snapshot
• Star Schema (Dimensional): Difficult
to reengineer fact tables for
granularity changes
• Difficult to get it right the first time
• Not adaptable to rapid business
change
• NOT AGILE!
WHAT IS DATA VAULT TRYING TO
SOLVE?

• The work on the Data Vault approach began in the early 1990s;
completed around 1999.
• Throughout 1999, 2000, and 2001, the Data Vault design was
tested, refined, and deployed into specific customer sites.
• In 2002, the industry thought leaders were asked to review the
architecture.
• Kent meets Dan at a Lunch & Learn in Denver!
• In 2003, Dan began teaching the modeling techniques to the mass
public.
• Kent & Team take their 1st Data Vault Modeling class
• In 2014, Dan introduced DV 2.0!
DATA VAULT EVOLUTION

WHAT DOES THE DATA VAULT
MODEL LOOK LIKE?

DATA VAULT – 3 SIMPLE
STRUCTURES
HUB
LINK
SATELLITE
EDW
Data Vault
1
2
3

DATA VAULT CORE
ARCHITECTURE
HUBS
-------------------
Unique List of
Business Keys
LINKS
-------------------
Unique List of
Relationships
across Keys
SATS
-------------------
Descriptive
Data
● Satellites have one and only one parent table
● Satellites cannot be parents to other tables
● Hubs cannot be child tables

© 2020 Empowered Holdings & Dan Linstedt. All Rights Reserved
• WHAT DOES A DATA VAULT MODEL LOOK LIKE?

1. HUB = BUSINESS KEYS
Hubs = Unique Lists of Business Keys
Business Keys are used to TRACK and IDENTIFY key information
DV 2.0 uses MD5 Hash of the BK for the PK
Natural Business Keys also acceptable for PK

2: LINKS = ASSOCIATONS
Links = Transactions and Associations
They are used to hook together multiple sets of information
In DV 2.0 the BK attributes may migrate to the Links for faster query

Tomorrow With DV
No need to change
EDW structure
The business rule can
change to a 1:M
Relationship is a 1:1
so
why model a Link?
MODELING LINKS - 1:1 or 1:M?
Today
You discover new
data later
Existing data is fine
New data is added

3. SATELLITES = DESCRIPTORS
Satellites provide context for the Hubs and the Links
Tracks changes over time - Like SCD 2
In DV 2.0 use HASH_DIFF to detect changes

• Think Type 2 SCD (Slowly Changing Dimensions)
• Old Way:
• Compare column by column
• Source value != Current value in DW table
• 20 columns, then 20 compares
• New Way:
• Concatenate all columns to one string
• Convert to one char(32) string with hash function
• Compare to hashed value (HASH_DIFF) in target table
• Does not matter how many columns
MD5-BASED CHANGE DETECTION

SNOWFLAKE
FEATURES

SEPARATION OF WORKLOADS
TO SUPPORT DATA VAULT
LOADING

Snowflake solution
• Assign each workload its own Virtual WH
• At a minimum, two WH: ETL/ELT and BI
• ETL can run continuously if it makes sense
for the business
• ETL / Business workload contention is
eliminated
• Further subdivide business or ETL
workloads into own clusters as needed
• Eliminate contention between Sales,
Marketing, Finance, Data Science
workloads
• Easy to add new workloads on demand!
SEPARATION OF
WORKLOADS
Virtual
Warehouse
Databases
Virtual
Warehouse
ETL & Data Loading
Business Workloads
Finance
Virtual
Warehouse
Test/Dev
Virtual
Warehouse
S Marketing
Virtual
Warehouse
Sales
Virtual
Warehouse
S
Research

OTHER AGILE WAREHOUSE
SCALING TECHNIQUES
Increase cluster size – on the fly!
• More data being analyzed
• More complex queries
• Get some concurrency boost
• No need to start, stop, or reboot!
Automatic scale out!
• Multi-cluster warehouse
concurrency
• Workload queries have usual weight
• But more of them, e.g. 20 dashboard
users rather than the usual 5
• Automatically scales back after
the load drops

DATA VAULT LOADING PATTERN
(BUSINESS / HASHED KEYS)
Processing:
• All processes within sets done in parallel
• Process sets ”wait” for previous set completion
• Sets run as soon as data is ready
• No other “waiting” needed
• Load dependencies maximally reduced
Vault Loads
Stage Loads
- Major Synchronization Points

APPLYING VIRTUAL WAREHOUSES TO DATA VAULT
Source to Stage Loads
•1 multi-cluster warehouse
Use as separate MCWH for each Data
Vault entity type
•Maximum parallelism
•Zero contention
•Maximum flexibility
•Each MCWH can be
independently sized

SEMI STRUCTURED
DATA & DATA VAULT

JSON SUPPORT WITH SQL – VARIANT
Apple 101.12 250 FIH-2316
Pear 56.22 202 IHO-6912
Orange 98.21 600 WHQ-6090
Structured data
(e.g. CSV)
Semi-structured data
(e.g. JSON, Avro, XML)
{ "firstName": "John",
"lastName": "Smith",
"height_cm": 167.64,
"address": {
"streetAddress": "21 2nd Street",
"city": "New York",
"state": "NY",
"postalCode": "10021-3100"
},
"phoneNumbers": [
{ "type": "home", "number": "212 555-1234" },
{ "type": "office", "number": "646 555-4567" }
]
}
Optimized storage
Flexible schema - Native
Relational processing
select v:lastName::string as last_name
from json_demo;
All Your Data!

Create Sats using a VARIANT
• Hub Key
• Load DTS
• Rec Source
• VARIANT
Load JSON doc into VARIANT column
Create a View on the Sat for Business Vault
• View presents only the JSON attributes needed for current use case
Benefits
• No change to Sat definitions if “schema” changes
• Business Vault views are unaffected
• Change or add new BV views over time as needed
COMBINE DATA VAULT & SCHEMA-IN-READ

BUILDING THE BIZ VAULT VIEW
CREATE OR REPLACE VIEW bizvault.vw_sat_account_type1 (
hub_account_hkey,
load_dts,
name,
account_type_code,
account_type,
account_type_description,
fund_bill_addr_code,
fund_bill_addr_name,
fund_bill_addr_description,
keywords,
kids,
addresses,
emails,
phones,
tags,
user_defined_fields,
attributes COMMENT '{
Accounts: "accountname INT, accountstatusid INT, accounttype INT, affiliations INT, alertids INT, announcement INT, authinfo
INT, bankaccounts INT, creditcards INT, dbid INT, extids INT, fndbilladdrid INT, fndmailaddrid INT, fndprimaryrepid BIGINT, fndrepresentatives INT,
genprimaryrepid BIGINT, genrepresentatives INT, householdincomeid INT, id INT, imageid STRING, keywords INT, kids INT, magstripeids INT,
mailingfl INT, onlinelockfl INT, orgaddresses INT, orgcomments INT, orgemails INT, orgindustryid INT, orgname INT, orgphones INT, orgtype INT,
orgurl INT, pin INT, preferredcontactmethod INT, preferredemail INT, preferredlanguageid INT, preferredphoneid INT, primarycontactid STRING,
profileids INT, salutations INT, sourceid INT, tags INT, tktbilladdrid INT, tktmailaddrid INT, tktprimaryrepid BIGINT, tktrepresentatives INT,
userdefinedfields INT, z_createduserid BIGINT, z_createduserip INT, z_createdusername INT, z_createdusersession INT, z_createdusertimestamp
TIMESTAMP, z_lastupdateduserid BIGINT, z_lastupdateduserip INT, z_lastupdatedusername INT, z_lastupdatedusersession INT,
z_lastupdatedusertimestamp TIMESTAMP, z_originationtimestamp TIMESTAMP, z_originationusername INT, z_status INT, z_tag INT, z_version
BIGINT",
Codes: "name STRING, description STRING"}'
)
COMMENT = 'Contains the latest active record for each key'

AS
WITH sat_cte AS (
SELECT
hub_account_hkey,
load_dts,
is_deleted,
attributes, -- This is the VARIANT
LEAD(load_dts) OVER (PARTITION BY hub_account_hkey
ORDER BY load_dts, load_seq) AS lead_dts
FROM
rawvault.sat_account
)

SELECT
sat.hub_account_hkey,
sat.load_dts,
sat.attributes:accountname::STRING,
sat.attributes:accounttype::STRING,
ref_account_type.attributes:name:en_US::STRING,
ref_account_type.attributes:description:en_US::STRING,
sat.attributes:fndbilladdrid::STRING,
ref_fund_bill_address.attributes:name:en_US::STRING,
ref_fund_bill_address.attributes:description:en_US::STRING,
sat.attributes:keywords::ARRAY,
sat.attributes:kids::ARRAY,
sat.attributes:orgaddresses::ARRAY,
sat.attributes:orgemails::ARRAY,
sat.attributes:orgphones::ARRAY,
sat.attributes:tags::ARRAY,
sat.attributes:userdefinedfields::ARRAY,
OBJECT_DELETE(sat.attributes,'accountname','accounttype','keywords','kids','orgaddresses’,
'orgemails','orgphones','tags','userdefinedfields’) as Attributes
FROM
sat_cte sat

LEFT JOIN
vw_ref_code_type1 ref_account_type ON
sat.attributes:accounttype::STRING = ref_account_type.code AND
ref_account_type.organization_id = '-1' AND -- Global code
ref_account_type.type = 'P' AND
ref_account_type.subtype = 'ACCT_TYPE'
LEFT JOIN
vw_ref_code_type1 ref_fund_bill_address ON
sat.attributes:fndbilladdrid::STRING = ref_fund_bill_address.code AND
sat.attributes:dbid::STRING = ref_fund_bill_address.organization_id AND
ref_fund_bill_address.type = 'P' AND
ref_fund_bill_address.subtype = 'ADDRESS'
WHERE
lead_dts IS NULL AND
is_deleted = FALSE;

“We compared before and after numbers and
found that with Snowflake there was a 90
percent reduction in the code used for the
ETL process. That’s a big win for us.”
- Paciolan’s Principal Software Engineer,
Ashkan Khoshcheshmi,

OTHER FEATURES

Allows loading several tables from one
source in one statement
USING MULTI-TABLE INSERT
INSERT ALL
WHEN (SELECT COUNT(*)
FROM dv_hub_country hc
WHERE hc.hub_country_key= stg.hash_key) = 0
THEN
INTO dv.hub_country
(hub_country_key, country_abbrv, hub_load_dts,
hub_rec_src)
VALUES (stg.hash_key, stg.country_abbrv, stg.load_dts,
stg.rec_src)
WHEN (SELECT COUNT(*)
FROM dv.sat_countries sc
WHERE sc.hub_country_key = stg.hash_key AND
sc.hash_diff = MD5(country_name)) = 0
THEN
INTO dv.sat_countries (hub_country_key, sat_load_dts,
hash_diff, sat_rec_src, country_name)
VALUES (stg.hash_key, stg.load_dts,
stg.hash_diff, stg.rec_src, stg.country_name)
SELECT MD5(country_abbrv) AS hash_key,
country_abbv,
country_name,
MD5(country_name) AS hash_diff,
CURRENT_TIMESTAMP AS load_dts,
stage_rec_src AS rec_src
FROM stage.country stg;

PERHAPS YOU WISH TO SPLIT FOR SECURITY REASONS?
From This
Schema layout
Single Snowflake Database
In Snowflake –
you can easily
split for security
reasons
To THIS!
DV Logical Partitioning
Non-sensitive
data
PII or sensitive data with
a Link to non-sensitive
data
Snowflake DB 1 Snowflake DB 2

VIRTUALIZING
INFORMATION MARTS

• Use the Hub/Link PK columns
• Filter on LEAD of LOAD_DTS - imbedded in a CTE
AS
WITH sat_cte AS (
SELECT
hub_account_hkey,
load_dts,
is_deleted,
attributes,
LEAD(load_dts) OVER
(PARTITION BY hub_account_hkey ORDER BY load_dts) AS expire_dts
FROM
rawvault.sat_account
)
TIP - EASILY GETTING CURRENT ROWS FROM A SAT

EXAMPLE – VIRTUAL CURRENT FLAG & EXPIRE DATE!
CASE
WHEN LEAD(stg.LOAD_DTS)
OVER (PARTITION BY stg.CDC_KEY
ORDER BY stg.LOAD_DTS) IS NULL
THEN 'Y'
ELSE 'N'
END CURR_FLG,
LEAD(stg.LOAD_DTS) OVER (PARTITION BY
stg.CDC_KEY ORDER BY stg.LOAD_DTS) EXPR_DTS

• Yes!
• Elastic and multi-cluster warehouses provide the power
• Dynamic optimization engine provides the performance
• Best practice
• Start with views!
• Find the right size VWH to support desired SLA
• How fast is fast enough?
• If still too slow THEN use traditional tables
• Allows agile development approach
• Fail fast!
WILL VIEWS PERFORM?

“In the past, our data vault had performance
limitations, since complex joins happened
on the same resource where data was
stored. With the separation of storage and
compute in Snowflake, that limitation goes
away.”
- GRAEME FORBES, Principal Data Engineer,
Sainsbury’s

SIMPLE REFERENCE
ARCHITECTURE

RAW Data Vault
Acquisition Business Data Vault Information Delivery
Snowflake Data Cloud
Data
Consumers
Data Lifecycle
Data Replication
Tools
Data Lake
(complement)
Bulk upload
Stream
Data Marketplace
User workspaces
MULTI-TIER
DATA VAULT ARCHITECTURE
Ad-hoc
Analysis
BI & Self-Service
Reporting
Data
Share
Data Science
Databases
Data
Providers
Bulk
File Upload
Data Lake
Apps & SaaS
Streaming data
Data Share
Customer managed
Snowflake managed
Information Marts
(physical or virtual)
Loading & Staging
Data
Share
Snowpipe, Kafka
External
Tables
Micro-batch
Direct Data Discovery
Governance, Security, DevOps
RAW Data
Structured
Semi-Structured
Unstructured
Data Share
Bridge, PIT tables
(optional)
derived DV objects
soft business rules
(physical or virtual)
RAW DV objects
hard business rules
+

Information Delivery
RAW Data Vault Business Data Vault
Data Lifecycle
DATA VAULT
VIRTUALIZATION
Loading & Staging
Data
Share
Snowpipe, Kafka
External
Tables
Micro-batch
...
Staging
Table
HUB
LINK
SAT
RAW Data
Structured
Semi-Structured
Unstructured
Data Share
Snowflake Data Cloud
Governance, Security, DevOps
Direct Data Discovery
New derived
relationship
...
View derivation
SAT
LINK
...
Any other delivery model
Optionally Materialized
XS
6XL
+
+
Bridge, PIT tables
optionally materialized
XS
6XL
business view
raw view
live raw data marts
live information marts
Data
Consumers
Ad-hoc
Analysis
BI & Self-Service
Reporting
Data
Share
Data Science

CRITICAL SUCCESS
FACTORS

IMPORTANT
SUCCESS COMPONENTS
All data
All workloads
At any scale
Simple
Training
Community
Consulting
Right Data Platform Automation
zz
Time to Value
Skills
200x
Certified!

• Several Snowflake Partners offer Data Vault classes
• Empowered Holdings (Dan Linstedt) – globally
• ScaleFree - in EMEA
• PerformanceG2 – in USA
• Certus – IN APAC
• Talk to your Snowflake account manager for contact information
• Join DataVaultAlliance.com to keep up and ask questions
DATA VAULT TRAINING & CERTIFICATION

CUSTOMER EXAMPLES

• Picnic (Amsterdam)
• Happy Money (US)
• Alpha Cruncher (EMEA)
• NYU (US)
• Sunoco (US)
• Emerald Performance
Materials (US)
• Finnair (Finland)
• Finnavia (Finland)
• NIB (Australia)
• NPS Medicinewise (Australia)
SNOWFLAKE CUSTOMERS USING DATA VAULT
• Scripps Health (US)
• PepsiCo (US)
• Hampshire Trust Bank (US)
• Paciolan (US)
• Micron (US)
• AXA (Germany)
• ResearchNow (US)
• JC Penny (US)
• ROLAND Rechtsschutz-
Versicherungs-AG (Germany)
• F+W Media (US)
• Sainsbury’s (London)
• Amica Insurance (US)

References

Inc. All Rights Reserved. Snowflake Proprietary. Not for Redistribution.
› Available on Amazon:
http://www.amazon.com/Better-
Data-Modeling-Introduction-
Engineering-ebook /dp/
B018BREV1C/
Intro ebook by
Kent Graziano:

› Available on Amazon.com
› Soft Cover or Kindle Format
› Now also available in PDF at
LearnDataVault.com
› Kent was the Technical Editor
Super Charge
Your Data Warehouse

› Available on Amazon:
http://www.amazon.com/Buildin
g-Scalable-Data-Warehouse-
Vault/dp/0128025107/
New DV 2.0 Book
from Dan Linstedt

THANK YOU

Introduction to Data Vault 2.0 on Snowflake

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Introduction to Data Vault 2.0 on Snowflake

Similaire à Introduction to Data Vault 2.0 on Snowflake (20)

Plus de Kent Graziano

Plus de Kent Graziano (13)

Dernier

Dernier (20)

Introduction to Data Vault 2.0 on Snowflake