Snowflake: The most cost-effective agile and scalable data warehouse ever!

© 2019 Visual BI Solutions, Inc. All rights reserved. www.visualbi.com
Snowflake: The Most Cost-Effective, Agile
and Scalable Data Warehouse Ever!
Chris Holliday
SVP, Client Management www.visualbi.com

© 2019 Visual BI Solutions, Inc. All rights reserved. www.visualbi.com 2
This presentation outlines our general product direction and should not be relied on in making a purchase decision. This presentation is not
subject to your license agreement or any other agreement with Visual BI Solutions. Visual BI Solutions has no obligation to pursue any course of
business outlined in this presentation or to develop or release any functionality mentioned in this presentation.
This presentation and Visual BI Solution’s strategy and possible future developments are subject to change and may be changed by Visual BI
Solutions at any time for any reason without notice. This document is provided without a warranty of any kind, either express or implied, including
but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. Visual BI Solutions assumes no
responsibility for errors or omissions in this document, except if such damages were caused by Visual BI Solutions intentionally or grossly
negligent.
Visual BI Solutions
LEGAL DISCLAIMER

ABOUT VISUAL BI
ALL-IN-ONE PARTNER FOR END-TO-END BI & ANALYTICS NEEDS
Trusted by
the industry
Our
Partners
Visual BI Solutions is a leading All-in-One Business Intelligence (BI) enablement firm specializing in BI &
Analytics services, solutions, training and products. We have proven expertise in enabling BI & Analytics for
100+ world’s leading brands.
Trusted by leading companies world-wide
CONSULTING SERVICES
• Strategy
• Architecture Implementation
• Managed Services
• Visualization
• Cloud Migrations
• Training
SOFTWARE PRODUCTS
• ValQ - Modern Digital Planning
• VBI View – Your Search Engine for All
Your BI Content
• VBX – Product Extensions for SAP Lumira
TRAINING
• Microsoft Power BI
• Tableau
• SAP Analytics Cloud
• SAP BW4HANA / BusinessObjects

Webinar
AGENDA
Introductions01
Snowflake – data warehousing, the Enterprise and platforms
dbt – rethinking the way data is transformed
Enterprise data flows – modern, efficient and agile
05
06
07
Q&A
End to end demo
02
03
04
05
06

INTRODUCTIONS
Ryan Goltz: Chesapeake Energy
Tristan Handy: dbt / Fishtown Analytics
Chris Holliday: Visual BI

SNOWFLAKE

What is Snowflake?
SNOWFLAKE
§ Completely new architecture from the
ground up
§ Complete SaaS – no tuning, indexing,
provisioning, upsizing, downsizing or
maintenance
§ 3 layers: Storage, Query Processing
and Cloud Services
§ Columnar MPP hybrid of shared-disk
and shared-nothing architecture
§ Seamlessly scales up and down

What is Snowflake?
SNOWFLAKE
§ Separation of storage
& compute
§ All workloads get
proper, isolated
resources
§ Handles spikes
§ No hardware
§ Fast time to value
§ Less maintenance
§ Pay as you use
§ Complete SQL
§ Unlimited queries
& users
§ Fresh and
complete data
§ Optimize revenue
& costs
All data centralized,
shareable
Simple, cost effective
infrastructure
Faster, better
decision making
§ All data, structured
and semi-structured
§ Single source of truth
§ Securely share live
data with partners
Unlimited, automatic
scale and speed

What Snowflake is not…
SNOWFLAKE
§ Kimball? Inmon? Data Vault? Reinvented Wheel? Snowflake doesn’t care.
§ Federated security? Centralized security? Row-level security? Snowflake supports what you need but Snowflake doesn’t
care.
§ Best practices? Worst practices? No practices? Snowflake doesn’t care.
Opinionated
§ GUI-based drag and drop table mappings? Sure.
§ Code-based versionable and testable SQL-based transforms? Sure.
§ 100% custom perl-scripts and chron jobs? Sure.
§ One table to rule them all and handle everything in the reporting layer? Sure (but please don’t)
An Integration
Platform
Expensive § It is not…. Remember?
Difficult to Use § Ask anyone. It just works and it works every time.
Reporting Tool
OLAP Layer
§ Technically you could, but please don’t!
§ If you need a cube, you will need to use something on top of Snowflake

Snowflake for the Enterprise
SNOWFLAKE
INTEGRATION TOOLS
Apps S3/Azure Blob
Store
Alteryx MuleSoft
TRANSFORM TOOLS
BODS Informatica dbt Talend
Big Data Tools
Reporting
Tools
Finance Sales Supply Chain Etc.,
Enterprises
Security Governance Management

Decisions, decisions – platforms, pipelines and tooling
SNOWFLAKE
Platforms: AWS, Azure and GCP (Q4)
AWS: S3-backed. More mature and with more zones.
Azure: Blob Store-backed. Near 100% parity with AWS.
GCP: Will be coming online for public use in Q4
Cloud native integration tools: Fivetran, Alooma, Stitch, Azure Data Factory
Cloud-based integration tools: Matillion, Talend, Wherescape
Server native tools: Informatica, SSIS,
Pipelines: Different apps and scenarios require different tools
§ Cloud SaaS apps (Salesforce, Netsuite, MS Dynamics)
§ Many cloud integration tools like Fivetran, Matillion, Alooma which
have pre-built integrations and subject areas
⇢ Pros – 5 minute deployment
⇢ Cons – General lack of flexibility
§ ERP apps
§ Some cloud

Decisions, decisions – platforms, pipelines and tooling
SNOWFLAKE
INGESTION
CSV ORC
JSON XML
S3/Blob Store
Google Cloud
Snowpipe
Cloud Storage Files Connectors
JAVA
NodePython .Net
Spark
Kafka
Connector
TRANSFORM
Scripts & Cron
Jobs
SQL
Informatica SSRS
Command Line/
Snowflake
worksheets
GUI Tools
Cloud Saas

What is dbt?
dbt
dbt (data build tool) is the “T” in the modern ELT stack. It takes SELECT statements and deploys them
into your warehouse as tables and views.
• SQL and Jinja (python template framework for more complex logic) so very short learning curve
• Natively graph-aware! Construct complex DAGs with ease.
• Everything is coded in your favorite IDE which makes it:
⇢ Versionable
⇢ Testable
⇢ CI/CD workflow incorporation (spawn events and processes from code commits)
⇢ Built in logging and environment management
• Includes a rich testing framework (schema and data tests)
• Unopinionated on the data model
• Has data model visualization and documentation built-in
• Open source! Lots of traction and velocity. github.com/fishtown-analytics/dbt

Who’s using dbt today?
dbt
§ ~ 1,000 companies ran dbt in
production in July, including these
folks.
§ Adoption is growing fast.
§ Snowflake is the most commonly
paired data warehouse with roughly
35% of dbt’s user base.
§ Snowflake + dbt fit together
incredibly well due to the separation
of storage and compute and the
elastic nature of Snowflake
resources.

dbt is not a data ingestion tool!
dbt
Get your data into Snowflake using…
⇢ Snowpipe
⇢ COPY commands
⇢ External tables
⇢ Stitch, Fivetran, etc.
Once the data is in Snowflake, transform it, snapshot it, test it, and document it with dbt.

Focus on: dbt Docs
dbt
⇢ dbt projects are fully self-documenting, including
graph visualization and data dictionaries.
⇢ dbt Docs provides a single source of truth for data and
a mechanism for users to explore data lineage: “Where
does this data come from?”
Graph viz
Data dictionary

CI/CD and dbt Cloud
dbt
dbt Cloud is a fully-managed
service to develop and deploy
dbt projects:
⇢ Schedule recurring jobs or trigger
via the API
⇢ View logs, get alerts
⇢ Natively integrates with git,
enabling full CI/CD
⇢ Full graphical IDE (Soon!)
⇢ Share documentation throughout
your org
⇢ Deployable on-prem or in our cloud

Challenges solved in Enterprise environments
dbt
Democratization of data engineering
⇢ Analysts, power users and anyone who can write SQL can now create and edit sophisticated data flows
⇢ Managing code using a git repository enables a natural and familiar governance model
⇢ Integrated testing ensures data quality, leading to greater trust from downstream users
⇢ Integrated documentation promotes transparency for all stakeholders
⇢ Open source prevents lock-in for this key organizational knowledge
In short:
⇢ Faster iteration
⇢ More reliable data
⇢ More trust
⇢ More transparency

ENTERPRISE DATA
FLOWS

Building an Enterprise Data Warehouse with Snowflake and dbt
Enterprise data flows
Snowflake and dbt can be used by your traditional data warehouse team to more quickly and easily build data the business will love
Make sure you can ingest legacy and cloud datasets
→ We did not anticipate demand for data from SaaS applications
Once you get past the exotic pipelines, get back to the basics of warehouse architecture!
§ Clearly separate different types of data (Source, Enterprise) and data in different states (Transform, Stage)
→ Databases and schemas are free, use them!
§ Build mechanisms for Data Quality
§ If you perform straight ELT, issues become “all or nothing”
§ Make sure Data Quality is:
→ Clearly traceable to the Transformation
→ Visible to Stewards
§ Develop your strategies for identification and relationships
Assign your workloads
→ Run separate workloads for ingest, transform, BI, etc.
→ Use clusters where things get chatty, such as ingest/source
Get past your prior assumptions
§ Incremental is overrated & DBT makes truncate/load easy
→ From Oracle in 48 hours to Snowflake in 3 minutes
Spend all your time on great data models, not ETL tuning!!!

Enabling Self Service with Snowflake and dbt
Enterprise data flows
With the right Architecture, Snowflake and dbt can democratize your data
Traditionally, the Enterprise Data Warehouse was the sole purveyor of data because:
§ The tools used to load the system were complicated dbt is the SQL your users know
§ Loads and queries compete for resources Snowflake provides isolation through “warehouses”
§ Performance required complex design choices Snowflake addresses most use cases without tuning
Align with Enterprise Architecture
§ Build “warehouses” for existing value chains or business groups
Define a governance model
§ Use Pull Requests to review for suitability
§ IT and/or Stewards should review changes
→ Are we using the right definitions? Should we create a shared definition??
§ Use CI/CD to address Change Authorization
§ Make your Audit team love you with Automated Testing
→ Don’t run complicated manual tests every quarter…
→ Run automated tests every time you load!
Use IT resources to enable and optimize self service
§ Train users
§ Develop best practices
§ Enforce standards

DEMO AND EXPLORE

Deck flow page
Demo
SAP -> Snowflake with transformation and scheduling
Demo ODP enabled data source
S/4
Connect with
Data Factory
Dump Results to
Blob Store
Load to
RAW
Stage
dbt
§ Run
§ Docs
§ Test
§ Update
§ Re-run
S/4
Power BI

Q & A

www.visualbi.com
THANK YOU!
Chris Holliday
chris@visualbi.com
www.visualbi.com
SVP, Client Mangement

Snowflake: The most cost-effective agile and scalable data warehouse ever!

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Snowflake: The most cost-effective agile and scalable data warehouse ever!

Similar to Snowflake: The most cost-effective agile and scalable data warehouse ever! (20)

More from Visual_BI

More from Visual_BI (7)

Recently uploaded

Recently uploaded (20)

Snowflake: The most cost-effective agile and scalable data warehouse ever!