SlideShare utilise les cookies pour améliorer les fonctionnalités et les performances, et également pour vous montrer des publicités pertinentes. Si vous continuez à naviguer sur ce site, vous acceptez l’utilisation de cookies. Consultez nos Conditions d’utilisation et notre Politique de confidentialité.
SlideShare utilise les cookies pour améliorer les fonctionnalités et les performances, et également pour vous montrer des publicités pertinentes. Si vous continuez à naviguer sur ce site, vous acceptez l’utilisation de cookies. Consultez notre Politique de confidentialité et nos Conditions d’utilisation pour en savoir plus.
• DevOps and Business Intelligence?
• CI/CD Pipelines: What are they?
• Database Deployments: State based vs Migration based
• Snowflake features for CI/CD
• Azure DevOps: Build and Release Pipelines
• Putting it all together: End to End solution
Using Azure DevOps
Data Engineer at ADESA (KAR Global)
Adorable picture so you
don’t have to look at me
● DevOps and Business Intelligence?
● CI/CD Pipelines: What are they?
● Database Deployments: State based vs Migration based
● Snowflake features for CI/CD
● Azure DevOps: Build and Release Pipelines
● Putting it all together: End to End solution
● Other Features we added
If you need to deploy a hot fix to a table in
production, what is your process?
Agile? DevOps? What’s the
It is about the Business!
Agile vs. DevOps
Business Developer Technology
**Image from: https://skookum.com/blog/agile-transformation-and-devops
Agile vs. DevOps
“ An agile culture that better
supports collaboration between
people that uses automation and
tooling in support of process.
- Grant Fritchey
PASS President (the man)
Continuous Integration + Continuous Deployment
Cool people call it CI/CD
● Small incremental changes
● Shift Left
● Test often and early
○ Unit Tests
○ Integration Tests
● Outcome: BUILD
● Hands Off deployments
● Consistent throughout
● Outcome: RELEASE
By source controlling your objects you can revert back
to a specific time/place.
Helps developers feel confident in their changes.
● Automation means less human mistakes.
● Helps developers feel confident in their changes.
● No one has access to production!
Build / Release
• Packages SQL to be
run on DB
• Runs Unit Tests
• Produces an Artifact
Artifact Release Stage
• File(s) that are
• Artifact will be
deployed into DB
• A release pipeline is
process that deploys to
• Holds environment
• A single deployment
into an environment
• Ex. 3 stages (DEV, TST,
State based vs Migration based
● Compare Database to
● Deploying tool makes
decision on what to deploy
based on differences.
● Source control objects in a
● Ex. CREATE TABLE AS…
● Ex. DACPAC
● Simply deploys scripts given by
● Source control objects in a
● Ex. ALTER TABLE AS…
Confused? Pictures help.
State Based Migration Based
State 1 State n
WOMP WOMP WOMP…
Database Management Tools
Available To Snowflake
● Liquibase (XML)
● FlyWay (XML/JSON)
● Sqitch (SQL, but complicated)
● Datical (2020?)
● Make your own!
○ Build the app/solution. Easy Peasy.
○ Not so fast. Building a script is not testing.
○ You have to Deploy!
○ Annnnnnd break all the DevOps rules we
How in the world do you
test a deployment?
Testing in Snowflake
● Create clones of tables,
schemas, or entire databases
● Requires NO additional
● Adds no extra load on the
What does that mean?
● I can Create an Artifact (in the Build Process) and Test it (still in
the Build) by Cloning production and trying to deploy to the
● I can know 100% that the artifact file that is being deployed in the
TESTING_BUILD clone will be deployed successfully to PRD.
How to Connect to Snowflake
What connector should we use?
SnowSQL (CLI) vs. Python
● SnowSQL is a command line interface (CLI) built off the Python
● Python cannot run multiple statements.
● SnowSQL can run files with over 100 separate queries and over
10K lines of code.
● Once installed, SnowSQL is super simple to use and easy to
● We choose SnowSQL!
SnowSQL Config File
● After installation of SnowSQL, a file called config is created.
● In this file, you can set up connections, variables, and other
SnowSQL Configuration Settings
● Helpful config settings:
■ exists the program immediately when there is an error.
■ If we deploy something and it fails. STOP! DROP! And ROLLBACK!
■ Instead of outputting to the terminal, it will write to a log file.
■ Output the Snowflake Query ID. Awesome for troubleshooting.
■ Change the look of how the output is formatted, grid vs. txt.
■ Snowflake greets you with hello and goodbye messages… But
Snowflake isn’t my only friend, so I don’t need this.
Planning / Repo / Pipelines all in one place!
One Stop Shop(s). sick rhyme
With the Azure Boards
web service, teams can
manage their software
projects. It provides a
rich set of capabilities
including native support
for Scrum and Kanban,
Azure Repos is a set
of version control
tools that you can
use to manage your
The testing suite
contains tools for
and manual testing
as well as
Azure Pipelines is a
cloud service that you
can use to
and test your code
project and make it
available to other
users. It works with
just about any
language or project
End to End
From Developer Typing Code to Deploy into Snowflake Production.
Pieces to the Puzzle
1. Source Control.
2. Put Process into place (PRs).
3. Decide how to build artifacts to deploy.
4. Decide stages and gates.
5. Deploy using SnowSql.
Set up your Repo
● It is crucial to use source control!
○ Log of every change to your database for all time.
○ Devs can work on the same objects at the same time.
○ Able to trace all changes in a project management tool
(Azure DevOps boards).
● Created when the developer is ready to promote their code.
● Reviewed by peers, architect, management, etc.
○ Asynchronous Code Reviews
○ Can add comments and follow up later.
○ Team knows what is going into Prod.
○ No one can merge into master directly!
Other Features we added
● Both Change and Rollback Scripts
● Run in Rollback-only Mode.
● Write all Deployment Logs to Storage Container via Azure CLI.
● Split CICD Pipeline into two: Lower and Upper.
■ Deploys to DEV.
■ Used for Devs to test deployments.
■ Can be run on-demand.
■ Deploys to TST and PRD.
■ Locked Down.
■ Only run by merge into master (PR).
● Next Big Enhancement:
○ Automated Testing!
Email me: firstname.lastname@example.org
Connect with me: https://www.linkedin.com/in/drew-hansen-09806289/