Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Snowflake Automated Deployments / CI/CD Pipelines

1 078 vues

Publié le

• DevOps and Business Intelligence?
• CI/CD Pipelines: What are they?
• Database Deployments: State based vs Migration based
• Snowflake features for CI/CD
• Azure DevOps: Build and Release Pipelines
• Putting it all together: End to End solution
• Demo

Publié dans : Données & analyses
  • Login to see the comments

Snowflake Automated Deployments / CI/CD Pipelines

  1. 1. Automated Deployments in Snowflake Using Azure DevOps
  2. 2. Hello! Drew Hansen Data Engineer at ADESA (KAR Global) Email: drew.hansen@adesa.com Twitter: @DrewHansen_9 2 Adorable picture so you don’t have to look at me
  3. 3. Agenda ● DevOps and Business Intelligence? ● CI/CD Pipelines: What are they? ● Database Deployments: State based vs Migration based ● Snowflake features for CI/CD ● Azure DevOps: Build and Release Pipelines ● Putting it all together: End to End solution ● Demo ● Other Features we added 3
  4. 4. How do you deploy? If you need to deploy a hot fix to a table in production, what is your process? 4
  5. 5. Agile? DevOps? What’s the Hype? It is about the Business!
  6. 6. Agile vs. DevOps 6 Business Developer Technology Agile DevOps
  7. 7. 7 **Image from: https://skookum.com/blog/agile-transformation-and-devops Agile vs. DevOps
  8. 8. “ An agile culture that better supports collaboration between people that uses automation and tooling in support of process. 8 DevOps is… - Grant Fritchey PASS President (the man)
  9. 9. Continuous Integration + Continuous Deployment Cool people call it CI/CD
  10. 10. Continuous Integration ● Small incremental changes ● Shift Left ● Test often and early ○ Unit Tests ○ Integration Tests ● Outcome: BUILD Definitions Continuous Deployment ● Hands Off deployments ● Consistent throughout environments ● Outcome: RELEASE 10 By source controlling your objects you can revert back to a specific time/place. Helps developers feel confident in their changes.
  11. 11. ● Automation means less human mistakes. ● Helps developers feel confident in their changes. ● No one has access to production! Guardrails 11
  12. 12. 12 All together now
  13. 13. Look a Pipeline! 13
  14. 14. Build / Release 14 Build • Packages SQL to be run on DB • Runs Unit Tests • Produces an Artifact Artifact Release Stage CI CD • File(s) that are irrespective of environment. • Artifact will be deployed into DB • A release pipeline is process that deploys to multiple stages. • Holds environment variables. • A single deployment into an environment (DB) • Ex. 3 stages (DEV, TST, PRD)
  15. 15. Database Deployments State based vs Migration based
  16. 16. State Based ● Compare Database to Repo/Source Control. ● Deploying tool makes decision on what to deploy based on differences. ● Source control objects in a consistent State. ● Ex. CREATE TABLE AS… ● Ex. DACPAC Definitions Migration Based ● Simply deploys scripts given by the developer. ● Source control objects in a evolving state. ● Ex. ALTER TABLE AS… 16
  17. 17. Confused? Pictures help. 17 DB Project/ DACPAC DB compare generate execute Difference Script State Based Migration Based State 1 State n Migration 1 Script Migration n Script Laptop Developer updates DB Project Laptop Developer creates new script Laptop
  18. 18. State Based ● None. WOMP WOMP WOMP… Database Management Tools Available To Snowflake Migration Based ● Liquibase (XML) ● FlyWay (XML/JSON) ● Sqitch (SQL, but complicated) ● Datical (2020?) ● Make your own! 18 😭 👍
  19. 19. Snowflake Features and Considerations
  20. 20. ● Application? ○ Build the app/solution. Easy Peasy. ● Database? ○ Not so fast. Building a script is not testing. ○ You have to Deploy! ○ Annnnnnd break all the DevOps rules we just discussed. ○ BUT… How in the world do you test a deployment? 20
  21. 21. Spoiler: Snowflake Wins… 21
  22. 22. Testing in Snowflake Zero-Copy Cloning ● Create clones of tables, schemas, or entire databases instantly ● Requires NO additional storage ● Adds no extra load on the Production users 22 PRD TESTING_BUILD
  23. 23. What does that mean? 23 ● I can Create an Artifact (in the Build Process) and Test it (still in the Build) by Cloning production and trying to deploy to the clone. ● I can know 100% that the artifact file that is being deployed in the TESTING_BUILD clone will be deployed successfully to PRD. ● BOOM.
  24. 24. How to Connect to Snowflake What connector should we use?
  25. 25. Snowflake Connectors ● Tons to choose from!
  26. 26. SnowSQL (CLI) vs. Python 27 ● SnowSQL is a command line interface (CLI) built off the Python Connector. ● Python cannot run multiple statements. ● SnowSQL can run files with over 100 separate queries and over 10K lines of code. ● Once installed, SnowSQL is super simple to use and easy to configure. ● We choose SnowSQL!
  27. 27. SnowSQL Config File 28 ● After installation of SnowSQL, a file called config is created. ○ C:Usersuser.name.snowsqlconfig ● In this file, you can set up connections, variables, and other configuration settings. ● Connections:
  28. 28. SnowSQL Configuration Settings 29 ● Helpful config settings: ○ exit_on_error ■ exists the program immediately when there is an error. ■ If we deploy something and it fails. STOP! DROP! And ROLLBACK! ○ output_file ■ Instead of outputting to the terminal, it will write to a log file. ○ sfqid ■ Output the Snowflake Query ID. Awesome for troubleshooting. ○ output_format ■ Change the look of how the output is formatted, grid vs. txt. ○ friendly ■ Snowflake greets you with hello and goodbye messages… But Snowflake isn’t my only friend, so I don’t need this.
  29. 29. Azure DevOps Planning / Repo / Pipelines all in one place!
  30. 30. Azure DevOps. One Stop Shop(s).  sick rhyme Boards With the Azure Boards web service, teams can manage their software projects. It provides a rich set of capabilities including native support for Scrum and Kanban, customizable dashboards, and integrated reporting. Repo Azure Repos is a set of version control tools that you can use to manage your code. Testing The testing suite contains tools for both exploratory and manual testing as well as automated continuous testing. 31 Pipelines Azure Pipelines is a cloud service that you can use to automatically build and test your code project and make it available to other users. It works with just about any language or project type.
  31. 31. End to End From Developer Typing Code to Deploy into Snowflake Production.
  32. 32. 33 How do we get there?
  33. 33. 34 Pieces to the Puzzle 1. Source Control. 2. Put Process into place (PRs). 3. Decide how to build artifacts to deploy. 4. Decide stages and gates. 5. Deploy using SnowSql.
  34. 34. Set up your Repo 35 ● It is crucial to use source control! ● Benefits: ○ Log of every change to your database for all time. ○ Devs can work on the same objects at the same time. ○ Able to trace all changes in a project management tool (Azure DevOps boards).
  35. 35. Pull Requests ● Created when the developer is ready to promote their code. ● Reviewed by peers, architect, management, etc. ● Benefits: ○ Asynchronous Code Reviews ○ Can add comments and follow up later. ○ Team knows what is going into Prod. ○ No one can merge into master directly! 36
  36. 36. Build Pipeline ● Purpose: Build Artifacts! ● Steps: 1. Download latest repo (master). 2. Create Artifact. 3. Install SnowSQL. 4. Clone PRD database. 5. Test Deploy Artifact. 6. Publish Artifact. 37
  37. 37. Release Pipeline 38 ● Purpose: Deploy Artifact to Stages! ● Steps: 1. Download Artifact. 2. Replace Environment Variables. 3. Install SnowSQL. 4. Deploy Artifact to DEV. 5. Deploy Artifact to TST. 6. Final Approval. 7. Deploy Artifact to PRD.
  38. 38. Demo Bout time, am I right?
  39. 39. Other Features we added ● Both Change and Rollback Scripts ● Run in Rollback-only Mode. ● Write all Deployment Logs to Storage Container via Azure CLI. 40
  40. 40. Other Features… ● Split CICD Pipeline into two: Lower and Upper. ○ Lower: ■ Deploys to DEV. ■ Used for Devs to test deployments. ■ Can be run on-demand. ○ Upper: ■ Deploys to TST and PRD. ■ Locked Down. ■ Only run by merge into master (PR). ● Next Big Enhancement: ○ Automated Testing! 41
  41. 41. 42 Thanks! Any questions? Email me: drew.hansen@adesa.com Connect with me: https://www.linkedin.com/in/drew-hansen-09806289/