Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
WIFI SSID:SparkAISummit | Password: UnifiedAnalytics
Premal Shah, Microsoft
Creating continuous integration pipelines on
Azure using Azure Databricks and Azure DevOps
#Unified...
What is DevOps?
3#UnifiedAnalytics #SparkAISummit
What is DevOps?
4#UnifiedAnalytics #SparkAISummit
People. Process. Products.
DevOps is the union of
people, process, and
p...
What is Azure Databricks?
5#UnifiedAnalytics #SparkAISummit
Increase productivity
Build on a secure, trusted cloud
Scale w...
Azure DevOps
6#UnifiedAnalytics #SparkAISummit
7#UnifiedAnalytics #SparkAISummit
Azure DevOps: Choose what you like
Any Language, Any Platform
8#UnifiedAnalytics #SparkAISummit
Azure
Databricks
Dev WS
Push
notebooks
to Azure
Devops
Azure
DevOps
Repo
Build Pipeline
...
9#UnifiedAnalytics #SparkAISummit
Azure
Databricks
Dev WS
Azure
DevOps
Repo
Build Pipeline
Artifact
Release
Pipeline
Run w...
Azure Databricks REST API/CLI
• Provides an easy-to-use interface to the Azure
Databricks platform. CLI (open source proje...
Demo
#UnifiedAnalytics #SparkAISummit
DevOps for ML: Goals
• Repeatability of model creation & behavior
• Evaluation of model predictions
• Managing different m...
13#UnifiedAnalytics #SparkAISummit
Demo
#UnifiedAnalytics #SparkAISummit
Summary
• Two approaches
• Implementation in Azure Notebooks
• Implementation in IDE
• Azure DevOps to build CI/CD pipelin...
Call to action
• Build a CI/CD pipeline using Azure DevOps
• Azure Databricks documentation
• Azure DevOps pipelines
• Inc...
THANK YOU!
“It is not the answer that enlightens, but the
question”
Eugene Ionesco
#UnifiedAnalytics #SparkAISummit
DON’T FORGET TO RATE
AND REVIEW THE SESSIONS
SEARCH SPARK + AI SUMMIT
Prochain SlideShare
Chargement dans…5
×

DevOps for Applications in Azure Databricks: Creating Continuous Integration Pipelines on Azure Using Azure Databricks and Azure DevOps

2 562 vues

Publié le

Working with our customers, developers and partners around the world, it's clear DevOps has become increasingly critical to a team's success. Continuous integration (CI) and continuous delivery (CD) which is part of DevOps, embody a culture, set of operating principles, and collection of practices that enable application development teams to deliver code changes more frequently and reliably. In this session, we will cover how you can automate your entire process from code commit to production using CI/CD pipelines in Azure DevOps for Azure Databricks applications. Using CI/CD practices, you can simplify, speed and improve your cloud development to deliver features to your customers as soon as they're ready.

Publié dans : Données & analyses
  • Login to see the comments

DevOps for Applications in Azure Databricks: Creating Continuous Integration Pipelines on Azure Using Azure Databricks and Azure DevOps

  1. 1. WIFI SSID:SparkAISummit | Password: UnifiedAnalytics
  2. 2. Premal Shah, Microsoft Creating continuous integration pipelines on Azure using Azure Databricks and Azure DevOps #UnifiedAnalytics #SparkAISummit
  3. 3. What is DevOps? 3#UnifiedAnalytics #SparkAISummit
  4. 4. What is DevOps? 4#UnifiedAnalytics #SparkAISummit People. Process. Products. DevOps is the union of people, process, and products to enable continuous delivery of value to your end users. “ ” Build & Test Continuous Delivery Deploy Operate Monitor & Learn Plan & Track Develop Donovan Brown, MSFT PM
  5. 5. What is Azure Databricks? 5#UnifiedAnalytics #SparkAISummit Increase productivity Build on a secure, trusted cloud Scale without limits Built with your needs in mind Enterprise grade Azure security Native integration with Azure services Live collaboration Enterprise-grade SLAs E2E data pipelines using ADF Integrated billing
  6. 6. Azure DevOps 6#UnifiedAnalytics #SparkAISummit
  7. 7. 7#UnifiedAnalytics #SparkAISummit Azure DevOps: Choose what you like Any Language, Any Platform
  8. 8. 8#UnifiedAnalytics #SparkAISummit Azure Databricks Dev WS Push notebooks to Azure Devops Azure DevOps Repo Build Pipeline Artifact Release Pipeline Deploy Notebook to staging Azure Databricks Staging WS Execute Tests Deploy Notebook to prod Azure Databricks Prod WS Implementation in Azure Databricks Notebooks
  9. 9. 9#UnifiedAnalytics #SparkAISummit Azure Databricks Dev WS Azure DevOps Repo Build Pipeline Artifact Release Pipeline Run with staging cluster Azure Databricks Staging WS Execute Tests Run with Prod cluster Azure Databricks Prod WS Implementation in IDE (PyCharm, IntelliJ) DB Connect
  10. 10. Azure Databricks REST API/CLI • Provides an easy-to-use interface to the Azure Databricks platform. CLI (open source project) is built on top of the REST APIs – Workspace API • Deploy notebooks from Azure DevOps to Azure Databricks – DBFS API • Deploy libraries from Azure DevOps to Azure Databricks – Jobs API • Execute notebooks and Spark code once deployed 10#UnifiedAnalytics #SparkAISummit
  11. 11. Demo #UnifiedAnalytics #SparkAISummit
  12. 12. DevOps for ML: Goals • Repeatability of model creation & behavior • Evaluation of model predictions • Managing different model versions and files • Operationalization of the model • Monitoring of training and scoring pipelines 12#UnifiedAnalytics #SparkAISummit
  13. 13. 13#UnifiedAnalytics #SparkAISummit
  14. 14. Demo #UnifiedAnalytics #SparkAISummit
  15. 15. Summary • Two approaches • Implementation in Azure Notebooks • Implementation in IDE • Azure DevOps to build CI/CD pipelines (you can selectively use) • REST/CLI APIs • Model CI/CD on Azure • Azure Databricks: Data preparation and model training • Azure ML: Model deployment and management • Azure DevOps: CI/CD pipeline 15#UnifiedAnalytics #SparkAISummit
  16. 16. Call to action • Build a CI/CD pipeline using Azure DevOps • Azure Databricks documentation • Azure DevOps pipelines • Incorporate DevOps in your Azure Databricks implementation 16#UnifiedAnalytics #SparkAISummit
  17. 17. THANK YOU! “It is not the answer that enlightens, but the question” Eugene Ionesco #UnifiedAnalytics #SparkAISummit
  18. 18. DON’T FORGET TO RATE AND REVIEW THE SESSIONS SEARCH SPARK + AI SUMMIT

×