Azure Key Vault, Azure Dev Ops and Techno Azure Data Factory ,how do these Azure Services work perfectly together!

1. Azure Key Vault, Azure Dev Ops and Azure Data Factory
how do these Azure Services work perfectly together!
Erwin de Kreuk
Microsoft Solution Architect
@erwindekreuk
https://erwindekreuk.com

2. AZURE DATA FACTORY

3. WHAT IS AZURE DATA FACTORY?
Hybrid data integration service
With visual tools, you can build, debug, deploy, operationalize and
monitor your (big) data pipelines
Provides a way to transform data at scale without any coding required.
ELT Platform

4. “DevOps is the union of people, process,
and products to enable continuous
delivery of value to your end users”
Donovan Brown

5. WHAT IS AZURE DEVOPS

7. WHAT IS AZURE KEYVAULT?
The safehouse to safeguard the cryptographic keys and secrets that are used
by your applications, servers and cloud applications
Fully integrated with Azure Active Directory
Secret management: Securely store and tightly control access to tokens, passwords, certificates,
API keys, and other secrets.
Key management: Create and control encryption keys that encrypt your data.
Certificate management: Provision, manage, and deploy public and private Secure Sockets
Layer/Transport Layer Security (SSL/TLS) certificates for use with Azure and your internal
connected resources.

8. ROLES IN AZURE KEYVAULT?
Deploys Application
Tells de application the URI of the key /
secret
Configures the application to use key /
secret (and may abuse), but never sees
the keys
AuditorDeveloperSecurity Operations
Manages Keys
Creates a Key Vault in Azure
Add keys / secrets to the Key Vault
Grants Permission to specific
applications to perform specific
operations using keys
Enables usage logs
Monitors access to keys
Reviews usage logs to confirm
proper key use and compliance with
data security standards

9. CREATING YOUR AZURE DATA FACTORY
demoedkazudvlmadfbicc01 demoedkazutestadfbicc01
Create a Data Factory for every environment
Define naming conventions
customername-azu-environment-azureservice-department
$SubscriptionName = “XXXXXXXXXXXXXXXXX"
$ResourceGroupName = "DEMOEDK-EUW-BICC-DVLM-RG-ENVIRONMENT"
$DataFactoryNameName="demoedkazudvlmadfbicc01"
$Location="westeurope"
Connect-AzAccount
Select-AzSubscription -SubscriptionName $SubscriptionName
New-AzDataFactoryV2 -Name $DataFactoryNameName -
ResourceGroupName $ResourceGroupName -Location $Location

10. SET UP CODE REPOSITORY

11. SET UP CODE REPOSITORY
Repository Settings

12. AZURE DATA FACTORY
Create linked Services:
Azure SQL DB
LS_ASQL_ENV01_WWI
Azure Blob Storage
LS_ABLB_ENV01
Directly saved in Azure DevOps !!

13. AZURE DATA FACTORY
Create linked Services:
Azure SQL DB
LS_ASQL_ENV01_WWI
Azure Blob Storage
LS_ABLB_ENV01
Directly saved in Azure DevOps !!
More naming conventions check:
https://erwindekreuk.com/2019/04/azure-data-factory-naming-conventions/

14. AZURE DATA FACTORY
Create Datasets:
DS_ABLB_ENV01
DS_ASQL_ENV01_WWI

15. AZURE DATA FACTORY
Create Pipeline:
PL_COPY_WWI_ABLB
DS_AQS
L_ENV01
_WWI
DS_ABL
B_ENV0
1

16. Demo

17. CREATING YOUR AZURE KEY VAULT
demoedkazudvlmadfbicc01 demoedkazutestadfbicc01

18. CREATING YOUR AZURE KEY VAULT
Demoedkazudtapkeyvault01
$SubscriptionName = “XXXXXXXXXXXXXXXXX"
$ResourceGroupName = "DEMO-EUW-DTAP-RG-ENVIRONMENT"
$KeyvaultName="Demoedkazudtapkeyvault02"
$Location="West Europe"
Connect-AzAccount
Select-AzSubscription -SubscriptionName $SubscriptionName
New-AzKeyVault -Name $KeyvaultName -ResourceGroupName $ResourceGroupName -location $Location

19. CONNECTING KEYVAULT IN ADF
Create a new Linked Service

20. CONNECTING KEYVAULT IN ADF
Grant the Azure Data Factory access to your Azure KeyVault.
Be aware that you need to publish the Azure KeyVault before you start using secrets !

21. CREATING SECRETS IN AZURE KEY VAULT
DVLM-ABLB-ENV01
TEST-ABLB-ENV01
DVLM-ASQL-ENV01-WWI
TEST-ASQL-ENV01-WWI
The use of naming conventions ensures that we can
configure everything more dynamically in the release
process later on !
ENVIRONMENT-CONNECTION-
REFERENCE
Environment
Development DVLM
Test TEST
Acceptance ACPT
Production PROD
Linked Service
Azure Blob Storage ABLB
Azure Cosmos DB SQL API ACSA
Azure Cosmos DB MongDB API ACMA
Azure Data Explorer ADEX
Azure Data Lake Storage Gen1 ADLS
Azure Data Lake Storage Gen2 ADLS
Azure Database for MariaDB AMDB
Azure Database for MySQL AMYS
Azure Database for PostgreSQL APOS
Azure File Storage AFIL
Azure Search ASER
Azure SQL Database ASQL
Azure SQL Database Managed Instance ASQM
Azure SQL Data Warehouse ASDW
Azure Table Storage ATBL
SQL Server MSQL
Reference
DatabaseConnection LogicalServer(Short)-DatabaseName
Storage
StorageName like
Audit/SSIS/Staging/DataLake

22. Demo

23. CREATING YOUR AZURE DATA FACTORY
demoedkazudvlmadfbicc01 demoedkazutestadfbicc01

24. CREATING LIBRARY IN AZURE DEV OPS
Configure the correct values

25. CREATE RELEASE IN AZURE DEV OPS

26. CREATE RELEASE IN AZURE DEV OPS
Create a Release based on an Empty Job

27. CREATE RELEASE IN AZURE DEV OPS
Create a Release based on an Empty Job
Create a Stage for each Environment
Test
Acceptance
Production

28. CREATE RELEASE IN AZURE DEV OPS
Create a Release based on an Empty Job
Create a Stage for each Environment
Add an Artifact

29. CREATE RELEASE IN AZURE DEV OPS
Create a Release based on an Empty Job
Create a Stage for each Environment
Add an Artifact
Enable Continuous Deployment

30. CREATE RELEASE IN AZURE DEV OPS
Create a Release based on an Empty Job
Create a Stage for each Environment
Add an Artifact
Enable Continuous Deployment

31. CREATE TASK IN RELEASE IN AZURE DEV OPS
Define an Agent

32. CREATE TASK IN RELEASE IN AZURE DEV OPS
Define an Agent
Add the task

33. CREATE TASK IN RELEASE IN AZURE DEV OPS
Define an Agent
Add the task
Configure the task

34. CREATE TASK IN RELEASE IN AZURE DEV OPS
Define an Agent
Add the task
Configure the task
Override the Template Parameters

35. RELEASE IN AZURE DEV OPS
Connecting Library
Release name format

40. Demo

41. CREATING YOUR AZURE DATA FACTORY
demoedkazudvlmadfbicc01 demoedkazutestadfbicc01

42. CUSTOM PARAMETERS
Not all the properties have parameters by default.
arm-template-parameters-definition.json
Adding accountkey for DataLake with secrets from Azure Key Vault
“accountkey": “|",
Download file from:
https://docs.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment#default-parameterization-template
= means keep the current value as the default value for the parameter.
- means don't keep the default value for the parameter.
| is a special case for secrets from Azure Key Vault for connection strings or keys.

43. REMARKS
You cannot publish while SSIS Integration Runtime is running
Triggers needs to be disabled and enabled afterwards
Every time you change something to your connection you need to update the
Template parameter
Deletes are handled manually or through Custom PS script or Marketplace extension
Resource:
https://docs.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment#linked-resource-manager-templates

44. HANDLING DELETES
Create a post and pre deployment step in your Release

45. Demo

46. Thank youDon’t forget to leave your feedback
Erwin de Kreuk
Microsoft Solution Architect
Axians
@erwindekreuk
https://erwindekreuk.com

47. QUESTIONS
Erwin de Kreuk
Microsoft Solution Architect
Axians
@erwindekreuk
https://erwindekreuk.com