Can we store our Connectionstrings or BlobStorageKeys or other Secretvalues somewhere else then in Azure Data Factory(ADF)? Yes you can! You can store these valuable secrets in Azure Key Vault(AKV). But how can we achieve this in ADF? And finally how do we deploy our DataFactories in Azure Dev Ops to Test, Acceptance and Production environments with these Secrets ? Can this be setup dynamically? During this session I will give answers on all of these questions. You will learn how to setup your Azure Key Vault, connect these secrets in ADF and finally deploy these secrets dynamically in Azure Dev Ops. As you can see a lot to talk about during this session.

1. Erwin de Kreuk
Microsoft Solution Architect
Azure Key Vault, Azure Dev Ops and AzureDataFactory
how do these Azure Services work perfectly together!

2. Sponsored by

3. AZURE DATA FACTORY

4. WHAT IS AZURE DATA FACTORY?
Hybrid data integration service
With visual tools, you can build, debug, deploy, operationalize and
monitor your (big) data pipelines
Provides a way to transform data at scale without any coding required.
ELT Platform

5. “DevOps is the union of people, process,
and products to enable continuous
delivery of value to your end users”
Donovan Brown

6. WHAT IS AZURE DEVOPS

8. WHAT IS AZURE KEYVAULT?
The safehouse to safeguard the cryptographic keys and secrets that are
used by your applications, servers and cloud applications
Fully integrated with Azure Active Directory
Secret management: Securely store and tightly control access to tokens, passwords,
certificates, API keys, and other secrets.
Key management: Create and control encryption keys that encrypt your data.
Certificate management: Provision, manage, and deploy public and private Secure Sockets
Layer/Transport Layer Security (SSL/TLS) certificates for use with Azure and your internal
connected resources.

9. ROLES IN AZURE KEYVAULT?
Deploys Application
Tells de application the URI of the key /
secret
Configures the application to use key /
secret (and may abuse), but never sees
the keys
AuditorDeveloperSecurity Operations
Manages Keys
Creates a Key Vault in Azure
Add keys / secrets to the Key Vault
Grants Permission to specific
applications to perform specific
operations using keys
Enables usage logs
Monitors access to keys
Reviews usage logs to confirm
proper key use and compliance
with data security standards

10. CREATING YOUR AZURE DATA FACTORY
demoedkazudvlmadfbicc01 demoedkazutestadfbicc01
Create a Data Factory for every environment
Define naming conventions
customername-azu-environment-azureservice-department
$SubscriptionName = “XXXXXXXXXXXXXXXXX"
$ResourceGroupName = "DEMOEDK-EUW-BICC-DVLM-RG-ENVIRONMENT"
$DataFactoryNameName="demoedkazudvlmadfbicc01"
$Location="westeurope"
Connect-AzAccount
Select-AzSubscription -SubscriptionName $SubscriptionName
New-AzDataFactoryV2 -Name $DataFactoryNameName -
ResourceGroupName $ResourceGroupName -Location $Location

11. SET UP CODE REPOSITORY

12. SET UP CODE REPOSITORY
Repository Settings

13. AZURE DATA FACTORY
Create linked Services:
Azure SQL DB
LS_ASQL_ENV01_WWI
Azure Blob Storage
LS_ABLB_ENV01
Directly saved in Azure DevOps !!

14. AZURE DATA FACTORY
Create linked Services:
Azure SQL DB
LS_ASQL_ENV01_WWI
Azure Blob Storage
LS_ABLB_ENV01
Directly saved in Azure DevOps !!
More naming conventions check:
https://erwindekreuk.com/2019/04/azure-data-factory-naming-conventions/

15. AZURE DATA FACTORY
Create Datasets:
DS_ABLB_ENV01
DS_ASQL_ENV01_WWI

16. AZURE DATA FACTORY
Create Pipeline:
PL_COPY_WWI_ABLB
DS_AQSL
_ENV01_
WWI
DS_ABLB
_ENV01

18. CREATING YOUR AZURE KEY VAULT
demoedkazudvlmadfbicc01 demoedkazutestadfbicc01

19. CREATING YOUR AZURE KEY VAULT
Demoedkazudtapkeyvault01
$SubscriptionName = “XXXXXXXXXXXXXXXXX"
$ResourceGroupName = "DEMO-EUW-DTAP-RG-ENVIRONMENT"
$KeyvaultName="Demoedkazudtapkeyvault02"
$Location="West Europe"
Connect-AzAccount
Select-AzSubscription -SubscriptionName $SubscriptionName
New-AzKeyVault -Name $KeyvaultName -ResourceGroupName $ResourceGroupName -location $Location

20. CONNECTING KEYVAULT IN ADF
Create a new Linked Service

21. CONNECTING KEYVAULT IN ADF
Grant the Azure Data Factory access to your Azure KeyVault.
Be aware that you need to publish the Azure KeyVault before you start using secrets !

22. CREATING SECRETS IN AZURE KEY VAULT
DVLM-ABLB-ENV01
TEST-ABLB-ENV01
DVLM-ASQL-ENV01-WWI
TEST-ASQL-ENV01-WWI
The use of naming conventions ensures that we can
configure everything more dynamically in the release
process later on !
ENVIRONMENT-CONNECTION-REFERENCE
Environment
Development DVLM
Test TEST
Acceptance ACPT
Production PROD
Linked Service
Azure Blob Storage ABLB
Azure Cosmos DB SQL API ACSA
Azure Cosmos DB MongDB API ACMA
Azure Data Explorer ADEX
Azure Data Lake Storage Gen1 ADLS
Azure Data Lake Storage Gen2 ADLS
Azure Database for MariaDB AMDB
Azure Database for MySQL AMYS
Azure Database for PostgreSQL APOS
Azure File Storage AFIL
Azure Search ASER
Azure SQL Database ASQL
Azure SQL Database Managed Instance ASQM
Azure SQL Data Warehouse ASDW
Azure Table Storage ATBL
SQL Server MSQL
Reference
DatabaseConnection LogicalServer(Short)-DatabaseName
Storage
StorageName like
Audit/SSIS/Staging/DataLake

24. CREATING YOUR AZURE DATA FACTORY
demoedkazudvlmadfbicc01 demoedkazutestadfbicc01

25. CREATING LIBRARY IN AZURE DEV OPS
Configure the correct values

26. CREATE RELEASE IN AZURE DEV OPS

27. CREATE RELEASE IN AZURE DEV OPS
Create a Release based on an Empty Job

28. CREATE RELEASE IN AZURE DEV OPS
Create a Release based on an Empty Job
Create a Stage for each Environment
Test
Acceptance
Production

29. CREATE RELEASE IN AZURE DEV OPS
Create a Release based on an Empty Job
Create a Stage for each Environment
Add an Artifact

30. CREATE RELEASE IN AZURE DEV OPS
Create a Release based on an Empty Job
Create a Stage for each Environment
Add an Artifact
Enable Continuous Deployment

31. CREATE RELEASE IN AZURE DEV OPS
Create a Release based on an Empty Job
Create a Stage for each Environment
Add an Artifact
Enable Continuous Deployment

32. CREATE TASK IN RELEASE IN AZURE DEV OPS
Define an Agent

33. CREATE TASK IN RELEASE IN AZURE DEV OPS
Define an Agent
Add the task

34. CREATE TASK IN RELEASE IN AZURE DEV OPS
Define an Agent
Add the task
Configure the task

35. CREATE TASK IN RELEASE IN AZURE DEV OPS
Define an Agent
Add the task
Configure the task
Override the Template Parameters

36. RELEASE IN AZURE DEV OPS
Connecting Library
Release name format

42. CREATING YOUR AZURE DATA FACTORY
demoedkazudvlmadfbicc01 demoedkazutestadfbicc01

43. CUSTOM PARAMETERS
Not all the properties have parameters by default.
arm-template-parameters-definition.json
Adding accountkey for DataLake with secrets from Azure Key Vault
“accountkey": “|",
Download file from:
https://docs.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment#default-parameterization-
template
= means keep the current value as the default value for the parameter.
- means don't keep the default value for the parameter.
| is a special case for secrets from Azure Key Vault for connection strings or
keys.

44. REMARKS
You cannot publish while SSIS Integration Runtime is running
Triggers needs to be disabled and enabled afterwards
Every time you change something to your connection you need to update the
Template parameter
Deletes are handled manually or through Custom PS script or Marketplace extension
You may run into the Azure Resource Manager template limits as your factory
grows bigger. => Linked Resource Manager templates usually have a master
template and a set of child templates linked to the master.
Resource:
https://docs.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment#linked-resource-manager-
templates

45. HANDLING DELETES
Create a post and pre deployment step in your Release

47. Erwin de Kreuk
Microsoft Solution Architect
Axians
@erwindekreuk
https://erwindekreuk.com

48. Session
Evaluation

49. Erwin de Kreuk
Microsoft Solution Architect
Axians
@erwindekreuk
https://erwindekreuk.com