Presentation from SQL Saturday Atlanta, 2017

1. #SQLSatATL
Introducing the Limitless DBA
Copy Data Management- DBA Style
Kellyn Pot’Vin-Gorman

2. #SQLSatATL
Kellyn Pot’Vin-Gorman
Technical Intelligence Manager for the Office of CTO,
Delphix
• Multi-platform DBA, (Oracle, MSSQL, MySQL, Sybase,
PostgreSQL…..)
• Oracle ACE Director, (Alumni), Oak Table
• APEX Women in Technology Award, CTA
• STEM education with Raspberry Pi and Python
• Liaison for Denver SQL Server User Group
• Rocky Mountain Oracle Training Days Conference Director
and Board Director
• Author, blogger, (http://dbakevlar.com)

3. #SQLSatATL
 Copy Data Management: Virtualized sanity for the DBA
Realist.
 Provisioning: Patching, refreshing and if you
ask me one more time!
 Cloud: Cloudy with a chance of failures.
 Security: Yo Developer- Is that the SA password
taped to your monitor??

4. #SQLSatATL
The Life of a DBA
 Provision Databases
 Refresh and provide data to reporting, testing and
development
 Secure database environments
 Optimize data access
 Collaborate to solve business challenges

5. #SQLSatATL
What is Copy Data Management, (CDM)
The management of all non-production databases.
 Broad Term- Physical and virtual clones
 Managed or unmanaged
 Command line or User Interface, (or both)
 Administrative, Infrastructure, security

6. #SQLSatATL
 Storage costs
 Thin-provisioning storage avoidance
 Data transfer costs
 Far less data transferred during provisioning/refresh
operations
 As opposed to the volume of data transferred using
traditional cloning techniques
 Simplifies Provisioning vs. archaic processes to copy data
Why Use Virtualization for CDM?

7. #SQLSatATL 7
▶▶▶
Virtualize and Deployed▶ ▶ ▶
Copies: 90%+ Repeated Data
Storage Pool for Delphix
QA
DEV PATCH TEST
PRODUCTION
Database/App Tier
1 TB
1 TB
0.6 TB
Read From Production
Spin a VIRTUAL database up a patch test, without having to remove a current development or test one.
TEST

8. #SQLSatATL
Example Environment- 1TB

9. #SQLSatATL
Data Virtualization: From Prod to
Virtual
Validated Sync Environment
Source Environment
SCSI/SSL

10. #SQLSatATL
Data Virtualization: From Prod to
Virtual
Validated Sync
Environment
Source Environment
SCSI/SSL
Any Storage
Create as many VDBs as needed!

11. #SQLSatATL
Data Virtualization: Space Savings
Delphix Virtualization EngineValidated Sync Target
Dozen Virtual Database Copies,
(VDBs)

12. #SQLSatATL
Data Virtualization: How is it possible?
Delphix Virtualization Engine
Any Storage
Virtual Database

13. #SQLSatATL
• Using any storage and only fraction of space
• Syncs with native or third-party SQL Server backups
• Can maintain two weeks of data changes
• Managed just like any SQL Server database
• Users can instantly provision a read/write virtual copy
of a database
• Can be used for replication, mirroring, and change data
capture (CDC).
This is Data Version Control

14. #SQLSatATL
Administration Ease

15. #SQLSatATL
Bookmarking and Branching
Release 1.2 Release 1.3 Branch 1.3
Branch 1.1

16. #SQLSatATL 16
Epiphany
e·piph·a·ny
əˈpifənē/
noun
a (1) : a usually sudden manifestation or perception of the essential nature or meaning of
something (2) : an intuitive grasp of reality through something (as an event) usually simple and
striking (3) : an illuminating discovery, realization, or disclosure
b : a revealing scene or moment

17. #SQLSatATL 17
Patching- Got the T-shirt

18. #SQLSatATL 18
How Do We Build Out? How Long?
Finance
HR
CRM

19. #SQLSatATL 19

20. #SQLSatATL 20
Patching and Upgrading Databases
 Each patch, would need to be applied to a
development database, requiring outage
to development teams and then tested
before applying to test, UAT and then
onto production.
 This has to be performed to EACH
environment, every SQL Server, each
quarter.
Finance
HR
CRM
UAT Test DevProd

21. #SQLSatATL 21
 Downtime for valuable resources.
 DBAs working afterhours
 Each database must have it done and…
 The tedious task must be performed over and over
again.
 Little opportunity for advanced learning.
 Each database may experience different bugs.
Risks/Challenges Of This Approach

22. #SQLSatATL 22
Spin up a new
VIRTUAL DB and SQL
bin files and apply
patch to it.
Environment Virtualization, DB Style
CRM
Finance
HR

23. #SQLSatATL 23
No need to keep the
extra VDBs post
patch to prod.
After Testing, Apply to Production
CRM
Finance
HR

24. #SQLSatATL 24
The Compressed Copies
in the Delphix Engine
are Upgraded!
Environment Virtualization, DB Style
HR
Finance
CRM

25. #SQLSatATL 25
In minutes,
refresh and
complete!
Environment Virtualization, DB Style
CRM
Finance
HR

26. #SQLSatATL 26

27. #SQLSatATL 27
• I didn’t have to take away a valuable resource’s database environment to test
the patches.
• I didn’t have to apply the patches to subsequent environments, as they are
virtualized copies of the source, simply requiring a refresh from production,
post final patch.
• I save significant time that commonly has to be allocated to quarterly and
annual maintenance for patching.
• I apply the patch twice- once to test, once to production. I only need to
refresh my environments after I’m done.
• For releases, this can be “containerized”, simplifying release and if required,
rollback.
Patching and Upgrading with
Virtualization

28. #SQLSatATL 28
The quickest way to complete a task is not
having to do it.

29. #SQLSatATL 29
The CLI, (Command Line Interface) for CDM should be robust and able to incorporate into DevOps
What if I’m Retro- Command Line?
• The name of the VDB you want to create
• The group in which to create the VDB
• The Oracle database name
• The Oracle database unique name
• The Oracle database instance number
• The Oracle database instance name
• The source dSource or VDB from which you wish to provision
• The SCN or timestamp of the point you want to provision from, (commands can be run
to get the list of snapshots or timeflow ranges.)

30. #SQLSatATL 30
https://docs.delphix.com/display/DOCS43/CLI+Cookbook%3A+Provisioning+a+VDB+from+a+TimeFlow+Bookmark
Choose Source Snapshot:
>snapshot list database=<dbname>
timeflow ”<dbname>" timeflowRanges;
commit;
Create A Virtual Database:
>database provision;
delphix database provision > defaults
delphix database provision defaults > set
container=<Vdbname>
delphix database provision defaults > commit;

31. #SQLSatATL 31
What a CLI Scripted Refresh Looks Like
> database
> select <VDB name>
> refresh
> set timeflowPointParameters.type= (one of TimeflowPointBookmark,
TimeflowPointBookmarkTag, TimeflowPointLocation,
TimeflowPointSemantic, TimeflowPointTimestamp as appropriate)
> set timeflowPointParameters.location= (the location, timestamp,
or bookmark you wish to refresh to)
> set timeflowPointParameters.timeflow= (the timeflow associated
with location)
> commit
This can all be called via a Powershell script…

32. #SQLSatATL 32

33. #SQLSatATL 33
• Monstrous deployments, (20+) benefit from a scripted, CLI option.
• Single or several VDB deployment.
• When investigating deep level issues, using the CLI can make details
easier to search through.
• When looking for “just the facts”, then a graphical UI is beneficial for high
level error messages.
• Self-service has great benefits and make it simple for those without deep
level skills to take advantage of virtualized environments for development
and testing.
• The CLI helps with DevOps automation and orchestration.
When Do I USE the CLI or Graphical UI?

34. #SQLSatATL
ProjectTimeline
Optimize
Migrate
Initial Testing
Requirements
Cloud Projects

35. #SQLSatATL
Just copy data and applications into the cloud…
• Straightforward approach
• Inefficient, non-incremental for large environments
• Open-source “bcp” uses encryption, compresses,
and multi-threads
• Archaic processes recommended by vendors
Start with backups to IaaS storage, then populate re-hosted
applications by restoring from those backups
• Cloud backups are easy, known technology
How Are Companies Migrating to
the Cloud?

36. #SQLSatATL
Cost Estimates for Azure
https://azure.microsoft.com/en-
us/pricing/details/storage/blobs/
Storage Capacity LRS
First 1 TB / Month $0.024 per GB
Next 49 TB (1 to 50 TB) / Month $0.0236 per GB
Next 450 TB (50 to 500 TB) / Month $0.0232 per GB
Next 500 TB (500 to 1,000 TB) / Month $0.0228 per GB
ZRS
$0.03 per GB
$0.0295 per GB
$0.029 per GB
$0.0285 per GB

37. #SQLSatATL
Backups cloud service: from on-
premises to the Cloud
Database server
Cloud Storage

38. #SQLSatATL
Migration Complete…Not so Fast…
• What if you only want dev and test in the cloud.
• Data is migrated, but this doesn’t count for ongoing data loads, application
connectivity across the network.
• To refresh will take considerable time to perform with traditional tools or cloning
methods.
• Rarely a consideration for the difference in cost structure for processing large
amounts of data from on-premise to the cloud.

39. #SQLSatATL https://blogs.msdn.microsoft.com/brunoterkaly/2013/09/26/how-to-migrate-an-on-premises-sql-server-
2012-database-to-windows-azure-sql-database/
What’s the right
configuration??

40. #SQLSatATL
Cloud Performance

41. #SQLSatATL
https://docs.microsoft.com/en-us/azure/sql-database/sql-database-cloud-migrate
• Database must be MSSQL 2005 or higher, (easy)
• Ensure that the database is compatible with Azure SQL DB, (correct any incompatible
functions, etc.)
• Must have identified all performance issues that will be impacted beforehand.
• Ensure there is as little physical distance between the cloud data center and bacpac files to
be used for migration.
• Disable management jobs that will hinder migration processing.
• Drop any objects or historical data that can impact migration time and can be performed
post migration.

42. #SQLSatATL
https://azure.microsoft.com/en-us/downloads/migration-assistant/?&wt.mc_id=AID559320_SEM_14UHplSj&gclid=CJWky8vB6tICFdG2wAod0SEJqQ
• Performs assessment of IIS environment and creates report of what can be migrated.
• Creates report to document what will require moving and any issues identified.
• Creates any websites and associated databases in preparation for the migration.
• If non-compatible products are in use, (which is in case of 90% of environments) then those
must be addressed manually.
• The tool is free to start, but will cost as project proceeds in duration.

43. #SQLSatATL
From on-premises to the Cloud, then
restore- Assumption
Database server
Cloud Storage
Database server

44. #SQLSatATL
Much Improved Option-
1. Data virtualization
Easy, secure, revolutionary
Optimized for faster agile development and testing
Optimized for new costing structures in IaaS
With a software appliance, no concerns about hardware or
software outside of virtualization lock-in.
Migrating from on-prem to IaaS with
Virtualization

45. #SQLSatATL
Data Virtualization: on-premise
Environment
Source DB server
SQL Server 2008-2016
1 TB
Delphix Virtualization Engine
2 TB storage
Target DB server
SQL Server 1008-2016
No database storage
SCCI/SSL
Backups
via SMB

46. #SQLSatATL
Data virtualization: from on-premises
into the Cloud
Source DB server
SQL Server 2008-2016
2TB storage
Delphix Virtualization Engine
2 TB storage
Target DB server
SQL Server 2008-2016
No database storage

47. #SQLSatATL
Optimized for the cloud in the first place…not after!
Different cost structures
Much smaller storage footprint, much less data-transfer
How Does Data Virtualization Enhance
this?

48. #SQLSatATL
Traditional copy data management techniques
Developed without concerns about infrastructure chargeback
This corresponds to higher cost.
IaaS vendors monitor storage and data transfers
Help meet SLAs, garner profits
It’s not just the data that exists in the end, so transformations can
equal big money for cloud vendors.
Different cost structures

49. #SQLSatATL
Know Thy Enemy…
• Tune SQL and Apps to perform efficiently as possible- natural
life of database is growth, (in processes, resources, etc.) before
migrating.
• The less network latency, the better- network tracing to
eliminate database blame is important.
• Many of the same tools and data provides value- DMVs provide
data internally to SQL Server.
• Look at management tools such as Cloudmonix, (formerly
AzureWatch) AppDynamics, Dynatrace, Zabbix or Logic Monitor.

50. #SQLSatATL
For Non-Production Systems…
Change the way you’ve always performed tasks.
Performing common tasks the same way as previously might
end up costing more.
Secure Data
All IaaS alternatives promote encryption for data in-flight and
for data at-rest, but encryption may not be the right
answer…
Secondary Considerations

51. #SQLSatATL
All IaaS solutions provide encryption in-flight and encryption at-rest
But encryption doesn’t protect data as much as it needs to be .
Europe already requires data masking, not just data encryption for any
confindential data, (GDPR):
http://ec.europa.eu/justice/data-protection/article-
29/documentation/opinion-
recommendation/files/2014/wp216_en.pdf
Confidential data

52. #SQLSatATL
Encryption is reversible data obfuscation, which is very different from
masking data.
• Data masking is non-reversible.
It solves the issue at the data level.
Is authentication and authorization in non-production in compliance with
security goals?
All organizations will soon need to review if critical data in non-
production environments be accessible to developers, testers and
users.
Confidential data

53. #SQLSatATL
Masking personally-identifiable, (PII, HIPPA, PCI, etc.) information
renders it useless from a security standpoint
Resolves both the technical and personal responsibility issue.
The data can be masked before it moves to non-production,
removing unnecessary risk.
Why Masking is Part of the Answer

54. #SQLSatATL
Confidential data
Production
Non-Production
Exposure

55. #SQLSatATL
Data virtualization: transformation by
masking
SQL Server
Validated Sync
Environment
Delphix Masking Engine
Delphix Virtualization Engine
2TB storage
SQL Server Target
No database storage

56. #SQLSatATL
Data virtualization: Masking and then to the
Cloud
Delphix Virtualization Engine
2 TB storage
SQL Server Target
Storage Only for
Masked Data
Delphix Virtualization Engine
2TB storage
Delphix Masking Engine
SQL Server
Validated Sync
Environment
SQL Server Target
Storage Only for
Masked Data

57. #SQLSatATL
Confidential data
Production
Non-
Production
Exposure
Encryption
Masking
Solution

58. #SQLSatATL
On-Prem, Masked, then Replicated to IaaS
Delphix Virtualization Engine
4 TB storage
SQL Server
Validated Sync
Environment
SQL Server Target
Storage Only for Masked
Data
SQL Server Target
Storage Only for Masked
Data
Delphix Virtualization Engine
4 TB storage
SQL Server Target
Storage Only for Masked
Data
Delphix Masking Engine

59. #SQLSatATL
Virtualization Makes Copy Data Management
Simple
Security with masking and encryption is best
Cloud migrations are more successful when
virtualized and planned accordingly.
Patching and Upgrading can be done with less
effort and resources.

60. #SQLSatATL
Want to try it out, download the Delphix AWS Trial- Available for Oracle, MSSQL Soon!
https://www.delphix.com/products/free-trial-request
Twittter: @DBAKevlar
Linked in: http://linkedin.com/in/kellynpotvin
Blog: http://dbakevlar.com

61. #SQLSatATL 61
Delphix with SQL Server- the Basics
https://docs.delphix.com/docs/delphix-administration/sql-server-environments-and-data-
sources/managing-sql-server-environments/overview-of-setting-up-sql-server-
environments
Delphix Upgrade Workflow: https://community.delphix.com/delphix/topics/tip-of-the-day-
upgrading-a-sql-server-dsource
Upgrading the Dsource after an Upgrade: https://docs.delphix.com/docs/delphix-
administration/sql-server-environments-and-data-sources/virtualizing-databases-using-
delphix-with-sql-server/managing-sql-server-dsources/additional-dsource-
topics/upgrading-a-dsource-after-a-sql-server-upgrade
Delphix in the Cloud
https://www.delphix.com/solutions/cloud-migration-virtual-data
References and Tips

62. #SQLSatATL