This joint webinar for DBmaestro (www.dbmaestro.com)and Delphix discuss the synergy between Delphix’s Database Virtualiztion and DBmaestro’s Database Enforced Change Management solutions.
The session discuss the challenges in database development and show in practice how Database Enforced
Change Management and Database Virtualization work together to create a version control, branching and merging method that addresses these challenges.
2. Presenters
Kyle Hailey @kylehhailey
• Technical Evangelist at Delphix
• Oracle ACE, member of the OakTable Network
Uri Margalit @UriMargalit
•
•
Director, Product Management
Presenter at world-wide conferences:
ODTUG, ilOUG, etc…
3. Before we start
•
You will be on mute for the duration of the event
•
We are now talking so please type a message
in the Questions box in the Control Panel if you can’t
hear us (please check your speakers and
GoToWebinar audio settings first)
•
There will be a Q+A session at the end but please
feel free to type your questions in the Questions
box in the Control Panel in advance
•
A recording of the full webinar will be put up online
4. About Delphix
•
•
•
•
Founded in 2008, launched in 2010
CEO Jedidiah Yueh (founder of Avamar: >$1B revenue))
Based in Silicon Valley, Global Operations
10% of Fortune 500
7. The Business Need
80%
More than
50%
of unplanned
downtime is due
to Change
of this, half is due
to human errors
40% of changes FAIL
Copyright@2008, Juniper Networks, Inc.
9. Dealing with Risk
Smaller and more focused changes are easier to manage (Agile…)
Automation of repeating tasks lowers risk of (human) error
Development and Operations should work in synergy (DevOps)
10. Source Control – Standard De Facto
Common version control tools:
GitHub
SVN
Perforce
TFS
RTC
VSS
11. The Database Challenge
•
•
•
•
•
•
•
The Database is a crucial part of the Application
— Schema Structure
— PL/SQL Code
— Lookup Content
The Database is a central resource
Business Data Must be preserved
The Database is not native to traditional
version control
Objects are not files on a file system
How can we manage Content?
How can we branch a Database?
14. What We’ve Seen
1.
2.
3.
4.
5.
Inefficient QA: Higher costs of QA
QA Delays : Greater re-work of code
Sharing DB Environments : Bottlenecks
Using DB Subsets: More bugs in Prod
Slow Environment Builds: Delays
15. 1. Inefficient QA: Long Build times
Build
QA Test
Build Time
96% of QA time was building environment
$.04/$1.00 actual testing vs. setup
16. 2. QA Delays: bugs found late require more code rework
Build QA Env
Sprint 1
Sprint 2
QA
Build QA Env
QA
Sprint 3
X Bug Code
Cost
To
Correct
Delay in Fixing the bug
Software Engineering Economics – Barry Boehm (1981)
17. 3. Full Copy Shared : Bottlenecks
Frustration Waiting
Old Unrepresentative Data
19. 4. Subsets : cause bugs
The Production ‘Wall’
Classic problem is that queries that
run fast on subsets hit the wall in
production.
Developers are unable to test against
all data
20. Data
5. Slow Environment Builds: 3-6 Months to Deliver
Data
Developers
Management
Submit
Request
Approve
Request $$
(2 Weeks)
Approve
Request $$
(2 Weeks)
Approve
Request $$
(1 Week)
(2 Days)
DBA
System Admin
(3 Days)
(3 Days)
Disk
Capacity?
Storage Admin
(3 Days)
Begin
Work
…….1-2 Weeks of Approvals, Delays, and Provisioning……
Request
Additional
Storage?
File System
Configured?
Provision
Capacity
20
Coordinate
Replication w/
Infrastructure
Configure LUNS &
Build File System
ReParameterize &
Configure DB
(3 Days)
Mount
Recovery DB
to Specific PIT
23. Poll
Which of the following have you run into at your
organization?
1. Inefficient QA driving up costs
2. QA Delays causing increased re-work of code
3. Sharing DB causing development bottlenecks
4. Subset DB database in development and QA
leading to bugs in production
5. Slow Environment Builds causing project delays
24. CIO Magazine Survey:
60% Projects Over Schedule and
Budget
Data is the problem
Solve the data problem.
TODAY.
33. One time backup of source database
Production
Instance
Database
File system
34. DxFS (Delphix) Compress Data
Production
Instance
Database
File system
Data is
compressed
typically 1/3
size
35. Incremental forever change collection
Production
Instance
Database
Changes
Time Window
File system
• Collected incrementally forever
• Old data purged
44. What We’ve Seen With Delphix
1.
2.
3.
4.
5.
Efficient QA: Low cost, high utilization
Quick QA : Fast Bug Fix
Every Dev gets DB: Parallelized Dev
Full DB : Less Bugs
Fast Builds: Fast Dev, Culture of Yes
45. 1. Efficient QA: Lower cost
Build
QA Test
Build Time
B
u
i
l
d
T
i
m
e
QA Test
1% of QA time was building environment
$.99/$1.00 actual testing vs. setup
46. 2. QA Immediate: bugs found fast and fixed
Build QA Env
Sprint 1
X
Sprint 2
Build QA Env
Sprint 3
Bug Code
QA
Sprint 1
QA
QA
Sprint 2
X
Bug Code
Sprint 3
QA
49. 5. Self Service: Fast, Efficient. Culture of Yes!
Developers
Management
Submit
Request
Approve
Request $$
(2 Weeks)
Approve
Request $$
(2 Weeks)
Approve
Request $$
(1 Week)
(2 Days)
(3 Days)
DBA
System Admin
Storage Admin
(3 Days)
Disk
Capacity?
(3 Days)
Begin
Work
…….1-2 Weeks of Approvals, Delays, and Provisioning……
Request
Additional
Storage?
File System
Configured?
Provision
Capacity
Coordinate
Replication w/
Infrastructure
Configure LUNS &
Build File System
ReParameterize &
Configure DB
(3 Days)
Mount
Recovery DB
to Specific PIT
50. What We’ve Seen With Delphix
1.
2.
3.
4.
5.
Efficient QA: Low cost, high utilization
Quick QA : Fast Bug Fix
Every Dev gets DB: Parallelized Dev
Full DB : Less Bugs
Fast Builds: Fast Dev, Culture of Yes
51.
52. Challenges of Development & Release to
Operation
Release
Management
Change
Management
Development
Organizing the
development of
changes
• Code
• Database
• Configuration
• Metadata
• Work Items
Test/ Staging/ UAT
Moving just the right
components needed
for the release
•
Release Approved
Items
Production
Enabling safe
migration into
production
• Moving the right
components
• Enabling
Rollback &
Recovery
Agility dictates frequent changes & new tools are needed to streamline
the process
Development
Operations
53. What is DBmaestro TeamWork
• Database Enforced Change Management
+ Database version control
+ Plugs into the ALM (change request, tickets &
work items)
+ Database change impact analysis
+ Database deployment automation
• DevOps Solution for databases
+ Deployment, rollback & recovery
+ Plugs into release management
54. The Challenges that DBmaestro Addresses
•
•
•
•
Development delays
Silos in development, DBA and operations
Delays in deployment (internally and to operations)
Errors in production
55. Development Delays
• Different methodologies for the application
& database
• Code overrides
• Lack of history of changes (who did what, where, when and
why)
• Manual writing of delta scripts
• Lack of automation
56. Silos in Development, DBA and Operations
• No sharing between the team
• No visibility
• Always looking for errors made by others
57. Delays in Deployment (Internally and to
Operations)
• Deployment automation does not really include
the database tier
• Database scripts generated out of
the scope of automation
• Lack of confidence in automation
58. Errors in Production
• Missing changes
• Deploying the wrong version of objects
• What about the reference data?
59. Poll
Which challenges have you experienced?
1. Development delays
2. Silos in development, DBA and operations
3. Delays in deployment (internally and to
operations)
4. Errors in production
60. How?
• Database version control
–
–
–
–
Enforced Check Out/In
Labels
Rollback/Undo
Audit trail reports
• Database impact analysis
– Utilizes version control repository information
– 3 way analysis
• Database deployment automation
–
–
–
–
API
Baselines
Conflict resolution
Customized business logic
61. Without DCM - Two isolated Processes
Version Control Process
Development Process
Check-Out
Script
?
Check-In
Script
?
?
Modify Script
Get updated
Script from DB
?
Compile
Script
in DB
Debug Script
in DB
62. With DCM - One Enforced Process
Development & Version Control Process
Check-Out
Object
Check-In
Object
Modify Object
in DB
Run
Applications’
Tests
63. Safety Net For Automation of Deployment
Simple Compare & Sync
Source vs.
Target
=
≠
Action
No Action
?
You do not have all
of the information
Baseline aware Deployment
Source vs.
Baseline
Target vs.
Baseline
Action
=
=
No Action
≠
=
Override
=
≠
Ignore
≠
≠
Merge
With Baselines and 3 way
analysis the unknown is
now known
64. Benefits - Development
• Database change repository
• Follow SCM best practices (Check-Out/CheckIn)
• All changes are documented
• Manage who can do what, where, when & why
66. Benefits - Management
• Complete visibility into changes in progress
• Management reports
• No silos
67. Live Demo
• Clone 2 virtual copies of the Trunk
1. Dev1
2. Dev2
• Make changes & merge them into the Trunk:
Developer1 modifies Dev1
Developer1 merges changes into the Trunk
Developer2 modifies Dev2
Developer2 merges changes into the Trunk
• Rely on enforced changes & automation
Founded in 2008, launched in 2010Jedidiah Yueh, President and CEOFounded Avamar in 1999, sold to EMC in 2006, VP Product Mgmt at EMCAvamar: >$1B revenue, 150 Employees: HQ in Menlo Park, SF, Boston, DC, London, NY and AtlantaGrowing 250% annually – 130+ customers including 100 Fortune1000 Customers
Founded in 2008Part of the Extreme Group which has about 180 IT professionals consultants
The topic of today webinar is how you can have your database development under version control principles and methods to catch up with your application development and how you can utilize the database virtualization in order to clone your database for parallel development
From the business point of view, they would like to have their system uptodate with no downtime.However researches show that:* 80% of outages, impact mission critical services* From the 80%, 50% are due to human errors* And In addition 40% of changes fail and require rollbackThis sums into many hours in which the systems are down and of course reflect the business.80%of outages impacting mission-critical services caused by people and process issues thru 2015, with the majority of those outages (50%+) caused by change/configuration/release integration and hand-off issues (Gartner RAS Core Research Note G00208328 Ronni J. Colville, George Spafford [October 27, 2010] – Strategic Planning Assumption(s) “Top Seven Considerations for Configuration Management for Virtual and Cloud )
From the technical point of view, Developers, DBA and basically no one like to repeat tasks over and over again.And we would like to have a system that will remember the changes we made 6 months ago and to allow us to focus on development tasks and not on overhead tasks and definitely not to repeat ourselves – this is why we want automation
In software development you cannot talk on automation and not to mention Agile & DevOps.So how we deal with the risk of changes, we release smaller and more focused changes while we believe they are easier to managed.We do it often, so we must have some automation (because a. we don’t want to repeat ourselves and b. we don’t want errors in this critical process)This is called Agile, but so far we were doing staff internally.With DevOps we can release the laser-focus changes to the operations often with confident.
No one today think on developing software without having any version control solution that will manage the changes, keep track on the history.But when considering the version control, a small but critical tier in the application is left aside – the database
Database is also part of the application.Code can be in several ways, schema structure, business code written in PL/SQL and reference data in lookup tables.The main reasons why database code is not native to traditional file based version control are that database is a central resource, objects within the database cannot being dropped and created as done to source code and as done in deployment to the executables.This brings us to a big challenge, how can we branch a database or clone it easily in order to be more agile? With this in mind, I’ll let Kyle answer.
You might be familiar with this cycle that we’ve seen in the industry:Where IT departments budgets are being constrainedWhen IT budgets are constrained one of the first targets is reducing storageAs storage budgets are reduced the ability to provision database copies and development environments goes downAs development environments become constrained, projects start to hit delays. As projects are delayed The applications that the business depend on to generate revenue to pay for IT budgets are delayedWhich reduces revenue as the business cannot access new applications Which in turn puts more pressure on the IT budget.It becomes a viscous circle
There is saying in the industry that we want “good, cheap, fast: choose two”Meaning we want to build applications quickly, ie fast, we want those applications to have good functionality and we want those applications to be \cheap to buildBut we can’t have all three.
I don’t knowIf these situations ring a bell at your organization orif you can imagine some of these situations But here are some of the issues we at Delphix are seeing in the industry with the companies we are talking to.Let’s look at the 5 points in more detail
We talked to Presbyterian HealthcareAnd they told us that they spend 96% of their QA cycle time building the QA environmentAnd only 4% actually running the QA suiteThis happens for every QA suitemeaningFor every dollar spent on QA there was only 4 cents of actual QA value Meaning 96% cost is spent infrastructure time and overhead
Because of the time required to set up QA environmentsThe actual QA tests suites lag behind the end of a sprint or code freezeMeaning that the amount of time that goes by after the introduction of a bug in code and before the bug is found increasesAnd the more time that goes by after the introduction of a bug into the codeThe more dependent is written on top of the bug Increasing the amount of code rework required after the bug is finally foundIn his seminal book that some of you may be familiar with, “Software Engineering Economics”, author Barry Boehm Introduce the computer world to the idea that the longer one delays fixing a bug in the application design lifescyleThe more expensive it is to to fix that bug and these cost rise exponentially the laterThe bug is address in the cycle
Not sure if you’ve run into this but I have personally experience the followingWhen I was talking to one group at Ebay, in that development group they Shared a single copy of the production database between the developers on that team.What this sharing of a single copy of production meant, is that whenever a Developer wanted to modified that database, they had to submit their changes to codeReview and that code review took 1 to 2 weeks.I don’t know about you, but that kind of delay would stifle my motivationAnd I have direct experience with the kind of disgruntlement it can cause.When I was last a DBA, all schema changes went through me.It took me about half a day to process schema changes. That delay was too much so it was unilaterally decided byThey developers to go to an EAV schema. Or entity attribute value schemaWhich mean that developers could add new fields without consulting me and without stepping on each others feat.It also mean that SQL code as unreadable and performance was atrocious.Besides creating developer frustration, sharing a database also makes refreshing the data difficult as it takes a while to refresh the full copyAnd it takes even longer to coordinate a time when everyone stops using the copy to make the refreshAll this means is that the copy rarely gets refreshed and the data gets old and unreliable
To circumvent the problems of sharing a single copy of productionMany shops we talk to create subsets.One company we talked to , RBS spends 50% of time copying databases have to subset because not enough storagesubsetting process constantly needs fixing modificationNow What happens when developers use subsets -- ****** -----
Stubhub (ebay) estimates that 20% of there production bugs arise from testing onSubsets instead of full database copies.
The biggest and most pervasive problem we see is slow build times.In order to set up an database copy for a development environmentsRequires submitting a request to management who has to review itThen if the request is granted, it is passed to the DBA who has to coordinate with the Sysadmin who has to coordinate with the storage admin.In such a situation it makes sense that copying a large database would take a long timeBut even when we talk to someone who uses netapp storage snapshots like Electronics Art, they said even using storage snapshot sit took2-4 days to get a database clone copy due to the coordination between DBA, sys admin and storage adminAt many of the customers we talk to provisioning a database clone copy takes weeks or months.One large global bank quotes us as taking typically 6 months to provision a database clone copy environment.Requirements: self service for app teamsRequirements: end-to-end automationMetrics: # people, process, time for deliverySo far we have talked about the weight of infrastructure on app delivery. Of course, to control and manage that infrastructure, firms layer on a large set of bureaucratic processes, change control, approvals, procurement, governance, etc etc. So the operational and organizational hurdles then create an even bigger drag on IT and app development.Here’s an example from one banking customer.Once the app developer puts in a request for a new development environment, there’s at least a week long wait for management approvals. Then project DBA work with the sysadmin and storage groups for capacity. If more capacity needs to be allocated, it’s 3 more days. If more needs to be purchased, weeks or months. If a copy of production data is needed, the process needs to wait on a production DBA, who might be busy with production issues. Recovering the database to a specific point in time and configuration can also take days.It is very common for two weeks to pass between a developer request and a ready environment. The process can be repeated for multiple environments, for data refreshes, and for integration across multiple systems.With Delphix, turns stop signs into green lights. Provisioning, refresh, rollback, and data integration happen nearly instantly and do not trigger approvals from production systems or require additional storage. That is why KLA is able to deliver 5 times the output from its SAP teams…Without Delphix, it’s impossible for organizations to implement the level of agile processes they desire. The management of data, and the bureaucracy of data management, slows things down too much.
Due to the constraints of building clone copy database environments one ends up in the “culture of no”Where developers stop asking for a copy of a production database because the answer is “no”If the developers need to debug an anomaly seen on production or if they need to write a custom module which requires a copy of production they know not to even ask and just give up.
The problem is getting the right data to the right people at the right time
As Vmware takes a single set of hardware and provisions many virtual machinesDelphix takes a set of datafiles and provisions many virtual database clones
In the physical database world, 3 clones take up 3x the storage.In the virtual world 3 clones take up 1/3 the storage thanks to block sharing and compression
Software installs an any x86 hardware uses any storage supports Oracle 9.2-12c, standard edition, enterprise edition, single instance and RAC on AIX, Sparc, HPUX, LINUX support SQL Server
EMC,Netapp, Fujitsu, Or newer flash storage likeViolin, Pure Storage, Fusion IO etc
Delphix does a one time only copy of the source database onto Delphix
Quote from a customer “Delphix GUI is what Oracle Enterprise Manager would look like if Apple had designed it”Delphix inter face is user friendly, polished and easy to use
Source Syncing* Initial backup once onlyContinual forever change collection Purging of old data Storage DxFSShare blocks snap shots , unlimited, storage agnosticCompression , 1/3 typically, compress on block boundaries. Overhead for compression is basically undetectable Share data in memory, super caching*Self Service AutomationVirtual database provisioning, rollback, refresh*, branching*, tagging*Mount files over NFSInit.ora, SID, database name, database unique nameSecurity on who can see which source databases, how many clones they can make and how much storage they can use
Presbyterian when from 10 hour builds to 10 minute buildsTotal Investment in Test Environment: $2M/year10 QA engineersDBA, storage team dedicated to support testingApp, Oracle server, storage, backupsRestore load competes with backup jobsRequirements: fast data refresh, rollbackData delivery takes 480 out of 500 minute test cycle (4% value)$.04/$1.00 actual testing vs. setup
For example Stubhub went from 5 copies of production in development to 120Giving each developer their own copy
Stubhub estimated a 20% reduction in bugs that made it to production
ExamlplesMacys 4000 hours/year cloning to 8 hours/yearKLA-Tencor over doubled project output, like taking 100 person team and making it a 200 person teamHealth Dialogue reduced storage from 720TB tos 8TB
Thanks Kyle, I'm sure that everyone will agree the opportunities with Delphix are very excited, the idea of spin up the databases very quickly it is exciting.
DBmaestro TeamWork is a solutions that enables you to take control your database development from the very beginning using version control.It is integrated with the existing ALM tools you already have.All changes are stored in a version control repository which is connected to the impact analysis module. More than just compare & sync.With automation in mind, you can complete your DevOps or Continuous Delivery or Continuous Deployment processes and if there is a slight change that something should not be promoted, TeamWork will make sure you know about it.
The challenges we’ve seen when we speak with people that there are:Many delay in developmentMany silos in the organization and within the development teamMany organizations still miss the due date of releasesThere are errors in productions