2. Why is Testing Important?
Performance issues not found in testing can cost 8-10x
more to resolve once in production*
75% of systems are upgraded or deployed without
effective and proper testing*
Average cost of downtime is $42,000/hour
87 Average downtime hours/yr = $3.65m/yr**
*Source: Gartner
**Source: CIO Magazine
Page 2
3. Business Case Background
Fortune 500 Insurance Company
10,000+ employees
Upgrading data warehousing system
from Version 9 to Version 10 of Oracle
Databases
Moving Operating System from Sun to
Linux
Information in business case, such as labor time,
System comprised of 53 different hourly rates, testing phases, phases effected by
applications, used by assorted RAT,and groups involved in testing provided by
customer
personnel spanning the organization
Page 3
4. Upgrade Process
Phases of Testing Roughly 60 people involved
• Defining requirements • Director-level Management
• Determine system usage* • Application Managers
• Identify areas to test*
• Application Users
• Building out test environment
• Development
• Data Completeness*
• Data Transformation* • Infrastructure
• Data Quality* • DBA
• System Testing* • Quality Assurance
• Integration Testing* • Enterprise Architecture
• Test modifications/Re-runs • Operations
• Reporting
Only 7 out of 53 applications in the system are tested
Testing of the system occurred over 4 months
*Areas where Real Application Test would help
Page 4
5. Labor Time
Test Scenario: Upgrade Oracle DB and the Operating System
Application Tested: Merlin Datawarehouse
Upgrade Frequency: Every 36 Months
Labor Labor Cost Annualized Percent
Expected Impact Hours @$66 Labor Cost Savings
Actual Without RAT 18,405 $ 1,214,730 $ 404,910
Planned With RAT 10,778 $ 711,348 $ 237,116
Estimated Cost
Savings 7,627 $ 503,382 $ 167,794 41%
Labor time tracked by internal Time Management Application
All hours, rates and cost provided to Oracle by the customer
Page 5
6. Additional Savings Areas
Monthly patch upgrades to the system
• Assumption – Patch upgrade cost 2% of major upgrade
Component upgrades
• Assumption – Component upgrade cost 10% of major upgrade
Hardware cost reduction
• Assumption – 50% hardware reduction in performance tests
Reduction in troubleshooting issues
• Assumption – Troubleshooting downtime = 15 hours per incident
• Assumption – 12 incidents were Tier 1 (critical) after upgrade
• Assumption – 50% reduction in support incidents
Costs in these areas were not available during case study
Page 6
7. Hard Dollar Savings Estimate
Without With Total
Category RAT RAT Savings Comments
Every 36 Months
Major Upgrade Cost $ 1,214,000 $ 711,000 $ 503,000 Based on detailed analysis
Component Upgrade Test $ 121,400 $ 71,100 $ 50,300 Assume 10% of major Upgrade
Annual Savings
M onthly Patch Testing
Patches Tested Per Year 12 12
Labor Cost per Patch Test $ 24,280 $ 14,220
Total Labor Cost for all Patch Tests $ 291,360 $ 170,640 $ 120,720 Assume patches require 2% of
major uprade effort
Production Support - Coverage Test
Relevant SRs per Year 12 6
Support Cost Per Incident $ 1,000 $ 1,000
Total Support Cost $ 12,000 $ 6,000 $ 6,000 Assume 50% reduction in support
incidences
Annual Hardware Cost Savings $ -
More information required
Page 7
8. Payback – Less than 12 months
Costs Year 1 Year 2 Year 3 Year 4
Oracle License $ 350,000 $ - $ - $ -
Annual Support Costs $ 77,000 $ 77,000 $ 77,000 $ 77,000
Training $ 10,000
Cumulative Costs $ 437,000 $ 514,000 $ 591,000 $ 668,000
Saving Category
Major Upgrade $ 503,000 $ 503,000
Component Upgrade $ 50,300
Patch Testing $ 120,720 $ 120,720 $ 120,720 $ 120,720
Prod Support $ 6,000 $ 6,000
Reduced Hardware
Cumulative Savings $ 629,720 $ 800,740 $ 921,460 $ 1,551,180
Cumulative Savings (net) $ 192,720 $ 286,740 $ 330,460 $ 883,180
Payback < 12 months
4 year ROI 232%
Oracle Real Application Test License – 40 CPU license with a 25% discount
Annual Software Maintenance – 22% of license cost (after discount)
Training – 5 Days
Page 8
9. What if…
Customer tried to test ALL 53 applications?
• Instead of only 7
There are Tier 1 issues on all patch upgrades?
• Instead of just during the major or component upgrades
How much more would they have saved?
Page 9
