EMC Proven Professional Knowledge Sharing 2009 Book of Abstracts
H10986 emc its-oracle-br-wp
1. EMC IT‘S ORACLE BACKUP AND
RECOVERY—4X CHEAPER, 8X
FASTER, AND 10X BETTER—
SYMMETRIX VMAX, DATA DOMAIN,
AND NETWORKER
DATA DOMAIN TRANSFORMS EMC IT‘S ORACLE
BACKUP AND RECOVERY INFRASTRUCTURE
ABSTRACT
Migrating from a legacy availability infrastructure for Backup and Recovery
creates challenges in terms of what are the best practices for a new Backup and
Recovery deployment with EMC‘s Oracle databases for Global Data Warehouse
and mission-critical Oracle applications. This white paper will illustrate the
transformation of EMC IT Oracle Backup and Recovery Infrastructure and
highlight how the Data Domain appliance transforms EMC IT Oracle Backup
infrastructure.
August 2012
WHITE PAPER
1
3. TABLE OF CONTENTS
EXECUTIVE SUMMARY ........................................................................................................................... 4
Audience ............................................................................................................................................ 4
EMC IT’S BACKUP AND RECOVERY JOURNEY ......................................................................................... 5
EMC IT Overview ................................................................................................................................ 5
EMC IT‘s Backup and Recovery business drivers ...................................................................................... 5
EMC IT‘s Backup and Recovery Legacy Architecture................................................................................. 5
EMC Backup Profile – Legacy EMC IT VTL infrastructure...................................................................... 5
Legacy Backup and Recovery Components........................................................................................ 5
Legacy Backup and Recovery Pain Points ............................................................................................... 8
EMC IT‘s New Backup and Recovery—Data Domain Infrastructure ............................................................. 8
Components ................................................................................................................................. 8
EMC Data Domain Deployment Models ............................................................................................. 9
GDW/CRM BACKUP AND RECOVERY COMPONENTS.............................................................................. 10
Backup and Recovery enablers ............................................................................................................ 10
EMC IT’S ORACLE BACKUP AND RECOVERY METHODS ......................................................................... 11
EMC IT‘s Offloaded Backup and Recovery Process for Oracle Databases .................................................... 11
Step 1- TimeFinder Clone .............................................................................................................. 11
Step 2- Clone Mounted on Proxy/Backup Host .................................................................................. 11
Step 3- RMAN/NetWorker Script to Backup up a Mounted Clone to Data Domain .................................. 11
EMC IT‘s Regular Backup Process ......................................................................................................... 12
EMC IT’S OFFLOADED BACKUP EXAMPLES ........................................................................................... 13
Global Data Warehouse (GDW) Backup Size .......................................................................................... 13
GDW Pain Points ................................................................................................................................ 13
Oracle CRM Backup and Recovery Problem Statement ............................................................................ 13
Advantages of Proxy Host Solution ....................................................................................................... 13
EMC TimeFinder Clone .................................................................................................................. 13
EMC TimeFinder Snapshot ............................................................................................................. 14
ADVANTAGES OF ORACLE BACKUP AND RECOVERY DATA DOMAIN DEPLOYMENT ............................... 15
Reuse of the legacy EMC IT Backup and Recovery process ...................................................................... 15
EMC Data Domain – Deduplication technology ....................................................................................... 15
Deduplication Benefits................................................................................................................... 15
Full Backups versus Incremental .................................................................................................... 15
ROI/TCO ........................................................................................................................................... 15
4X Cheaper.................................................................................................................................. 15
10X Better ................................................................................................................................... 15
EMC IT LESSON LEARNED .................................................................................................................... 16
Bottlenecks ....................................................................................................................................... 16
Disks .......................................................................................................................................... 16
Proxy Server: 8X Faster ............................................................................................................... 16
EMC NetWorker Server .................................................................................................................. 16
Utilization versus Vulnerability ....................................................................................................... 16
CONCLUSION ....................................................................................................................................... 17
References ........................................................................................................................................ 18
Acknowledgments .............................................................................................................................. 18
3
4. EXECUTIVE SUMMARY
EMC IT has seen explosive growth over the last five years accelerating the need to move from its legacy, Virtual Tape Library
(VTL) Backup infrastructure to a new EMC Data Domain Backup infrastructure.
This creates challenges in terms of what are the best practices for a new Backup and Recovery deployment with EMC‘s Oracle
databases for the Global Data Warehouse and mission-critical Oracle applications.
EMC IT implemented a phased approach because of the size of the Oracle environments:
Over 670 Oracle databases in all environments:
o Production
o Test/Development
o QA
o Performance
o Patch
Greater than 30 TB of Oracle Redo Logs generated per day
Mission-Critical environments (databases) backed up daily
Moving to the new EMC IT Data Domain infrastructure, for EMC‘s mission-critical Oracle Global Data Warehouse (GDW) and
Oracle CRM production environments has delivered the following advantages:
4X Cheaper - The Data Domain appliances are a quarter the cost of the legacy EDL/VTL
8X Faster – Move from an incumbent EDL/VTL speed of 500 MB/hour to Data Domain speed of 4 TB/hour
10X Better – Based on the following:
o Reliability - More reliable than tapes – guaranteed that backups can be restored.
o Density - Stores more backups for a longer period of time, even old backups can be quickly restored.
o Protection – Production Data Domain appliances are replicated to a remote Data Domain appliance off-site.
o Speed – Both backups and restores are significantly faster. Backups are now completed within the Service Level
Agreement (SLA) with fewer support resources.
o Complexity – Standard full backups are much easier to execute and restore than incremental backups.
o Cost – Data Domain units are one-fourth the cost with up 10 times the capacity, after deduplication.
This white paper will illustrate the journey and benefits of EMC IT‘s Oracle Backup and Recovery Infrastructure and highlight how
the Data Domain appliance transformed EMC IT Oracle Backup architecture. It will also include lessons learned from EMC IT on
this migration from a legacy VTL infrastructure to a Data Domain infrastructure.
Audience
This white paper is intended for CIOs, Oracle architects, Backup and Recovery architects, storage architects, Oracle Database
Administrators (DBAs), and server and network administrators.
4
5. EMC IT’S BACKUP AND RECOVERY JOURNEY
This section covers the drivers and the size of the journey, with respect to EMC‘s Backup and Recovery profile. It details the
legacy Backup and Recovery infrastructure, as well as the new EMC IT Backup and Recovery Data Domain infrastructure
deployment models.
EMC IT Overview
EMC is a company with over 53,000 users of IT services. It supports over 400,000 customers and partners in 5 Data Centers
with over 16 PB of storage. EMC IT has a portfolio with over 500 business applications and tools and over 8000 OS images with
more than 90 percent of all servers virtualized in 80 countries and 20 languages.
EMC IT’s Backup and Recovery business drivers
The following are the business drivers for EMC IT‘s Backup and Recovery:
Inability to meet EMC‘s backup or restore SLAs with the legacy EMC VTL infrastructure
o Erratic backup windows, some spanning over 24 hours
o Daily ―hand-holding‖/‖fire fighting‖ of the backup process by EMC IT Backup and Oracle SMEs
o Limitations on the number of backup images stored on the legacy EMC VTL infrastructure
EMC IT’s Backup and Recovery Legacy Architecture
This section describes EMC‘s Backup and Recovery profile as well as the legacy Backup and Recovery components.
EMC Backup Profile – Legacy EMC IT VTL infrastructure
The following is the current EMC IT Oracle Backup and Recovery profile:
Over 670 Oracle databases
Over 100 Terabytes of Oracle Databases backed up per day
Mission-Critical production environments backed up daily FULL
Mission-Critical Dev and Test environments backed up every two days FULL
Non Mission Critical and dev/test environments backed up FULL 2X per week
Archive log backups are run daily and when triggered by space alerts
Legacy Backup and Recovery Components
The following are the baseline Backup and Recovery components used in the legacy (VTL) infrastructure:
EMC NetWorker
NetWorker delivers complete Oracle protection through the NetWorker Module for Oracle.
NetWorker integrates with Oracle Recovery Manager (RMAN)—Oracle‘s recommended Backup and Recovery interface for its
databases. Through this integration, NetWorker simplifies Backup and Recovery for Oracle and is able to integrate Oracle server
protection into several core features, including wizard-driven configuration and push installation.
The EMC NetWorker Module for Oracle has value-added features available to reduce backup impact and required storage. This
includes integrated deduplication and snapshot management functionality.
Additionally, an exclusive feature called event-based backup helps deliver more effective backup operations based on probing for
conditions rather than starting merely by time.
To gain the benefits of consolidation and flexible availability, the world is moving increasingly to operation in virtual server
environments. NetWorker, and its Oracle support, integrate with VMware, delivering in guest, application-consistent protection
and support for environments that leverage VMware technologies such as Distributed Resource Scheduler, vMotion, and High
Availability that move and migrate virtual machines in a VMware environment.
Another benefit of NetWorker is that it is tested, certified and supported in a broad range of operating systems and
environments.
5
6. Oracle Recovery Manager (RMAN)
Oracle RMAN is a built-in tool that allows the database administrator (DBA) to easily backup and recovery data in an Oracle
database. RMAN handles the coordination required to ensure that transaction integrity is preserved, and sufficient information is
maintained to recover the database to any appropriate point. RMAN can create backup sets that comprise as much or as little
recovery information as the DBA requires but usually include information from the database datafiles, control files, redo files and
archived log files.
EMC NetWorker/RMAN Integration
NetWorker delivers full integration and is fully compliant with Oracle RMAN. Through this integration, NetWorker delivers
complete Backup and Recovery for Oracle databases, including the ability to capture and protect databases, individual datafiles,
tablespaces, and logs—all of the things that comprise an Oracle installation.
NetWorker, through the RMAN integration, is able to provide backups for Oracle whether the database is online or offline. Other
highlights include the ability to deliver block-level incremental backups, which helps to ensure backups are fast and capturing
the unique changes to the data. Other RMAN features include block corruption detection and surgical recovery of any detected
corrupted blocks to help ensure data is accurate and recoverable.
The solution also compresses any unused blocks to save time, space, and bandwidth. In addition, RMAN delivers restartable
backups from the point of failure for interrupted backups—very helpful, especially for very large databases. Through Oracle‘s
Data Recovery Advisor, any issues encountered and diagnosed are automatically repaired.
All of these capabilities help ensure recovery, which is the primary goal. By providing automatic log management, NetWorker is
able to bring the database back to a point-in-time, thereby restoring all data and replaying any needed logs to facilitate a
complete restore. Finally, not only can you restore back to where the data came from, but also in cases where there is other or
new hardware in place, NetWorker will restore the data to an alternate location.
EMC Disk Library (EDL)/Virtual Tape Library (VTL)
A virtual tape library is a disk based storage array front-ended by servers that emulate tape drive systems. Disk based VTLs
have significant advantages over their tape counterparts as they dramatically improve throughput speeds. This is even more
significant when it comes time for a restore. Since the backups already exist on disk, restores do not suffer from the mechanical
limitations of either the tape drive streaming speeds or the robotic arms or even the fast forward and rewind actions associated
with finding the backup set.
The following is a high-level illustration of the legacy Backup and Recovery infrastructure:
NetWorker
Virtual Tape Library Database Server
Figure 1. Legacy Backup and Recovery infrastructure with VTL
6
7. Proxy Host
A Proxy Host is a server used to mount a cloned copy of a database for backup purposes. It is primarily used to offload the
backup overhead from the production database. It can be used by many databases and has the advantage that the VTL can be
directly mounted to a single Proxy Host, giving significantly more throughput with SAN protocols, vs. backups over NFS. This is
much more important when performing partial restores, as the database can be rapidly restored to the Proxy Host and a partial,
or surgical restore can be performed very rapidly. Of course, if the data required is from the most recent backup, the database
does not even need to be restored from the VTL, as it still exists on the Proxy Host.
NetWorker
Proxy Host
Virtual Tape Library Database Server
Figure 2. Legacy Backup and Recovery infrastructure with Proxy Host
The configuration of the proxy server does not require significant CPU resources. The focus of the server is strictly for IO
throughput. For this reason, EMC IT selects a server with a large number of expansion slots. As an example, for EMC IT‘s CRM
environment the proxy server configuration is shown in Table 1.
Quantity Hardware
1 Cisco UCS C460 M2 - 128GB RAM - 2 x 73GB HDD
8GB DDR3-1333MHz
2 10 GB Network Card - Bonded
4 lpe1200E dual-port 8GB HBA – 4 ports zoned for VMAX, 4 ports zoned to VTL device
SOFTWARE
Oracle RDBMS – single instance
RedHat 5.6
ASM
Table 1. CRM Proxy Host configuration
TimeFinder Clone
EMC TimeFinder is an EMC replication technology that is used by EMC Symmetrix VMAX storage arrays to instantly create an
exact copy of a set of LUNS. For a database, these LUNS would represent the database files for building a standby or backup
copy and optionally include the redo files, for creating a reporting copy. This capability allows the backup or clone of a database
without putting the database in a ―HOT‖ backup mode.
7
8. Legacy Backup and Recovery Pain Points
The legacy Backup and Recovery infrastructure, described above, created some ―pain points‖ for EMC IT.
The following are the two that most impacted EMC IT:
Limited retention for old backups
Like all organizations, EMC IT has a need for retaining backups for extended periods. This generally consists of writing backups
off to tape and sending them to an offsite storage facility. Given the fragility of tape backup media, EMC IT could never be 100
percent sure that it was possible to restore these tapes and the cost of limited use media was expensive to maintain. By placing
these EDLs at a data center nearby, EMC IT was solving the offsite requirement, but at a cost of backup throughput.
Storage needed (large overhead)
With the introduction of EDL devices, EMC‘s version of a VTL, EMC IT quickly adopted them and began deploying the backups to
these, disk-backed, virtual tape libraries. It did not take long to outgrow these EDLs, as the amount of disk required to retain
EMC IT‘s long-term backups was massive. This required EMC IT to continue to rely on tapes for long-term backup retention.
Fortunately, the introduction of deduplication technology and devices like Data Domain‘s VTL, Virtual Tape Library helped
address this challenge.
EMC IT’s New Backup and Recovery—Data Domain Infrastructure
This section describes the legacy Backup and Recovery tools that are re-used in the new Backup and Recovery Data Domain
infrastructure. This includes: the Data Domain appliance Backup and Recovery; the deployment model needed for the EMC IT‘s
shared GDW/CRM Backup and Recovery process; and components for their Oracle Database Proxy Host Backup and Recovery
process.
Components
The new Backup and Recovery infrastructure consists of EMC IT‘s current Backup and Recovery tools:
EMC NetWorker - 7.6.3
Oracle RMAN
EMC NetWorker Module for Oracle (NMO) – 5.0
Proxy Hosts
EMC TimeFinder Clones
The major difference, between the new infrastructure and the legacy infrastructure, is the destination of the backups. In the new
infrastructure, EMC IT writes to the Data Domain appliances. The Data Domain appliances, though the deduplication feature,
allows EMC IT to store many more backups in a fraction of the space previously required. This capability has eliminated the
need for the EDL/VTL and more importantly the need for any fragile tapes.
For the offsite requirements, EMC IT uses Data Domain replication to replicate the backups to a secondary datacenter 600 miles
away, completely eliminating the need for any tapes.
Data Domain
EMC Data Domain deduplication storage systems are designed and optimized, specifically for backup and archive of data.
Support for any conventional backup or archive application through generalized support for network-attached storage (NAS)
interfaces over Ethernet; a virtual tape library (VTL) interface option over Fibre Channel; and product-specific interfaces such
as NetBackup OpenStorage and EMC Data Domain Boost
High-speed, inline deduplication using small, variable-sized sequences to identify and eliminate redundant data segments
before storing to disk
Integrated data protection technologies such as RAID 6, post-backup data verification, and periodic validation checks of
existing data sets
Automated replication of backup data for disaster recovery (DR) using cost-effective, low-bandwidth WAN links, which
enables faster ―time-to-DR‖ readiness
8
9. EMC Data Domain Deployment Models
EMC Data Domain systems can be deployed as a NAS device or as a VTL:
NAS deployment
When the Data Domain system is deployed as NAS, RMAN can write backups directly to it as a disk backup. This can be
extremely useful for environments that have only a few databases to backup. As the number of databases grows, the
management all of these disparate mounts can be overwhelming. For larger organizations, integration with an enterprise
backup solution, such as EMC NetWorker, is recommended. This integration provides many benefits, such as a centralized
location for managing and cataloging backups. EMC NetWorker, along with many of the other solutions, also includes
technology that can significantly improve backup throughputs.
Data Domain Database Server
Figure 3. Data Domain deployed as NFS mount to Host
VTL deployment
When the Data Domain system is deployed as a VTL, RMAN must be integrated with an enterprise backup application like EMC
NetWorker. This is because, like all VTLs, Data Domain systems operate with VTL or Open Storage protocols and RMAN is not
capable of interfacing with them directly.
With a VTL deployment, Data Domain systems can be connected though a 1 GB or 10 GB network typically dedicated to backup
traffic. They can also be connected directly to database server HBA ports achieving much higher rates of throughput.
Networker
Data Domain Database Server
Figure 4. Data Domain deployed as a VTL device
9
10. GDW/CRM BACKUP AND RECOVERY COMPONENTS
The following illustrate the new EMC IT Backup and Recovery infrastructure components, when offloading backups to a
Proxy Host:
8 Gbs SAN
Fabric A
8 Gbs SAN O
Fabric B
N
T
A
C
T
Data Domain Proxy Backup Production
U
Server
S
Figure 5. Oracle Database Proxy Host Backup and Recovery Process
To
le
ar
As the diagram illustrates, EMC IT deploys a shared Backup and Recovery infrastructure using the following components and
best practices: n
m
VMAX TimeFinder Clones
or
Deployment of a Proxy Host or EMC NetWorker Storage node e
ab
o Oracle database – single instance (backup license) ou
o EMC NetWorker Module for Oracle
t
ho
o RedHat AS 5.6 w
E
Data Domain appliance deployed as a VTL device – DD890
M
Backup and Recovery enablers C
pr
The following technologies enable EMC IT‘s Oracle Backup and Recovery today:
od
8 Gigabit Storage Area Network uc
ts,
10 Gigabit dedicated Local Area Network se
EMC Data Domain 890 Appliances rvi
ce
EMC NetWorker – Version 7.6.3 s,
an
EMC NetWorker Module for Oracle (NMO) – Version 5.0
d
Oracle RMAN – 11.2 so
lut
EMC VMAX TimeFinder Clone
io
Proxy Clone Hosts ns
ca
n
he
lp
so
lv
10 e
yo
ur
11. EMC IT’S ORACLE BACKUP AND RECOVERY METHODS
EMC IT’s Offloaded Backup and Recovery Process for Oracle Databases
The following are the high-level steps for the offloaded Backup and Recovery Proxy Host backup process:
1. Create a TimeFinder Clone
2. Include the redo logs so that the database can be verified prior to the backup. This is not necessary, but adds an extra
level of confidence that the database backup is good. It also allows the database to be opened, read-only for
investigation, reporting or extract purposes.
3. Mount the Clone on the Proxy Host server
4. Invoke the NetWorker/RMAN scrip to backup the TimeFinder Clone to Data Domain infrastructure
Step 1- TimeFinder Clone
EMC IT uses EMC TimeFinder/Clone Technology as the first step in the backup process. The Clone is a full size point-in-time
replica of the source database. Once the Clone is originally created, subsequent ―recreate‖ operations only apply the
incremental changes made to the source since the last ―recreate‖ of the Clone. This reduces the first step of the backup
process. EMC IT‘s daily clone typically takes 40 minutes for a 20TB database.
After 40 minutes, the database has a viable Backup and Recovery solution. This backup, is now available for surgical or
complete restores.
Step 2- Clone Mounted on Proxy/Backup Host
EMC IT uses a Proxy Host, or backup host, to offload the RMAN backup to a virtual tape that is the Data Domain appliance. With
a Proxy Host, a separate host is used, configured with Oracle Binaries and ASM installed. The ASM diskgroups on the Proxy Host
are made up of the source database‘s cloned devices.
Deploying a backup host has many benefits:
Allows full backup to run on the Proxy Host producing no performance impact to the production database and production
user community
Enables surgical restores from the Proxy Host from the most recent backup or from a restored backup from any point in
time. Having a point-in-time restore on a separate host allows individual tables or tablespaces to be restored using
export/import or transportable tablespace or even database links for very specific requirements.
Enables ad-hoc querying for end-users. Queries can run from the clone for ad-hoc reporting – allowing full autonomy for
reports running from the clone. This can stop less efficient or resource intensive reports from interfering with the source
database (Production).
Reverse Clone – For databases that require a quick and efficient RTO (Recovery Time Objective) the Reverse Clone allows
super quick restores back to the source.
Offloaded Stats Generation – before the next backup run, the database can be opened and stats generated on the cloned
version of the database. This allows for complete level of generation, with no impact to the production system.
Step 3- RMAN/NetWorker Script to Backup up a Mounted Clone to Data Domain
Once the Clone operation is complete, the Clone is opened as a read only database, thus verifying a valid copy, and backed up
to Data Domain using RMAN/NetWorker. EMC IT re-used the same legacy scripts that were already in place when EMC IT
moved from the EDL to the Data Domain appliance deployed as a VTL.
The only additional consideration was to change the ―filesperset‖ setting, which was changed to 1. This contributes to the most
favorable compression ratio and efficiently stores subsequent full backs to the Data Domain with a very small storage footprint.
Using the Data Domain appliance to store backups has enabled many advantages including:
Virtually no change to existing backup scripts, RMAN catalogs or pre-determined scheduling – the only consideration or
potential change being a change to the filesperset parameter.
Integrates easily with EMC NetWorker infrastructure.
11
12. The deduplication benefits of Data Domain allow many more images of full backups to be stored with minimal storage
overhead. This is especially significant for databases that carry a lot of unchanged history as the bulk of the data, which is
the case in all data warehouses and most OLTP databases.
Data Domain enables EMC IT to eliminate the need for a traditional Oracle incremental backup strategy. This simplifies the
restore steps and shortens the recovery time window.
RMAN Script Example
Here is an RMAN script excerpt from EMC IT‘s backup script used to backup EMC IT‘s offloaded databases.
The script uses standard Oracle RMAN commands:
connect rcvcat DatabaseA/xxxxxx@rmancatolog
connect target xxxxxxx/xxxxxx@DatabaseA
run {
set command id to 'rman database';
allocate channel t1 type 'SBT_TAPE' format '%d_full_%U';
allocate channel t2 type 'SBT_TAPE' format '%d_full_%U';
allocate channel t3 type 'SBT_TAPE' format '%d_full_%U';
….
(EMC IT uses 24 channels)
…..
send 'NSR_ENV=(NSR_DATA_VOLUME_POOL=DBORAP)';
backup tag = ‗DatabaseAbkp_hot_0209_214224' filesperset 1 database;
backup current controlfile channel t1 format '%d_cf_%U';
release channel t1;
release channel t1;
release channel t1;
……
}
Table 2. EMC IT legacy RMAN script example
EMC IT’s Regular Backup Process
For the majority of EMC‘s databases offloading to a proxy server is not required.
For those databases, where speed is not an absolute necessity and the overhead of an RMAN backup directly on the database
server does cause performance issues, EMC IT will not offload the backup to a proxy server.
In this case, EMC IT executes the backup directly from the database server. Backups are scheduled every other day as full
database backups to the same Data Domain appliances as the offloaded backups. Since the deduplication benefits are the
same, as offloaded backups, there is no need to do incremental backups, which are impacted by the extra complexity, and
restore times associated with incremental backups.
For EMC IT‘s larger databases, using this regular backup process EMC IT installs a dedicated network card to the database
server. This additional network card in the database server reduces the overhead to the public network that a large backup
would add, and assists with overall throughput.
In the future, EMC IT is looking at Data Domain‗s Boost technology to both alleviate that overhead as well as reduce the backup
window by distributing the deduplication process between the backup server and database server.
12
13. EMC IT’S OFFLOADED BACKUP EXAMPLES
Global Data Warehouse (GDW) Backup Size
EMC‘s Global Data Warehouse (GDW) is a central source of relational data that provides business intelligence to many key
business groups within EMC including Sales, Marketing, Finance, Manufacturing, Customer Service, Professional Services, Human
Resources and Legal departments.
It is a critical component of intellectual property and it‘s supported and maintained accordingly. One key and highly important
support requirement is the ability to backup and provide realistic and flexible recovery options to the business groups that rely
on this data.
GDW runs on a six-node Linux-based Oracle Real Application (RAC) Cluster that went live in November of 2009, as a 10TB
database. Since 2009, GDW has significantly grown and as of May of 2012 GDW has doubled to 20 TB with archive log
generation totaling from 500 to over 1500 GB daily.
GDW Pain Points
In the summer of 2011, the time to complete an online RMAN/NetWorker backup of the GDW to an EMC Disk Library (EDL)
degraded. The throughput was erratic and at times would exceed 24 hours, ―wrapping‖ over the scheduled start time of the
daily TimeFinder Clone used as part of the backup solution.
This undermined EMC IT‘s ability to successfully provide a backup of the database.
There were periods where it took a week to successfully backup the database to EDL, as the scheduled clone would cause the
online backup to fail. This resulted in daily handholding of backups and undermined the ability of IT to protect this valuable
investment.
The backup/recovery solution was in need of remediation and some redesign as: backups ran too long and would sometimes fail
leaving data protection capability limited and vulnerable. Additional staff-hours were needed to ―fire-fight‖ daily challenges.
A strict limitation to the number of full backup images that could be stored on the Disk Library, due to the large storage footprint
each backup imposed. Archived log backups became increasingly important and the number of logs needed to satisfy recovery
points became suspect.
Oracle CRM Backup and Recovery Problem Statement
EMC IT‘s Oracle 11i eBusiness Suite Database grew from 8 TB to 10 TB over a period of two years. During this two-year period,
the backup time increased from 6 hours to over 22 hours, causing significant challenges in getting a full, successful, daily
backup. This also meant that restores, to this environment, would take longer than 22 hours. These times were well beyond
EMC IT‘s SLA for the databases Recovery Time Objective (RTO).
This, over 3X increase in the backup window, was due to more than just growth in database size. It also was due to the
increase in the number of databases that needed to backup to this shared infrastructure, and was guaranteed not to be solved
on its own. As this was EMC IT‘s mission-critical database, with a zero data loss policy, 11i Database had to be backed up daily.
The increase in backup time, violated EMC IT‘s restore SLAs as well.
To resolve this challenge, EMC IT embarked on a new solution with EMC VMAX Clone and Data Domain.
Advantages of Proxy Host Solution
EMC IT uses a Proxy Host to run backups for its Oracle 11i eBusiness Suite database. The copy that is mounted to the Proxy
Host is built using EMC TimeFinder clone technology.
EMC TimeFinder Clone
TimeFinder Clone provides single or multiple point-in-time copies of the full database within the same storage array.
The cloned database is mounted on the Proxy Host for backup. A database clone requires the same disk space as the source
database. Clones are extremely fast, as the data never leaves the storage array.
The initial clone for a 20 TB database takes about 4 hours, with a low quality of service, QOS. EMC IT uses the low QOS to
eliminate the impact to production. After the initial clone, all subsequent clones use a differential clone, only copying the
changed tracks to the cloned devices. For example, EMC‘s GDW 20 TB database clone takes 40 minutes to complete. From the
Proxy Host, backup of this 20 TB database to Data Domain appliance takes 4 hours.
13
14. EMC TimeFinder Snapshot
Another option is to use EMC TimeFinder snapshots.
A snapshot is a point-in time "virtual" copy that occupies very little disk space. A snap requires only 10 to 20 percent of source
database space for storing only changed tracks.
In this case, backup will actually read most of the data from production disks. This is a very efficient method of offloading the
backups, but does incur a small overhead to the production database, since the database and backup are competing for the
same physical disks. A good way to minimize this is to use virtual pools in the storage array. This provides many more disks to
spread out the I/O load.
The full clone solution offloads the entire backup load from the production database server and doesn‘t impact performance for
EMC IT‘s OLTP online users. This is especially useful when backup runs for a long time.
Proxy Host copy also provides much better Recovery Time Objectives (RTO). In the event you need to restore a production
database, restore from backup media will take hours. Since proxy copy is already on disk, it takes significantly less time to
restore. You can do Reverse Clone to restore a production database and then apply missing archive logs.
This method also provides some protection against accidental data corruption/loss due to human errors. For example, a few
critical tables in the production database were dropped one hour ago due to human error. Application will work fine if you can
restore only these tables. You cannot afford to restore complete production database and suffer 1-hour data loss. Instead, you
can restore proxy database point in time just before table drop, and then copy tables back to production manually. Proxy copy is
extremely useful in such surgical repair scenarios.
14
15. ADVANTAGES OF ORACLE BACKUP AND RECOVERY DATA DOMAIN
DEPLOYMENT
The following are the advantages the EMC IT gained when deploying the new Backup and Recovery Data Domain infrastructure
Reuse of the legacy EMC IT Backup and Recovery process
The first gain was virtually no change to existing EMC IT Backup and Recovery processes such as backup scripts and RMAN
catalogs. There was one consideration potential change to ―filesperset‖ as defined in the EMC white paper, Oracle RMAN Design
Best Practices with EMC Data Domain.
Another advantage was that the Data Domain infrastructure integrated seamless with the legacy EMC IT backup software, EMC
NetWorker.
EMC Data Domain – Deduplication technology
Deduplication technology allowed EMC IT to move its long-term backups to the Data Domain infrastructure with complete
confidence that EMC IT would be able recover those backups, if the need ever arose.
It also solved the issue with off-site storage as these Data Domain devices have the ability to replicate all backups to a Data
Domain unit at a remote site. For EMC IT, the replicated Data Domain unit is located at EMC IT‘s DR site, giving us additional
protection.
Deduplication Benefits
Data Domains deduplication technology has enabled EMC IT is to retain many more backup images with minimal storage
overhead. EMC IT can now restore backup images from several months past—without having to go through the difficult process
of locating off-site tapes and hoping that they are still valid.
Full Backups versus Incremental
Due to Data Domain‘s ability to deduplicate the backups, EMC IT was able to use daily full backups versus having to do
incremental backups. This allows a much quicker recovery process since incremental backups require restoring the full backup
then restoring the incrementals on top of that. This not only reduces the complexity of the restore process, but it also reduces
EMC IT‘s recovery window.
ROI/TCO
4X Cheaper
Typically, it is difficult to calculate actual dollars saved when utilizing shared infrastructure and the costs to administer an
environment that is not meeting SLA‘s. In EMC IT‘s case, the actual cost of the Data Domain units EMC IT replacing its EDL
with are one fourth the cost and between double and quadruple the capacity—depending on the actual de-dupe rates.
In addition, to these savings, Data Domain technology is also increasing EMC IT‘s ability to retain backups longer, and
completely eliminate the need for tapes. When factoring in all of these additional cost reductions, it is clear that overall costs
greatly exceed the 4x cheaper claim.
10X Better
The following is a mix of quantitative/qualitative measurements of ―Better‖ that include people, process, and technology:
1) Faster—8X faster with Proxy server deployment with 8GB FC deployment
2) Reliability—More reliable than tapes – guaranteed backup can be restored
3) Cost—Costs less for hardware and environments (power and cooling)
4) Density—Stores more backups creating a greater retention period
5) Protection—Replicated remote Data Domain appliance can be deployed off-site
15
16. EMC IT LESSON LEARNED
As with all functions in IT, following is a set of best practices that will yield better performance, scalability and stability.
Bottlenecks
Disks
When configuring for high speed backups, there are many potential locations for bottlenecks. The most obvious of these is the
disk layout, in the storage array. When it comes to I/O, the capability of the disks is most often the bottleneck.
To give some idea, a 15K fibre channel drive is capable of driving between 47 and 53 GB/hour. This means that to support an
8TB/hour backup, you would need 155 of these disk drives.
For the Data Domain Backup and Recovery environments, EMC IT does not need 8 TB/hour. This means EMC IT configured its
disk pool with only 76 disks and is achieving 4 TB/hour.
Proxy Server: 8X Faster
Next up, in the stack, is the connectivity between the storage array and the proxy server. This was initially overlooked when
EMC IT migrated from the EDL to the Data Domain. The Backup and Recovery infrastructure had 8 GB SAN switches, HBAs and
ports on the SAN; however, the Backup and Recovery infrastructure initially connected it to the 2 GB SAN switch.
Even after re-cabling to the 8 GB SAN, EMC IT still found the ports on the storage array had been configured for 4 GB. After
fixing these issues, the new Backup and Recovery infrastructure effectively doubled the initial throughput of 2 TB/hour to 4
TB/hour. When compared to EMC IT‘s legacy EDL backup speeds, EMC IT improved its backup speeds by a factor of 8 from
about 500 MB/hour on the legacy EMC DL4206 EDLs to about 4 TB/hour on the new Data Domain VTL
EMC NetWorker Server
After relieving that bottleneck, the new Backup and Recovery infrastructure began to have a problem with the NetWorker server.
The NetWorker server that was deployed was already overloaded with backup requests and was not able to keep up the large
number of backup sets that were now being requested to be processed. Keep in mind that EMC IT set files per set to 1, for best
results in deduplication. Since this node was part of the original shared infrastructure and was clearly overloaded, configuring
and deploying another NetWorker server relieved the bottleneck and improved overall backup performance throughout the data
center.
Utilization versus Vulnerability
One of the constant challenges EMC IT faces with their backup infrastructure is how to maximize the utilization without creating
vulnerability.
In order to maximize the investment in the new backup infrastructure EMC IT moved from a dedicated model to shared model.
This has allowed EMC IT‘s Backup and Recovery team to become much more efficient. However, there was no mechanism to
measure the effect of consolidation on individual backup and restore times. This is what had lead EMC IT into the problem of
backups not completing. It is extremely important to monitor backup times and investigate individual backups that start taking
longer than was originally configured. An obvious example of this would be to use the proxy backup server for more than just
one environment—but without coordinated backup windows, both backups could suddenly fall out of the configured times.
16
17. CONCLUSION
EMC IT implemented a Backup and Recovery solution to deal with the growing size of its Oracle environments:
Over 670 Oracle databases in all environments (Test/Development/QA/Performance/Patch/Production)
Over 30 TB of Oracle Redo Logs per day
Mission-critical environments (databases) backed up daily
Migrating from a legacy EMC EDL/VTL solution to Data Domain brought the following advantages to EMC IT‘s Backup and
Recovery deployment:
Moving away from limited Backup and Recovery backup retention to enabling of simple full backup to reduce the complexity
of incremental restore process
Accommodates months of full Backup and Recovery backups enabled by the Data Domain appliance
Deployment of a Proxy host architect in the Backup and Recovery architect delivers the following:
o Off load backup process off of production enabling EMC business users experience no disruption or performance issues
on the CRM production server
o Ability to reuse legacy Backup and Recovery processes/scripts
Implementation of a shared infrastructure for EMC IT‘s mission critical CRM and GDW environments
EMC TimeFinder technology, which was able to create database clones in minutes (18 TB in 40 minutes) off the production
server and mount on the proxy server and backup in less than four hours
EMC IT also learned valuable lessons from the EMC EDL/VTL to Data Domain Backup and Recovery journey:
Disks—To meet the throughput in a Data Domain the source database Array needs the correct number of drives. To drive to
8 TB/hour the array would need 155 drives
Proxy Server—Confirm the infrastructure cabled correctly and enough HBAs
NetWorker Server—Identify a near capacity NetWorker server and plan for enough nodes to meet your total Backup and
Recovery community needs
Utilization versus Vulnerability—Monitor the backup and restore times in a shared deployment to increase the Backup
and Recovery infrastructure utilization while diminishing vulnerability of long running backups , failed backed or long running
restores or failed restores
Moving to the new EMC IT Data Domain infrastructure for EMC‘s mission-critical Oracle Global Data Warehouse (GDW) and
Oracle CRM production environments has enabled the following:
4X Cheaper—The Data Domain appliances are a quarter the cost
8X Faster—Move from a legacy EDL speed of 500 MB/hour to an EMC Data Domain speed of 4 TB/hour
10X Better—Based on the following:
o Reliability—More reliable than tapes; guaranteed backup restoration
o Density—Stores more backups for a longer period of time; even restores for old backups can restored quickly
o Protection—Production Data Domain appliances are replicated to a remote Data Domain appliance offsite
o Speed—Both backups and restores are significantly faster; backups are completed within the SLA with fewer support
resources
o Complexity—Standard full backups are much easier to execute and restore than incrementals
o Cost—EMC Data Domain appliances are one-fourth of the cost with up 10 times the capacity, after deduplication
17
