This document discusses configuring Oracle Enterprise Manager Cloud Control 12c for high availability. It outlines three levels of high availability - Level 1 uses a single OMS and repository, Level 2 uses an active/passive OMS with a local Data Guard repository, and Level 3 uses multiple active/active OMS instances behind a load balancer with a RAC Data Guard repository. It provides recommendations for configuring high availability for the repository, OMS instances, agents, and software library. The presentation also covers backup and recovery procedures.

1. Configuring Oracle Enterprise
Manager Cloud Control 12c for
High Availability
Session ID# 326
Leighton L. Nelson
Mercy

2. About Me
•Oracle Certified DBA with 10+ years
experience
•Oracle RAC SIG US Events Chair/IOUG Liaison
•Co-Author (Expert Oracle Enterprise Manager
12c- Apress)

3. Agenda
•Architecture
•HA Overview
•Level 1 HA – Single OMS & Repository
•Level 2 HA – Active/Passive OMS with Local
Data Guard
•Level 3 HA – Multiple OMS behind SLB with
RAC Data Guard
•General Recommendations for HA
•Configuration Steps
•Backup & Recovery

4. What is High
Availability?
99.9999999999999999999999
“Seven Nines” = 3.15 seconds
downtime per year

5. What is High
Availability?

6. Architecture

7. High Availability Levels

8. Level 1- Single OMS & Repository

9. Level 1- Single OMS & Repository
•Separate servers for OMS & Repository
•Keeps resource usage separate
•memory_target or sga_size on database
repository
•Java VM Heap Size - Xmx

10. Level 1- Single OMS & Repository
Pros Cons
Simple Setup No redundancy
Low cost
Scalable

11. Level 2 – Active/Passive OMS w/
local Data Guard

12. Level 2 – Active/Passive OMS
with local Data Guard
OMS binaries installed on shared filesystem
NFS/OCFS2
Uses virtual hostname
Managed by Oracle Clusterware or 3rd party solution
Virtual hostname should resolve to unique IP address
Configure repository database using local physical standby
w/ Data Guard

13. Level 2 – Active/Passive OMS
with local Data Guard
Pros Cons
Protects against More complex
repository and OMS host
failure
Switchover time minimal No storage
redundancy for OMS
Requires additional
hardware

14. Level 3 – Active/Active OMS w/ SLB
and RAC Data Guard Repository

15. Level 3 – Active/Active OMS w/ SLB
and RAC Data Guard Repository
•Primary OMS with RAC Repository
•SLB for multiple OMSs
•All access to OMS done via SLB
•EM Console
•EMCLI
•Agents
•Standby Database (RAC or Single-Instance)

16. Level 3 – Active/Active OMS w/ SLB
and RAC Data Guard Repository
Pros Cons
Protects against Very complex
repository and OMS host
failure
Switchover time minimal More hardware
Protects against storage
failure

17. Recommendations for
Repository HA
•Use dedicated database
•Use ASM for database storage
•Configure RAC Repository
•Configure Standby Database
•Configure Fast Start Failover
•Review MAA Advisor after OEM installation

18. Configure DB service for
Repository
11gR2 method
srvctl add service -d emrep -s emrepsrvc -l
PRIMARY -q FALSE -e NONE -m NONE -w 0 -z 0
Pre-11gR2 method
•Add service using
DBMS_SERVICE.CREATE_SERVICE
•Start Service using DBMS_SERVICE.START SERVICE
•Create trigger to start service when in primary role

19. Recommendations for
OMS HA
•Configure Multiple OMSs
•Configure SLB in Active/Active configuration
•Install OMS on separate hosts from repository
•Use “Add Management Service” Deployment
Procedure to Add Primary OMS
•Use Modified “Add Management Service” Procedure
to Add Standby OMSs

20. Configure OMS HA
•Active/Passive configuration uses virtual hostname
•Create Application VIP using Clusterware
•11gR2 uses appvipcfg command
•Pre-11gR2 uses crs_profile command
•Clusterware manages failover of virtual hostname
•OEM can also be configured as Clusterware
resource

21. Configure OMS HA
•Active/Active configuration requires SLB
•SLB requirements
•Virtual Servers
•Persistence (Stickiness)
• Health Monitoring
•Register virtual hostname and IP in DNS

22. Configure OMS HA

23. Configure OMS HA
1 Create Pools
• Group of servers receiving on same port
• One pool per service
1 Create Virtual Servers
• Redirects requests to servers in pool
• Clients access virtual hostname and IP
1 Create Health Monitors
• Checks health of services at an interval

24. Configure OMS HA
•Secure OMS with SLB
$OMS_HOME/bin/emctl secure oms -sysman_pass
-reg_pwd regpass
-host slb.example.com
-secure_port 4904
-slb_port 4904
-slb_console_port 443
-reset
-console
-lock_console

25. Configure OMS HA
•Re-secure Agents with SLB
•emctl secure agent –emdWalletSrcUrl
https://slb.example.com:4899/em
•Re-configure EMCLI
•emcli setup –url=https://slb.example.com/em –
username=sysman

26. Configure OMS HA
•Configure OMS to use SCAN name in 11gR2 or
VIPs in pre-11gR2 for RAC
emctl config oms -store_repos_details -repos_conndesc "
(DESCRIPTION= (ADDRESS=(PROTOCOL=TCP)
(HOST=emrep-scan.example.com)(PORT=1521) )
(CONNECT_DATA=(SERVER=DEDICATED) (SERVICE_NAME
= emrep)))"
-repos_user sysman

27. Configure OMS HA
•Configure OMS to use service name when
database role changes in 11gR2
emctl config oms -store_repos_details -repos_conndesc
'(DESCRIPTION=(FAILOVER=ON)
(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=oemhost1)
(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=oemhost2)
(PORT=1521)))
(CONNECT_DATA=(SERVICE_NAME=emrepsrvc.smrcy.com))
(FAILOVER_MODE=(TYPE=select)(METHOD=basic)))' -repos_user sysman

28. Configure OMS HA
Add Management Service

29. Configure OMS HA

30. Recommendations for
Agent HA
•Configure to start automatically on boot
•Install on redundant storage
•Configure restart environment variables for
watchdog daemon
•EM_MAX_RETRIES
•EM_RETRY_WINDOW

31. Recommendations for
Software Library
•Replicate to secondary system
•Storage based replication
•rsync

32. Backup & Recovery
•Repository
•OMS
•Agent
•Software Library

33. Backup & Recovery
•Enable Archivelog mode
•Backup database using RMAN
•Enable Flashback database
•Enable Flash/Fast Recovery Area

34. Backup & Recovery
Repository Recovery
•Stop OMS
•Restore and Recover database using RMAN
•Reconfigure OMS if DB restored to diff host
•Complete Repos Resync if Incomplete Recovery
•Restart Admin Server and OMS

35. Backup & Recovery
•Backup OMS filesystem and instance home:
•Export static configuration
$OMS_HOME/bin/emctl exportconfig oms
-sysman_pwd oracle12c -dir
/mnt/backup

36. Backup & Recovery
OMS Recovery
•Restore OMS HOME from backup
•Software-Only install
•Re-install plug-ins using PluginInstall.sh
•$OMS_HOME/bin/omsca recover –as –ms -nostart
–backup_file <exportconfig file>
•Restart OMS

37. Backup & Recovery
Agent Backup
•No special considerations
•Maintain reference image

38. Backup & Recovery
Agent Recovery
Reinstall Agent Software from Reference Image

39. Summary
HA Level should match business needs
Complexity increases cost
HA for Enterprise Manager consists of HA for all
components

40. Who are we?
•Sixth largest Catholic health care system in the U.S.
•Serves more than 3 million people annually.
•Highly integrated organization that includes 32 hospitals, 300 outpatient locations, 39,000 co-
workers and 1,700 integrated physicians in Arkansas, Kansas, Missouri and Oklahoma.
•Among 10 percent of health care organizations nationwide to have a comprehensive,
integrated electric health record that provides real-time, paperless access to patient information
•MyMercy, a free service, allows patients to connect online with their doctors, plus see medical
information, view test results, renew prescriptions an schedule appointments using a personal
computer or smart phone
•Health Information and Management Systems Society, a global leader in health care
technology, designated Mercy a stage six of a seven-stage process, joining the ranks of health
care leaders such as Johns Hopkins and Mayo Clinic
•$60 million data center provides fail-safe protection and power. When an EF5 tornado struck
Mercy's Joplin facility in 2011, patient records electronically backed up at offsite data center
were immediately available
•Mercy SafeWatch, the largest electronic ICU in the nation, provides 24-hour vigilance to
critically ill patients. Alongside on-site medical teams, high-definition cameras and sophisticated
technology allow specialized ICU physicians and nurses in a central command center to expand
each patient's care team

41. Questions?
Comments?

42. THANK YOU
blogs.griddba.com
linkedin.com/pub/leighton-nelson/4/6b8/a7a
@leight0nn
leighton.nelson@mercy.net