2. About Me
• Oracle DBA for 10+ years
• Oracle RAC SIG US Events Chair/IOUG Liaison
• Blogs at blogs.griddba.com
• Started presenting in 2012 (Collaborate 12)
5. Enterprise Manager 12c Upgrade
Paths
• 1-System Upgrade
• Upgrade done on same host as 10.2.0.5/11.1 EM
• All agents, OMS and repository upgraded at once
• Requires significant downtime
• 2-System Upgrade
• New OMS created on separate host (platform migration
possible)
• Both 10.2.0.5/11.1 and 12.1 EM run concurrently
• Requires almost zero downtime
• Migrates existing metrics, jobs, report etc.
6. Which Path to Choose?
• Migrate/consolidate onto
VMware
• EM 10.2.0.5 primary monitoring
tool for Oracle databases.
Minimal downtime required
• User-defined metrics, scheduled
reports and jobs in EM 10.2.0.5.
Don’t want to recreate
7. Upgrade Overview
1. Apply Upgrade Console patch to 10.2.0.5 OMS
2. Deploy 12.1 agents to monitored targets from
Upgrade Console
• Only 12c agents monitored by EM 12c
3. Backup repository database. This will be used as the
12.1 repository.
• DBCA, RMAN can be used for backups (same platform)
• Datapump, Cross-Platform tablespaces/databases (different
platforms)
4. Install and upgrade OMS on new RHEL 6.2 host
5. Shutdown 10.2.0.5 agents and startup 12c agents
6. Decommission 10.2.0.x agents/OMS or rollback
9. Pre-requisites
• Patches
• OMS requires PSU 3 or higher
• opatch lsinventory –bugs_fixed | grep PSU
• Pre-Upgrade Console
• Certification Matrix
• Database, OMS, Agent and OS
• Plug-ins and agent software
• Plug-ins required can be identified from the Upgrade
Console Manage Software
• Plug-ins will be migrated/upgraded to new OMS
• Agents required for each OS being monitored
12. 2-System Upgrade
Repository Database Upgrade
• The EM installer does not
upgrade the database
• Database can be upgraded
before EM upgrade or after
repository backup
11.1.0.6
11.2.0.2
18. 2-System Upgrade
Backup Repository
• DBCA or RMAN recommended
• Unlock SYSMAN account if using DBCA
• Duplicate from Active feature in 11g makes cloning DB easy
• For cross-platform migration take a look at note
388090.1
• Datapump instructions not specific to EM12c repository
upgrade (Open SR with Oracle Support)
• Cross platform transportable tablespaces instructions not
specific to EM12c repository upgrade (Open SR with Oracle
Support)
• RMAN Convert database also possible for supported
platforms
19. 2-System Upgrade
Backup Repository
• Using an existing database via Datapump
• Enterprise Manager schema needs to be removed from
target database first
$OMS_HOME/sysman/admin/emdrep/bin/RepManager
omshost port SID -sys_password <password> -action
drop
• Not all objects dropped in some cases
• See MOS NOTE [ID 1365820.1]
• EM 12c R2 has a database template
20. 2-System Upgrade
Backup Repository
• Copy emkey to repository before taking backup
emctl config emkey –copy_to_repos –sysman_pwd <password>
• Re-secure emkey after taking the backup
• Verify that SYSMAN schema doesn’t contain invalid
object
SELECT count(*) from user_objects
WHERE status=‘INVALID’
22. 2-System Upgrade
Upgrade OMS and Repository Manually
• Check pre-requisites for the intended 12.1 repository
using EM Prereq Kit
• EM Prereq Kit can validate database parameters and
configuration prior to upgrade
e.g. install/requisites/bin/emprereqkit -executionType
upgrade -prerequisiteXMLLoc install/requisites/list -
connectString
"(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)
(HOST=dbhost1)(PORT=1521)))(CONNECT_DATA=(SID=emrep)))"
-dbUser SYS -dbPassword -dbRole sysdba -showPrereqs
• Some parameters can be automatically adjusted (e.g.
process)
• The utility is also run automatically by OUI
23. 2-System Upgrade
Upgrade OMS and Repository Manually
Repository Database Pre-requisites
• Verify your initialization parameters
• PROCESSES parameter is at least 300. The installer can also make the
changes if desired.
• Set JOB_QUEUE_PROCESSES to 0 (temporarily)
• Set SHARED_POOL_SIZE to 700MB or greater
• PGA_AGGREGATE_TARGET should be at least 1GB
• SGA_TARGET should be at least twice PGA_AGGREGATE_TARGET
• Grant execute on DBMS_RANDOM to DBSNMP and SYSMAN
• Unlock ORACLE_OCM account
• Increase your redo log size to at least 300MB due to
the high level of transactions during the repository.
• Check to make sure your emkey is in the repository
24. 2-System Upgrade
• Check to make sure your emkey is in the repository
SQL> select count(*) from MGMT_REPOS_TIME_COEFFICIENT;
COUNT(*)
----------
1
TIP
Create guaranteed restore point on both the old and
new repository database so that changes can be
easily flashed back;
create restore point PRE_OMS_UPGRADE guarantee
flashback database;
25. 2-System Upgrade
Upgrade OMS and Repository Manually
OMS Server Verification
• Verify that server meets pre-requisites
• Check ulimit to make sure it’s at least 4096
• ulimit –n
• If using RHEL 6 or Oracle Linux 6 – make sure you
have the following package installed or else your
installation may fail at OMS Configuration Assistant.
• glibc-2.12-1.47.0.2.el6_2.12.x86_64
• See MOS Note: EM12c: Installation Fails On Linux 6 at "Start
Oracle Management Service" step (Doc ID 1472216.1) for
workaround
33. 2-System Upgrade
Upgrade OMS and Repository Manually
• EM Prereq Kit generates false warnings for ORACLE_OCM
account if it has expired and processes parameter if less
than 1000!!
• Run EM Prereq Kit manually and double check database
39. 2-System Upgrade
Upgrade OMS and Repository Manually
• At the end of the installation you can verify whether if
the OMS is running.
/u01/app/oracle/Middleware/oms/bin/
emctl status oms -details
43. 2-System Upgrade
Post-Upgrade
• Create a DB link between the old and new
repositories
• Upgrade Console in old EM creates/recreates link
• The link between the two repositories allows
operations performed on the old repository to be
propagated to the new repository.
• Link also enables DDMP and ADDMP
44. 2-System Upgrade
• Create DB Link
• Connect descriptor found in MW_HOME/gc_inst/em/EMGC_OMS1/
emgc.properties
e.g. /u01/app/oracle/Middleware/gc_inst/em/EMGC_OMS1/emgc.properties
• Remove escape characters
$ cat emgc.properties |grep EM_REPOS_CONNECTDESCRIPTOR
EM_REPOS_CONNECTDESCRIPTOR=(DESCRIPTION=(ADDRESS_LIST
=(ADDRESS=(PROTOCOL=TCP)(HOST=host)(PORT=1521)))
(CONNECT_DATA=(SID=sid)))
45. 2-System Upgrade
Agent Upgrade Steps
• EM12c only communicates with 12c agents
• Deploy and configure 12c agents
• Generate and sign off on health check reports
• Switch over the agents to the new EM System.
• Brief outage incurred while 10g agents are stopped and 12c
agents are started
56. 2-System Upgrade
Post-Upgrade Steps
• Stop the OCM Scheduler in the 10.2.0.5 OMS
• Delete Obsolete targets (via console or EMCLI)
• EM Website
• EM Website System
• Oracle Application Server target of 10g OMS ORACLE
HOME consisting of:
• Oracle Application Server
• OC4J
• Oracle HTTP Server
• Web Cache
• Delete default incident Rule sets
57. 2-System Upgrade
Post-Upgrade Console
• Location changed in 12.1.0.2.0
• In 12.1.0.1.0 go to Setup -> Post Upgrade Tasks
• In 12.1.0.2.0 go to Setup->Manage Cloud Control->Post
Upgrade Tasks
12.1.0.1.0
12.1.0.2.0
58. 2-System Upgrade
Post-Upgrade Console
• Track the status of Accrued Data Migration (ADMP)
job
• ADMP is data accrued since the repository backup
and agent switchover
59. 2-System Upgrade
Post-Upgrade Console
• Track the status of Deferred Data Migration (DDMP)
job
• DDMP is historical metrics, configuration data which
is moved over time (lazy migration)
63. How far did the rabbit hole go?
• Host level jobs on 10.2.0.5 EM should work before
starting agent deployment
• Configure PAM on Linux/UNIX for 3rd party authentication if
in use. E.g. LDAP, Kerberos, SecureID
• Configure Privilege Delegation Provisioning (PDP). i.e. sudo
or Powerbroker for easier deployment
• Verify that OMS host server can send emails for
notifications
• Some servers require authentication for SMTP relay
• Cross-platform repository migration is your
responsibility!
• Complex but not scary. Read instructions carefully.
• Test, Test and Test Again!
66. References
• OTN Enterprise Manager Page
• Announcing Enterprise Manager Cloud Control 12c Release 12.1.0.1
Bundle Patch 1(BP1) and 12.1.0.2 Plug-ins [ID 1395505.1]
• Upgrade Advisor: Enterprise Manager 2-System Upgrade from 10.2.0.5
or from 11.1.0.1 to 12.1.0.1 [ID 327.1]
• EM 12c: Upgrading to Enterprise Manager Cloud Control 12.1.0.1 on
Linux x86_64 with the 2-System Method [ID 1367919.1]
• Grid Control Repository: Steps for Cross Platform Migration of the 10g
Grid Control Repository [ID 388090.1]
• EM12c R2: How to Upgrade to EM 12c R2 - Two System Method [ID
1363162.1]
• EM 12c R2: How to Apply Upgrade Console Patch for Upgrading to EM
12.1.0.2 [ID 1362537.1]
• Grid Control Repository: How to Recompile Invalid SYSMAN Objects?
[ID 372706.1]
67. SAVE THE DATE!
COLLABORATE 13
April 7-11, 2013
Colorado Convention Center
Denver, Colorado
http://collaborate13.ioug.org