Optimizing the Enterprise Manager 12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Optimizing the Enterprise Manager
12c Environment
Enterprise Manager 12c, Release 4
Kellyn Pot’Vin-Gorman
Consulting Member of Enterprise Manager Technical Team
Strategic Customer Program
May, 2015
Oracle Confidential

Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.
Oracle Confidential 4

Program Agenda
1
2
3
4
5
Oracle Cloud Control User Interface, (Console)
Oracle Management Service, (OMS)
Oracle Management Repository, (OMR)
Metric Collections and Jobs
Weblogic and Agent Tuning
Summary
6

The Challenge
•Slow Login Page Response
•Slow EM Page Response
•No Data Results to Screen
•Slow Refresh of Data Upon Request
•Job Delays
•Upload or Collection Delays/Not Uploading

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 7
Architecture

Program Agenda
1
2
3
4
5
Oracle Cloud Control User Interface, (Console)
6

Health Overview
Setup Manage Cloud Control Health Overview

Console Performance Page
Setup  Manage Cloud Control  Health
Overview
OMS and Repository  Monitoring  Page
Performance

Page Performance
OMS and Repository  Monitoring  Page Performance

Page Correlation vs. Network/OMR/OMS

Which Pages Are Highest Impacting
OMS Repository  Monitoring  Page Performance

Java or Agent?

SQL and Plan
SQL from Page
Plans
Compare to awrsqrpt.sql
(aka sql_id specific awr)

Job Run
Not to be confused
with user sessions

Load Balancers

Why a Load Balancer?
Load Balancer setup and configuration
– Users require session affinity to a particular OMS
This allows for new connections to perform Round-Robin vs. Most Requested
which is less efficient.
Allows for simpler scaling for environments to support future growth.

Distributing Work Load to Multiple Oracle Mgmt Servers
Setup  Manage Cloud Control  Repository

Using the Access Log
$OMS_HOME/EMGC_OMS1/access.log
•Tracks each login, along with app/module logged in from:
Use NSLookup to locate workstation name.

Using EM CLI to Track UI Page

Controlling Login and Login Time
•Automatically controls logout of inactive sessions
•Single configuration setting for ALL OMS in environment-
•default is 45 minutes, but can be changed with EMCTL command:
$emctl set property -name
oracle.sysman.eml.maxInactiveTime -value 30

Have You Succeeded?
Bottom of Page Performance Page:
OMS and Repository  Monitoring  Page Performance

Program Agenda with Highlight
1
2
3
4
5
Oracle Management Service, (OMS)
6

What We Think Of When We Think of EM12c

Enterprise Level- Enterprise Manager Environment

EM12c Project Consideration
•Although it requires licensing, consider RAC-One Node for any production
repository to allow for easy scalability in future.
•8Gb for a small OMR, 12Gb+ for medium and 16Gb+ for large.
•Consider ASM to assist with tablespace maintenance and enhanced
management. Remember, everything is in the MGMT_TABLESPACE.
•Consider installing the OMS and OMR on separate hardware. Each requires
different resources, but can compete with each other if constrained.

Initialization Parameters for the Repository Database
Note the Compliance Checks

Why ODL Format Log Files Are Important
Oracle Diagnostic Log Files provide a lot of data in an aggregated format that
allows you to locate what you need if you know what you are looking for!
Located in following directories for EM12c:
$OMS_HOME/EMGC_OMS1
$OMS_HOME/EMCG_ADMINSERVER
$OMS_HOME/emoms
$OMS_HOME/nodemanager
$OMS_HOME/opmn_logs, etc., etc.
*Agent has similar directory structures

EMGC AdminServer Diagnostics Log File

Format Translations
Timestamp
Source and Type
Weblogic Server Mgmt Product
Component ID
Error Message
Execution Context ID
Transaction ID

Google is our Friend
Search using the error
message first, then add in
filters with the context or
type information to assist.

Tuning Java Heap Size
JAVA heap size for the OMS is a crucial setting. This value is used to
accommodate memory for each user logged in.
$OMS_HOME/user_projects/domains/GCDomain/bin/startEMServer.sh
– Use OMS_HEAP_MAX, OMS_HEAP_MIN, OMS_PERMGEN_MAX,
OMS_PERMGEN_MIN parameters to change the heap settings
$emctl set property -name OMS_HEAP_MAX -value 2560
– Stay between JAVA Heap size between 2Gb and 4Gb
Default values:
OMS_HEAP_MAX: 1740M
OMS_HEAP_MIN: 56M
OMS_PERMGEN_MAX: 768M
OMS_PERMGEN_MIN: 128M

Using EM Diagnostics
Repvfy(Repository Verify)
•Run level 9 –details
•Review Jmap output file.

EMGC_AdminServer.out
Includes Java Heap Settings in first five lines.
$OMS_HOME/EMGC_ADMINSERVER/EMGC_AdminServer.out
JAVA Memory arguments: -Xms256m -Xmx512m -
XX:MaxPermSize=512m

1
3
2
4
5
Oracle Management Repository, (OMR)
6

Repository Support

The OMR Schema

Repository Space Usage in Table View

Partition Retention

Retention Killing Your Space?
Change at Table/Partition Level
> execute gc_interval_partition_mgr.set_retention('SYSMAN', <table name>, <number
of partitions to retain>);
Modify retention or view impact of changing.
> repvfy send set_purge_retention -name {ALERTS|ERROR} -id <num days>

Management Server Incidents

OMR High Redo
Very common, consider at least 500M-1G Redo logs with at least 5 groups
for high demand EM environments and consider retention data
>repvfy send set_purge_retention -name CONFIGURATION -id <num days>

The “Weight” of Different Tasks

Most Important Graphs for OMR
Setup  Manage Cloud Control  Repository Repository Tab

Upload Performance
•Uploads are collections of data from
agents to the OMS.
•If backlogs occur, experiences of-
•Delayed notifications
•Unresponsive agents
•Inaccurate information to console
•Delayed response to incidents from
jobs

SQL Baselines
For the SYSMAN schema, no SQL plan baselines should get captures, and no
baselines should be used. The use of SQL plan baselines can severely impact
the performance of the OMS and the repository (in a very negative way).
SQL> ALTER SYSTEM SET
optimizer_capture_sql_plan_baselines=false
SCOPE=both;
SQL> ALTER SYSTEM SET
optimizer_use_sql_plan_baselines=false SCOPE=both;

Repository Performance
•Average Collection Duration in Seconds
•Number of Collections in Backlog**
•Throughput per Second
•Metric Data Rollup Performance, (metric extensions included)

Task Worker Threads
These workers process tasks and messages from the internal subsystems.
These internal housekeeping jobs are executed by 2 main DBMS_SCHEDULER
jobs in the repository database:
One to execute small jobs (SHORT worker pool), and one to execute larger,
more time consuming jobs (LONG worker pool).
The number of 'workers' for each of these pools is configurable, and can be
changed. To set the minimum of 2 workers for both the short and long
running pool, use this command:
>repvfy send set_workers

1
5
2
3
4 Metric Collections and Jobs
6

All Target Metrics Are Not Created Equal- Uploads by Day
Exadata Storage Server Target: 27K rows, 9.5M
Cluster Target: 23K rows, 29M
Infiniband Switch: 27K rows, 11M
Database Instance: 122K rows, 7M
Oracle Home: 8K rows, 3M

How Much Metric Data Am I Receiving?
Data can be gathered by running an EMDiag Report:
>repvfy dump metric_stats
Or….

Top Data Loading Metrics
Setup  Manage Cloud Control  Repository  Metrics Tab

Demands and Consequences

Cluster Detail Bubble Map

Metric Collection Errors

Collection Performance Reconfigure
Update from Short Running to Long Running Graphed

Metric Raw Data
Metric data represents all the metric collections that are enabled against a
target (3 main settings).
Raw Data, 7 days by default. Largest data, take care in changing retention.
Once per day, raw data is aggregated by day, month and 366 days. To change
this retention:
>repvfy send set_data_retention -name {RAW|HOUR|DAY} -id <num days>

Advanced Troubleshooting
•Highlight a metric alert type
•Hover cursor to view options to:
•Problem Analysis
•Metric Details
•Target Home
OMS and Repository  Monitoring  Metrics Collections

Problem Analysis

Files Processed Vs. Backoff
Outage

Collection Performance- Improvement?

Repository Job Activity
>repvfy send set_job_retention -name <job module> -id <num days>

Optimizing OMS Jobs
•Backload of Jobs, consider updating the parameter job_queue_processes
up to 10.
•Edit the schedule for the daily jobs to run in off-peak time or more often to
lessen impact.

Job Pool Tuning
In any environment relying on EM to execute jobs and deployment
procedures, the size of the job connection pool from the OMS to the
repository will have to get tuned.
To change the site of the job pool connection pool, run this command on
each OMS in the environment:
>emctl set property -name
oracle.sysman.core.conn.maxConnForJobWorkers
-value <recommended> -module emoms

1
5
2
3
4
Weblogic and Agent Tuning
6

Weblogic Server- Stuck Threads
Stuck Thread Timeout (BEA-337 incidents in the WebLogic log/trc files)
From OMS in Admin Console:
Stuck Thread Timeout = 1800
Stuck Thread Timer Interval = 120rom OMS in
Admin Console:
Admin Console → Environment → Servers → EMGC_OMSx→ Tuning

Weblogic: LDAP Configuration Flag
For LDAP and SSO setups, always make sure the local LDAP can be used.
Without this set, users are unable to login, (even as SYSMAN) remotely. The
console will report remote authorization and authentication as down or
unavailable.
To correct, set in the Admin Console:
LDAP Provider Control Flag = Sufficient
Admin Console → Security Realm -> myrealm -> Providers → LDAP Provider

Agents
•Separate Management Area from all OMS/OMR
Performance Screens.
•How many targets are monitored by the agent.
•Incidents involving the agent or dependent targets.
•Last successful upload to the OMS.

Agents
Setup  Manage Cloud Control  Agents

Agent Down Detection
To eliminate staleness of agents after downtime and allow OMS's to catch up, force the agents to send heartbeats with
a more effective grace period by defining in the system. This grace period serves as a time period right after an outage
during which the OMS’s will NOT report any Agent as 'down'.
SQL> exec gc_diag2_ext.SetPingGracePeriod;
To set the grace period to a specific value, use this command:
SQL> exec gc_diag2_ext.SetPingGracePeriod(90);
The value for the grace period should be between 60 and 300 seconds. Higher values should only be used if specifically
instructed by Oracle.
Setup  Manage Cloud Control  Agents

View by Name Default

Agent Heartbeat
Incoming heartbeats from agents are “batched” to allow for bulk updates in
the repository. For optimal performance, the size of such a bulk operation
performed by a single thread in the OMS should never exceed 1000 Agents.
The default number of threads configured per OMS is two, which should be
sufficient to handle up to 2000 Agents per OMS. If the number of Agents is
above this, the number of threads to do the Agent heartbeat processing
should be increased:
>emctl set property -name
oracle.sysman.core.omsAgentComm.ping.heartbeatPingRe
corderThreads –value "3"

Auto Tuning Set in Agents
To make sure the auto tuning is enabled check the value of the
'enableAutoTuning' parameter:
>emctl getproperty agent -name enableAutoTuning
If the value is not set to true, change it, and restart the Agent.

Agents with Variety of Target Types
If the Agent is monitoring a various number of different target types, the
overall timeout value can be changed, to avoid having to specify the timeout
for each individual target type one by one:
>emctl setproperty agent -allow_new -name
dynamicPropsComputeTimeout -value 180

Agent Tuning on Incoming to OMS
Similarly, if there are a lot of targets (+50) of a given type discovered on the
Agent, and those targets are administered in bulk (by applying templates,
submitting jobs, or performing repetitive tasks), the number of incoming
connections from the OMS to the Agent can exceed the default of 25. For
those cases, the default value can be increased. Values above 50 should be
extreme cases, and need to be validated and verified by Oracle.
>emctl setproperty agent -allow_new -name
MaxInComingConnections –value 40

Agent Patching Tips
Pre-stage patches
To apply a set of patches automatically to an Agent while deploying it via the console, stage the patches
in the
>$OMS_HOME/install/oneoffs/<version>/Generic/ (generic platform
independent patches)
and
>$OMS_HOME/install/oneoffs/<version>/<platform>/ (platform specific
patches)
Every Agent path to apply will have to get copied on each of the OMS’s, to make sure that regardless
which OMS does the Agent install, the pre-staged patches will get applied.
Download Agent Software and Opatch
The Agent software for the managed platforms can be downloaded using the Self Update pages in the
console.
Before installing additional agents for other platforms, ensure that the OPatch download job has
completed. This job runs on a 24hr cycle, and will get the latest OPatch from My Oracle Support.

1
5
2
3
4
Summary
6

Summary
•Best Performance requires different skills for optimizing Enterprise Manager
Cloud Control than for a database or engineered system.
•There are impressive enhancements and bug fixes in the PSU patches and
yes, they exist for EM12c- the OMS, OMR and Agent. Address this FIRST!
•Don’t get overwhelmed- use the tools provided in the Manage Cloud
Control and rep/agt/oms vfy to inspect the environment on a quarterly basis.
•Don’t just settle with the default values for the environment- many times
these are set by development and you most often use your EM12c
differently than they imagined or plugins, etc. can change usage.

Staying on Top of PSU Patching
96% of issues experienced are addressed by applying OMS and Agent
patches. Please remember to provide tender loving care to your Enterprise
Manager… 

Tip Working With MOS and Service Requests
omsvfy “Diagnostic dump” contains a file called “env”.
File contains all lib files, all environment information, ports, etc. that will be
required for an service request. Upload this one file and save a LOT of time..


Want to Learn More?
DBAKevlar Blog Posts:
http://dbakevlar.com/2014/10/em12c-release-4-health-overview/
http://dbakevlar.com/2014/11/em12c-rel-4-oms-and-omr-health-part-ii/
http://dbakevlar.com/2014/12/oms-and-omr-performance-part-iii-metrics-
page/
http://dbakevlar.com/2014/11/agents-management-via-em12c-release-4-
console/
Change Retention:
http://docs.oracle.com/cd/E24628_01/doc.121/e24473/repository.htm#EM
ADM12785

Connect with me via Social Media:

Optimizing the Enterprise Manager 12c

Optimizing the Enterprise Manager 12c

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Optimizing the Enterprise Manager 12c

Similar to Optimizing the Enterprise Manager 12c (20)

More from Kellyn Pot'Vin-Gorman

More from Kellyn Pot'Vin-Gorman (20)

Recently uploaded

Recently uploaded (20)

Optimizing the Enterprise Manager 12c

Editor's Notes