Contenu connexe
Similaire à Cloud Consolidation with Oracle (RAC) - How much is too much? (20)
Plus de Markus Michalewicz (20)
Cloud Consolidation with Oracle (RAC) - How much is too much?
- 1. 5/14/2012
1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Cloud Consolidation with Oracle (RAC)
– How much is too much?
Markus.Michalewicz@oracle.com Nitin.Vengurlekar@oracle.com
Senior Principal Product Manager Technical Director, Oracle RAC
2 Copyright © 2012, OracleRAC, Oraclereserved.
Oracle and/or its affiliates. All rights America Development, Oracle America
1
- 2. 5/14/2012
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features
or functionality described for Oracle’s products remains at the sole discretion of Oracle.
3 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Agenda
• Database Cloud Architectures
• General Considerations
• Database Consolidation
– Memory Management
– CPU(_COUNT) Management
• A word on Instance Caging
– Database Resource Management
– Summary Database Consolidation
• Oracle RAC-specific Considerations for Consolidation
– Per Server Limits
– Real Time (RT) Processes
– Cluster Limits
• Summary and Q&A
4 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
2
- 3. 5/14/2012
Database Cloud Architectures
Common building blocks are shared server and storage pools
Infrastructure Cloud Database Cloud
Database Cloud
DW CRM ERP DW ERP CRM DW ERP CRM
DB
DB
DB
DB
DB
DB
DB
OS OS OS
Hypervisor Hypervisor OS OS OS OS
Server Database Schema
5 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
General Considerations
6 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
3
- 4. 5/14/2012
Oracle (RAC) Database Consolidation
Registered vs. Running
• Registered databases and instances could Registered:
potentially start and run at the same time. • n DB instances
are defined to
run on a machine
• Oracle’s Quality of Service Management (potentially)
or scripts can be used to model policies
to run certain databases only at certain
times; e.g. geographic region over time vs.
• Assume the cluster is PST based:
Running:
• EMEA based DBs run 10:00pm – 8am PST • Registered databases
• APAC based DBs run 6:00pm – 3am PST and instances are
(concurrently) running
• USA based DBs run 8:00am – 6pm PST (active workload)
7 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Database Consolidation
General Considerations
• Simplest of all consolidation cases: Database Cloud
• One database Instance per Server
DW ERP CRM
• When consolidating more than one
DB
database on a server, consider the
server capacity with any DB added.
OS OS
• More details:
Best Practices for Database Consolidation in Private Clouds
http://www.oracle.com/technetwork/database/focus-
areas/database-cloud/database-cons-best-practices-1561461.pdf
• Exadata Database Machine-specific: Schema
http://www.oracle.com/technetwork/database/features/
availability/exadata-consolidation-522500.pdf
8 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
4
- 5. 5/14/2012
General Considerations
Consider Workload Characteristics during Capacity Planning
Existing Workload
Utilization
Peak
The smaller this
Average gap, the better.
Time
New Workloads
Workload A OR Workload B
Utilization
Utilization
Time Time
9 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
General Considerations
Consider Workload Characteristics during Capacity Planning
Existing Workload Resulting Workload
Utilization
Peak
The smaller this
gap, the better.
Average
Peak Poor match:
Utilization
Time
Gap increases
(antagonistic)
+ Average
Workload B
Time
Utilization
Time
10 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
5
- 6. 5/14/2012
General Considerations
Consider Workload Characteristics during Capacity Planning
Existing Workload Resulting Workload
Utilization
Peak
The smaller this
gap, the better.
Average
Utilization
Time Good match:
Peak
Gap decreases
Average
+
(complimentary)
Workload A
Time
Utilization
Time
11 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
General Considerations
Consider Workload Characteristics during Downtime
Utilization Normal Operation
Utilization
Peak
HR DW ERP CRM
Average
DBA
Time DBA
DBB
OS OS OS OS
12 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
6
- 7. 5/14/2012
General Considerations
Consider Workload Characteristics during Downtime
Utilization Normal Operation
Utilization
Peak
HR DW ERP CRM
Average
DBA
Time DBA
DBB
During Downtime
Peak OS OS OS OS
Utilization
Average
Time
13 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
The Benefits of Standardization
Easier deployment and better predictability
• Standardization of software and
hardware simplifies planning
• Standardized hardware means a
Nodes
predictable behavior should demand
increase and additional hardware needs
to be added (horizontal scaling approach)
New
• Using “application profiling” (template Application A
based deployment) based on current
system(s) and performance baselines Large
OLTP
Medium
allows for a predictable deployment of OLTP
Small
new applications on the same system OLTP
using existing profiles. Then deploy
14 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
7
- 8. 5/14/2012
Database Consolidation
15 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Database Consolidation
Starting block: One Database instance per server
• Components to consider:
• Memory
• CPU
DB
• I/O
• Processes
OS
• Network
16 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
8
- 9. 5/14/2012
Database Consolidation
One instance per server as the basis
• An Oracle database by default assumes that there is
only one database instance running on the server:
• Instance parameters are based on this assumption
• Consolidation changes that premise
• Main resources used:
DB
• Memory
• CPU
• I/O
• Resources “regulated by default”:
• Memory OS
• SGA / PGA Targets
• CPU
17 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Database Consolidation
Recommendation 1: Manage Memory carefully (and dynamically)
• Avoid memory starvation and swapping
as it has negative impact on the system.
• Do not oversubscribe memory resources
• Define memory settings carefully – rule of thumb:
• For the OS in general:
• Shared Memory identifiers and segments
DB
• Use Hugepages, if possible – details: 80%
• MOS notes 361323.1 and 401749.1
• For OLTP applications:
• SUM (sga_target + pga_aggregated_target)
<= 80% of physically available memory per DB server
OS 20%
• For DW / BI applications:
• SUM (sga_target + 3* pga_aggregated_target)
<= 80% of physically available memory per DB server
18 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
9
- 10. 5/14/2012
Database Consolidation
Recommendation 2: Use CPU_COUNT to “cage instances”
• CPU usage should be regulated.
• The OS scheduler schedules CPU as
requested by each individual instance.
• The OS scheduler does not know about the
DBB
priority of the various instances on the server.
• Use CPU_COUNT or ideally Instance Caging
DBA
• Instance Caging is configured in just 2 steps:
• 1. Set “cpu_count” parameter
• Max. number of CPUs the instance can use at any time
• 2. Set “resource_manager_plan” parameter OS
• Enables CPU Resource Manager
• E.g. out-of-box plan “DEFAULT_PLAN”
19 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Database Consolidation
Using CPU_COUNT as a “central knob”
• CPU_COUNT regulates CPU usage and Data Structures
dependent resources (to a certain degree): Concurrency
Parallelism
• Parallelism (PQ operations) fx(CPU_COUNT)
• Processes Processes
DBB
• Load Calculation Memory Allocation
Load Calculation
• Processes and PQ
DBA
operations should be 80%
considered explicitly.
16 OS 20%
20 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
10
- 11. 5/14/2012
A Word on Instance Caging
21 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Database Consolidation
Instance Caging: Partitioning Approach
• Provides maximum isolation
16 CPUs 32
28
• For performance-critical databases
DBA DBB DBC DBD
24
• If one database instance is idle, 20
its CPU allocation is unused 16
Instance D: 4 CPUs
12
• The rule of thumb for the partitioning Instance C: 4 CPUs
8
approach is to set a general limit: Instance B: 4 CPUs
• SUM (CPU_COUNT) < 75% x Total CPUs 16 OS 4
Instance A: 4 CPUs
0
22 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
11
- 12. 5/14/2012
Database Consolidation
Instance Caging: Over-Provisioning Approach
• Best used for non-critical databases
16 CPUs 32
that are typically well-behaved
28
DBA DBB DBC DBD
• Contention for CPU if database 24
instances are sufficiently loaded 20
Instance D: 8 CPUs
16
• Typically not enough contention
12
to destabilize OS or DB instances Instance C: 6 CPUs
8
Instance B: 4 CPUs
• Best approach if the goal 16 OS 4
Instance A: 4 CPUs
is fully utilized CPUs 0
23 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Database Consolidation
Instance Caging – under the covers
• If cpu_count is set to 4 on a 16 CPU server Over-Provisioning
• All foreground processes make progress Partitioning Approach Approach
• But only 4 foregrounds are running at any time 32 32
28 28
• Most backgrounds are not managed 24 24
• Critical and use very little CPU
20 20
• MMON, Job Scheduler slaves are managed Instance D: 8 CPUs
16 16
Instance D: 4 CPUs
• No CPU affinity 12 12
Instance C: 6 CPUs
Instance C: 4 CPUs
• Not meant for hard-partitioning or licensing 8 8
• All CPUs may be used Instance B: 4 CPUs Instance B: 4 CPUs
4 4
• CPU utilization averaged across all CPUs ≤ 25% Instance A: 4 CPUs Instance A: 4 CPUs
0 0
• More information:
http://www.oracle.com/technetwork/database/focus-areas/performance/instance-caging-wp-166854.pdf
24 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
12
- 13. 5/14/2012
Database Consolidation
Over-Provisioning Approach – It’s still hardware that’s the limit
• Best used for non-critical 140
120
databases that are typically 100
80
well-behaved – examples:
CPU Util.
60
Average
40
• Complimentary workload 20
0
• Systems with little contention t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 t12
for CPU if database instances 32
16 CPUs
are sufficiently loaded. 28
DBADBBDBCDBD
24
• Do not use, 20
Instance D: 8 CPUs
16
• If the load is significant and
of longer duration, as system 12 Instance C: 6 CPUs
stability can get impacted. 8
Instance B: 4 CPUs
• For highly critical systems 16 OS 4
Instance A: 4 CPUs
0
25 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Database Consolidation
How much over-provisioning is OK?
• As CPU_COUNT does not consider the
“quality of a CPU” an absolute maximum
32
is hard to determine / depends on the system. 16 CPUs
28
DBA DBB DBC DBD
• The general rule of thumb is:
24
• SUM (CPU_COUNT) <= up to 2x Total CPUs
20
• Consider different system types: Instance D: 8 CPUs
16
Threaded Core based Engineered
12
“Total CPUs” Instance C: 6 CPUs
# of threads # of cores # of threads
are based on 8
Max. over- Instance B: 4 CPUs
provisioning
1.5 (HT*)
2.0
3.0 (DBM) 16 OS 4
1.0 (hHT**) 2.0 (ODA) Instance A: 4 CPUs
factor
0
(HT*): ratio 1:2; (hHT**): ratio 1:n, with n >2
26 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
13
- 14. 5/14/2012
Database Resource Management
27 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Database Consolidation
Recommendation 3: Use DB Resource Manager
3 steps to use Resource Manager:
1. Group sessions with similar
performance objectives into
Consumer Groups
2. Allocate resources to consumer
groups using Resource Plans
3. Enable Resource Plan
28 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
14
- 15. 5/14/2012
Summary Database Consolidation
29 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Database Consolidation
Summary
1. Define memory settings carefully:
• For OLTP applications: 32
• SUM (sga_target + pga_aggregated_target) 28
<= 80% of physically available memory per DB server
DBB
• For DW / BI applications: 24
• SUM (sga_target + 3* pga_aggregated_target) 20
<= 80% of physically available memory per DB server 80% Instance D: 8 CPUs
DBA
16
2. Use CPU_COUNT or ideally Instance Caging 12
Instance C: 6 CPUs
• The general rule of thumb is:
• SUM (CPU_COUNT) <= up to 2 x Total CPUs 8
OS 20% Instance B: 4 CPUs
4
• Consider different system types. Instance A: 4 CPUs
0
3. These are the most crucial per server limits.
30 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
15
- 16. 5/14/2012
Oracle RAC-specific
Considerations for Consolidation
31 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Oracle RAC based Consolidation
Server limits are mostly reached before cluster limits apply
• Most customers will experience “per
server limits” before “cluster limits” apply.
• Oracle RAC introduces a few more
Per Server Limits
DBA1 DBB1 DBC1
DBA2 DBB2 DBC2
processes (potential limits) to consider.
• Oracle RAC DBs use LMS Real
Time (RT) processes per instance.
• LMS RT processes need to
be considered in particular
Clusterware Clusterware
16 OS 16 OS
Cluster Limits
32 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
16
- 17. 5/14/2012
Real Time (RT) Processes
33 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Oracle RAC based Consolidation
Considerations for Real Time (RT) Processes in general
• A Real Time process can only
run on one CPU (core) at a time.
• The usage of the CPU is typically short.
DBA1 DBB1 DBC1
DBA2 DBB2 DBC2
• The general rule of thumb is:
• The aggregated number of
RT processes per server should
not exceed the number of cores per server
• One Oracle RAC instance has typically at Clusterware Clusterware
least one RT process (LMS) per default
16 OS 16 OS
• An Oracle ASM instance has one RT process
• Oracle Clusterware uses various RT processes
34 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
17
- 18. 5/14/2012
Oracle RAC based Consolidation
Background for LMS Real Time (RT) Process recommendation
• The number of LMS RT processes per
instance is determined by a function on
CPU_COUNT.
• In order to guarantee optimized
DBA1 DBB1 DBC1
DBA2 DBB2 DBC2
performance and reliability, the
general rule of thumb for RAC is:
• The aggregated number of LMS
RT processes per server should
not exceed [cores per server]-1
• See MOS note: 558185.1 for details
Clusterware Clusterware
• This leaves one core free for additional RT
processes to be assigned as needed, as 16 OS 16 OS
LMS RT can stay on a core for a moment
35 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Oracle RAC based Consolidation
Automatic adjustment of LMS process priority in 11.2.0.3
• With 11.2.0.3 the number of LMS RT
processes are monitored and adjusted
according to the number of cores on
DBA2 DBB2 DBC2 DBD2
DBA1 DBB1 DBC1 DBD1
the node periodically.
Per Server Limits
DBF1 DBF2
• The goal is to keep
RT LMSs per server <= # cores per server
• For details, see MOS note 1392248.1 –
Auto-Adjustment of LMS Process Priority
in Oracle RAC with 11.2.0.3 and later DBE1 DBE2
• This excludes any ASM instance running
on the system as well as any pre-11.2.0.3
database instance 4 OS 4 OS
• See also MOS note 1439551.1 –
Oracle (RAC) Database Consolidation Guidelines
for Environments using mixed Database Versions
36 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
18
- 19. 5/14/2012
Oracle RAC based Consolidation
Recommendation 4: Over-provision only based on CPU_COUNT
• Do not over-provision the number 48
.
of LMS RT processes on one server. .
.
.
DBA2 DBB2 DBC2 DBD2
DBA1 DBB1 DBC1 DBD1
.
Per Server Limits
.
• Limit the number of RT LMSs by: DBF1 DBF2
32
• Using CPU_COUNT
28
• Directly reducing the number of LMS 24
RT processes (gcs_server_processes)
DBE1 DBE2
20
• Downgrading additional LMS RT Instance D: 8 CPUs
16
processes to time share (TS).
12
• In 11.2.0.3 and later, the RT to CPU OS OS 8
Instance C: 6 CPUs
rule will be enforced by automatically Instance B: 4 CPUs
downgrading subsequently started 4
LMS RT processes to TS. Instance A: 4 CPUs
0
37 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Cluster Limits
38 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
19
- 20. 5/14/2012
Oracle RAC based Consolidation
Cluster Limits – Starting DBs are currently the main concern
• In most cases, per server limits will be Starting:
reached before cluster limits are reached. • Registered databases
and instances start
• Cluster limits apply to: • Default is “starting
• Registered resources (databases) at the same time”.
DBC1 DBD1
DBC2 DBD2
DBA1 DBB1
DBA2 DBB2
• Starting databases in the cluster – reason:
• Starting databases need to register
with the cluster (Oracle Clusterware).
• Currently and for example, 100 Clusterware Clusterware
starting Oracle RAC databases
on a 4 node cluster are supported. OS OS
• The Number assumes each Oracle RAC
database uses an instance on each node.
Cluster Limits
39 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Summary
40 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
20
- 21. 5/14/2012
(Oracle RAC) Database Consolidation
Summary
• For a successful database consolidation,
consider the following as rules of thumb: Existing Workload
Peak
DB
Utilization
80%
1. General considerations for capacity planning Average
2. Manage Memory carefully (and dynamically) Time
+ OS 20%
3. Use CPU_COUNT
4. Use DB Resource Manager
Data Structures
5. Over-provision only based on CPU_COUNT
Concurrency
Parallelism
Processes
Memory Allocation
Load Calculation
=
41 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
42 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
21