Multiple Shared Processor Pools In Power Systems

© 2009 IBM Corporation

Multiple Shared-Processor Pools
in Power Systems

Session: SMP34

Consulting IT Specialist
Christopher Hales IBM Corporation Outstanding Technical Achievement Award 2008
Invention Achievement Award 2009
High Value Patent Application Award 2009

STG Technical Conferences 2009 © 2009 IBM Corporation

Trademarks
The following are trademarks of the International Business Machines Corporation in the United States, other countries, or both.

Not all common law marks used by IBM are listed on this page. Failure of a mark to appear does not mean that IBM does not use the mark nor does it mean that the product is not
actively marketed or is not significant within its relevant market.
Those trademarks followed by ® are registered trademarks of IBM in the United States; all others are trademarks or common law marks of IBM in the United States.

For a complete list of IBM Trademarks, see www.ibm.com/legal/copytrade.shtml:
*, AS/400®, e business(logo)®, DBE, ESCO, eServer, FICON, IBM®, IBM (logo)®, iSeries®, MVS, OS/390®, pSeries®, RS/6000®, S/30, VM/ESA®, VSE/ESA,
WebSphere®, xSeries®, z/OS®, zSeries®, z/VM®, System i, System i5, System p, System p5, System x, System z, System z9®, BladeCenter®

The following are trademarks or registered trademarks of other companies.

Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries.
Cell Broadband Engine is a trademark of Sony Computer Entertainment, Inc. in the United States, other countries, or both and is used under license therefrom.
Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.
Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.
Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel
Corporation or its subsidiaries in the United States and other countries.
UNIX is a registered trademark of The Open Group in the United States and other countries.
Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both.
ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office.
IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency, which is now part of the Office of Government Commerce.
* All other products may be trademarks or registered trademarks of their respective companies.

Notes:
Performance is in Internal Throughput Rate (ITR) ratio based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput that any user will
experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed.
Therefore, no assurance can be given that an individual user will achieve throughput improvements equivalent to the performance ratios stated here.
IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply.
All customer examples cited or described in this presentation are presented as illustrations of the manner in which some customers have used IBM products and the results they may have achieved. Actual
environmental costs and performance characteristics will vary depending on individual customer configurations and conditions.
This publication was produced in the United States. IBM may not offer the products, services or features discussed in this document in other countries, and the information may be subject to change without
notice. Consult your local IBM business contact for information on the product or services available in your area.
All statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.
Information about non-IBM products is obtained from the manufacturers of those products or their published announcements. IBM has not tested those products and cannot confirm the performance,
compatibility, or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.
Prices subject to change without notice. Contact your IBM representative or Business Partner for the most current pricing in your geography.

2 Multiple Shared-Processor Pools in Power Systems 2009 IBM Server Systems Technical Conference Christopher Hales © 2009 IBM Corporation

Note: Statements regarding SMP servers do not imply that IBM will introduce a system with this capability

Multiple Shared-Processor Pools
in POWER6 Systems

Abstract: With virtualization becoming more ubiquitous in server estates, the facility of Multiple Shared-
Processor Pools (MSPPs) in the Power Systems severs is an enhancement to the extensive PowerVM
portfolio. Using MSPPs as can make capacity management and administration easier, and contain software
license exposure. After a short primer on PowerVM partitioning, the co-inventor of the MSPP technology
with take you through the basic concepts, the implementation and configuration of MSPPs, and conclude
with some considerations/decision-points when architecting virtualized services with MSPPs.

Bio: Christopher Hales is a Systems Architect and Consulting IT Specialist based in the UK and currently
works in the IBM Systems and Technology Group - Major Contracts Team. Chris has been designing and
implementing IT solutions on behalf of customers for over 22 years. He specializes in virtualization
technologies and has assisted several large IBM customers to deploy virtualized infrastructures across a
number of platforms. In 2007, Chris was uniquely invited on an internship at IBM Austin development labs
where he worked on the high-end POWER6 servers, and early in 2008 was awarded an Outstanding
Technology Achievement Award by the IBM corporation for his work on Multiple Shares-Processor Pools.
Chris holds an honours degree in computer science.

Contact:
CHRISTOPHER HALES BSc(Hons), MBCS, CITP, MIET
SYSTEMS ARCHITECT : Consulting IT Specialist
Mobile +(44)07967 275825 (Mobex 37-275825)
email: Christopher_Hales@uk.ibm.com



Contents

Opening Remarks
PowerVM Partitioning Primer
Deployment Sets
MSPPs – POWER6 implementation
MSPP Considerations
Further Info. and Next Steps



POWER6 Systems range

BladeCenter Power 520 Power 550 Power 560 Power 570 Power 595
JS12/22/23/43

Size: Blade Size: 4U Size: 4U Size: 4/8U Size: 4/16U Size: 24” Rack
CPU: 2/4/8 CPU: 1/2/4 CPU: 2/4/6/8 CPU: 4/8/16 CPU: 2/4/8/12/16 CPU: 8-64
Speed: 3.8 to Speed: 4.2 to Speed: 3.5 to Speed: 3.6GHz 4//8/16/24/32 Speed: 4.2 GHz
4.2 GHz 4.7 GHz 5.0GHz Mem: 384GB Speed: 3.5GHz to 5.0 GHz
Mem: 64/32GB Mem: 64GB Mem: 256GB PCI-X: 83 5.0 GHz Mem: 4TB
Slots: 2 PCI-X: 58 PCI-X: 58 Disk: 73GB Mem: 768GB PCI-X: 640
Eth: 1000Mbps Disk: 73GB Disk: 73GB 5.4TB PCI-X: 212 Disk: 73GB
Disk: 73GB 1.8TB 1.8TB Disk: 73GB 28.1TB
10.8TB

1-way to 64-way UNIX Servers Binary compatible throughout
Supports a scale-out or scale-up Very-high built-in reliability & availability
Scale-within via virtualization Industry leading robust virtualization
High Performance and throughput



Technology Roadmap for IBM POWER
path to current product technology
1 Planned - All statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.

Binary Compatibility
POWER5 & POWER5+ POWER6 POWER71

POWER5 POWER6 POWER7
130 nm 65nm 45nm
1.5GHz 4.7GHz Multicore

Enhanced
?GHz
L2 L2
Cache Cache Advanced Cache
2.3GHz 5.0GHz Multicore
structure

System Interconnect System Interconnect
Enhanced

Enhanced
?GHz
L2 L2
Shared L2 Cache
Cache Cache Advanced Cache
POWER5+ POWER6+ structure
90 nm 65nm
Distributed Switch System Interconnect
Distributed Switch Distributed Switch
System Interconnect

2004 and 2006-7 2008 and 2009 20101

Autonomic Computing Enhancements
64-way SMP Enhanced virtualization ???-core SMP
Simultaneous multi-threading Partition Grouping Enhanced virtualization
Micro-partitioning (254 partitions) Partition Mobility Partition Grouping
Virtual Storage, Virtual Ethernet Advanced memory subsystem Enhanced Partition Mobility
Dynamic firmware updates Enhanced error detection/recovery Ensemble management
Enhanced memory subsystem Enhanced error detection/recovery


Contents

Opening Remarks
Deployment Sets
MSPP Considerations



Mapping hardware resources to LPARs
dLPAR granularity of 1 processor, 16MB, or 1 I/O slot

Central Electronics Complex

CPU Media &
CPU Memory Disk
CPU Packs
CPU
IO Slots

Memory

IO
Slots



Power Systems Logical Partitioning:
run completely isolated images and different workloads

LPAR #1 LPAR #2 LPAR #3

CPU CPU CPU CPU

Memory

IO
Slots

AIX 6.1 AIX 5.3 Linux

HMC POWER Hypervisor




LPAR #1 LPAR #2 LPAR #3 LPAR #4

CPU CPU CPU CPU

Memory

IO
Slots

AIX 5.3 AIX 5.2 AIX 6.1 Linux





LPAR #1 LPAR #2

CPU CPU CPU CPU

Memory

IO
Slots

AIX 6.1 Linux





LPAR #1

CPU CPU CPU CPU

Memory

IO
Slots

AIX 6.1





LPAR #1 LPAR #2 LPAR #3

CPU CPU CPU CPU

Memory

IO
Slots

AIX 5.3 AIX 6.1 Linux




Architecture for CPU Capacity Entitlement

Dedicated-processor
Micro-Partitions
partitions

AIX V5.2

AIX V5.3

AIX V6.1

AIX V6.1

AIX V5.3

AIX V6.1

AIX V6.1

AIX V5.3
AIX V5.3

Linux

Linux
0.75 CPU

0.75 CPU

0.85 CPU
0.5 CPU

1.6 CPU

0.5 CPU

0.6 CPU

0.5 CPU
2 CPU 3 CPU 1 CPU

POWER Hypervisor

cpu
cpu cpu
cpu cpu
cpu
cpu
cpu cpu
cpu cpu
cpu
cpu
cpu cpu
cpu cpu
cpu
Dedicated processors Physical shared-processor pool



POWER6: Optimised CPU Capacity Focus
cycle redistribution, improved throughput/server-utilisation

Capped Micro-Partition
Defined CPU entitlement

AIX V6.1

AIX V5.3

AIX V6.1

AIX V6.1

AIX V5.3
AIX V5.3
Entitlement is guaranteed

Linux

Linux
Partition can utilise CPU
Entitlement
capacity up to the
entitlement limit (never
more)

0.75 CPU

0.75 CPU

0.85 CPU
workload

0.5 CPU

1.6 CPU

0.5 CPU

0.6 CPU

0.5 CPU
Unused CPU capacity is
ceded back to pHype

Redistributed
Unused
Entitled
Cycles

Cycles

Cycles
Uncapped Micro-Partition
Defined CPU entitlement
Entitlement is guaranteed
POWER Hypervisor
Partition can utilise more
Entitlement
CPU capacity than
entitlement if available cpu
cpu cpu
cpu
cpu
cpu cpu
cpu
Additional capacity cpu
cpu cpu
cpu
workload

allocated on weighted
basis Physical shared-processor pool
Unused CPU capacity is
ceded back to pHype


Calculating extra cycle allocations
unused cycles are harvested from the pool and reallocated
W : Weighting factor (0 - 255)
RμP : Runnable μP
WμPn : Weight factor of the target μP
HCk : Harvested processor Cycles (k)
RμP

∑W : Sum of the weighting factors for all runnable μP' s
0

WμPn Formula for the actual number of processor cycles allocated to the target µPartition on
HCk ( RμP )
this pHype partition table inspection cycle
∑W 0

Calculation for allocation of harvested processor cycles
Values for the RμP : 8 in this pHype cycle to eligible (runable) µPartitions
calculation of
harvested WμPn : 75 WμPn 75
processor cycles to
HCk :1,500,000 HCk ( 0
) ≈ 1,500,000 x ≈ 251,116
be allocated in this
∑W
RμP
448
pHype cycle to
eligible (runable) ∑W : 448 μ
R P
µPartitions 0



Capped and Uncapped microPartitions
Power Systems Servers

Micro-Partions

Processor capacity

1 2 3 4 5 6 7 8
Set of micro-partitions



Capped and Uncapped Micro-Partitions
Power Systems

WμPμPn
Wn
HCkμPRμP )
R
(
Harvested Cycles = 1000 ∑∑W W
0 0

Runable µPartitions Total Weight=450 Additional Cycles

75
µPa W=75 = 0.1667 167
450

µPb W=150 150 333
= 0.3333
450

225
µPc W=225 = 0.5000 500
450



Power Systems: Partition Capacity Utilisation
pHype satisfies demand for extra CPU resource

Micro-Partions

Processor capacity

1 2 3 4 5 6 7 8



Contents

Opening Remarks
Deployment Sets
MSPP Considerations



Power Systems: Deployment Sets
Sets of functional-servers deployed as Partitions
Deployment Set
Par0
Protocol Dispatcher
Par1

Par2

Linear Partition layout & Management
Par3
HTTP HTTP HTTP
Server Server Server Par4

Par5

Scope of EMA0
Par6

Par7
Appln Appln Appln
Server Server Server Par8

Par9

Par10

Par11
DB DB
Server Server

Par254



Deployment Set

Par
Protocol Dispatcher
Par
Par
Par
HTTP HTTP HTTP
Sales & Server Server Server Par
Par
Marketing
Par
Par

Linear Partition layout & Management
Appln DeploymentAppln
Set Appln
Server Server Server Par
Protocol Dispatcher
Par
Par
Par
DBHTTP DBHTTP HTTP
Server
Server Server
Server Server Par
Par

Scope of EMA0
Par
Manufacturing Par
Appln Appln Appln
Par
Deployment Set
Par
Protocol Dispatcher
Par
DB DB
Server Server Par
Par
HTTP HTTP HTTP
Server Server Server
Par
Par
Par
Head Office Par
Appln Appln Appln
Par
Par
Par
DB
Server
DB
Server
Par
Par

Par



Par
Par
Par

Scope of EMA0
Par
Sales & Par
Par
Marketing
Par
Par
Par
Par
Par
Par
Par

Scope of EMA0
Par

Scope of EMA1
Par
Manufacturing Par
Par
Par
Par
Par
Par
Par
Par

Scope of EMA0
Par
Par
Head Office Par
Par
Par
Par
Par
Par
Par

Par




Micro-Partions

Processor capacity

1 2 3 4 5 6 7 8




Partition Group Partition Group Partition Group
Processor capacity

1 2 3 4 5 6 7 8 1 2 3 4 5 1 2 3 4 5
Set of micro-partitions Set of micro-partitions Set of micro-partitions



Contents

Opening Remarks
Deployment Sets
MSPP Considerations



Please be aware !
Terminology alert

Database Server

Database Server
Database Server
Database Server

Web Server

Web Server

Web Server

Web Server
Web Server

Web Server
Web Server

Web Server

App Server

App Server

App Server

App Server

App Server

App Server
App Server

App Server

POWER Hypervisor

cpu cpu cpu cpu cpu cpu cpu cpu cpu cpu cpu cpu cpu cpu cpu cpu

Pool 0 Pool 1 Pool 2
POWER6 processors Pool 3



Please be aware !
Terminology alert

IBM’s function of “Multiple Shared-Processor Pools”
are implemented logically
MSPPs are not different physical pools of shared-
processors
MSPPs a name given to a function
● implies a set of behaviour / configuration interface

POWER Hypervisor

cpu cpu cpu cpu cpu cpu cpu cpu cpu cpu cpu cpu cpu cpu cpu cpu

Pool 0 Pool 1 Pool 2
POWER6 processors Pool 3

Think of MSPPs as “Micro-Partition Groups/Sets”



POWER6: Multiple Shared-Processor Pools
part of PowerVM™ virtualization capabilities

Supported on POWER6 servers only
● Licensed as part of PowerVM
● Standard Edition and Enterprise Edition



POWER6: Multiple Shared-Processor Pools
multiple deployment-sets within a single server
MSPP definition from the Redbook
● “capability allows a system administrator
to create a set of micro-partitions with the
purpose of controlling the processor
capacity that can be consumed [by the set]
from the physical shared-processor pool”

Physical Shared-Processor Pool
● Physical processors installed in a Power
Systems server that are used to run micro-
partitions
● Only one physical pool of processors in a
Power System
Note: definition of processor capacity
available to the physical shared-processor
in Redbook



POWER6: MSPPs Architecture

Level1 capacity
resolution
POWER Hypervisor
Level1 capacity
harvests unused resolution
processor capacity
from Shared-
Processor Pools and
redistributes it SPP0 SPP1 SPP2 SPPn
across all uncapped
micro-partitions

Micro-partition n
Micro-partition0

Micro-partition1

Micro-partition2

Micro-partition3
regardless of the
Shared-Processor
Pool structure

Level0
capacity SPP0 capacity SPP1 capacity SPP2 capacity SPPn capacity
resolution resolution resolution resolution resolution

Level0 capacity POWER Hypervisor
resolution
Resolution of
the Entitled Pool p60 p61 p62 p63 p64 p65
Capacity within
the same
Shared- Physical Shared-Processor Pool
Processor Pool



POWER6: MSPP Characteristics & Definitions
Definition
MPC 5
Reserved Pool Capacity
Shared-Processor Pooln CPU capacity allocated to
the MSPP but not to any

EC 1.0
specific Micro-Partition
Processor capacity

MSPP guaranteed to
receive this CPU capacity

EC 0.5

EC 0.5

EC 0.5

EC 0.5

EC 0.5

Database Server
Entitled Pool Capacity
Web Server

Web Server

Web Server

Sum of Pars’ entitlements
App Server

App Server

PLUS the Reserved Pool
RPC 1.0
Capacity
1 2 3 4 5 6

Up to MaxPoolCap

MPC = 5 Maximum Pool Capacity
RCP = 1.0
EPC = 4.5 Maximum CPU capacity of
Σmicro-partition entitled capacity = 3.5
the MSPP (upper boundary
KEY:
EC Entitled Capacity for the micro-partition
of the capacity)
RPC
MPC
Maximum Pool Capacity
Up to capacity of system
n Shared-Processor Pool identifier



POWER6: Dynamic adjustment of capacity
Definition
MPC
Shared-Processor Pooln Reserved Pool Capacity
Dynamic change via HMC
Entitled Pool Capacity
Processor capacity

Changes in Partitions CPU
Additional Partitions
Removal of Partitions
Database Server
Changes to ResPoolCap
Web Server

Web Server

Web Server

App Server

App Server

Cannot > MaxPoolCap
RPC Max Pool Capacity
1 2 3 4 5 6

Dynamic change via HMC
Up to max. physical
KEY: processor-pool capacity
RPC Reserved Pool Capacity Can “over-commit” as this is
MPC Maximum Pool Capacity
n Shared-Processor Pool identifier an upper limiting threshold



POWER6: “Capping” an MSPP
Definition
MPC 4.5
Shared-Processor Pooln MSPP “Capping”
● EntPoolCap = MaxPoolCap

EC 1.0
Processor capacity

Unused cycles within the
EC 0.5

EC 0.5

EC 0.5

EC 0.5

EC 0.5

Database Server
MSPP are harvested by
Web Server

Web Server

Web Server

App Server

App Server

pHype
1 2 3 4 5 6 RPC 1.0
● pHype redistributes
Set of micro-partitions harvested cycles
MPC = 4.5
RCP = 1.0
EPC = 4.5
Σmicro-partition entitled capacity = 3.5

KEY:
RPC Reserved Pool Capacity



Special characteristics of default MSPP0
default settings and POWER5 (p5Hype) compatibility
Par
POWER5 operational Par0 Par
compatibility is provided by Par

Scope of EMA0
Par1 Par
the Default MSPP Par
Par
Par2
Fixed characteristics Par
Par
MSPP identifier = 0 Par3 Par

MaxPoolCap = physical Par
Par
Par4
shared-processor pool Par

MaxPoolCap cannot be Par

Scope of EMA0
Par5 Par
changed

Scope of EMA1
Scope of EMA0
Par
ResPoolCap =0 Par6 Par

EntPoolCap = as normal Par7
Par
Par
(sum of Par capacities) Par
Par
Par8 Par
All Partitions created in Par

MSPP0 by default unless Par9 Par

Scope of EMA0
Par
created in alternate active Par10
Par
Par
MSPP Par
Par11 Par
Partitions can be Par
Par
transferred in/out of MSPP0 Par
Par
Par254
Par



MSPP: Creating/Removing MSPPs

64 MSPPs defined and “exist” 64 Shared-Processor Pools defined
Shared-Processor Pooln
Only MSPP0 is active (by default) Dormant

● All others are inactive Shared-Processor Pooln

Processor capacity
Create/Activate MSPP

● Set MaxPoolCap > 0 [zero]

Database Server
Web Server
Web Server

Web Server

App Server

App Server
● Activates MSPP and will accept
Partitions
1 2 3 4 5 6
● (can be transferred from MSPP0 or created Set of micro-partitions

within this MSPP) Shared-Processor Pooln

Dormant

Remove/Deactivate MSPP Shared-Processor Pooln

● Remove all Partitions Dormant

● Set MaxPoolCap = 0 [zero]
Dormant


Dormant



MSPP: Dynamically move partitions
between Shared-Processor Pools
Web Facing Intranet facing
MPC 5 MPC 5
Shared-Processor Pooln Shared-Processor Pooln

EC 1.0

EC 1.0
Processor capacity

Processor capacity

EC 0.5

EC 0.5

EC 0.5

EC 0.5

EC 0.5

EC 0.5

EC 0.5

EC 0.5

EC 0.5

EC 0.5

Database Server
Database Server

Web Server

Web Server

Web Server
Web Server

Web Server

Web Server

App Server

App Server

App Server

App Server
1 2 3 4 5 6 RPC 1.0 1 2 3 4 5 6 RPC 1.0

Set of micro-partitions Set of micro-partitions

MPC = 5 MPC = 5
RCP = 1.0 RCP = 1.0
EPC = 4.5 EPC = 4.5
Σmicro-partition entitled capacity = 3.5 Σmicro-partition entitled capacity = 3.5

KEY:
RPC Reserved Pool Capacity



Multiple Shared Processor Pools In Power Systems

Multiple Shared Processor Pools In Power Systems

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Multiple Shared Processor Pools In Power Systems

Similar to Multiple Shared Processor Pools In Power Systems (20)

More from Andrey Klyachkin

More from Andrey Klyachkin (9)

Recently uploaded

Recently uploaded (20)

Multiple Shared Processor Pools In Power Systems