CIBER Managed Services GmbH built a dynamic Cloud Infrastructure for SAP landscapes, based on a combination of IBM server, storage, database, and system management software. The new storage infrastructure, which consists of the IBM XIV Storage System and IBM SAN Volume Controller, offers individual service classes to suit different customer needs, maximizing operational efficiency and minimizing expenses. The SAP application landscapes run on both IBM Power Systems servers and IBM System x Blade Server technologies. Visit http://bit.ly/KWh5Dx to 'Follow' the official Twitter handle of IBM India Smarter Computing.

1. IBM SAP International Competence Center
CIBER creates next-generation
IT infrastructure solution for
SAP hosting services

2. “Through the 100 percent
virtualization you gain a dynamic
infrastructure and you get the ability
for an easy billing system.”
Carl-Philipp Müller
Lead Sales Operations Services
CIBER Managed Services GmbH
“XIV is simply amazing. The usage of
XIV real time copy, which is able to
copy many terabytes of data very
fast, reduces the downtime for
provisioning drastically. The main
focus as a service provider is the
sector of customer satisfaction, which
is especially improved by the usage
of the new XIV system.”
Markus Peltz
Data center Manager
CIBER Managed Services GmbH

3. CIBER creates next-generation IT infrastructure
solution for SAP hosting services
About this paper
CIBER Managed Services GmbH built a dynamic Cloud Infrastructure for SAP landscapes, based on a combination of IBM server,
storage, database, and system management software. The new storage infrastructure, which consists of the IBM XIV Storage System
and IBM SAN Volume Controller (SVC), offers individual service classes to suit different customer needs, maximizing operational
efficiency and minimizing expenses. The SAP application landscapes run on both IBM Power Systems servers and IBM System x
Blade Server technologies.
ter
nce Cen
IB M SA P
Intern ational
Compete
“IBM was chosen as a partner,
because of the very good long-
n
neratio term experience with them.
next-ge
creates solution for
CIBER Especially their integrated
ructure
IT infrast service
s concepts and their highly
SAP h osting
skilled consultants for very
special challenges, plus the
capability to implement the
required standards within
time and budget convinced
us to do this project with
IBM.”
Mats Johansson, Head of Operations, CIBER
Managed Services GmbH

4. Customer Objectives • Installed six IBM System x3650 M2 systems, powered by
• Provide a highly scalable and dynamic infrastructure to host quad-core 2.26 GHz Intel Xeon Processor L5520 series
customer-oriented SAP landscapes for small- and medium- processors, running Novell SUSE Linux Enterprise Server
sized businesses in a private cloud environment that will 10 and 11 as well as Windows Server 2003 and 2008
support the future growth of the business • Implemented a next generation three service class
• Establish high-performance infrastructure, featuring storage solution based on IBM SAN Volume Controller 5.x,
transparent and easy billing models for high customer IBM XIV Storage System and IBM System Storage DS3400
satisfaction • Implemented IBM Tivoli Storage Manager and two IBM
• Reduce CIBER’s IT operational costs without sacrificing System Storage TS3500 tape library systems, each with
service quality five LTO4 tape drives, 260 slots, and about 200 cartridges
• Operate a complete range of SAP Business Suite (migration speed to tape of 350 GB/h).
components on an integrated and shared storage
infrastructure and powered by either IBM Power servers or Customer Benefits
IBM System x BladeServer technology depending on • Ability to provide faster and more flexible and scalable
customer requirements SAP hosting services to businesses, with 99.9 percent
• Introduce reliable and comprehensive backup, resilience, virtualization and improved Service Level Agreements.
and disaster recovery solutions • Improved business continuity and higher availability even
• Increase the speed of new SAP application provisioning, to across two locations, using IBM SVC storage virtualization
help respond faster to business requests. and the IBM XIV Storage System
• Reduced workload for IT staff through simplified setup and
IBM Solution administration of the XIV Storage Systems
• Worked with IBM and SVA, an IBM Premier Business Partner, • Reduced operating costs with improved service quality
to design and implement a highly flexible and cost-efficient supported by three levels of storage service classes, an
infrastructure and a next-generation storage solution for SAP integrated and virtualized storage architecture, and new
in only four months server hardware
• Set up two IBM Power 560 servers, each with two POWER6 • Reduced time for provisioning and deployment of new
processors with four cores, and two IBM Power 750 servers, systems from weeks to hours with the new storage solution
each with four POWER7 processors with eight cores, running based on IBM XIV technology, using the XIV Snapshot
SAP applications on IBM AIX in about 40 LPARs, sharing functionality and the SVC FlashCopy functionality.
processor resources and direct mappings of LPARs to • Improved database performance with DB2 by a factor of
LUNs. The two POWER7 machines provide 170,000 SAPS eight in online operations
and process 12,500 sales order positions per hour between • Achieved higher level of automation in database
them environment
• Installed two BladeCenter H, both with six HS22 • Gained disk space savings and reduced data growth by
BladeServers powered by quad-core 2.40 GHz Intel Xeon deploying DB2 compression and the reclaimable Storage
L5530 processors, and four HS22 Blade servers powered by features (reduced size of tablespaces from about 1 TB
six-core 2.60 GHz Intel Xeon X5640 processors, running down to 300 MB for SAP NetWeaver Business Warehouse)
VMware vSphere 4, forming several SAP clusters and • The BladeCenter technology allows an option to upgrade
hosting guest operating systems Novell SUSE Linux online to current BladeServer models, such as HX5 and
Enterprise Server 10 and 11 as well as Windows Server 2003 also offers hot swap technology for blades to ensure high
and 2008 availability with the BladeCenter Open Fabric Manager, if
• DB2 for Linux, UNIX and Windows is the most appropriate needed.
database platform to support the SAP applications at CIBER
4

5. About CIBER
Background, starting point and objectives
CIBER is a full service IT partner founded in 1974 and Application Services provide comprehensive SAP-functional
headquartered in Denver, USA. It employs more than 8,500 staff and business process-related support. With Applications
at 85 locations in 18 countries, and generates annual revenues of Services, CIBER supports customers in their daily business.
about $1.037 billion. CIBER specializes in implementing and Furthermore Application Services support the continuous
supporting complex SAP and similar solutions for a broad improvement process. Additionally, CIBER renders SAP
customer base in multiple industry sectors. Software Support for all SAP Software Maintenance customers.
For these customers, the CIBER support organization works as
an integral part of the Global SAP support organization.
Service Management ensures comprehensive and reliable
service delivery. CIBER’s customers benefit from the efficiency
of individually assigned Service Teams delivering the agreed
services. A dedicated Service Manager coordinates the delivery
More than 400 of CIBER’s total of 1,300 SAP consultants work for of the agreed services, provides transparent reporting of all
CIBER Germany, which has particular expertise in retail and activities and KPIs, and represents the customer and their needs
consumer products, chemicals and pharmaceuticals, inside the CIBER service organization.
automotive industry, and financial services. With more than 20
years of SAP solution implementation experience, CIBER
Germany achieved sales of €55 million in 2009.
CIBER Managed Services GmbH is based in Freiburg,
Germany, and was established in January 2010 to provide SAP
application and operation services for small- and medium-sized
businesses from its two data centers, with more than 120
experienced consultants.
The Managed Services business specializes in post-
implementation-services, designed to preserve the initial
investments that customers make in their SAP implementations. Figure 1: CIBER Services
Managed Services consist of Operations Services, Application
Services, Software Support and comprehensive Service As a VMware Solution Provider with a special expertise in SAP
Management. implementations, CIBER has competencies and proven
processes in architecture and sizing, and also in deployment
Operations Services provide customers with a full range of SAP and analyzing complex virtualized SAP landscapes.
system maintenance capabilities, covering the provision and
operation of modern data centers, the necessary IT
infrastructure, and the administration and maintenance of SAP
and IT-related applications.
5

6. Initial IT environment
In order to offer operation services for SAP as a cloud provider, full service desk based on ITIL® V3. The CIBER tool and SAP
CIBER Managed Services needed to build up two completely Solution Manager enable the creation of a virtual server (either
new data centers. The new and flexible technologies allow using VMware vSphere 4 technology on IBM System x or using
CIBER to satisfy growing customer needs. As a future service, LPARs on IBM Power servers), the installation of the operating
CIBER wants to implement a self-service portal to allow system, assignment of the LUNs for storage and the deployment
customers to request SAP services. Subsequently, the of the SAP applications. The CIBER team needed next to
necessary server capacity, operating system, applications and consider the best infrastructure to support this business strategy.
storage could be provisioned and charged back automatically.
This would keep costs low and enable CIBER to serve smaller With very positive experience of IBM System x and IBM Power
clients cost-effectively with enterprise-class solutions. Systems servers, CIBER chose to continue on the same strategic
path, and selected IBM XIV Storage Systems with IBM SAN
CIBER selected SAP Solution Manager and its own CIBER Volume Controller to address performance, high availability
Solution Suite to provide the central interface, which includes a scenarios, and premium support for data center management.
Redundant FC
connection
Figure 2: SAP private cloud
6

7. Business challenges and project objectives
The thinking behind the technology strategy was based on the If an additional Java Stack is needed, this is installed in a
view that although customer workload was growing separate LPAR. The largest SAP environment provides up to
continuously, the nature of the demand was highly volatile. 170,000 SAPS in a total of 20 instances, processing up to 12,500
Providing an industry-standard IT services solution with server sales order positions per hour. On customer request the
capacity sold in fixed amounts would not deliver the kind of environment is set up as a Power HA for AIX cluster to support
flexibility that customers wanted, and CIBER wanted to create a high availability of logical partitions for business critical systems
solution that could scale up and down in performance – and in a two-node cluster with automatic failover.
allow customers to pay on a more cost-effective usage basis.
Continuing the theme of continuous availability, CIBER uses Live
As a starting point, CIBER needed an infrastructure capable of Partition Mobility, available on Power servers with POWER6
hosting 120 SAP instances with about 8,000 SAP users, and up processors and upwards. Live Partition Mobility enables the
to 20 systems for one customer in a private cloud environment. migration of entire running AIX partitions and hosted applications
The customers are offered free choice among different operating from one physical server to another without disrupting services
systems and platforms, from IBM AIX on IBM Power Systems to and loads. Combined with LPARs, SAN Volume Controller and
SUSE Linux Enterprise Server or Windows Server on IBM other virtualization technologies, Live Partition Mobility will
System x BladeServer technology. While CIBER recommends further enhance CIBER’s ability to respond to customer
that its customers use IBM DB2 for Linux, UNIX and Windows, demands.
some customers run MaxDB, Oracle, and MS SQL Server
databases. The infrastructure is capable of running all these Alongside the scale-up scale-down flexibility for individual
operating systems and databases, and the CIBER team can customers, CIBER planned for overall systems growth that would
support all of these systems. equip them to respond to the total business demand. The
solutions must therefore be able to expand in capacity as a
CIBER operates the complete range of SAP ERP software whole, while preserving the ability to offer each customer exactly
components, as well as SAP applications such as Supplier the right services.
Relationship Management (SRM), SAP ERP Human Capital
Management (HCM), Advanced Planning and Optimization Knowledge of new customer contracts, the customer queue,
(APO), Customer Relationship Management (CRM), SAP and planned growth predicted that the infrastructure at the
NetWeaver Business Warehouse (BW) and SAP NetWeaver current data centers would reach capacity in less than six
Portal. Additionally, CIBER offers its own solutions based on SAP months. CIBER planned to build the new data center in less than
All-in-One software, including Rapid Retail, QuickWear, four months to allow for sufficient transition time and maintain the
Fabricated Metals, Industrial Machinery and Components, delivery of high service levels for its SAP hosting customers.
Aerospace and Defense, and CIBER Solution Suite, a new
intuitive front-end for SAP Solution Manager. Each SAP instance
runs in its own LPAR.
7

8. Technical solution
Server architecture
Working with SVA and IBM, the CIBER team designed and Ultimately, however, CIBER’s clients select the platform: either
implemented a highly flexible and fully virtualized environment, Power Systems and AIX or System x and Linux or Windows,
capable of hosting almost every kind of SAP application depending on the combination of reliability, scalability and
landscape. predicted workload.
SVA, an IBM Premier Business Partner, provided implementation
services. SVA installed the SAN Volume Controller, XIV, x3650 IBM System x and IBM BladeCenter
and BladeCenters. Ten SVA and IBM technicians completed the CIBER runs IBM BladeCenter H chassis featuring IBM
implementation on time and on budget in a six-week period – BladeServer HS22, each of them powered by two 2.40
during part of which one of the data centers did not have a fully GHz quad-core Intel Xeon Processor E5530 series
functioning electricity supply. processors and four HS22 Blade servers powered by
hexa-core Intel Xeon Processor X5640 series processors
In essence, everything is virtualized – the servers, storage and at 2.6 GHz. The blades run multiple virtual servers –
network. When an application grows in workload and functional collectively around 500 systems – using a combination of
reach, its processes are distributed among multiple virtual VMware vSphere 4 as hypervisor, and Novell SUSE Linux
machines to ensure performance is maintained, using either Enterprise Server, and Windows Server as the SAP
VMware virtual machines on System x or logical partitions platform.
(LPARs) on Power servers. VMware delivers the freedom to be
fully supported for SAP production environments on both Linux CIBER uses IBM System x3650 M2 machines powered
and Windows operating system and multiple x86-Hardware. by 2.26 GHz quad-core Intel Xeon Processor L5520
series processors. These x3650 M2 servers provide
VMware virtual machines, whereby each allows at most 8 virtual infrastructure services, running Novell SUSE Linux
CPUs and 255 GB RAM, are smaller than the virtual machines Enterprise Server 10 and 11, as well as Microsoft
available in LPARs, which support maximum of 256 virtual CPUs Windows 2003 and 2008. These servers run without
and 1 TB RAM. When applications grow, this scale-out virtualization, and monitor the virtualized systems with
distribution of processes happens earlier on System x than on applications such as IBM Director, cloud monitoring
Power servers. tools, and SAP monitoring applications.
The result is that CIBER has gathered a great deal of experience IBM Power Systems
in the scale-out of SAP applications, particularly on System x. CIBER uses two IBM Power 560 and two IBM Power 750
Examples include separation of database and application server servers, running the IBM AIX operating system and IBM
(that is, moving from two-tier to three-tier configurations), DB2 for Linux, UNIX and Windows. Both Power 750
separation into multiple application servers, and separation of systems are equipped with four POWER7 processors
SAP Central Services. The SAP Central Services include the with 8 cores each and 512 GB of memory. The IBM Power
unique SAP application system services, including enqueue 560 servers are equipped with two POWER6 processors
server and message server. with 4 cores each and 128 GB of memory. The servers
each host around 40 logical partitions (LPARs).
8

9. Data center 1 Data center 2
RESOURCE POOL
Non Non
SAP SAP SAP SAP SAP SAP SAP SAP SAP SAP SAP SAP SAP SAP SAP SAP
SAP SAP
Win/MaxDB Linux/DB2 Non Win/Oracle Win/MS SQL
SAP SAP SAP Intel/DB2
SAP SAP SAP SAP Intel/DB2 SAP
ERP 6.0 ERP 6.0 SAP
SAP SAP
ERP 6.0
SAP SAP
ERP 6.0 SAP
SAP SAP
Non Non
SAP SAP SAP SAP SAP SAP SAP SAP SAP SAP SAP SAP SAP SAP SAP SAP
SAP SAP
ESX ESX ESX ESX ESX <<< ESX HA / VMWare Vmotion >>> ESX ESX ESX ESX ESX
LPAR LPAR LPAR LPAR LPAR PowerVM LPAR LPAR LPAR LPAR LPAR
Mirrored vDisks Direct Mapping
Direct Mapping
Productive non productive
non productive
1 TB 2.5 TB
2.5 TB
SVC MDISK SVC MDISK
SAP CRM SAP ERP
SAP BI DEV SAP EP DEV
SAP SolMan D SAP NW QA
SAP ERP P SAP ..
… …
Figure 3: High Availability Concept and storage cloud operating system
IBM BladeCenter and System x landscapes Virtualization using VMware in the Intel processor-based systems
For many customers, CIBER provides SAP hosting services on Almost all applications, perhaps 99.9 percent, on the Intel
IBM System x BladeServers. processor-based platforms are virtualized using VMware
vSphere 4. The only exceptions are monitoring and systems
For systems powered by Intel processors, CIBER follows a rule, management applications.
based on experience: There is an optimum relationship between
processor capacity consumed and memory required, which The Blade Servers host guest operating systems such as Novell
means that more memory does not necessarily increase the SUSE Linux Enterprise Server 10 and 11, as well as Microsoft
performance of the SAP applications hosted on that server. Windows Server 2003 and 2008.
CIBER found that 8 GB memory per core is sufficient. In the
future, CIBER may upgrade the blades with IBM BladeServer For the System x servers, CIBER recommends the Linux
HX5 technology. operating system to customers for the following reasons:
Provisioning of new SAP instances is completed using VMware • Lower license costs
templates, making it fast and easy to set up standardized • Linux installation can be configured to comprise just the
systems ready for immediate customer deployment. Additional software components needed, creating a low-footprint
tools such as TSM backup agents, monitoring agents, and operating system with relatively low maintenance
special SAP configurations are already integrated for quick requirements (Windows includes many software
deployment. components not actually required by the applications,
increasing the operating system footprint and increasing the
CIBER uses VMware vCenter Converter (also named patch requirements).
vConverter) to automate physical to virtual machine conversions • CIBER believes that Linux Logical Volume Manager (LVM) is
as well as conversions between virtual machine formats. In more flexible than the Microsoft Logical Disk Manager; for
particular, the converter is used to migrate native physical example, Linux LVM allows data replication (mirroring) at
systems into CIBER’s virtual environments. server level
9

10. The conversion takes place either at the customer site or in For disaster tolerance and disaster recovery capabilities, as well
CIBER’s central data center. CIBER uses Acronis True Image to as for high availability across locations, CIBER implemented an
create a backup image of the customer’s application systems enterprise-class storage design. It consists mainly of the IBM
environment and transport the image files from the client’s site to SAN Volume Controller (SVC) and the IBM XIV Storage System.
the data center, which eliminates the need for risky physical The SVC has been set up with the Split I/O Group design to
movement of hardware from place to place. achieve high availability even if one location fails completely.
Once customer systems have been virtualized, CIBER uses In 2011, CIBER plans to deploy VMware vCloud Director data
VMware Virtual Symmetric Multi-Processing (SMP) to enable a center management software, which pools virtual infrastructure
single virtual machine to use multiple physical processors resources and delivers them to users as a catalog-based
simultaneously. This allows CIBER to maximize utilization of its service. The target users are internal consultants who administer
physical processor resources, and provide the virtual machines the local data center systems.
with the capacity required easily and quickly.
CIBER also plans to implement VMware vCenter Chargeback,
With the applications and servers hosted by CIBER, the next which logs resource requests and consumption and enables
question is how to deliver them to the customers. VMware View is usage-based chargeback and accounting for hosted data
built on VMware vSphere for Desktops. CIBER runs its own data center operations.
center management and administration applications using
Customer 1
VMware View virtualized desktops.
VMware vCloud Director
For data storage, security and management, CIBER uses CIBER Solution Suite Catalogs Security
VMware vStorage VMFS (virtual machine storage file system), a Win/MaxDB
ERP 6.0
Linux/ DB2
BS7
Linux/SQL
BI
high-performance cluster file system optimized for VMware
virtual machines. A VMware virtual machine stores the contents Datacenter Configuration(Gold)
VMware
vShield
Datacenter Configuration (Silver)
of a virtual machine’s hard disk drive in a virtual disk file with the
extension .vmdk (virtual machine disk). The vmdk specification
describes and documents the virtual machine environment, and
how the contents of the hard disk drives is stored. CIBER uses
.vmdk files rather than storing the virtual disks on raw devices to VMware VMware
vCenter Server vCenter Server
ensure isolation of the data stored in separate virtual disks. VMware vSphere VMware vSphere
VMware Storage vMotion enables Live Migration of .vmdk files – Figure 4: Providing and consuming IT as a service
in other words, a .vmdk file can be moved from one location to
another even while it is in use. This ability helps CIBER to
minimize downtime during disk array migrations and storage
upgrades, or when the virtual machines require larger-capacity
storage devices.
10

11. IBM Power Systems landscapes
Typically, for larger customers CIBER hosts SAP ERP and SAP The VIOS provides full virtual paths from each LPAR to the
NetWeaver applications on IBM Power systems. For existing switched fabric in the data center. Switched fabric network
clients, CIBER has two IBM Power 560 and two IBM Power 750 topology allows nodes to connect with each other via one or
servers, running the IBM AIX operating system and IBM DB2 for more network switches, which provides both high total capacity
Linux, UNIX and Windows. and a high degree of resilience if a network switch fails.
LPARs are subsets of a computer’s hardware resources, CIBER has assigned a separate LPAR for storage management,
virtualized as a separate computer using the hypervisor. A with a dedicated Converged Network Adapter to support
physical machine can be partitioned into multiple LPARs, each concurrent LAN (TCP/IP) and SAN (FCoE, iSCSI) traffic over a
hosting a separate AIX operating system. The hypervisor shared 10 Gb Ethernet link. CIBER currently uses only TCP/IP
assigns a specific range of virtual processors to each of the and iSCSI protocols, and wants to be ready for FCoE in the
LPARs. CIBER does not configure dedicated assignments of future.
processors, and shares uncapped processor resources across
all the LPARs. PowerHA for AIX (previously named High Availability Cluster
Multiprocessing HACMP) provides high availability of logical
partitions (LPARs) in a two-node cluster by way of automatic
failover. When one server node goes down, the other server
node takes over. Depending on clients’ requirements, most
production systems and some of the quality assurance systems
running on AIX use PowerHA to ensure continuous availability.
For large SAP systems, CIBER prefers to use POWER CIBER uses IBM Virtual I/O Server (VIOS) to enable the sharing
technology and virtualization over VMware vSphere. This is for of physical I/O resources between client logical partitions within
two reasons: first, because AIX LPARs can scale to higher total the server. The total physical bandwidth can be allocated in fixed
processor and memory limits; and second, because AIX amounts to each server, or policies set to enable full capacity to
enables much higher processor utilization. By using AIX LPARs, be used by high-priority workload at peak times.
the SAP application can grow much larger before it is necessary
to scale-out over multiple servers.
The migration from IBM POWER5 to POWER6 and POWER7 was
without any problems, CIBER simply connected the new servers
to the SAN storage system and restarted the new servers and the
SAN storage system.
11