To maintain its position at the forefront of international research, the Institute for Computational Cosmology at Durham University wanted to develop a new high-performance computing cluster that would enable even more sophisticated simulations of the universe. With constraints on data centre space, power and cooling, the Institute looked to base the new cluster on compact, energy-efficient technologies. They Worked with OCF and IBM to design and deploy COSMA4, a 25 teraflop cluster based on water-cooled IBM System x iDataPlex servers with 2,640 Intel Xeon cores. IBM System Storage DS3500 hardware provides 610 TB of storage capacity, which is managed by the IBM General Parallel File System (GPFS) and IBMTivoli Storage Manager. For more information on Storage System, visit http://ibm.co/LIg7gk. Visit http://on.fb.me/LT4gdu to 'Like' the official Facebook page of IBM India Smarter Computing.

1. IBM Systems and Technology Group Education
Deep Computing
Durham University gains
new insight into the
evolution of the universe
Building an eco-friendly high-performance computing
cluster for cosmology research with IBM System x
iDataPlex
Durham University is the third oldest university in England after
Overview Oxford and Cambridge, and has been a leading European centre of
learning for over 1,000 years. With more than 14,000 students and
Business challenge 3,800 staff, the University’s 16 colleges provide degree courses in a wide
To maintain its position at the forefront
of international research, the Institute for
range of subjects across three faculties: Arts and Humanities, Social
Computational Cosmology at Durham Science and Health, and Science.
University wanted to develop a new
high-performance computing cluster that Established in 2002, the University’s Institute for Computational
would enable even more sophisticated
simulations of the universe. With Cosmology has become a leading international centre for research
constraints on data centre space, power into the origin and evolution of the universe, using high-performance
and cooling, the Institute looked to base computing (HPC) clusters to simulate cosmological events and answer
the new cluster on compact, energy-
efficient technologies.
some of the most fundamental questions in science: What were the first
objects in the Universe? How do galaxies form? What is the nature of
Solution dark matter and dark energy? Where does the large-scale structure of
Working with OCF and IBM, the Institute
designed and deployed COSMA4, a 25
the universe come from? What is the fate of the Universe?
teraflop cluster based on water-cooled
IBM® System x® iDataPlex® servers with “Along with engaging the general public and helping people to
2,640 Intel Xeon cores. IBM System understand their place in the cosmos, our research helps raise the
Storage® DS3500 hardware provides 610
TB of storage capacity, which is managed profile of science in general,” says Professor Carlos Frenk, Director of
by the IBM General Parallel File System™ the Institute for Computational Cosmology. “It serves as an important
and IBM Tivoli® Storage Manager. factor in motivating young people to become scientists. There are also
technological spin-offs of our work, in the form of better numerical
algorithms and novel solutions to problems which have applications in
many areas other than cosmology.”
Computational cosmology
Since its foundation, the Institute has run its simulations on a series of
HPC clusters known as COSMA. COSMA2 and COSMA3 were both
based on Sun servers and storage and the Solaris operating system,
but as the field of computational cosmology developed, the need for a
larger-scale solution became evident.
“The level of detail in the models we use is continually increasing, so
there’s a perpetual need for more processor power and data storage,”
says Dr Lydia Heck, Senior Computer Manager. “In addition to
running our own projects, we are also part of an international

2. IBM Systems and Technology Group Education
Deep Computing
cosmology consortium called Virgo, which unites HPC resources from
Business Benefits around the world to run simulations based on hundreds of terabytes
of data. To meet these needs, we decided to build COSMA4 – a new
• Enables simulations of universe models cluster that would provide at least 500 TB of storage and 25 teraflops of
that provide unprecedented levels of
detail – down to the level of galaxies, stars
processing power.”
and potentially even planets.
• Performs seven times faster than
Choosing the right partners
COSMA3, and 50 times faster than The Institute received proposals from eight leading vendors, including
COSMA2, which has now been Dell, HP and Viglen, as well as a joint bid from IBM and OCF, a
decommissioned. specialist provider of HPC server and storage clusters.
• Uses the same amount of electricity as
COSMA2, and does not require air cooling “What we liked about the IBM offer was that it could deliver all the
– reducing overall data centre energy infrastructure – servers, storage and software – from a single vendor,”
consumption by 60 kW.
comments Dr Heck. “We had allowed vendors to bid for the different
• Achieves a Linpack benchmark of 91 elements separately, but the advantages of a coherent single-vendor
percent efficiency, delivering more than strategy were clear. We had also worked with OCF before, and we had
400 megaflops per Watt. It is currently the
UK’s greenest supercomputer and would
been impressed with the level of service they offered.”
make 19th place on the November 2010
worldwide Green500 list. Rapid, successful delivery
• Helps Durham play a leading role in The OCF team worked with the Institute’s in-house HPC specialists to
national and international supercomputing deliver, install and test the COSMA4 cluster. OCF used the PRINCE2
consortia such as Virgo – utilising multiple methodology to manage the project, which helped to accelerate the
supercomputers to work on very large-
scale cosmology projects.
testing phase and prepare COSMA4 for go-live in record time. OCF
also provided comprehensive training on the new IBM technologies,
and will continue to support the infrastructure in accordance with a
detailed service level agreement.
“The hardware and software integration from OCF was very smooth,”
says Dr Heck. “The OCF team is very knowledgeable, and provided
four days of invaluable intensive training on the system.”
Exploring the infrastructure
From the server perspective, COSMA4 is built on 220 IBM System x
iDataPlex dx360 M3 servers, each of which contains two Intel® Xeon®
X5650 processors running at 2.67 GHz. Each processor has six cores,
which add up to a total of 2,640 cores across the whole cluster. The
servers are supported by a storage infrastructure based around eight
IBM System Storage DS3500 disk systems, which provide 620 TB of
storage capacity. The IBM General Parallel File System (GPFS) is used
to provide rapid parallel access for multiple users, while IBM Tivoli
Storage Manager handles backups, and may also be used for policy-
based archiving for long-term storage in the near future.
“The IBM infrastructure has two vital advantages,” comments Dr
Heck. “First, the iDataPlex architecture provides a very dense compute
capacity in a small physical footprint, so it gives us a huge increase
in performance compared to our previous clusters without requiring
a move to a larger data centre. Second, it is water-cooled, which
significantly reduces the need for air conditioning, reducing energy
costs and improving our green computing profile.”

3. IBM Systems and Technology Group Education
Deep Computing
Space and energy
The iDataPlex and DS3500 racks are fitted with IBM Rear Door Heat
Solution Components
eXchanger technology, which passes the hot air generated by the disks
Software and processors over a series of sealed coils filled with chilled water. As
• IBM® General Parallel File System™
a result, the air expelled from the back of the rack is 3 °C colder than
• IBM Tivoli® Storage Manager
the data centre’s ambient temperature, enabling the Institute to retire
Servers three of its seven air conditioning units and use the remaining units less
• IBM System x® iDataPlex® intensively.
• IBM System Storage® DS3500
IBM Business Partner “Our data centre has an upper limit on energy consumption so it was
• OCF essential we procured a powerful but well balanced and energy-efficient
machine,” says Dr Heck. “Although COSMA4 is 50 times faster than
the COSMA2 cluster which it replaced, it only draws about the same
“Our new server and amount of power. It also doesn’t require any air conditioning, so we
save about 60 kW in total data centre energy consumption.”
storage cluster gives us
the ability to experiment Green computing
with vast, sophisticated According to Linpack benchmarks, COSMA4 is currently running
at 91 per cent efficiency, and has an energy efficiency of over 400
models of the universe
megaflops per Watt of energy consumed. This level of energy efficiency
and answer fundamental makes it currently the greenest supercomputer in the UK, and would
questions about our cosmic put it in 19th place on the November 2010 worldwide Green500 list.
environment.”
Massive performance
Most important of all, COSMA4 is seven times faster than its
— Professor Carlos Frenk, Director of the
Institute for Computational Cosmology, Durham predecessor, COSMA3, which means that researchers at the Institute
University and their international collaborators can run more complex and
sophisticated simulations of the cosmos.
Professor Frenk comments: “It is very difficult to test theories about
the universe using traditional methods, as the scale is so large and the
algorithms are so complex. Our new server and storage cluster gives us
the ability to experiment with vast, sophisticated models of the universe
and answer fundamental questions about our cosmic environment –
how does gravity operate, for example, and how does the Universe
expand?”
Making an international contribution
COSMA4 already has 100 registered users and 240 TB of data, and
is preparing for a number of major international projects with other
supercomputing centres both within the UK and internationally.
Dr Heck concludes: “The launch of COSMA4 helps Durham
University maintain its status as one of the world’s leading cosmology
research centres, and enables us to play a major role in national
and international HPC initiatives such as the Virgo consortium.
The projects we run on COSMA4, whether they are internal or in
partnership with other academic institutions, are helping us to reveal
some of the universe’s oldest mysteries.”

4. For more information
To learn more about IBM software, contact your IBM sales
representative or visit: ibm.com
To learn more about products, services and solutions from OCF, visit:
ocf.co.uk
© Copyright IBM Corporation 2011
IBM United Kingdom Limited
PO Box 41
North Harbour
Portsmouth
Hampshire
PO6 3AU
Produced in the United Kingdom
May 2011
All Rights Reserved
IBM, the IBM logo, ibm.com, General Parallel File System, iDataPlex, System
Storage, System x and Tivoli are trademarks of International Business Machines
Corporation, registered in many jurisdictions worldwide. A current list of other IBM
trademarks is available on the Web at “Copyright and trademark information” at:
ibm.com/legal/copytrade.shtml.
Intel, the Intel logo, and Xeon are trademarks of Intel Corporation in the U.S. and/or
other countries.
Other company, product or service names may be trademarks, or service marks of
others.
IBM and OCF are separate companies and each is responsible for its own products.
Neither IBM nor OCF makes any warranties, express or implied, concerning the
other’s products.
References in this publication to IBM products, programs or services do not imply
that IBM intends to make these available in all countries in which IBM operates. Any
reference to an IBM product, program or service is not intended to imply that only
IBM’s product, program or service may be used. Any functionally equivalent product,
program or service may be used instead.
All customer examples cited represent how some customers have used IBM products
and the results they may have achieved. Actual environmental costs and performance
characteristics will vary depending on individual customer configurations and
conditions.
IBM hardware products are manufactured from new parts, or new and used parts.
In some cases, the hardware product may not be new and may have been previously
installed. Regardless, IBM warranty terms apply.
This publication is for general guidance only.
Photographs may show design models.
Please Recycle
DCC03012-GBEN-01