The document discusses five keys to achieving ultra-low PUEs (Power Usage Effectiveness) in data centers: 1) Be brave and embrace hardware failure by operating servers in higher temperature and humidity environments to reduce energy costs, 2) Ensure proper high-efficiency mechanical and electrical equipment and power distribution, 3) Maintain precise temperature control through environmental monitoring and adjustments, 4) Increase voltage distribution to reduce transmission losses, and 5) Choose equipment and designs that reduce pressure drops in air flow. The document argues these strategies can lower PUEs below 1.15 and significantly reduce total cost of ownership for data centers.
In the Data Center: Five Keys to Achieving Ultra-Low PUEs
1. Enterprise Data Center Solutions
Five keys to achieving
ultra-low PUEs
Chris Yetman
Senior Vice President, Operations
2. Ultra-low
PUEs
Getting From Here To There
2
You’re using the latest high-efficiency servers. You keep your server room at a higher
temperature. You’ve virtualized to cut down on the number of servers. It seems like you’re
doing everything right. But you’re still stuck at a less than enviable PUE.
3. 33MW of data center, 18 acres
Commissioned PUEs as low as 1.12
15% reduction in TCO due to highly
efficient facility designs
Largest LEED Platinum data center in
North America
I feel your PUE pain
Today I’d like to share five things we’ve learned
about getting to ultra-low PUEs. Check it out:
we’ve been able to hit 1.15 PUEs for our
customers.
We get it. We know this problem. We live and
breathe it every day.
4. PREVIOUS JOB
Chief Information
Officer (CIO)
As a former CIO, a current Chief of
Operations, and, most importantly,
someone who is passionate about building
awesome data centers, I’d like to call on all
CIOs.
Too often we are needlessly cautious when
it comes to hardware. Too often we make
IT decisions based on fear.
Why are we afraid to allow equipment to
fail? Why are we working so hard to avoid
breaking a server?
These decisions cost us more than we
suspect.
1. Be Brave
5. 1. Be Brave
Operating
Envelope
ASHRAE
Allowed
Temp: 59°-90°
Humidity: 20-80%
Dew Point:
Up to 63°
Temp: 65-80
Humidity: 42-60%
Dew Point:
Up to 59°
The truth is, studies have shown IT equipment
can handle higher temperatures than what the
industry currently operates under.
The industry needs to diversify equipment-level
risk. Why should the loss of any single piece of
equipment compromise your business?
For Vantage, being located in Santa Clara, we
utilize the free-cooling hours throughout the fall,
winter, spring, and temperate summer days.
We help each customer build the data center
they want to build. If one feels comfortable and
wants to move out of the recommended
ASHRAE zone into allowable conditions, they
create greater cost efficiency by utilizing natural
air temperature and flow up to 80% of the year.
Even if you’re timid about rising to 90-degrees,
being more flexible on humidity can have a
positive effect.
ASHRAE
Recommended
6. 1. Embrace Failure
Hello, my
name is
failure.
But you can
call me
recovery.
Stop trying to prevent hardware failure. It’s
not worth the cost. If you build with the
knowledge that hardware will fail, you can
instead focus on recovery.
This will save you money.
Operating at a higher operating envelope
may mean more hardware failure, but you
will save so much in energy efficiency and
costs.
Those savings will cover the money needed
to replace hardware, ten-fold.
Don’t let hardware failure be the worst-case-
scenario. Hardware failure is nothing to fear.
7. 1.3 1.2
PUE
Operating
Envelope
$500K
7
Check out the cost savings!
A large data center running five-plus
megawatts can spend $500,000 more in
energy per month.
But let’s say you run the temperature up to
80-degrees in the cold aisle, taking your
PUE down from 1.3 to 1.2.
In this scenario, you will save over
$500,000 per year in energy, and break a
whole lot less than that in hardware.
Imagine the temperature at 85, 90-
degrees.
Focus on extending the envelope.
1. Bravery and Recovery = Cost Savings
8. 8
Take the example of Netflix. Here’s a company that’s
challenged its system to plan for recovery and resilience.
They created Chaos Monkey, a software tool that forces
their engineers to deal with small failures, ones that, once
eliminated, will keep from turning into major outages.
Netflix says the name “Chaos Monkey” comes from the
idea of “unleashing a wild monkey with a weapon in data
center or cloud region to randomly shoot down instances
and chew through cable—all while we continue serving
our customers without interruption.”
When Netflix loses a server—no big deal. The software
reassigns the data elsewhere.
The challenge to developers everywhere is to write
applications to be resilient.
1. Bravery and Recovery = Cost Savings
9. 2. Get the Foundation Right
High-efficiency
mechanical /
electrical
equipment
High-voltage
power
distribution
Measurement
&
Control
Proper
containment of
hot and cold
aisles
9
Achieving low PUEs begins with right, high-efficiency equipment. For Vantage that means these four pillars
listed above.
A solid foundation also includes minute attention to mechanical infrastructure like high-efficiency fan motors,
direct drive units, and variable frequency drives on all motors. Did you know we don’t use fans with belts? That
way we don’t have to worry about slippage and losing efficiency to those incremental, inevitable belt slips. All
the little things add up.
10. Monitor Adjust Recirculate
Precise temperature control
3.Maintain Infinite Control
It’s critical to choose an
environmental-electrical-monitoring
system that allows for precise high-
speed industrial process control.
The system tracks air conditions
both inside and outside the data
center—temperature, humidity,
dew point, flow volume.
This allows the system to make
adjustments with precision.
As the outside air temperature drops, the system is able to recirculate more heated air from
the data module into the supply air stream, allowing for precise temperature control.
This allows the system to modulate anywhere from 100% outside air economization down to
0% full recirculation.
The system can provide the minimal amount cooling to meet the envelope!
11. Losses
Current
Conductors
480V to
the rack
4. Increase the Voltage
11
Higher
Voltage
High voltage distribution to the IT racks reduces
transmission loss, decreases the amount of copper running
into the building, and minimizes transformation losses.
If you combined this with an open compute platform
approach, where you transform down from 480V at the rack
and share power supplies—think of all the energy you’d
retain!
Here’s the straight electric talk:
• The losses due to typical transformation from 480V to
208/120V of a typical Power Distribution Unit
(PDU) account for anywhere from 1.5%-3.5%
• Providing 400V to the Busway Distribution Units
(BDU’s, equivalent to PDU) line to neutral voltage of 230V
can be supplied to the IT equipment
• The result is reduced losses, lower current supplied to the
IT equipment, and smaller conductors required to power
the servers
• Extending the higher voltages out to the rack saves on
transformation losses and conductor costs
12. 5.Reduce Pressure Drop
12
Choose mechanical cooling equipment that enables the
lowest available pressure drop.
Start with the outside air damper array and filter bank.
Choose components to minimize air velocity. These
range from oversized dampers and filters to higher cross-
section cooling coils.
Look at your fans. Belt drives inevitably lead to slippage
and energy loss. Direct drive supply fans eliminate such
loss, and large diameter fan rotors increase flow volume
while maintaining reasonable equipment footprint.
Use natural cycles. Data modules utilize vertical hot-
aisle, allowing for convective air flow to return the hot air
from the IT equipment to the cooling plenums.
Avoid leakage by creating a strong barrier between hot
and cold aisles.
13. 1. This is not an all-or-nothing environment.
Going from open/shut cases to varying
degrees of open-to-shut provides optimization.
2. Moving a lot of air is turbulent. Rigid filtering is
better. Don’t let your filters disintegrate.
3. Air will mix before it hits the equipment. Add
lots of measurement points along the air flow
path. The air the racks ingest matters the most.
Lessons Learned
14. • Runs at 1.15 PUEs
• Fully redundant 2N
electrical configuration
• 9 megawatt, expandable
to 18 megawatts
This is already a reality today.
15. 1. Be brave. Embrace failure.
2. Maintain infinite control.
3. Pick the right equipment.
4. Increase the voltage.
5. Reduce pressure drop.
Getting to Ultra-Low PUEs
Let’s connect: