Mark Skinner's presentation to #lspe at http://www.meetup.com/SF-Bay-Area-Large-Scale-Production-Engineering/events/15481232/

1. Architecting
a Scale Out Cloud Storage Solution

2. Open versus Commercial
Implementation
• Commercial – comes at a price
• Open architecture – with support
• Whatever your choice, HW must be flexible.

3. Appliance vs. SW on Commodity HW
Appliance model wins when… SW on commodity HW wins when…
Workloads Workloads are fairly standard across Workloads vary tremendously
organizations across multiple dimensions (e.g.
performance, availability, capacity,
I/O requirements, etc.)
Economics/Growth Savings from commodity hardware Savings from commodity hardware
are limited to the savings from one stretch across multiple “boxes” and
“box” and therefore small relative to are massive
the overall TCO savings from easy
deployment and management
Flexibility All needed flexibility can be Significant need for flexibility in
accomplished by configuration hardware to optimize capacity,
changes in a single box performance, availability, etc.
Cloud Cloud not a factor Cloud a major factor, driving need to
be able to run on multiple disparate
devices across the internet
Ben Golub, Computerworld , October 11, 2011 “Storage is a hard problem with a soft(ware) solution”

4. Open Software Choices
• GlusterFS
• Open Stack (Swift)
• NexentaStor
• OS Nexus
• Open-E
• OpenStorage Software
• Openfiler
• FreeNAS
• ………

5. Three Leading Open Architectures
• Lustre/Gluster (bought by Red Hat)
– HPC – the others don’t have the throughput
– If something fails, it takes days to bring it back
– Gluster fixed HA and failover and now they’ve been acquired…?
• Open Stack (Swift) (numerous contributors)
– Compute, Storage and Networking Management
– Many firms are trying to commercialize it, Rackspace one of the largest
– Pure cloud storage = pure block level replication, not file level
– Still at least 6-12 months from being production ready
• ZFS/NexentaStor (Nexenta)
– OpenSolaris developed at Sun, Nexenta added GUI and redundancy for HA
– Swiss army knife = rich feature set, highly configurable platform for unified storage
management (iSCSI, NAS, DAS)
– GlusterFS and Swift need dedicated server head node plus storage head nodes,
NexentaStor only requires one head node

6. Case Study – Korea Telecom
• Storage Cloud – mobile & online storage
• NexentaStor for unified storage management
• Cirrascale for consolidation (space savings),
cooling efficiency (power savings and
reliability), and lowest TCO
– Decouple server and storage
– Flexibility to configure/reconfigure on the fly

7. Case Study – Korea Telecom
Problem
• 6 x 4u storage chassis per rack
– 24 x 2TB HDDs per chassis
– 288TB per rack
• Due to overheating, could not
install more than 6 chassis in
each rack
• Each rack occupied 1 x 2 floor
tiles
• High cost of wasted
infrastructure became a
problem for the data center

8. Case Study – Korea Telecom
Problem Resolved
• Cirrascale provided Storage Bricks
in a 1:4 Head Node to Storage
Blade configuration
– 12 x 2TB HDDs per Storage Blade
– 96TB per Storage Brick
– 14 Storage Bricks per rack
– 1.3PB storage capacity per rack
(compared to 288TB and lots of heat)
• Reduced almost 5 racks to 1
• Rack still had 8U of space available
for switching and other equipment
• All in the same two floor tiles

9. Vertical Cooling Technology
Server Blades
Close up of the air gap
Blades x 12
between HDDs and blade
chassis
Fans x 8
Storage Blades
Blades x 12
Air Flow
Fans x 8
Ethernet and I/O
Blades x 12 cabled from the
backplane to the
Cooling Fans foot of the rack.
Fans x 8 I/O in separate axis
than air flow.
Horizontal 8u
Switches
Patch Panels
Serial Concentrators

10. Storage Systems
 “Storage Brick” is the base product
 1 Head Node (Control Logic/Power Supply)
 1 - 4 Storage Blades (12 – 48 HDDs)
 12 - 36 Storage Bricks per frame
 2PB per Rack in a 1:3 Configuration
 Flexible Configuration
 Mix and match storage media, modify head node to
HDD ratios, upgrade or downgrade server types
 Rack or blade level redundancy depending on cost
and application requirements
 Flexible Management
 IPMI 2.0 support for integration with leading tools
 Individual HDDs hot swappable

11. SB1315 7 SB1315 4 SB1315 1
SB1058 7 SB1058 4 SB1058 1
FAN
FAN
FAN
FAN
SB1058 7 SB1058 4 SB1058 1
SB1058 7 SB1058 4 SB1058 1
SB1315 8 SB1315 5 SB1315 2
FAN
FAN
FAN
FAN
SB1058 8 SB1058 5 SB1058 2
SB1058 8 SB1058 5 SB1058 2
Front
SB1058 8 SB1058 5 SB1058 2
FAN
FAN
FAN
FAN
SB1315 9 SB1315 6 SB1315 3
SB1058 9 SB1058 6 SB1058 3
SB1058 9 SB1058 6 SB1058 3
FAN
FAN
FAN
FAN
SB1058 9 SB1058 6 SB1058 3
SB1315 16 SB1315 13 SB1315 10
SB1058 16 SB1058 13 SB1058 10
FAN
FAN
FAN
SB1058 16 SB1058 13 SB1058 10
SB1058 16 SB1058 13 SB1058 10
SB1315 17 SB1315 14 SB1315 11
FAN
FAN
FAN
SB1058 17 SB1058 14 SB1058 11
SB1058 17 SB1058 14 SB1058 11
Back
• FAN Servers
SB1058 17 SB1058 14 SB1058 11
FAN
FAN
FAN
18 FAN FAN
SB1315 18 SB1315 15 SB1315 12
• Up to 1.9PB
SB1058 18 SB1058 15 SB1058 12
SB1058 18 SB1058 15 SB1058 12
FAN
FAN
• Optional SSD FAN
FAN
SB1058 18 SB1058 15 SB1058 12
ratio at any time
• 2.5” or 3.5” HDDs
• 1,2,or 3TB SATA or SAS
• 54 Disk Blades (648 Drives)
• Reconfigure storage and compute
• User defined performance and features
Modular Design
• Up to 2.2PB
• 1:14 HeadNode to Storage Blade Ratio
• Same 2 SKUs was before, with additional
• 4 Servers
SAS Switch Blade using LSI 6160
• 56 Disk Blades (672Drives)

12. Why Separate Head Node from Storage?
• Easily configure the server independent of
storage
• Optimize the compute to storage
configuration
• Storage blades can be SSD, SAS, or SATA or
any combination

13. Storage Server Capacity/Configurability
Feature Cirrascale Standard 4U High Density 4U
Rackmount Rackmount
Floor Tiles 2 2 2
Height 87.5” 84.0” 84.0”
Maximum Capacity 2PB 720TB 1.38PB
Floor Raised or Concrete Raised or Concrete Raised or Concrete
Maximum Storage 672 drives 240 drives 432 drives
Up to 3TB HDDs Up to 3TB HDDs Up to 3TB HDDs
14 Server heads 10 Server heads 10 Server heads
Configuration Granularity
Drives 12 Drives 24 Drives 45 Drives
Server:Drive Ratio 1:12, 1:24, 1:36, 1:48 1:24 1:45

14. Case Study: SAN for Cloud Deployment
Cirrascale HW Configuration Dell HW Configuration
• Head Node: • Head Node: PowerEdge R510 2U
– Dual Xeon X5606 2.13GHz quad core CPUs – Dual Xeon X5530 2.4GHz quad core CPUs
– 48GB DDR3-1333MHz ECC memory – 32GB DDR3-1333MHz ECC memory
– 2 x 10GbE ports – 2 x 10GbE ports
– 2 x 250GB, 7200 RPM 2.5” HDD (hot swap) – 2 x 250GB, 7200 RPM 3.5” HDD
• Storage Blade: • Storage JBOD: MD1220 4U
– 24 x 2.5” 300GB SAS 15K RPM HDDs – 24 x 2.5” 300GB SAS 10K HDDs
7.4 TB per Blade 7.4 TB per JBOD
• Head Node to Storage Blade Ratio 1:2 • Head Node to JBOD Ratio 1:2
– 1 Head Node: 48 HDDs – 1 Head Node: 48 HDDs
Bottom line
2.6PB, 7 Racks, $2.4M 2.6PB, 27 Racks, $4.3M

15. Summary
• Every storage architecture balances trade offs among:
• Availability • Manageability
• Backup/Recovery • Compliance
• Performance • Scalability
• Capacity • Density
• Power Consumption • Cost
• Compelling benefits among SAN, iSCSI, NAS, and DAS are
driving Cloud Service Providers and enterprises to install
converged and hybrid solutions
• Open Software and Hardware platforms provide the most
flexible, cost effective, future proof platforms for scale