In this deck from the Rice Oil & Gas Conference, John Fragalla from Cray presents: Data Storage & I/O Performance: Solving I/O Slowdown: The "Noisy Neighbor" Problem "In Oil and Gas, when using shared storage, mixed workloads can have a big impact on I/O performance causing considerable slowdown when running small I/O alongside large I/O on the same storage system.  With the increase adoption of Flash, and new features in Lustre, such as Progressive File Layout (PFL) and Data on Metadata (DoM), Storage can be tuned to automatically isolate different workloads to different storage media to accelerate small I/O without disrupting large sequential I/O performance, using the right mix of Flash and Disk transparently.  In this presentation, Cray will share real benchmark results on the impacts of the "Noisy Neighbor" application has on sequential I/O, and with the right storage tuning and flash capacity, how to optimize the storage to meet the demanding workloads of Oil and Gas to accelerate performance for a mixed workload environment." Watch the video: https://wp.me/p3RLHQ-jYL Learn more: http://cray.com and http://rice2019oghpc.rice.edu/ Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

1. © 2018 Cray Inc. 1
Solving I/O the Slowdown:
The "Noisy Neighbor" Problem
Rice Oil and Gas Conference 2019
John Fragalla
Principal Engineer
Cray, Inc.

2. © 2018 Cray Inc. 2
• Today’s I/O challenge with shared I/O
• New Lustre Features enabling Flash to improve Shared Application Performance
• Performance results isolating “Noisy Neighbor” Applications
• Summary
Agenda

3. © 2018 Cray Inc. 3
Today’s I/O Challenge
• When multiple users share a high speed parallel filesystem, ”bad applications”
will effect “good application” performance
• Bad applications: Lots of Small Files, Random Small I/O, Unaligned I/O
• Good applications: Stream large I/O, Sequential performance, Aligned I/O
• Recent features available in Lustre help automate I/O isolation and placement
with transparent use of Flash and HDD Devices in a Single Namespace
• Progressive File Layout with Lustre Storage Pools
• Data on Metadata
• Distributed Namespace (DNE) 2 – Clustered Metadata

4. © 2018 Cray Inc. 4
Hybrid File System Architecture
CN
CN
CN
CN
CN
CN
CN
CN
CN
CN
CN
CN
CN
CN
CN
CN
CN
CN
CN
CN
HDD HDD HDD HDD
HDD HDD HDD HDD
SSD SSD SSD SSD
SSD SSD SSD SSD
Parallel File System
Single Namespace
Compute Cluster
Flash Tier :
• Optimize for throughput and IOPs - $/GB/sec.
• Improved Performance for Intermediate Results
• Small/random I/O performance improved
High Performance HDD Tier:
• Optimize Throughput/Capacity - $/GB/Sec
• Optional Flash/Cache to accelerate small block
IO within the HDD Tier
HDD HDD HDD HDD
HDD HDD HDD HDD
HDD HDD HDD HDD
Capacity HDD Tier:
• Optimize cost - $/GB
• Lower performance, longer term data retention
MDS
MDS
MDS
MDS
Scalable MDS Flash Tier :
• Large number of Inode Support per FS
• Improved Metadata Operations
• Improved Small I/O Latency

5. © 2018 Cray Inc. 5
• Progressive File Layouts (PFL)
• Optimized striping based on file size
• Layout changes at specific thresholds
• Can locate components on specific pools
• Fixed amount of file on flash, the rest on disk
Lustre | Flexibility and Usability

6. © 2018 Cray Inc. 6
• Lustre Pools historically was used for debugging to isolate performance issues,
for example, for a subset of OSTs or OSS Nodes.
• Now, with PFL, Lustre Storage Pools can be a powerful tool to create automatic
data placement on different storage medias
• Flash Pool
• High Performing Disk Pool
• Slow Performing Disk Tier (e.g. focus on capacity)
Lustre Storage Pools are more relevant Now

7. © 2018 Cray Inc. 7
Lustre | Improved Small File Performance
• Data on Metadata
• Ideal for small file workloads
• File data stored directly on metadata storage
• Lower communication overhead for data access
• Scales with Distributed Namespace (DNE)
• Avoids contention by not placing small files on OSTs

8. © 2018 Cray Inc. 8
Lustre | Data on Metadata and PFL
• Leverage DoM and PFL for more flexible solution
• Small files land on MDT Component
• Medium files land on flash with larger files growing to disk (for example)
• Compatible with Progressive File Layouts

9. © 2018 Cray Inc. 9
DNE Phase 2
• Allows a user to spread a single large directory across multiple MDTs using the
DNE striped directory feature
• Note due to some overhead, this should only be done to very large
directories with file counts in +50K range.
DNE phase 1 DNE phase 2

10. © 2018 Cray Inc. 10
System Setup for Benchmarks
Hardware:
• 4 Flash MDTs with RAID-10
• 2 Flash IOPS Optimized OSTs with
RAID-10
• 4 GridRAID OSTs (Parity De-
Clustered RAID-6 equivalent data
protection)
• Up to 64 Client nodes (FDR
Connectivity)
• EDR InfiniBand Non-Blocking Fabric
Software:
• Lustre 2.11.0 clients and server
• CentOS Linux release 7.5 (server and
client)
• Spectre/Meltdown-enabled kernels on
Clients, S/M disabled on Server
• Client: 3.10.0-862.el7.x86_64
• Server: 3.10.0-693.21.1.x3.1.9.x86_64

11. © 2018 Cray Inc. 11
LUSTRE PFL STREAMING PERFORMANCE
Flash MDTs -> HDD OST Tier (DoM)
0
5,000
10,000
15,000
20,000
25,000
30,000
No DoM DoM=64K DoM=256K DoM=1024K DoM=4096K
Performance(MB/sec)
Progressive File Layout Small Component Size
Write Mean
Read Mean
Progressive File Layout maintains peak performance for streaming workloads
We want no change in performance across various sizes

12. © 2018 Cray Inc. 12
LUSTRE PFL NOISY NEIGHBOR ISOLATION
Lustre
HDD OST
File 1 File 2 File 3 File 4
File
Small File Workload
Streaming Workload
HDD OST
File 1 File 2 File 3 File 4
File
Small File Workload
Streaming Workload
Flash
OST or MDT
Two Competing Workloads on Same Resources
Two Workloads Separated using PFL

13. © 2018 Cray Inc. 13
12,000
14,000
16,000
18,000
20,000
22,000
24,000
None None (1MB) 1024K (1MB)
MB/s
1 MB FILES
Write Mean Read Mean
12,000
14,000
16,000
18,000
20,000
22,000
24,000
None None (4MB) 1024K (4MB) 4096K (4MB)
MB/s
4 MB FILES
Write Mean Read Mean
LUSTRE PFL NOISY NEIGHBOR ISOLATION
Flash Tier (OST or DoM with MDTs) -> HDD OST Tier
PFL ISOLATION OF IOPS FROM STREAMING IMPROVES PERFORMANCE
Baseline
Interfered
Isolated Baseline Isolated
Interfered
X-Axis Legend
PFL Size on Flash (Noisy Neighbor File Size)

14. © 2018 Cray Inc. 14
• New Lustre Features such as PFL, DoM, and DNE2 help improve mixed I/O
performance on high speed shared parallel filesystem
• Transparent data placement on Flash MDTs and/or OSTs and HDDs for various
I/O sizes to optimize throughput and IOPS
• Isolate small files or small I/Os from streaming I/O to solve the “Noisy Neighbor”
slow down for sequential performance
Summary

15. THANK YOU
Q U E S T I O N S ?
cray.com