How to design ZFS systems using the performance envelope method

1. STORAGE SYSTEMS
Architecting ZFS
Systems
Richard.Elling@DEY-sys.com

2. Architect’s Dilemma
How do we know a system will meet
its performance goals?
ZFS Day October 2, 2012
STORAGE SYSTEMS

3. Architect’s Dilemma
How do we know a system will meet
its performance goals?
... before we build it ...
... before we buy a rack full of stuff ...
... before we build the next datacenter ...
ZFS Day October 2, 2012
STORAGE SYSTEMS

4. Hybrid Storage Pool
Log Main Pool L2ARC
Minimal HDD
Good SSD HDD
HDD
mirror, raidz
Better SSD
SSD
HDD HDD HDD SSD
mirror raidz2, raidz3
SSD SSD HDD HDD HDD SSD SSD
Best SSD
mirror
SSD
mirror
HDD
mirror
HDD
mirror
HDD
mirror stripe
stripe stripe
ZFS Day October 2, 2012
STORAGE SYSTEMS

5. Reaching for the Peak
Log Main Pool L2ARC
Better SSD
SSD
HDD HDD HDD SSD
mirror raidz2, raidz3
SSD SSD HDD HDD HDD SSD SSD
Best SSD
mirror
SSD
mirror
HDD
mirror
HDD
mirror
HDD
mirror stripe
stripe stripe
Extreme SSD SSD SSD
raidz2, raidz3
Over the Top SSD
HDD SSD
HDD SSD
HDD
mirror mirror mirror
stripe
ZFS Day October 2, 2012
STORAGE SYSTEMS

6. Design Trade-offs
Log Main Pool L2ARC
Performance $ / TB
Good HDD HDD HDD SSD Best
SSD
SSD
mirror
Space,
raidz2, raidz3
Dependability,
Performance:
SSD
SSD SSD
SSD HDD Pick 2 HDD
HDD HDD
HDD HDD SSD SSD
Better mirror mirror mirror mirror mirror stripe Better
stripe stripe
Best SSD
HDD SSD
HDD SSD
HDD Good
mirror mirror mirror
stripe
ZFS Day October 2, 2012
STORAGE SYSTEMS

7. Promoting Good Design
OK, so I get it, Hybrid Storage Pools are cool
ZFS Day October 2, 2012
STORAGE SYSTEMS

8. Promoting Good Design
OK, so I get it, Hybrid Storage Pools are cool
... how should I configure my thumper?
ZFS Day October 2, 2012
STORAGE SYSTEMS

9. Previous Modeling
Best worst-case IOPS
WARNING: this perf model does not consider caching
... and this is intentional
ZFS Day October 2, 2012
STORAGE SYSTEMS

10. IOPS ~ Latency
ZFS Day October 2, 2012
STORAGE SYSTEMS

11. Performance Envelope
10,000,000
4KB Random Read IOPS
1,000,000
100,000
10,000
1,000
0
250
500
750
1000
Working Set Size (GB)
4KB random read IOPS
Expected Max Performance
ZFS Day October 2, 2012
STORAGE SYSTEMS

12. Performance Envelope
10,000,000
4KB Random Read IOPS
1,000,000
100,000
A
R
C L2ARC
10,000
Pool Disk
1,000
0
250
500
750
1000
Working Set Size (GB)
4KB random read IOPS
Expected Max Performance
ZFS Day October 2, 2012
STORAGE SYSTEMS

13. Performance Envelope
10,000,000
ARC Hit Performance
4KB Random Read IOPS
1,000,000
L2ARC Hit
Performance Pool Performance
100,000
A
R
C L2ARC
10,000
Pool Disk
1,000
0
250
500
750
1000
Working Set + L2ARC Size
ARC Size ARC Size (GB)
4KB random read IOPS
Expected Max Performance
ZFS Day October 2, 2012
STORAGE SYSTEMS

14. Read Performance Envelope
1,500,000
1,250,000
4KB Random Read IOPS
1,000,000
Small Config Expected Performance
Medium Config Expected Performance
750,000 Large Config Expected Performance
10 GbE wire speed
500,000
250,000
0
0 250 500 750 1000
Working Set Size (GB)
Configuration Small Medium Large
RAM size (GB) 24 96 192
100% ARC hit rate performance 600,000 900,000 1,300,000
L2ARC size (GB) 0 250 480
L2ARC device small random read IOPS 0 30,000 60,000
Pool small random read IOPS 1,400 3,600 8,000
ZFS Day October 2, 2012
STORAGE SYSTEMS

15. Performance Testing
Local SSS-PTS test, single zvol
ZFS Day October 2, 2012
STORAGE SYSTEMS

16. Simple Example
100,000,000
10,000,000
IOPS
1,000,000
100,000
ZFS ZFS
ZFS 10,000
0 5,000 10,000 15,000 20,000
WSS (GBytes)
Max Achievable (llama’s view)
Network
Storage
ZFS Day October 2, 2012
STORAGE SYSTEMS

17. Complex Example
100,000,000
10,000,000
IOPS
1,000,000
$ $
100,000
10,000
0 5,000 10,000 15,000 20,000
WSS (GBytes)
ZFS ZFS
Max Achievable (llama’s view) Cache Layer
Storage Effective Storage
ZFS Client Network Storage Network
Cache Network
ZFS Day October 2, 2012
STORAGE SYSTEMS

18. What Else?
• Space - trivial, count the bytes
• Cost - special deal, just for you
• Dependability - stay tuned for new models
• Writes
• write-back cache models are not readily
available for generic workloads
• caches in many parts of the systems
• Working Set Size - help wanted
ZFS Day October 2, 2012
STORAGE SYSTEMS

19. STORAGE SYSTEMS
Thanks
Richard.Elling@DEY-sys.com