This document discusses optimizing performance for EC2 instances and EBS volumes. It provides guidance on provisioning IOPS for different types of storage workloads and database software. The key recommendations are to use EBS-optimized instances with Provisioned IOPS (PIOPS) volumes for random I/O workloads like databases, size volumes appropriately based on the needed IOPS and throughput, and architect for consistent low latency by adjusting the queue depth.

1. STG302: Maximizing EC2 and Elastic Block
Store Disk Performance
Miles Ward, Sr. Manager Solutions Architecture, Amazon Web Services
November 12, 2013
© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.

2. EBS =

3. So.. what storage workloads?

4. A few definitions…

5. A “Normal” Hard Drive

6. Review: Provisioned IOPS Volumes
❶ Select a new type of Provisioned IOPS volume
❷ Specify the volume capacity
❸ Specify the number of I/O operations per
second your application needs, up to 4000
IOPS per volume. The volume will deliver the
specified I/O operations per second.
Minimum ratio of capacity to IOPS = 1:30
$aws ec2 create-volume --availability-zone us-east-1a --size 134 --volume-type io1 --iops
4000

7. Amazon EBS Standard
• IOPS: ~100 IOPS steady-state, with best-effort bursts to hundreds
Amazon Elastic
Block Storage
(EBS)
• Throughput: variable by workload, best effort to 10s of MB/s.
• Latency: Varies, reads typically <20 ms writes typically <10 ms
• Capacity: As provisioned, up to 1 TB

8. EBS PIOPS
• IOPS: Within 10% of up to 4000 IOPS,
99.9% of a given year, as provisioned.
Amazon Elastic
Block Storage
(EBS)
*
• Throughput: 16 KB per I/O = up to 64 MB/s, as provisioned.
*
• Latency: low and consistent. Second/IOPS, at recommended QD
• Capacity: As provisioned, up to 1 TB

9. USE PIOPS AND EBS OPTIMIZED
PLEASE

10. Why the
*
?
An I/O
EC2
EBS

11. Why the
*
?
A bigger I/O
EC2
EBS
>16 KB? = more than one I/O

12. Why the
*
?
A smaller I/O
EC2
EBS
< 16 KB? = no “bonus” random I/Os

13. Why the
*
?
An I/O
EC2
EBS
Once it’s gone, it’s gone.

14. Why the
*
?
Just because Amazon EC2 sends more
work doesn’t mean there’s enough
bandwidth to handle it!
An I/O
EC2
EBS

15. Why the
*
?
An I/O
EC2
Without more bandwidth,
more EBS volumes or higher PIOPS won’t help!

16. Why the
*
?
Just because EC2 sends more work
doesn’t mean there’s enough
IOPS to handle it!
an I/O
EC2
EBS

17. Why the
*
?
Adding bandwidth alone
or increasing instance size won’t help!
An I/O
EBS-Optimized
EC2
EBS

18. *
*
Oh, YEAH!!
A “boatload” of I/O
EBS-Optimized
EC2

19. Architecting for Performance
•
•
•
IOPS consistency requires EBSoptimized instances
Maximum throughput delivered by
Amazon EBS is limited by Amazon
EC2 bandwidth
EBS throughput =
EBS IOPS × Block size
–
Ex: 64 MB/s = 4000 IOPS × 16 KB
EBS
Instance vCPU Optimized Max MB/s Max 16k IOPS
t1 micro
1
No
32MB/s
2000
m1.small
1
No
64MB/s
4000
m1.medium 1
No
64MB/s
4000
m1.large
2
Yes
64MB/s
4000
m1.xlarge
4
Yes
128MB/s
8000
m3.xlarge
4
Yes
64MB/s
4000
m3.2xlarge
8
Yes
128MB/s
8000
c1.medium
2
No
32MB/s
2000
c1.xlarge
8
Yes
128MB/s
8000
cc2.8xlarge 32
NA
800MB/s
50,000
m2.xlarge
2
No
64MB/s
4000
m2.2xlarge
4
Yes
64MB/s
4000
m2.4xlarge
8
Yes
128MB/s
8000
cr1.8xlarge 32
NA
800MB/s
50,000
hi1.4xlarge 16
NA
800MB/s
50,000
cg1.4xlarge 16
NA
800MB/s
50,000
Max 8k =
2x
Max 4k =
4x*
Max 2k =
 8x*
*Maximum IOPS is also limited to ~100,000 per 32 vCpu,
irrespective of block size/throughput.

20. EBS-Optimized
Network interference tests
Row Labels
AvgBW
AvgIOPs
m3.2xlarge (EBS-optimized)
•
no network load
EBS-optimized offers a “SAN-like” experience
random
•
No impact on IOPS or
Amazon EBS
throughput
read
57,542
3,596
write
61,713
3,857
rw (70/30)
Network interference results:
66,997
4,186
read
61,708
3,856
write
61,651
3,853
rw (70/30)
66,996
4,187
read
59,835
3,739
write
63,407
3,962
rw (70/30)
68,859
4,303
read
61,736
3,858
write
63,360
3,959
rw (70/30)
68,859
4,302
sequential
with network load-test1
random
sequential

21. I/O Characteristics
•
–
•
•
PIOPS delivers same number of IOPS for
sequential and random I/O
•
PIOPS delivers same number of IOPS for
reads or writes
Sequential and random
I/O type
–
PIOPS always measures I/O in terms of
16 KB or smaller
4 KB to 64 MB
I/O pattern
–
•
•
I/O size
Read and write
PIOPS is optimized for database workloads

22. Smaller I/O (4 KB, 16 KB)
Results for 400 GB volume with 4000 IOPS at QD 8; EBS-optimized instances
•
•
•
•
Why are 4 KB I/O size in
sequential operations
driving greater than 4000
IOPS?
Why is m1.large and
m3.xlarge IOPS at 16 KB
less than 4000 IOPS?
Database needs 5000
ops/second. How many
IOPS do I need to
provision?
What happens when
customers want to burst
beyond provisioned IOPS?
m1.large
IOPS and BW
performance at QD
Avg BW
8
AvgIOPs ( KB)
M3.xlarge
Avg BW
AvgIOPs ( KB)
m3.2xlarge
Avg
BW
AvgIOPs ( KB)
Write
sequential
4K
4146
16,587
5997
23,990
7767 31,068
16K
3712
59,402
4157
55,461
4153 60,332
4K
4082
16,329
4433
17,733
4178 16,712
16K
3713
59,422
3743
53,813
4153 60,332
4K
5301
21,205
9232
36,929
13450 53,802
16K
3533
56,535
4796
56,824
4153 60,332
4K
4538
18,154
5864
23,457
4177 16,711
16K
3510
56,168
3583
51,246
4153 60,332
Write random
Read
sequential
Read random

23. Larger I/O (128 KB, 512 KB)
Results for 400 GB volume with 4000 IOPS at QD 8
•
Why am I seeing only 462
IOPS on a volume?
m1.large
IOPS and BW
performance at QD 8 AvgIOPs
M3.xlarge
Avg
BW(KB)
AvgIOPs
Avg
BW(KB)
m3.2xlarge
AvgIOPs
Avg
BW(KB)
Write sequential
•
Why there is no difference
in performance for
random and sequential
workloads?
How should I configure
500 MB/s read or write
throughput using PIOPS
volumes
462
59,268
462
59,145
522
66,843
512K
115
59,292
115
59,278
130
66,804
128K
462
59,265
462
59,241
522
66,843
512K
•
128K
115
59,291
115
59,272
130
66,843
128K
455
58,240
454
58,225
522
66,843
512K
113
58,003
114
58,589
130
66,843
128K
455
58,236
454
58,215
522
66,843
512K
113
57,960
114
58,496
130
66,805
Write random
Read sequential
Read random
4000, 16 KB read/write per second, or 2000 32
KB read/write per second, or 1000 64 KB
read/write per second…

24. Write Latency
•
•
16 KBk random WRITE- M3.2Xlarge EBSoptimized
Database applications care
about latency as much as IOPS
delivered
There is an Interdependency
among IOPS, queue depth, and
latency
Current guidance is queue
depth of 1 for every 200 IOPS,
but if latency-bound and writeheavy, 1:500 – 1:1000 is better.
4500
9
4000
Write IOPS
•
8
7.71
7.48
4152
3500
7
6.18
3000
6
5.54
2500
5
2000
4
3.62
3.56
3.13
1500
3
2.03
1000
2
1.47
QD500
0
1
845
AvgIOPS ( Count)
1
845
4
4152
8
4153
12
4177
16
4152
20
4176
24
4177
28
4177
32
4151
AvgTP90 ( ms)
3.13
1.47
2.03
3.56
3.62
5.54
6.18
7.48
7.71
0
L
a
t
e
n
c
y

25. Read Latency
16 KB random READ - M3.2Xlarge EBS-optimized
•
4500
Reads can take advantage of a
deeper queue
100
4153
4120
4000
91.14
93.18
93.70
90
80
•
Current guidance is queue
depth of 1 for every 250 IOPS
EBS-optimized provides
predictable latency
Read
•
IOPS
3500
70
3000
60
2500
50
1965
2000
1864
40
1500
30
1000
20
500
10
5.18
AvgIOPS ( Count)
3.88
3.43
2.15
1.46
0.68
1
4
8
12
16
20
1864
4153
4153
4177
4120
2800
24
1965
28
1213
32
1089
AvgTP90 ( ms)
0.68
91.14
93.18
93.70
QD
0
1.46
2.15
3.43
3.88
5.18
0
L
a
t
e
n
c
y

26. Architecting for Performance: Latency
• Performance requirements may be driven by IOPS or
latency or both
• Recommendation is to start with queue depth of 4
and tune based on IOPS and latency requirement
– Some customers may need lowest possible latency; this can be
achieved at queue depth of 1 or 2
• Very high queue depths ( >24) may decrease IOPS
count as well as increase latency

27. Pre-warming EBS volumes
• Typically 5%, extreme worst case of 50% performance
reduction in IOPS and latency when volumes are used without
pre-warming
– Performance is as provisioned when all the chunks are accessed
• Recommendation if testing or you have spare setup time:
– Write to every 4 MB block before using new volumes
• Linux: DD
• Windows: NTFS Full format
– Takes roughly an hour to pre-warm 1TB 4KB PIOPS volume
– Be warned, can take up to a day for a 1 TB standard EBS volume

28. What about RAID?
Amazon Elastic
Compute Cloud
(EC2)
Amazon Elastic
Compute Cloud
(EC2)
Amazon Elastic
Block Storage
(EBS)
Amazon Elastic
Block Storage
(EBS)

29. Architecting for Performance: RAID
•
Customers stripe number of
volumes to drive higher
IOPS and throughput
–
•
RAID 0 or RAID 10
IO Pattern
How should customers think
about taking snapshots on a
striped volume?
–
–
–
12×400 GB PIOPS volumes, pre-warmed,
RAID 0 LVM, Stripe size 64 KB, attached to
CR1 instance
Quiesce file systems and
take snapshot
Unmount file system and
take snapshot
Use OS-specific tools
Sequential
Random
Block Sizes
4K
16K
1M
Thread
8
8
1
4K
16K
1M
8
8
1
Write IOPS Write BW (MB) Read IOPS Read BW (MB)
33,500
134
48,250
193
13,875
222
48,063
769
247
247
823
823
35,250
13,875
496
141
222
496
48,250
42,125
795
Check out STG402 for new advanced snapshot tools!!
193
674
795

30. Cheat Sheet! Storage Workloads on AWS
Workload/
Software
Typical block
size
Random
/Seq?
Typical
correct
EBS type
Max EBS @
500 MB/s
Max EBS @
1 GB/s
Max EBS @ CC/H
Oracle DB
Configurable:2 KB –
16 KB
random
PIOPS
RAID 0, 2 × 4000
PIOPS:
~7800 IOPS
RAID 0, 4 × 4000
PIOPS:
~15,600 IOPS
RAID 0, 24 × 4000 PIOPS:
~96,000 IOPS
Default 8 KB
random
PIOPS
RAID 0, 2 × 4000
PIOPS:
~7800 IOPS
RAID 0, 4 × 4000
PIOPS:
~15,600 IOPS
RAID 0, 20 × 4000 PIOPS:
~80,000 IOPS
MySQL
16 KB
random
PIOPS
1 x 4000 PIOPS:
~4000 IOPS
RAID 0, 2 × 4000
PIOPS:
~7,800 IOPS
RAID 0, 12 × 4000 PIOPS:
~48,000 IOPS
8 KB
random
PIOPS
RAID 0, 2 × 4000
PIOPS:
~7,800 IOPS
RAID 0, 4 × 4000
PIOPS:
~15,600 IOPS
RAID 0, 24 × 4000 PIOPS:
~96,000 IOPS
IBM DB/2
RDBMS
8 KB w/ 64 KB
extents
PostgreSQL
NewSQL
Microsoft
SQL Server
Configurable:2 KB –256 KB
random
PIOPS
1 x 4000 PIOPS:
~2000 IOPS
RAID 0, 4 × 4000
PIOPS:
~8,000 IOPS
RAID 0, 12 × 4000 PIOPS:
~24,000 IOPS
Default 32 KB
NuoDB
8 KB
random
PIOPS
RAID 0, 2 × 4000
PIOPS:
~7800 IOPS
RAID 0, 4 × 4000
PIOPS:
~15,600 IOPS
RAID 0, 24 × 4000 PIOPS:
~96,000 IOPS
Clustrix
4 KB
random
PIOPS
RAID 0, 4 × 4000
PIOPS:
~15,600 IOPS
RAID 0, 8 × 4000
PIOPS:
~31,000 IOPS
RAID 0, 24 × 4000 PIOPS:
~96,000 IOPS
GenieDB
16 KB
random
PIOPS
1 × 4000 PIOPS:
~4000 IOPS
RAID 0, 2 × 4000
PIOPS:
~7,800 IOPS
RAID 0, 12 × 4000 PIOPS:
~48,000 IOPS

31. Cheat Sheet! Storage Workloads on AWS
Typical
block
size
Random
/Seq?
Typical
correct
EBS type
Max EBS @
500 MB/s
Max EBS @ 1 GB/s
Max EBS @ CC/H
MongoDB
4 KB
serialized
PIOPS
RAID 0, 4 × 4000
PIOPS:
~15,600 IOPS
RAID 0, 8 × 4000
PIOPS:
~31,000 IOPS
RAID 0, 24 × 4000
PIOPS:
~96,000 IOPS
Cassandra
NoSQL
Workload/
Software
4 KB
random
PIOPS
RAID 0, 4 × 4000
PIOPS:
~15,600 IOPS
RAID 0, 8 × 4000
PIOPS:
~31,000 IOPS
RAID 0, 24 × 4000
PIOPS:
~96,000 IOPS
CouchBase
4 KB
random
PIOPS
RAID 0, 4 × 4000
PIOPS:
~15,600 IOPS
RAID 0, 8 × 4000
PIOPS:
~31,000 IOPS
RAID 0, 24 × 4000
PIOPS:
~96,000 IOPS
Hadoop /
Hbase
64 KB
random
PIOPS
1 × 4000 PIOPS:
~1000 IOPS
RAID 0, 2 × 4000
PIOPS:
~2000 IOPS
RAID 0, 12 × 4000
PIOPS:
~12,000 IOPS
Riak /
LevelDB
4 KB
random
PIOPS
RAID 0, 4 × 4000
PIOPS:
~15,600 IOPS
RAID 0, 8 × 4000
PIOPS:
~31,000 IOPS
RAID 0, 24 × 4000
PIOPS:
~96,000 IOPS

32. Cheat Sheet! Storage Workloads on AWS
DN File System
Typical
block
size
Random
/Seq?
Typical
correct
EBS type
Max EBS @
500 MB/s
Max EBS @ 1 GB/s
Max EBS @ CC/H
AeroSpike
128 KB
random
PIOPS
1 × 4000
PIOPS:
~500 IOPS
RAID 0, 2 x 4000
PIOPS:
~1000 IOPS
RAID 0, 12 x 4000 PIOPS:
~6,000 IOPS
(768MB/s)
Vertica
1 MB
sequential
PIOPS
1 × 4000
PIOPS:
~68 IOPS
RAID 0, 2 × 4000
PIOPS:
~125 IOPS
RAID 0, 12 × 4000 PIOPS:
~750 IOPS
(768MB/s)
TeraData
Warehouse
Workload/
Software
128 KB
sequential
PIOPS
1 × 4000
PIOPS:
~500 IOPS
RAID 0, 2 × 4000
PIOPS:
~1000 IOPS
RAID 0, 12 × 4000 PIOPS:
~6,000 IOPS
(768MB/s)
GlusterFS
128 KB
sequential
PIOPS
1 × 4000
PIOPS:
~500 IOPS
RAID 0, 2 × 4000
PIOPS:
~1000 IOPS
RAID 0, 12 × 4000 PIOPS:
~6,000 IOPS
(768MB/s)
OrangeFS
256 KB
sequential
PIOPS
1 × 4000
PIOPS:
~250 IOPS
RAID 0, 2 × 4000
PIOPS:
~500 IOPS
RAID 0, 12 × 4000 PIOPS:
~3,000 IOPS
(768MB/s)
Riak (CS)
1 MB
sequential
PIOPS
1 × 4000
PIOPS:
~68 IOPS
RAID 0, 2 × 4000
PIOPS:
~125 IOPS
RAID 0, 12 × 4000 PIOPS:
~750 IOPS
(768MB/s)

33. Architecture – What Else Uses Network?

34. Performance – Extra-large Production Scale
•
Leverage SSD instance type
(hi1.4xlarge)
o 2 × 1 TB SSD storage (ephemeral
storage)
o Perfect for replicas
•
If replicas on SSD instance types, disable
integrity features such as fsync and
full_page_writes on those hosts to
improve performance

35. Testing Random 4 KB Reads
+
PIOPS
Stable
EBS
SSD

36. What About Performance Cost?
hi1.4xlarge
cc2.8xlarge
hi1.4xlarge
VS.
24 @ 4 KB
PIOPS
If >10 KB write IOPS, TEST,
but probably choose PIOPS
If >20 KB IOPS read, choose hi1
If 3 YR, and >8 KB IOPS, choose hi1
On demand, If <20 KB read IOPS, choose PIOPS
$11773 on-demand,
$10589 effective 3 YR reserved
$4538 on-demand,
$1539 effective 3 YR reserved

37. What about Capacity Cost?
hs1.8xlarge
cc2.8xlarge
hs1.8xlarge
VS.
48x
1TB
EBS
$7312 on-demand,
$6128 effective 3 YR reserved
If >43TB, or > 800MB/s, choose hs1
If 3 year, and >18TB, choose hs1
$6734 on-demand,
$2408 effective 3 YR reserved

38. Performance / Stability Tips
• Ext4 or XFS (understand journal impact!)
• nobarrier, noatime, noexec, nodiratime
• Raise file descriptor limits
• Set read-aheads low
• SNAPSHOT SNAPSHOT SNAPSHOT

39. From 1000 to 98,000: What does that mean?

40. So.. all storage workloads!

41. Please give us your feedback on this
presentation
STG 302
As a thank you, we will select prize
winners daily for completed surveys!
@milesward