This document discusses how Amazon Web Services (AWS) provides high-performance computing (HPC) capabilities in the cloud. It notes that HPC workloads require large amounts of data, compute, and people. AWS offers elastic, on-demand infrastructure services like Amazon EC2 and S3 that allow users to pay for only what they use. The document outlines how users can run HPC workloads on AWS using EC2 instance types optimized for compute and GPU-accelerated instances. It shares examples of organizations using AWS for applications like computational fluid dynamics and molecular modeling.

1. HPC on Amazon Web Services
Deepak Singh
Amazon Web Services
Dec 17, 2010

2. Image: Simon Cockell under CC-BY

3. the new reality

4. lots and lots and lots
and lots and lots of data

5. lots and lots and lots
and lots and lots of
compute

6. lots and lots and lots
and lots and lots of
people

7. lots and lots and lots
and lots and lots of
places

8. constant change

9. goal

10. innovate

11. innovate in a new reality

12. optimize the most
valuable resource

13. compute, storage,
workflows, memory,
transmission, algorithms,
cost, …

14. people drive innovation
Credit: Pieter Musterd a CC-BY-NC-ND license

15. make people productive
Credit: Pieter Musterd a CC-BY-NC-ND license

16. challenges

17. Successful
Your Idea
Product

18. Successful
Your Idea
Product

19. Successful
Your Idea
Product
Great Idea Not Prioritized

20. Successful
Your Idea
Product
Great Idea Not Prioritized

21. Fully Utilized Clusters
Tight Budgets
Shared Resources

22. enter the cloud

23. infrastructure services

24. building blocks

26. Undifferentiated
Heavy Lifting

27. pay as you go

28. pay for what you use

29. on demand

30. programmable

31. import boto
import boto.emr
from boto.emr.step import StreamingStep
Connect to Elastic MapReduce
from boto.emr.bootstrap_action import BootstrapAction
import time
# set your aws keys and S3 bucket, e.g. from environment or .boto
AWSKEY=
SECRETKEY=
S3_BUCKET=
NUM_INSTANCES = 1
conn = boto.connect_emr(AWSKEY,SECRETKEY)
bootstrap_step = BootstrapAction("download.tst",
"s3://elasticmapreduce/bootstrap-actions/download.sh",None)
Install packages
step = StreamingStep(name='Wordcount',
mapper='s3n://elasticmapreduce/samples/wordcount/wordSplitter.py',
cache_files = ["s3n://" + S3_BUCKET + "/boto.mod#boto.mod"],
reducer='aggregate',
input='s3n://elasticmapreduce/samples/wordcount/input',
output='s3n://' + S3_BUCKET + '/output/wordcount_output')
Set up mappers &
jobid = conn.run_jobflow(
name="testbootstrap",
reduces
log_uri="s3://" + S3_BUCKET + "/logs",
steps = [step],
bootstrap_actions=[bootstrap_step],
num_instances=NUM_INSTANCES)
print "finished spawning job (note: starting still takes time)"
state = conn.describe_jobflow(jobid).state
print "job state = ", state
print "job id = ", jobid
while state != u'COMPLETED':
print time.localtime() job state
time.sleep(30)
state = conn.describe_jobflow(jobid).state
print "job state = ", state
print "job id = ", jobid
print "final output can be found in s3://" + S3_BUCKET + "/output" + TIMESTAMP
print "try: $ s3cmd sync s3://" + S3_BUCKET + "/output" + TIMESTAMP + " ."

32. elastic

33. Capacity
Elastic
capacity
Real
demand
Time

35. On demand Faster to market
Pay as you go Maintain focus
Pay to play Efficiency
Elastic resources Capacity planning

36. Computing with
Amazon EC2

37. Credit: Angel Pizzaro, U. Penn

38. Credit: Tom Fifield: U. Melbourne

40. s
pe
ty
e
nca
standard “m1”
st
in
2
EC
high cpu “c1”
high memory “m2”
http://aws.amazon.com/ec2/instance-types/

41. listening to customers

42. new EC2 instance type

43. cluster compute instances
text
http://aws.amazon.com/ec2/instance-types/

44. 2 * Xeon 5570 (“Intel Nehalem”)
23 GB RAM
10 gbps Ethernet
1690 TB local disk
HVM-based virtualization
$1.60 / hr

45. 10gbps

46. Placement
Group

47. full bisection bandwidth

48. HPC on EC2 =
EC2 instance
+
high bandwidth, low latency
networking
http://aws.amazon.com/hpc-applications/

49. Linpack benchmark
880-instance CC1 cluster
Performance: 41.82 TFlops*
*#231 in the most recent Top 500 rankings

51. CFD
Molecular Modeling
Sequence Analysis
Engineering Design
Energy Trading
…
high I/O applications

52. s
pe
ty
e
standard “m1”
nca
st
in
high cpu “c1”
2
EC
high memory “m2”
cluster compute “cc1”
http://aws.amazon.com/ec2/instance-types/

53. HPC is evolving

55. cluster GPU instances
http://aws.amazon.com/ec2/instance-types/

56. HPC on EC2 =
EC2 instance
+
high bandwidth, low latency
networking
+
GPU
http://aws.amazon.com/ec2/hpc-applications/

57. 2 * Xeon 5570 (“Intel Nehalem”)
2 * Tesla M2050 GPU
22 GB RAM
10 gbps Ethernet
1690 TB local disk
HVM-based virtualization
$2.10 / hr

58. s
pe
ty
nc
e
a
standard “m1”
st
high cpu “c1”
in
2
EC
high memory “m2”
cluster compute “cc1”
cluster GPU “cg1”
http://aws.amazon.com/ec2/instance-types/

59. CFD
Molecular Dynamics
Financial Modeling
Rendering
Video Processing
…
What is your interest?

60. “90 percent scaling efficiency on
clusters of up to 128 GPUs”
-- mental images iRay

61. Getting Started

62. http://aws.amazon.com/hpc-applications

64. 4 steps

65. 15 minutes

66. http://aws.amazon.com/ec2

72. ecosystem

74. ISV ecosystem

75. Mathworks
mental images
Revup Render
Elemental Technologies
...

76. HPC with AWS

77. E2 instance
+
high bandwidth, low
latency networking
+
Tesla GPU*
*optional

78. On demand Faster to market
Pay as you go Maintain focus
Pay to play Efficiency
Elastic resources Capacity planning

79. make people productive
Credit: Pieter Musterd a CC-BY-NC-ND license

80. Successful
Your Idea
Product
Great Idea Not Prioritized

81. Successful
Your Idea
Product

82. http://aws.amazon.com/hpc-applications

83. deesingh@amazon.com
Twitter:@mndoci
http://slideshare.net/mndoci
http://mndoci.com
Inspiration and ideas from
Matt Wood, James Hamilton
& Larry Lessig
Credit” Oberazzi under a CC-BY-NC-SA license