1. Big Data Analytics
with Amazon Web Services
Dr. Matt Wood
An Online Seminar. Tuesday 16th October.

2. Hello, and thank you.

3. Big Data Analytics
An introduction

4. Big Data Analytics
An introduction
The story of analytics on AWS

5. Big Data Analytics
An introduction
The story of analytics on AWS
AWS Marketplace

6. Big Data Analytics
An introduction
The story of analytics on AWS
AWS Marketplace
Success story: Brightcove

7. 1
INTRODUCING BIG DATA

8. Data for competitive
advantage.

9. Using data
Customer segmentation,
financial modeling,
system analysis,
line-of-sight,
business intelligence.

10. Generation
Collection & storage
Analytics & computation
Collaboration & sharing

11. Cost of data generation
is falling.

12. lower cost,
increased throughput
Generation
Collection & storage
Analytics & computation
Collaboration & sharing

13. Generation
HIGHLY CONSTRAINED
Collection & storage
Analytics & computation
Collaboration & sharing

14. Very high barrier to turning
data into information.

15. Move from a
data generation challenge to
analytics challenge.

16. Enter the Cloud.

17. Remove the constraints.

18. Enable data-driven innovation.

19. Move to a distributed data
approach.

20. Maturation of two things.

21. Software for distributed
storage and analysis
Maturation of two things.

22. Software for distributed
storage and analysis
Maturation of two things.
Infrastructure for distributed
storage and analysis

23. Software
Frameworks for
data-intensive workloads.
Distributed by design.

24. Infrastructure
Platform for
data-intensive workloads.
Distributed by design.

25. Support the
data timeline.

26. Generation
HIGHLY CONSTRAINED
Collection & storage
Analytics & computation
Collaboration & sharing

27. Generation
Collection & storage
Analytics & computation
Collaboration & sharing

28. Lower the
barrier to entry.

29. Accelerate time to market
and increase agility.

30. Enable new business
opportunities.

31. Washington Post
Pinterest
NASA

32. “AWS enables Pfizer to explore
difficult or deep scientific
questions in a timely, scalable
manner and helps us make better
decisions more quickly”
Michael Miller, Pfizer

33. 2
THE STORY OF ANALYTICS

34. EC2
Utility computing.
6 years young.

35. Scale out systems
Embarrassingly parallel problems.
Queue based distribution.
Small, medium and high scale.

39. Cost optimization.
EC2
Utility computing.
6 years young.

40. Achieving economies of scale
100%
Time

41. Achieving economies of scale
100%
Reserved capacity
Time

42. Achieving economies of scale
100%
On-demand
Reserved capacity
Time

43. Achieving economies of scale
UNUSED CAPACITY
100%
On-demand
Reserved capacity
Time

44. Spot Instances
Bid on unused EC2 capacity.
Very large discount.
Perfect for batch runs.
Balance cost and scale.

45. <$1000 per hour

46. Map/reduce
Pattern for distributed computing.
Software frameworks such as
Hadoop.
Write two functions. Scale up.

47. Map/reduce
Pattern for distributed computing.
Software frameworks such as
Hadoop.
Write two functions. Scale up.
Complex cluster configuration
and management.

48. Amazon Elastic MapReduce
Managed Hadoop clusters.
Easy to provision and monitor.
Write two functions. Scale up.
Optimized for S3 access.

49. S3
Input data

50. S3
Input data
Code Elastic
MapReduce

51. S3
Input data
Code Elastic Name
MapReduce node

52. S3
Input data
Code Elastic Name
MapReduce node
Elastic
cluster

53. S3
Input data
Code Elastic Name
MapReduce node
HDFS
Elastic
cluster

54. S3
Input data
Code Elastic Name
MapReduce node
Queries
HDFS
+ BI
Via JDBC, Pig, Hive
Elastic
cluster

55. S3
Input data
Code Elastic Name Output
MapReduce node S3 + SimpleDB
Queries
HDFS
+ BI
Via JDBC, Pig, Hive
Elastic
cluster

56. S3
Input data
Output
S3 + SimpleDB

67. Performance

68. Performance
Compute performance

69. Cluster Compute
Intel Xeon E5-2670
10 gig E non-blocking network
60.5 Gb
Placement groupings

70. Cluster Compute
Intel Xeon E5-2670
10 gig E non-blocking network
60.5 Gb
Placement groupings
+ GPU enabled instances

71. Performance
Compute performance

72. IO performance
Performance
Compute performance

73. NoSQL
Unstructured data storage.

74. DynamoDB
Predictable, consistent performance
Unlimited storage
Single digit millisecond latencies
No schema for unstructured data
Backed on solid state drives

75. ...and SSDs for all.
New Hi1 storage instances.

76. hi1.4xlarge
2 x 1Tb SSDs
10 GigE network
HVM: 90k IOPS read, 9k to 75k write
PV: 120k IOPS read, 10k to 85k write

77. “The hi1.4xlarge configuration is
about half the system cost for the
same throughput.”
Netflix
http://techblog.netflix.com/2012/07/benchmarking-high-performance-io-with.html

78. Generation
Collection & storage
Analytics & computation
Collaboration & sharing

79. Performance + ease of use

80. 3
AWS MARKETPLACE

81. Extend platform with
partners

82. Innovate on behalf of
customers

83. Remove undifferentiated
heavy lifting

84. AWS Marketplace
aws.amazon.com/marketplace

87. Generation
Collection & storage
Analytics & computation
Collaboration & sharing

88. Generation
Collection & storage
Analytics & computation
Collaboration & sharing

89. Collection & storage
Acunu Reflex
Apache Cassandra NoSQL database
MongoDB
With and without EBS RAID storage
Couchbase
Community and Enterprise editions
ScaleArc
MySQL load balancing

95. Generation
Collection & storage
Analytics & computation
Collaboration & sharing

96. Generation
Collection & storage
Analytics & computation
Collaboration & sharing

97. Analytics & computation
KarmaSphere Analytics
for Amazon Elastic MapReduce
MapR M5
Hadoop Distribution
Metamarkets
Event based data processing

98. Analytics & computation
StackIQ Rocks+
HPC clusters with MPI, Grid Engine
Univa Grid Engine
One click cluster deployment
Quantivo
Data association analytics

99. Generation
Collection & storage
Analytics & computation
Collaboration & sharing

100. Generation
Collection & storage
Analytics & computation
Collaboration & sharing

101. Collaboration & sharing
Aspera Faspex
20 Mbps data transfer

102. 4
SUCCESS STORY