Title: Scaling and Managing Big Data Apps on Public Clouds Abstract: The massive computing and storage resources that are needed to support big data applications make on-demand, elastic cloud environments an ideal fit. However, managing your big data app on the cloud is no walk in the park - configuration, orchestration, H/A, auto-scaling are all quite complex when it comes to choosing the right cloud for you, whether it’s public, private or a hybrid cloud - which is where Cloudify and Eucalyptus come together. In this session, you'll learn how to deploy, manage, monitor and scale your big data apps on the open source Eucalyptus cloud platform using Cloudify, as well as easily test drive your apps locally and then migrate the workload to Amazon Web Services EC2.

1. @natishalom

2. About GigaSpaces
2

4. The Reality of Big Data..
2.7 ZB
Global Digital Data
0.5 Petabytes
43% think that data
analytics could be improved in their
Two years tweets organization if data analytics was part of
cloud services
66%
Plan to use Big Data/Cloud
4

5. Large ISV Case Study
• Application
– Call Center surveillance
• Background
– Previously – voice data
• Goal for a new system
– Monitor data & voice
– Multiple data sources
– Advanced correlations

6. Ever Growing Data
Deeper Correlation
Tight Performance

7. A Classic Case for...

8. A Typical Big Data System…

9. Business
Cost
Impact

10. Big Data
in the Cloud

11. Big Data in the Cloud - 3 Reasons
• Skills
– Do you really need/want this all in-
house?
• Huge amounts of external data
Holger Kisker – Does it make sense to move and
manage all this data behind your
firewall?
• Focus on the value of your data
– Instead of big data management

12. • Auto start VMs
• Install and configure
app components
• Monitor
• Repair
• (Auto) Scale
• Burst…

13. Use Eucalyptus for private
data , AWS for sporadic
workloads ..

14. • Consistent
Management
• Automation Through
the Entire Stack

16. Consistent Management
Recipes consistent description for running any app:
 What middleware services to run
 Dependencies between services
 How to install services
 Where application and service binaries are
 When to spawn or terminate instances
 How to monitor each of the services
16

18. Choosing the Right Cloud for the Job
compute {
template "SMALL_LINUX"
}
SMALL_LINUX : template{ SMALL_LINUX : template
imageId linuxImageId imageId "us-east-1/ami-76f0061f“
remoteDirectory "/home/user/gs-files" remoteDirectory "/home/ec2-user/gs-files“
machineMemoryMB 1600 machineMemoryMB 1600
hardwareId “m1.medium” hardwareId "m1.small"
locationId “us-west-1” locationId "us-east-1"
localDirectory "upload“ localDirectory "upload"
keyFile “myEucaKeyFile.pem” keyFile "myKeyFile.pem"
username "user" options ([
options ([ "securityGroups" : ["default"]as
"securityGroups" : ["default"] as String[],
String[], "keyPair" : "myKeyFile"
"keyPair" : keyPair ])
]) overrides (["jclouds.ec2.ami-query":"",
overrides ([“endpoint” : "jclouds.ec2.cc-ami-query":""])
“http://communitycloud.eucalyptus.com”]) privileged true
privileged true }
}
18

19. Automation across the stack
1 Upload your recipe.
2 Cloudify creates VM’s & installs agents
3 Agents install and manage your app
4 Cloudify automate the scaling

21. Big Data On Demand with Cloudify
Relational DB Clusters NoSQL Clusters Hadoop
MySQL MongoDB Hadoop (Hive, Pig,..)
Postgress Cassandra Storm
Couchbase ZooKeeper
ElasticSearch

22. Demo Time: Storm Cluster
® Copyright 2011 Gigaspaces Ltd. All Rights
22
Reserved

23. Large ISV Case Study
• Application
– Call Center surveillance system
• Background
– Previously – voice data
• Goal for a new system
Monitor data & voice
Multiple data sources
Advanced correlations Mission
Accomplished

24. Additional Benefits
• True Cloud Economics
• One product -> Any
Customer Environment
• Increased Agility

25. http://www.cloudifysource.org
http://github.com/CloudifySource