A short introduction to Apache Whirr. What is it and how does it relate to the cloud ? How can it be used with Hadoop ?

1. Apache Whirr
●
What is it ?
●
How does it work?
●
The Cloud
●
Architecture
●
Use with Hadoop
www.semtech-solutions.co.nz
info@semtech-solutions.co.nz

2. Apache Whirr – What is it ?
●
A library based cloud service system
●
API libraries for cloud providers
●
Choose a configuration file to define a cluster
●
High level interaction
●
Service based on roles
www.semtech-solutions.co.nz
info@semtech-solutions.co.nz

3. Apache Whirr – How does it work ?
●
Libraries provided to offer high level API
●
Based on JClouds
●
Recipe based approach for cloud providers i.e. EC2
●
It is cloud neutral
●
Create clusters as you need them for
–
Dev / test etc
www.semtech-solutions.co.nz
info@semtech-solutions.co.nz

4. Apache Whirr – How does it work ?
●
Automatically start instances on the Cloud
●
Configure and start Hadoop
●
Add applications like
–
Hive
–
Hbase
–
Yarn / MapReduce
www.semtech-solutions.co.nz
info@semtech-solutions.co.nz

5. Apache Whirr – Why go virtual ?
●
Whirr gives independence from Cloud vendor
●
Makes it easier to move vendors later
●
Save money by only using what you need
●
Expand the cluster as demand requires
●
Reduce the cluster when possible
●
Compress data as much as possible to reduce costs
●
Virtual cost < physical cost ( should be ) until
–
Data sizes in high Tbyte – low Pbyte range
www.semtech-solutions.co.nz
info@semtech-solutions.co.nz

6. Apache Whirr – Supported
●
What cloud suppliers are available
–
–
●
Amazon EC2
Rackspace Cloud Services
What services do they support ?
–
Cassandra
–
Hadoop
–
Zoo Keeper
–
Hbase
–
Elastic Search
–
Voldemort
–
Hama
www.semtech-solutions.co.nz
info@semtech-solutions.co.nz

7. Apache Whirr – Example config
Example Whirr configuration - .whirr/credentials
whirr.provider=aws-ec2
whirr.identity=your-aws-key
whirr.credential=your-aws-secret
Whirr Hadoop configuration - hadoop.properties
whirr.cluster-name=myhadoopcluster
whirr.instance-templates=1 hadoop-jobtracker+hadoop-namenode,1
hadoop-datanode+hadoop-tasktracker
whirr.provider=aws-ec2
Start Whirr
whirr launch-cluster --config hadoop.properties
www.semtech-solutions.co.nz
info@semtech-solutions.co.nz

8. Contact Us
●
Feel free to contact us at
–
www.semtech-solutions.co.nz
–
info@semtech-solutions.co.nz
●
We offer IT project consultancy
●
We are happy to hear about your problems
●
You can just pay for those hours that you need
●
To solve your problems