Big Data Analytics on AWS - Carlos Conde - AWS Summit Paris
•
4 j'aime•1,548 vues
This document discusses using AWS for big data analytics. It notes that as data volumes grow, collecting, storing, analyzing and sharing data is essential. AWS provides services like S3, DynamoDB and EMR on Hadoop to help with collecting, storing, analyzing and sharing large volumes of data cost effectively. It also discusses using tools like Pig and parallelizing workloads to analyze data more efficiently.
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
En vedette
En vedette (7)
Similaire à Big Data Analytics on AWS - Carlos Conde - AWS Summit Paris
Similaire à Big Data Analytics on AWS - Carlos Conde - AWS Summit Paris (20)
Plus de Amazon Web Services
Plus de Amazon Web Services (20)
Dernier
Dernier (20)
Big Data Analytics on AWS - Carlos Conde - AWS Summit Paris
- 1. BIG DATA ANALYTICS WITH AWS Carlos Conde │Solutions Architect
- 2. BIG DATA When innovation is required to collect, store, analyze, and share your data
- 3. YOU DON’T HAVE THE CHOICE…
- 4. 27 TB per day Large Hadron Collider – CERN
- 6. The more data you collect The more VALUE you can derive from it
- 7. Big Data Verticals Social Media/Advertising Oil & Gas Retail Life Sciences Financial Services Security Network/Gaming User Monte Anti-virus Demographics Targeted Recommendations Carlo Advertising Simulations Seismic Genome Fraud Usage Analysis Analysis Detection analysis Image and Transaction Risk Video Analysis Analysis Image In-game Processing Recognition metrics
- 8. 2.7 Zetabytes in 2012 Over 90% will be unstructured
- 10. COLLECT │ STORE │ ANALYZE │ SHARE
- 11. COLLECT │ STORE │ ANALYZE │ SHARE
- 15. COLLECT │ STORE │ ANALYZE │ SHARE
- 16. AMAZON S3
- 17. AMAZON DYNAMODB
- 19. COLLECT │ STORE │ ANALYZE │ SHARE
- 22. CLUSTER GPU QUADRUPLE EXTRA LARGE Intel Xeon X5570, quad-core 2x Nehalem architecture NVIDIA Tesla Fermi 2x M2050 GPUs 22 GB of memory – 1.7 TB of storage $2.1 PER HOUR
- 24. PARALELIZATION
- 25. ON A SINGLE INSTANCE COST: 4h x $2.1 = $8.4 RENDERING TIME: 4h
- 26. ON MULTIPLE INSTANCES COST: 2 x 2h x $2.1 = $8.4 RENDERING TIME:
- 27. "Hadoop is a reliable storage and data analysis system" HDFS MapReduce
- 28. Deploying a Hadoop cluster is hard http://eddie.niese.net/20090313/dont-pity-incompetence/
- 29. AMAZON EMR HADOOP + AWS
- 31. Doing analytics in Eclipse is wrong…
- 32. PIG
- 33. A real Pig script (used at Twitter)
- 37. USE THE RIGHT TOOL FOR THE RIGHT JOB RDBMS Hadoop Interactive Reporting Affordable (<1sec) Storage/Compute Multistep Transactions Structured or Not (Agility) Lots of Updates/Deletes Resilient Auto Scalability
- 38. Data Warehouse (Batch Processing) Data Warehouse Data Warehouse (Steady State) (Steady State) Shrink to Expand to 9 instances 25 instances
- 40. OPERATIONAL HADOOP EXPERIENCE Operated over a million Hadoop clusters last year
- 41. COLLECT │ STORE │ ANALYZE │ SHARE
- 42. PUBLIC DATA SETS http://aws.amazon.com/publicdatasets
- 46. COLLECT │ STORE │ ANALYZE │ SHARE
- 47. INNOVATE
- 48. « Want to increase innovation? Lower the cost of failure » Joi Ito
- 49. AWS LOWERS THE COST OF INNOVATION Testing a new idea is cheap