Big data: understanding the present
•
0 j'aime•645 vues
Slides for an introductory course on Big Data Tools for Artificial Intelligence. This first set of slides introduces the concept of big data and the current context.
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à Big data: understanding the present
Similaire à Big data: understanding the present (20)
Dernier
Dernier (20)
Big data: understanding the present
- 1. Big Data Understanding the present Víctor Sánchez Anguix Universitat Politècnica de València MSc. In Artificial Intelligence, Pattern Recognition, and Digital Image Course 2014/2015
- 2. Big Data: Understanding the present. MSc. in Artificial Intelligence, Pattern Recognition and Digital Image Definitions “Big Data is the opportunity space created by new open source, distributed systems from the consumer internet space” Hortonworks, http://hortonworks.com/blog/big-data-defined/
- 3. Big Data: Understanding the present. MSc. in Artificial Intelligence, Pattern Recognition and Digital Image Definitions “Big data is the term increasingly used to describe the process of applying serious computing power – the latest in machine learning and artificial intelligence – to seriously massive and often highly complex sets of information.” Microsoft, http://news.microsoft.com/2013/02/11/the-big-bang-how-the-big-data-explosion-is-changing-the-world/
- 4. Big Data: Understanding the present. MSc. in Artificial Intelligence, Pattern Recognition and Digital Image Definitions “Big data is a collection of data from traditional and digital sources inside and outside your company that represents a source for ongoing discovery and analysis” Forbes, http://www.forbes.com/sites/lisaarthur/2013/08/15/what-is-big-data/
- 5. Big Data: Understanding the present. MSc. in Artificial Intelligence, Pattern Recognition and Digital Image Definitions “Big data is what happened when the cost of storing information became less than the cost of making the decision to throw it away” George Dyson, technology historian
- 6. Big Data: Understanding the present. MSc. in Artificial Intelligence, Pattern Recognition and Digital Image Definitions The 5 V’s in Big Data: Volume Variety Velocity Variability Veracity Based on Doug Laney’s 3 V’s
- 7. Big Data: Understanding the present. MSc. in Artificial Intelligence, Pattern Recognition and Digital Image Definitions “Big data is like teenage sex: everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it...” Dan Ariely, MIT
- 8. Big Data: Understanding the present. MSc. in Artificial Intelligence, Pattern Recognition and Digital Image What is exactly big data?
- 9. Big Data: Understanding the present. MSc. in Artificial Intelligence, Pattern Recognition and Digital Image Big Data ➢ There is not an agreed definition ➢ The term first appeared in a NASA paper: “... an interesting challenge for computer systems: data sets are generally quite large, taxing the capacities of main memory, local disk, and even remote disk. We call this the problem of big data. When data sets do not fit in main memory (in core), or when they do not fit even on local disk, the most common solution is to acquire more resources” http://dl.acm.org/citation.cfm?id=266989. 267068&coll=DL&dl=GUIDE ➢ Say no to definitions. It’s better to understand the present
- 10. Big Data: Understanding the present. MSc. in Artificial Intelligence, Pattern Recognition and Digital Image Understanding the present Large Synoptic Survey Telescope ~ 20 TB per night http://www.lsst. org/lsst/science/concept_data
- 11. Big Data: Understanding the present. MSc. in Artificial Intelligence, Pattern Recognition and Digital Image Understanding the present Internet exponential growth http://www.internetlivestats. com
- 12. Big Data: Understanding the present. MSc. in Artificial Intelligence, Pattern Recognition and Digital Image Understanding the present U.S. congress library ~ 235 TB of data http://blog.getsatisfaction.com/2011/07/13/big- data/?view=socialstudies
- 13. Big Data: Understanding the present. MSc. in Artificial Intelligence, Pattern Recognition and Digital Image Understanding the present 1 million transactions per hour. Estimated 2.5 PB data http://www.sas. com/resources/whitepaper/wp_46345.pdf
- 14. Big Data: Understanding the present. MSc. in Artificial Intelligence, Pattern Recognition and Digital Image Understanding the present Many successful stories getting value from data
- 15. Big Data: Understanding the present. MSc. in Artificial Intelligence, Pattern Recognition and Digital Image Understanding the present http://www.mocom2020.com/data/2009/05/computer-power-future.gif
- 16. Big Data: Understanding the present. MSc. in Artificial Intelligence, Pattern Recognition and Digital Image Understanding the present ➢ More data than ever ➢ There is hidden value in data ➢ Computers are cheaper and faster ➢ They are not enough → Algorithms!! ○ Distributed file systems ○ Map Reduce ○ Locality-sensitive hashing ○ Lempel-Ziv-Oberhumer (LZO) ○ etc...
- 17. Big Data: Understanding the present. MSc. in Artificial Intelligence, Pattern Recognition and Digital Image Understanding the present ➢ And of course, tools
- 18. Big Data: Understanding the present. MSc. in Artificial Intelligence, Pattern Recognition and Digital Image What is our role? http://datasciencedojo.com/what-are-the-key-skills-of-a-data-scientist/
- 19. Big Data: Understanding the present. MSc. in Artificial Intelligence, Pattern Recognition and Digital Image In the next 10h ... ➢ Overview of the Big Data ecosystem ➢ Core algorithms → Map Reduce & DFS ➢ ETL tools ➢ AI tools for Big Data ➢ Future trends
- 20. Big Data Understanding the present Víctor Sánchez Anguix Universitat Politècnica de València MSc. In Artificial Intelligence, Pattern Recognition, and Digital Image Course 2014/2015