Cloud architectures for data science
•
3 j'aime•569 vues
Data Science covers the complete workflow from defining a question, finding the most suitable data source, identifying the right tools and finally presenting the best possible answer in a clear, engaging manner. But it all starts with having access to the data. In these slides I will walk your through some examples of how to collect, store and access data in the Cloud with the use of different APIs.
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
En vedette
Similaire à Cloud architectures for data science
Similaire à Cloud architectures for data science (20)
Plus de Margriet Groenendijk
Plus de Margriet Groenendijk (11)
Dernier
Dernier (20)
Cloud architectures for data science
- 1. @MargrietGr Margriet Groenendijk, PhD Developer Advocate for IBM Cloud Data Services O’Reilly Software Architecture Conference San Francisco 16 November 2016 Cloud Architectures for Data Science
- 2. @MargrietGr About me • Developer Advocate at IBM Cloud Data Services, UK •Data science •Python, Spark, R, Cloudant, dashDB • Research Fellow at University of Exeter, UK •Worked with very large observational datasets and the output of global scale climate models • PhD at Vrije Universiteit Amsterdam, the Netherlands •Explored large observational datasets of carbon uptake by forests
- 3. @MargrietGr A Brief History of Data Science • Computer Science • Data Technology • Visualization • Mathematics • Statistics http://www.datascienceassn.org/content/history-data-science
- 10. @MargrietGr 2016
- 11. @MargrietGr https://blog.rjmetrics.com/2015/10/05/how-many-data-scientists-are-there/ How many Data Scientists are there?
- 15. @MargrietGr Data Engineers Data Scientists Business Analysts App Developers Data Science is a Team Effort Data
- 16. @MargrietGr
- 20. @MargrietGr Collect Data APIs Open Data Maps Web Scraping Time Series
- 21. @MargrietGr Store Data Object Store - binary files Relational database Document store - json
- 25. @MargrietGr The Spark Stack from Karau et al.: Learning Spark
- 26. @MargrietGr RDDs : Resilient Distributed Datasets • Data does not have to fit on a single machine • Data is separated into partitions • Creation of RDDs •Load an external dataset •Distribute a collection of objects • Transformations construct a new RDD from a previous one (lazy!) • Actions compute a result based on an RDD
- 27. @MargrietGr Run Spark locally in a Python notebook https://www.continuum.io/downloads http://spark.apache.org/downloads.html Create a new kernel to use in a Jupyter notebook
- 28. @MargrietGr Jupyter Notebooks! • Server-client application to edit and run notebook documents via a web browser • Cells with: •Code •Figures and tables •Rich text elements • Different kernels: Python, R, Scala, Spark In the Cloud:
- 30. @MargrietGr
- 31. @MargrietGr
- 32. @MargrietGr
- 34. @MargrietGr Define Question What will the weather be next weekend? https://unsplash.com/search/autumn?photo=LSF8WGtQmn8 https://unsplash.com/search/rain?photo=19tQv51x4-A
- 36. @MargrietGr Explore Data Python packages • requests and json •API credentials and latitude/longitude of San Francisco •json data returned • pandas, numpy and datetime •convert json to pandas DataFrame (table with multiple indices) •add time as index
- 37. @MargrietGr Weather forecast for San Francisco https://developer.ibm.com/ clouddataservices/2016/10/06/ your-own-weather-forecast-in-a- python-notebook/ Visualize Data Python packages • pandas - rolling mean • matplotlib • Basemap
- 38. @MargrietGr Weather map - example for UK https:// developer.ibm.com/ clouddataservices/ 2016/10/06/your-own- weather-forecast-in-a- python-notebook/ Python packages • matplotlib • Basemap • itertools • urllib
- 39. @MargrietGr Run as a daily cron job cloudant
- 40. @MargrietGr
- 41. @MargrietGr
- 43. @MargrietGr Weather, Twitter and Sentiment • Where to find the data? • Where to store the data? • Where to analyse the data? • Quick tools to explore
- 45. @MargrietGr Add sentiment - example
- 46. @MargrietGr • watson tone analyser Emotion Language style Social propensities Analyze how you are coming across to others
- 47. @MargrietGr Simpler Workflow Weather Company Data crontab -e 0 23 * * * /path/to/file/do_something.sh python do_something.py Tweets Weather Sentiment Watson Tone Analyser Insights for Twitter Cloudant NoSQL
- 49. @MargrietGr PixieDust: an Open Source Library that simplifies and improves Jupyter Python Notebooks • PackageManager • Visualizations • Cloud Integration • Scala Bridge • Extensibility • Embedded Apps https://developer.ibm.com/clouddataservices/2016/10/11/pixiedust-magic-for- python-notebook/ @DTAIEB55
- 50. @MargrietGr Install Spark packages or plain jars in your Notebook Python kernel without the need to modify configuration file Uses the GraphFrame Python APIs Install GraphFrames Spark Package
- 51. @MargrietGr One simple API: display() Call the Options dialog Panning/Zooming options Performance statistics
- 52. @MargrietGr Easily export your data to csv, json, html, etc. locally on your laptop or into a cloud-based service like Cloudant or Object Storage
- 53. @MargrietGr Scala Bridge Define a Python variable Use the Python var in Scala Define a Scala variable Use the Scala var in Python
- 54. @MargrietGr Easily extend PixieDust to create your own visualizations using HTML/CSS/JavaScript Customized Visualization for GraphFrame Graphs
- 55. @MargrietGr Encapsulate your analytics into compelling User Interfaces better suited for Line of Business Users
- 56. @MargrietGr
- 58. @MargrietGr • Mission: To explore, understand and explain the origin and nature of life in the universe • Origins: Started in 1959 by two physicists at Cornell • NASA became interested in 1970, started working with SETI in 1988, funding cut in 1993 SETI@IBMCloud http://www.seti.org/node/861
- 59. @MargrietGr • The Allen Telescope Array •198 million radio events detected in the last decade •400,000 candidate signals identified •5TB data generated in 10 hours • No modern analysis or machine learning has been performed on this data • 5 TB of special observations on IBM Object Store SETI@IBMCloud - the Data https://github.com/ibm-cds-labs/ibmseti/
- 60. @MargrietGr Public Spark@SETI 4 TB of SETI Data stored in Object Storage Web API provides Bluemix users access to download SETI data Object Storage Web API Spark Object Storage Public Spark@SETI Bluemix Account My Bluemix Account Spark using Jupyter Notebook and IBM SETI Python Library Goal: Amateur scientists/data scientists download and analyze SETI data
- 61. @MargrietGr IBM Watson Data Platform • Data Science Experience • Watson Data Platform • Machine Learning • Sign up for beta: http://datascience.ibm.com/features#machinelearning
- 62. @MargrietGr Data Science in the Cloud • Flexible and quick to iterate, play and explore data • APIs •Streaming data •Cloud databases •Watson • Scaling up - add storage or Spark kernels • Easy collaboration and presentation •Store Data •Share your analyses in notebooks • Some useful packages: pandas, pyspark, requests, matplotlib, cloudant • Notebooks can be extended! PixieDust
- 63. @MargrietGr https://developer.ibm.com/clouddataservices/author/ mgroenen/ Thanks! Slides will be here: http://www.slideshare.net/MargrietGroenendijk