Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
1© Copyright 2016 Pivotal. All rights reserved.
Data Science-Powered Apps for
the Internet of Things
Chris Rawles1 and Jar...
2© Copyright 2016 Pivotal. All rights reserved.
‘By the year 2025, $4 to $11 trillion of
economic value could be created t...
3© Copyright 2016 Pivotal. All rights reserved.
IoT
Platform
Applications
Data Science
4© Copyright 2016 Pivotal. All rights reserved.
New business models
Improve efficiencies
Personalized experiences
________...
5© Copyright 2016 Pivotal. All rights reserved.
Today’s Speaker
Chris Rawles
Senior Data Scientist
6© Copyright 2016 Pivotal. All rights reserved.
Today’s talk
1.  A real-time data science app
A.  The app: a live demonstr...
7© Copyright 2016 Pivotal. All rights reserved.
Today’s talk
1.  A real-time data science app
A.  The app: a live demonstr...
8© Copyright 2016 Pivotal. All rights reserved.
App
9© Copyright 2016 Pivotal. All rights reserved.
Today’s talk
1.  A real-time data science app
A.  The app: a live demonstr...
10© Copyright 2016 Pivotal. All rights reserved.
Training

app
Model
Scoring as
a service
API Call
Model
Training as
a ser...
11© Copyright 2016 Pivotal. All rights reserved.
here is my source code
run it on the cloud for me
-  Onsi Fakhouri
@onsij...
12© Copyright 2016 Pivotal. All rights reserved.
cf push
!  CF determines app type (Java, Python, Ruby, …)
!  Installs nec...
13© Copyright 2016 Pivotal. All rights reserved.
Data ingestion: Accelerometric data
!  Accelerometric data streamed from
...
14© Copyright 2016 Pivotal. All rights reserved.
!  For real-time applications, low-latency data ingestion into
the data s...
15© Copyright 2016 Pivotal. All rights reserved.
Data storage
!  We are using a redis store for:
–  Storing training data
...
16© Copyright 2016 Pivotal. All rights reserved.
Modeling
Scalable machine learning applications in Pivotal
Cloud Foundry
...
17© Copyright 2016 Pivotal. All rights reserved.
Modeling – Training app
!  Goal: build a data-driven model that learns ac...
18© Copyright 2016 Pivotal. All rights reserved.
Model building
!  20 seconds per
training activity
!  Two second moving
w...
19© Copyright 2016 Pivotal. All rights reserved.
Model training approaches
1.  Near-real-time model training
–  Use small ...
20© Copyright 2016 Pivotal. All rights reserved.
Feature Engineering
•  Time-domain
transformations
•  Fast Fourier Transf...
21© Copyright 2016 Pivotal. All rights reserved.
Scaling the model scoring application
$	cf	scale	–i	10	 Scoring App
Scori...
22© Copyright 2016 Pivotal. All rights reserved.
1.  Application auto-scaling
–  As the data grows, the model scales
2.  B...
23© Copyright 2016 Pivotal. All rights reserved.
Today’s talk
1.  A real-time data science app
A.  The app: a live demonst...
24© Copyright 2016 Pivotal. All rights reserved.
App
25© Copyright 2016 Pivotal. All rights reserved.
Today’s talk
1.  A real-time data science app
A.  The app: a live demonst...
26© Copyright 2016 Pivotal. All rights reserved.
Gene Sequencing
Smart Grids
COST TO SEQUENCE
ONE GENOME
HAS FALLEN FROM
$...
27© Copyright 2016 Pivotal. All rights reserved.
How can we use data
to help prevent
accidents like the Macondo
Disaster ?
28© Copyright 2016 Pivotal. All rights reserved. 28© Copyright 2016 Pivotal. All rights reserved.
…by creating a Smart App...
29© Copyright 2016 Pivotal. All rights reserved.
Training

app
Model
Scoring as
a service
API Call
Model
Training as
a ser...
30© Copyright 2016 Pivotal. All rights reserved.
Training

app
Model
Scoring as
a service
API Call
Model
Training as
a ser...
31© Copyright 2016 Pivotal. All rights reserved.
Training

app
Model
Scoring as
a service
API Call
Model
Training as
a ser...
32© Copyright 2016 Pivotal. All rights reserved.
Blogs on Building Data Science Apps
Blogs
!  Scoring-as-a-Service To Oper...
33© Copyright 2016 Pivotal. All rights reserved.
Prochain SlideShare
Chargement dans…5
×

Learn How to Operationalize IoT Apps on Pivotal Cloud Foundry

620 vues

Publié le

The Internet of Things (IoT) holds promise for both consumers and enterprises alike. To succeed, any IoT project must concern itself with how to ingest machine and sensor data, how to build actionable models, and how to react to the output of models in real-time.

Join Pivotal Data Scientist Chris Rawles, as he illustrates how to build and operationalize an IoT application running on Pivotal Cloud Foundry that scores and reacts to streaming data in real-time. In this webinar, you will learn how to:

- Collect streaming IoT data
- Build and train machine learning models in real-time
- Score streaming data in real-time in an application

Chris Rawles, Data Scientist, Pivotal

Publié dans : Technologie
  • DOWNLOAD THAT BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Full EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Full doc Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download doc Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book that can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer that is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story That Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money That the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths that Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici

Learn How to Operationalize IoT Apps on Pivotal Cloud Foundry

  1. 1. 1© Copyright 2016 Pivotal. All rights reserved. Data Science-Powered Apps for the Internet of Things Chris Rawles1 and Jarrod Vawdrey2 1. Sr. Data Scientist in New York, New York 2. Sr. Data Scientist in Atlanta, Georgia
  2. 2. 2© Copyright 2016 Pivotal. All rights reserved. ‘By the year 2025, $4 to $11 trillion of economic value could be created through the Internet of Things.’ Michael Chui Partner, McKinsey & Company
  3. 3. 3© Copyright 2016 Pivotal. All rights reserved. IoT Platform Applications Data Science
  4. 4. 4© Copyright 2016 Pivotal. All rights reserved. New business models Improve efficiencies Personalized experiences _____________________ Trillions $ Economic Value
  5. 5. 5© Copyright 2016 Pivotal. All rights reserved. Today’s Speaker Chris Rawles Senior Data Scientist
  6. 6. 6© Copyright 2016 Pivotal. All rights reserved. Today’s talk 1.  A real-time data science app A.  The app: a live demonstration B.  How can a data scientist build a data science application? C.  Revisiting the app 2.  Generalizing the framework: Solving new data science challenges A.  Internet of Things – Creating a smart app to prevent oil spill disasters B.  Financial data - How can retail banks influence their cardholders’ behavior?
  7. 7. 7© Copyright 2016 Pivotal. All rights reserved. Today’s talk 1.  A real-time data science app A.  The app: a live demonstration B.  How can a data scientist build a data science application? C.  Revisiting the app 2.  Generalizing the framework: Solving new data science challenges A.  Internet of Things – Creating a smart app to prevent oil spill disasters B.  Financial data - How can retail banks influence their cardholders’ behavior?
  8. 8. 8© Copyright 2016 Pivotal. All rights reserved. App
  9. 9. 9© Copyright 2016 Pivotal. All rights reserved. Today’s talk 1.  A real-time data science app A.  The app: a live demonstration B.  How can a data scientist build a data science application? C.  Revisiting the app 2.  Generalizing the framework: Solving new data science challenges A.  Internet of Things – creating a smart app B.  Financial data - How can retail banks influence their cardholders’ behavior?
  10. 10. 10© Copyright 2016 Pivotal. All rights reserved. Training
 app Model Scoring as a service API Call Model Training as a service Sensor
 app Scoring
 app Dashboard
 app Data science workflow: Movement classification 1.  Sensor + Dashboard 2.  Redis 3.  Training app 4.  Scoring app
  11. 11. 11© Copyright 2016 Pivotal. All rights reserved. here is my source code run it on the cloud for me -  Onsi Fakhouri @onsijoe i do not care how
  12. 12. 12© Copyright 2016 Pivotal. All rights reserved. cf push !  CF determines app type (Java, Python, Ruby, …) !  Installs necessary environment !  Provisions and binds data services !  Creates domain, routing, and load balancing !  Continual app health checks and restarts
  13. 13. 13© Copyright 2016 Pivotal. All rights reserved. Data ingestion: Accelerometric data !  Accelerometric data streamed from mobile phone at 15 Hz (15x / second) !  Other sensor data: gyroscopic data, magnetometer data, lon/lat, etc. Accelerometer axes
  14. 14. 14© Copyright 2016 Pivotal. All rights reserved. !  For real-time applications, low-latency data ingestion into the data store is essential !  WebSocket protocol - socket.io –  Mobile phone " Webserver –  Webserver " Dashboard !  socket.io " redis Data ingestion Training
 app Sensor
 app
  15. 15. 15© Copyright 2016 Pivotal. All rights reserved. Data storage !  We are using a redis store for: –  Storing training data –  Model persistence –  Storing a micro-batch of scoring data !  Other storage systems include Pivotal Cloud Cache, GemFire, HAWQ/Hadoop, Greenplum Database, PostgreSQL, …
  16. 16. 16© Copyright 2016 Pivotal. All rights reserved. Modeling Scalable machine learning applications in Pivotal Cloud Foundry 1.  Training app 2.  Scoring app
  17. 17. 17© Copyright 2016 Pivotal. All rights reserved. Modeling – Training app !  Goal: build a data-driven model that learns accelerometric motions associated with each activity Feature Engineering •  Time-domain transformations •  Fast Fourier Transform analysis Machine Learning Classification Model •  Random Forest Model using 2 second time windows (30 samples) … Training data Trained model
  18. 18. 18© Copyright 2016 Pivotal. All rights reserved. Model building !  20 seconds per training activity !  Two second moving window on training data !  Features: time- domain summary statistics and Fourier transform coefficients
  19. 19. 19© Copyright 2016 Pivotal. All rights reserved. Model training approaches 1.  Near-real-time model training –  Use small batches to train model 2.  Real-time model training –  Online machine learning algorithm : continually update model using each new data point 3.  Offline model training –  Build a model offline using batches –  Useful for models requiring finer model tuning and calibration
  20. 20. 20© Copyright 2016 Pivotal. All rights reserved. Feature Engineering •  Time-domain transformations •  Fast Fourier Transform analysis Machine Learning Classification Model •  Random Forest Model using 2 second time windows (30 samples) Trained model Streaming input window Model Prediction API Call Model prediction PCF App: Scoring app •  Real-time model scoring •  The dashboard initiates a request via an API call and receives a model prediction { "channel": "1234", "label": ”walking", ”score": 0.746 }
  21. 21. 21© Copyright 2016 Pivotal. All rights reserved. Scaling the model scoring application $ cf scale –i 10 Scoring App Scoring App Scoring App Scoring App Horizontal scaling
  22. 22. 22© Copyright 2016 Pivotal. All rights reserved. 1.  Application auto-scaling –  As the data grows, the model scales 2.  Building a model factory–evaluate many models in production 3.  Application autonomy –  The model application is independent of other applications = faster development iterations –  Faster development = rapid feedback loop 4.  Multiple applications can access model scoring app Operationalizing scalable data science applications Model scoring as a service Why?
  23. 23. 23© Copyright 2016 Pivotal. All rights reserved. Today’s talk 1.  A real-time data science app A.  The app: a live demonstration B.  How can a data scientist build a data science application? C.  Revisiting the app 2.  Generalizing the framework: Solving new data science challenges A.  Internet of Things – creating a smart app B.  Financial data - How can retail banks influence their cardholders’ behavior?
  24. 24. 24© Copyright 2016 Pivotal. All rights reserved. App
  25. 25. 25© Copyright 2016 Pivotal. All rights reserved. Today’s talk 1.  A real-time data science app A.  The app: a live demonstration B.  How can a data scientist build a data science application? C.  Revisiting the app 2.  Generalizing the framework: Solving new data science challenges A.  Internet of Things – Creating a smart app to prevent oil spill disasters B.  Financial data - How can retail banks influence their cardholders’ behavior?
  26. 26. 26© Copyright 2016 Pivotal. All rights reserved. Gene Sequencing Smart Grids COST TO SEQUENCE ONE GENOME HAS FALLEN FROM $100M IN 2001 TO $10K IN 2011 TO $1K IN 2014 READING SMART METERS EVERY 15 MINUTES IS 3000X MORE DATA INTENSIVE Stock Market Social Media FACEBOOK UPLOADS 250 MILLION PHOTOS EACH DAY In all industries billions of data points represent opportunities for the Internet of Things Oil Exploration Video Surveillance OIL RIGS GENERATE 25000 DATA POINTS PER SECOND Medical Imaging Mobile Sensors
  27. 27. 27© Copyright 2016 Pivotal. All rights reserved. How can we use data to help prevent accidents like the Macondo Disaster ?
  28. 28. 28© Copyright 2016 Pivotal. All rights reserved. 28© Copyright 2016 Pivotal. All rights reserved. …by creating a Smart Application
  29. 29. 29© Copyright 2016 Pivotal. All rights reserved. Training
 app Model Scoring as a service API Call Model Training as a service Sensor
 app Scoring
 app Dashboard
 app Data science workflow: Movement classification
  30. 30. 30© Copyright 2016 Pivotal. All rights reserved. Training
 app Model Scoring as a service API Call Model Training as a service Sensor
 app Scoring
 app Dashboard
 app Data science workflow: Creating a smart app to prevent oil spill disasters •  Alert operator •  Send signal to control system to change operating parameters •  Replace old machinery •  Shut down plant
  31. 31. 31© Copyright 2016 Pivotal. All rights reserved. Training
 app Model Scoring as a service API Call Model Training as a service Sensor
 app Scoring
 app Dashboard
 app Data science workflow: How can retail banks influence their cardholders’ behavior? •  Provide customized services and promotions •  Next best offer •  Characterize and improve customer satisfaction
  32. 32. 32© Copyright 2016 Pivotal. All rights reserved. Blogs on Building Data Science Apps Blogs !  Scoring-as-a-Service To Operationalize Algorithms For Real Time !  How to Scale a Machine Learning Model Using Pivotal Cloud Foundry !  Data Science How-To: Text Analytics as a Service crawles@pivotal.io
  33. 33. 33© Copyright 2016 Pivotal. All rights reserved.

×