Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

IoT - Data Management Trends, Best Practices, & Use Cases

1 187 vues

Publié le

With billions of new devices, IoT is transforming how businesses capitalize on data. Data driven organizations are using IoT as as a means to improve their customer experience, drive operational efficiencies, and enable new business models. However, without the right data management strategy and tools, investments in IoT can yield limited results.

Join Cloudera and 451 Research for a joint webinar to learn more about some of the data management best practices and how organizations are using advanced analytics and machine learning to enable IoT use cases.

Publié dans : Technologie

IoT - Data Management Trends, Best Practices, & Use Cases

  1. 1. 1© Cloudera, Inc. All rights reserved. - Trends, Best Practices and Key Use Cases IoT Data Management WEBINAR
  2. 2. 2© Cloudera, Inc. All rights reserved. Your Speakers for Today… Vijay Raja Solutions Marketing Lead, IoT Christian Renaud Research Director, Internet of Things
  3. 3. Number of Current and Planned Enterprise IoT Initiatives IoT Respondents 4 Q. How many IoT initiatives does your organization have in the following phases of implementation? (Mean) n=346 Source: 451 Research, Voice of the Enterprise: Internet of Things, Vendor Evaluations 2016 Current State of IoT Adoption
  4. 4. 5
  5. 5. Key Use Cases Gaining Traction Today 611% 12% 13% 15% 22% 39% 50% 74% Smart Grid Smart City Health/Patient Monitoring Retail/Point-of-Sale Environmental Monitoring (Weather) Mobile Device Tracking Surveillance/Security Management/Automation (Fleet, Factory, Supply Chain) Source: 451 Research, Voice of the Enterprise: Internet Of Things, Budgets and Outlook 2016: “Which of the following best describes the IoT/projects enabled by these connected endpoints?”. Base: IoT-familiar respondents. Multiple select.
  6. 6. 7 By Morio - Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=9951305 A typical Formula One car already carries between 150 and 300 sensors Copyright (C) 2016 451 Research LLC
  7. 7. 8 By Morio - Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=9951305 Today, those couple of hundred sensors already capture data in milliseconds. Race cars generate 100-200Kb of data per second Copyright (C) 2016 451 Research LLC
  8. 8. 9 By Morio - Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=9951305 Each individual reading might translate into a relatively small amount of data but there are hundreds or thousands of them being generated each second Copyright (C) 2016 451 Research LLC
  9. 9. What is the Data of Things? 10 • Metrics and measures (Metadata and State). This type of data consists of the data that comes from the ‘things’ themselves – measures from sensors such as temperature, humidity, acceleration, vibration, speed, video feeds, biometric data, and so on. • Transactions (Commands). They could include an interaction between two machines, or between a system and a human being. They could include an adjustment to the parameters of a machine or system, such as an alteration to a generator or air conditioning unit. • Diagnostics (Telemetry). Provides an insight into the overall health of a machine, system or process. Diagnostic data might not only show the overall health of a system, but also serve as an alert that a system is no longer functioning within normal parameters and might need further analysis to determine the root cause.
  10. 10. 11 IoT Data – Unique Attributes Frequency of interaction Volume of data (per interaction) Traditional Enterprise applications IoT • Traditionally, most transactional systems were designed to be able to cope with one or two transactions every few minutes – at the most • A sensor or smart device could potentially generate data that needs to be handled by backend systems in some way every millisecond. • Each individual reading from a sensor might translate into a relatively small amount of data
  11. 11. 12 IoT Data – Unique Attributes Frequency of interaction Volume of data (in total) Traditional Enterprise applications IoT • Each individual reading from a sensor might translate into a relatively small amount of data, but there are hundreds or thousands being generated each second.
  12. 12. IoT Data Processing Requirements In order to gain insight and value from data generated by the IoT, enterprises need to: 13 Capture and process data coming from sensors and other devices Ensure interoperability of data coming from multiple sensors with multiple data formats and multiple protocols Analyze data in real-time to compare it with historical trends Ensure that appropriate responses are built in to operational application workflows and business processes CAPTURE INTEROPERATE ANALYZE ACT
  13. 13. INTERNET OF THINGS: ORGANIZATIONAL DYNAMICS 2016 Source: 451 Research, Voice of the Enterprise: Internet of Things, Budgets and Outlook 2016 Q41. Which of the following technologies or processes are high priorities for your organization to deploy in 2017 for your Internet-of- Things (IoT) initiatives? 14 47.0% 33.7% 31.2% 30.1% 26.5% 22.6% 19.4% 18.3% 14.0% 3.9% 14.3% IoT Security Big Data Analytics for IoT IoT Infrastructure Equipment IoT Applications IoT Network Edge IT Staff To Support IoT IoT Storage Aligning Corporate Policies, Procedures and Compliance To Support IoT Aligning IoT Across Multiple IT Groups Other None Percent of Sample n = 279 High-Priority Technologies and Processes for IoT Initiatives IoT-Familiar Respondents Security & Analytics – High Priority Areas for IoT Adoption
  14. 14. INTERNET OF THINGS: ORGANIZATIONAL DYNAMICS 2016 Source: 451 Research, Voice of the Enterprise: Internet of Things, Budgets and Outlook 2016 Q29. Which skills or capabilities will these new IoT staff need? 15 69.9% 54.8% 54.8% 45.2% 41.1% 38.4% 35.6% 28.8% 24.7% 20.5% 19.2% 1.4% Data Analytics Security Cloud Computing Network Edge/Perimeter Software Development Virtualization Standards and Protocols Storage Management General Management Compliance/Licensing Distributed Computing Other Percent of Sample n = 73 Required Skills for New IoT Staff Respondents Adding Dedicated IoT Staff Big Data Analytics will be a Critical Success Factor for IoT
  15. 15. 4 High-level IoT Data Architecture
  16. 16. Dramatic workload migration over the next two years: from 41% currently to 60% expected in two years On-premises to off-premises shift: from 35% to 52% Significant expansion of public clouds (IaaS and SaaS) as workload execution venues Source: 451 Research, Voice of the Enterprise: Cloud Transformation, Workloads & Key Projects 2016 51.6% 33.5% 7.4% 7.0% 13.8% 14.2% 7.8% 11.1% 5.5% 11.7% 13.8% 22.5% 2016 2018 Software-as-a-Service (SaaS) Infrastructure-as-a-Service (IaaS) Hosted Private Cloud On-Premises Private Cloud Off-Premises Non-Cloud On-Premises Non-Cloud 2016 2018 IT Workload Migration 17
  17. 17. INTERNET OF THINGS: ORGANIZATIONAL DYNAMICS 2016 Source: 451 Research, Voice of the Enterprise: Internet of Things, Budgets and Outlook 2016 Q37. Which deployment locations do you plan to use to store and analyze IoT data in 2017? 18 58.0% 37.2% 34.4% 28.1% 22.9% 17.0% Company-Owned/Leased Datacenter Facilities IT Infrastructure Located Where The IoT Data Is Generated Public Cloud Infrastructure (IaaS, PaaS) Software-as-a-Service (SaaS) Managed Services/Hosted Services Third-Party Colocation Facilities Percent of Sample n = 288 Deployment Locations Planned for 2017 IoT-Familiar Respondents On Prem/ Datacenter still the epicenter for Data Analytics
  18. 18. Infrastructure on-premise fastest growing category for IoT workload analytics SaaS growing rapidly y/y Public cloud and hosted services experiencing strong y/y uptake by nascent IoT verticals Source: 451 Research, Voice of the Enterprise: Internet of Things, Budgets and Outlooks, 2016 (Multi-answer) 57.7% 58.0% 18.9% 17.0% 28.8% 37.2% 18.0% 22.9% 32.4% 34.4% 20.7% 28.1% 2016 2017 Software-as-a-Service (SaaS) Infrastructure-as-a-Service (IaaS) Hosted Private Cloud Infrastructure on-premises Colocation On-Premises Non-Cloud 2016 2017 IoT Workload Migration 19
  19. 19. 20 IoT Analytics Continuum – Edge, Near Edge, Cloud
  20. 20. 21 Manufacturing Optimize Operations Reduce Risk New/ Enhanced Existing Products/ Services Customer Targeting/ Increase Sales 80% 69% 45% 22% Source: 451 Research, Voice of the Enterprise IoT Budgets and Outlook 2016
  21. 21. 22 Transportation 93% 47% 87% 27% Optimize Operations Reduce Risk New/ Enhanced Existing Products/ Services Customer Targeting/ Increase Sales Source: 451 Research, Voice of the Enterprise IoT Budgets and Outlook 2016
  22. 22. 23 Utilities Optimize Operations Reduce Risk New/ Enhanced Existing Products/ Services Customer Targeting/ Increase Sales 86% 81% 48% 23% Source: 451 Research, Voice of the Enterprise IoT Budgets and Outlook 2016
  23. 23. 24© Cloudera, Inc. All rights reserved. IoT Data Characteristics - The Foundation of Hadoop’s Potential IoT data comes from a variety of different sources Massive volumes of intermittent data streams Generated from a variety of data sources Predominantly time-series Can come in streams (real-time) or batches Diverse data structures and schemas Some of it may be perishable Combining sensor data with contextual data is the key to value creation from IoT
  24. 24. 25© Cloudera, Inc. All rights reserved. The IoT Ecosystem & Architecture IoT Gateway Gateway • Edge-Processing • Edge-Analytics IoT Data Storage, Processing & Analytics Centralized IoT Analytics • Time Series Data, Trends • Machine Learning • Context Enrichment • Deeper business insights Distributed Data Processing & Analytics • Cloud & On- Premise Connected Things • Analytics at the edge • For immediate response Data Center Cloud IoT Analytics Enterprise Data Sources
  25. 25. 26© Cloudera, Inc. All rights reserved. A Platform That Just Won’t Stop Growing… NEW PROJECTS EXISTING PROJECTS *CDH SUPPORTED Core Hadoop (HDFS, MapReduce) Solr Pig Core Hadoop HBase ZooKeeper Solr Pig Core Hadoop Hive Mahout HBase ZooKeeper Solr Pig Core Hadoop Sqoop Avro Hive Mahout HBase ZooKeeper Solr Pig Core Hadoop Flume Bigtop Oozie HCatalog Hue Sqoop Avro Hive Mahout HBase ZooKeeper Solr Pig YARN Core Hadoop Spark Tez Impala Kafka Drill Flume Bigtop Oozie HCatalog Hue Sqoop Avro Hive Mahout HBase ZooKeeper Solr Pig YARN Core Hadoop Parquet Sentry Spark Tez Impala Kafka Drill Flume Bigtop Oozie HCatalog Hue Sqoop Avro Hive Mahout HBase ZooKeeper Solr Pig YARN Core Hadoop Knox Flink Parquet Sentry Spark Tez Impala Kafka Drill Flume Bigtop Oozie HCatalog Hue Sqoop Avro Hive Mahout HBase ZooKeeper Solr Pig YARN Core Hadoop Kudu* RecordService* Ibis* Falcon Knox Flink Parquet* Sentry* Spark* Tez Impala* Kafka* Drill Flume* Bigtop* Oozie* Hcatalog* Hue* Sqoop* Avro* Hive* Mahout* Hbase* ZooKeeper* Solr* Pig* YARN* Core Hadoop* 2006 2008 2009 2010 2011 2012 20132007 2014 Present
  26. 26. 27© Cloudera, Inc. All rights reserved. Cloudera Enterprise – The Data Mgmt. Platform for IoT Connected Devices/ IoT Data Sources Enterprise Data Sources External Data Sources BI Solutions Real-Time Apps Search Data Science Workbench SQL Machine Learning Data Center Hybrid Cloud Sensor/ IoT Data • Data Storage • Data Processing • Machine Learning • Real-time Analytics OPERATIONS Cloudera Manager Cloudera Director DATA MANAGEMENT Cloudera Navigator Encrypt and KeyTrustee Optimizer BATCH Sqoop REAL-TIME Kafka, Flume PROCESS, ANALYZE, SERVE UNIFIED SERVICES RESOURCE MANAGEMENT YARN SECURITY Sentry, RecordService FILESYSTEM HDFS RELATIONAL Kudu NoSQL HBase STORE INTEGRATE BATCH Spark, Hive, Pig MapReduce STREAM Spark SQL Impala SEARCH Solr SDK Partners Other Enterprise Data Sources
  27. 27. 28© Cloudera, Inc. All rights reserved. The Cloudera Platform for IoT – Data Mgmt. Value Chain Data Sources Data Ingest Data Storage & Processing Serving, Analytics & Machine Learning ENTERPRISE DATA HUB Apache Kafka Stream or batch ingestion of IoT data Apache Sqoop Ingestion of data from relational sources Apache Hadoop Storage (HDFS) & deep batch processing Apache Kudu Storage & serving for fast changing data Apache HBase NoSQL data store for real time applications Apache Impala MPP SQL for fast analytics Cloudera Search Real time searchConnected Things/ Data Sources Structured Data Sources Security, Scalability & Easy Management Deployment Flexibility: Datacenter Cloud Apache Spark Stream & iterative processing, ML
  28. 28. 29© Cloudera, Inc. All rights reserved. HDFS Fast Scans, Analytics and Processing of Stored Data Fast On-Line Updates & Data Serving Arbitrary Storage (Active Archive) Fast Analytics (on fast-changing or frequently-updated data) Kudu – Fast Analytics on Fast Data Real Time Use cases that fall between HDFS and HBase were difficult to manage Unchanging Fast Changing Frequent Updates HBase Append-Only Real-Time Complex Hybrid Architectures Analytic Gap Pace of Analysis PaceofData
  29. 29. 30© Cloudera, Inc. All rights reserved. Cloudera for IoT – Key Enabling Capabilities Ideal for real-time analytics on IoT and time series data. Simplifies Lambda architectures for running real-time analytics on streaming data Preserve business flexibility and data portability and minimize cloud lock-in by running in any one of the three major public cloud providers or in private cloud Kudu: Real-Time Analytics Multi-Cloud Portability Data Science Workbench Collaborative hub for enterprise data science and an integrated development environment for running Python, R, & Scala with support for Spark
  30. 30. 31© Cloudera, Inc. All rights reserved. IoT - Key Customer Use Cases
  31. 31. 32© Cloudera, Inc. All rights reserved. Powering a Variety of IoT Use Cases… Connected Vehicles Usage Based Insurance Industrial IoT Predictive Maintenance Smart Cities & Ports Oil & Gas Aerospace & Aviation Smart Healthcare
  32. 32. 33© Cloudera, Inc. All rights reserved. Using Predictive Maintenance to Improve Performance and Reduce Fleet Downtime • Real-time visibility of 300,000+ trucks in order to improve uptime and vehicle performance • OnCommand Connection is collecting telematics and geolocation data across the fleet • Reduced maintenance costs to $.03 per mile from $.12-$.15 per mile • Centralizing data from 13 systems with varying frequency and semantic definitions TRANSPORTATION » PREDICTIVE MAINTENANCE » IMPROVED SERVICE » DATA DRIVEN PRODUCTS IOT & Connected Products CASE STUDY
  33. 33. 34© Cloudera, Inc. All rights reserved. Predictive Maintenance on industrial- grade turbines for hydro power stations Challenge: • Gather, store and analyze noise levels from turbines for anomaly detection Solution: • Cloudera platform used to gather and analyze acoustic data/audio files coming from the turbines in real-time • Diagnostic solution to monitor the health of turbines and predict failures in advance • Prevent downtimes and failures PREDICTIVE MAINTENANCE » INDUSTRIAL IoT » LOWERED DOWNTIME » LOWERED COSTS Predictive Maintenance - Turbines DATA-DRIVEN PROCESS CASE STUDY IOT & Connected Products
  34. 34. 35© Cloudera, Inc. All rights reserved. #1 Telematics provider with 130 billion miles of driving data collected from black boxes in connected cars Challenge: • Drive analytics on 12 million miles of driving data collected every hour Solution: • Telematics solution based on Cloudera to process data from black boxes • Analytics around driving behavior, risks, location, braking patterns, contextual elements and crash information • Provide Usage Based Insurance services TELEMATICS » CONNECTED VEHICLES » INSURANCE TELEMATICS » PREDICTIVE ANALYTICS Connected Car Telematics for Insurance CASE STUDY DATA-DRIVEN PROCESS IOT & Connected Products
  35. 35. 36© Cloudera, Inc. All rights reserved. Ensuring Zero Down Time & lowered energy costs on industrial-grade robots Challenge: • Gather, store and analyze sensor data from 10,000 robots in order to minimize downtime Solution: • Cloudera platform used to gather and analyze sensor data coming from robots in real-time • Diagnostic solution predicts potential failures and alerts the operators in advance ZERO DOWN TIME » INDUSTRIAL IoT » LOWERED DOWNTIME » LOWERED COSTS Zero Down Time – Industrial Robotics DATA-DRIVEN PROCESS CASE STUDY DATA-DRIVEN PRODUCTS
  36. 36. 37© Cloudera, Inc. All rights reserved. Enabling the State of Kentucky optimize management of snow and ice events in real time Challenge: • Needed more efficient approach to inclement weather road management Solution: • Real-time weather response system that incorporates real-time data from Waze, HERE, ESRI’s GeoEvent processor, and Automatic Vehicle Locations (sensor data from salt trucks). • KYTC aggregates 15-20 million records every day and process more than a million records per second. Smart Cities 2016 Data Impact Award Winner State of Kentucky Department of Transportation CASE STUDY
  37. 37. 38© Cloudera, Inc. All rights reserved. Using sensors & IoT to improve efficiencies in cargo handling Challenge: • Bring together data streams from millions of cargo equipment to enable predictive maintenance Solution: • Sensor Data Analytics Framework based on Cloudera and TCS to collect, store and analyze data collected from port equipment & machinery • Improve utilization, reduce unplanned equipment downtime Smart Ports & Cargo Handling DATA-DRIVEN PROCESS CASE STUDY DATA-DRIVEN PRODUCTS TRAVEL & TRANSPORTATION » INTERNET OF THINGS » PREDICTIVE MAINTENANCE » ADVANCED ANALYTICS Leading Cargo Handling Providers in Europe
  38. 38. 39© Cloudera, Inc. All rights reserved. MINING & HEAVY MACHINERY » ASSET OPTIMIZATION » PREDICTIVE ANALYTICS » INDUSTRIAL IOT IoT enabled Asset Optimization CASE STUDY DATA-DRIVEN PROCESS DATA-DRIVEN PRODUCTS Optimize equipment performance and costs using real-time IoT analytics • Connected machinery includes some of the largest mobile mining equipment used in surface and underground mining • Data growth anticipated to reach 30 TB per month • Cloudera on Azure to easily analyze data from connected machines and third party sources • Doubled the utilization of a longwall system for one of their Clients
  39. 39. 40© Cloudera, Inc. All rights reserved. To Learn More… https://www.cloudera.com/solutions/iot.html Cloudera Booth # 225
  40. 40. 41© Cloudera, Inc. All rights reserved. Thank you Questions?
  41. 41. 42© Cloudera, Inc. All rights reserved. A Data Management Platform for IoT Handle real-time data ingest from diverse sources Fundamentally Secure Data Streams Machine Learning Capabilities Diverse Analytical Options Enterprise Data Sources Scale easily & Cost effectively Batch or Real- time Data Streams A comprehensive data management platform to drive business insights from IoT data Data Sources Data Storage & Processing Serving, Analytics & Machine Learning Data Ingest Connected Machines/ Data Sources Cloudera Enterprise Data Hub
  42. 42. 43© Cloudera, Inc. All rights reserved. Cloudera Enterprise – Data Management & Analytics for IoT BI Solutions Real-Time AppsSearch SQL Analytics Machine Learning Deployment Flexibility Spark Streaming Leadership in Spark Integrated with EDH Flexible Storage Store any and all Data. Kudu – Real-Time Analytics on Streaming Data Real-Time Data Processing Data Security Four pillars of security: Perimeter, Access, Visibility, and Data + Record Service Streaming Ingest Kafka & Flume - Real-Time Data Ingest for streaming, high volume data Sensor/ IoT Data Internal Systems External Sources Data Science Cloudera Data Science Workbench - Collaborative hub for enterprise data science Manage Multiple Clusters – On Premise or Cloud environment - On Premise or Hybrid Cloud OPERATIONS Cloudera Manager Cloudera Director DATA MANAGEMENT Cloudera Navigator Encrypt and KeyTrustee Optimizer BATCH Sqoop REAL-TIME Kafka, Flume PROCESS, ANALYZE, SERVE UNIFIED SERVICES RESOURCE MANAGEMENT YARN SECURITY Sentry, RecordService FILESYSTEM HDFS RELATIONAL Kudu NoSQL HBase STORE INTEGRATE BATCH Spark, Hive, Pig MapReduce STREAM Spark SQL Impala SEARCH Solr SDK Partners Data Science Workbench
  43. 43. 44© Cloudera, Inc. All rights reserved. Cloudera in the Cloud - Hybrid Cloud Deployments Flexible Deployments • Multi-cloud: AWS, Azure, GCP • Fast cluster deployments • Scaling of CDH clusters • Spot instance support Easy Administration • Dynamic cluster lifecycle management • Single pane of glass: multi-cluster view Enterprise-grade • Integration across Cloudera Enterprise • Management of CDH deployments at scale Cloudera Director

×