SlideShare une entreprise Scribd logo
1  sur  47
Télécharger pour lire hors ligne
Orientierung im Nebel:
Unterschiede zwischen Stream
Processing in Edge und Cloud
• IIoT vs IoT
• (Stream) processing at the Edge
• How Apache Edgent can help
What I’ll talk about
• Julian Feinauer
• Phd in mathematics
• Founder of pragmatic minds GmbH and pragmatic industries GmbH
• Apache PLC4X & Apache Edgent (P)PMC member
• Twitter: @jfeinauer2
Who am I?
Industry 4.0, Digitalization and the
Industrial Internet of Things?!
• Everybody wants Industrie 4.0 or Digitalization
• „Can we do something with Big Data?“
• „We need a Data Lake“
• Can AI solve this for us?“
• This means that the Shopfloor is confrontend with
• ... connectivity
• ... modern IT infrastructure
• ... Cloud technology
Current Trend in the Industry
• #Neuland for the Industry
• Industry is always conservative regarding IT infrastructure
• Suppliers try to close the gap but are themselves faced with a
completly new situation and new technologies
• Very dynamic situation but Industry likes to plan
• In many industrial sites we have bad bandwidths
But...
• „Universal Protocol
Adapter“
• Connectivity for PLCs
• Bridging the GAP between
Shopfloor and IT
However...
The Industrial Internet of Things!
The Industrial Internet of Things
PLCs
Sensors
Actors
The Industrial Internet of Things
PLCs
Sensors
Actors
Hundreds of internal variables
• 10-100 Sensors
• Scaling
• Data types
• 10-100 Actors
• Different technologies
• Metadata is important
The Industrial Internet of Things
PLCs
Sensors
Actors
• No discrete Events
• Timeseries!
• Events have to be extracted based
on
• ... absolute values (thresholds)
• ... (bit-)triggers / value changes
• ... derivatives
• ...
• è Stream Processing at the Edge!
Stream Processing at the Edge
(Classic-)Definition:
“A data stream can be seen as a continuous and
potentially infinite stochastic process in which events
occur independently from another”
Streaming is easy!
The Industrial Internet of Things
PLCs
Sensors
Actors
• No discrete Events
• Timeseries!
• Events have to be extracted based
on
• ... absolute values (thresholds)
• ... (bit-)triggers / value changes
• ... derivatives
• ...
• è Stream Processing at the Edge!
A simple Example…
Example: Cycle Detection
• A (boolean) marker is true during cycle, false otherwise
• We want to detect cycles as events, and record their start / end
Example: Cycle Detection
Example: Cycle Detection
• A (boolean) marker is true during cycle, false otherwise
• We want to detect cycles as events, and record their start / end
• „Oh, we sometimes jitter, so a cycle is only a real cycle if its
longer than 10 seconds“
Example: Cycle Detection
Example: Cycle Detection
• A (boolean) marker is true during cycle, false otherwise
• We want to detect cycles as events, and record their start / end
• „Oh, we sometimes jitter, so a cycle is only a real cycle if its
longer than 10 seconds“
• „Ah, we forgot, it would be good to also store the pressure at the
end of the cycle“
Example: Cycle Detection
Example: Cycle Detection
• A (boolean) marker is true during cycle, false otherwise
• We want to detect cycles as events, and record their start / end
• „Oh, we sometimes jitter, so a cycle is only a real cycle if its longer than
10 seconds“
• „Ah, we forgot, it would be good to also store the pressure at the end
of the cycle“
• „In fact, we also need the current, rpm and temperature for the cycle“
Example: Cycle Detection
...
Example: Error Logging
• Exceptions are stored as bit mask
• About 2300 bits / exceptions on the machine
• Task
• Read all bits
• See which toggled
• Generate Event for Exception on / off
...
Example: Error Logging
Example: Complex Actions
• We need production parameters, at defined points
• First, wait until bit0 is set, then take the start time
• Then, wait until bit1 is set, then take the pressure
• Then, wait until bit2 is set, then take the rpm
• ...
Example: Complex Actions
...
Part II – Which Engine to use?
Streaming is easy!
Stream Processing is not Stream
Processing
Property Cloud Edge
Distribution Yes No
Scalable Yes No
Fast Yes Easy
Stateless Yes* No
Message Delivery Exactly-Once At least once sufficient
From Imperative to Declarative
CRUNCH it!
• CRUNCH - A framework for declarative industrial data analysis
• Not another streaming framework, ABOVE the streaming layer
• Created by pragmatic minds GmbH
• Open Source sind end of 2018
• Apache 2.0 License
• Visit: https://github.com/pragmaticminds/crunch
CRUNCH it!
• CRUNCH - A framework for declarative industrial data analysis
• Not another streaming framework, ABOVE the streaming layer
• Created by pragmatic minds GmbH
• Open Source sind end of 2018
• Apache 2.0 License
• Visit: https://github.com/pragmaticminds/crunch
Introducing Apache Edgent (incubating)
Best practices... λ-Architecture
λ-Architecture at the Edge?
λ-Architecture at the Edge?
• Incubating project at the Apache Software Foundation
• Established 2016
• Edge runtime with
• a streaming runtime
• real-time analytics capabilities
• and cloud communication
for constrained devices (i.e., with small footprint), at the edge.
Apache Edgent
Apache Edgent
Apache Edgent
Edge to Cloud...
But...
• All have their own specific API
• Partly open source, but usually very vendor driven
• Vendor-Lock in
• Entrance barrier is low but this should be a strategic decision
• Can become very expensive, long term
Edgent to the rescue!
• All Open Source (Apache 2.0 Licence)
• Cloud Connectors for different providers, no vendor lock-in
• Tight integration with other Apache Frameworks, like
• Apache PLC4X (incubating)
• Apache IoTDB (coming soon)
• Complete toolkit to build analytic applications at the edge!
What‘s (probably) Next?
• More Cloud Connectors!
• Tight integration of CRUNCH
• Edge Integration Patterns!
• Improve Routing / Rule Engine at the Edge („Camel at the Edge“)
• Revive the Community
Interested?
• Visit www.edgent.apache.org
• Sign up to our mailing-list: dev@edgent.apache.org
• Follow us on twitter: @ApacheEdgent
• Start using Edgent
• File bugs: https://issues.apache.org/jira/projects/EDGENT
• Source code: https://github.com/apache/incubator-edgent
• Examples: https://github.com/apache/incubator-edgent-samples
• Join our meetup in Frankfurt on Mai 6th in Frankfurt (am Main)
Questions?
Julian Feinauer
T +49 7022 24933 01
M +49 176 2488 3977
E j.feinauer@pragmaticminds.de
PRAGMATICMINDS GmbH
Zementwerk 1
72622 Nürtingen
www.pragmaticminds.de

Contenu connexe

Dernier

CNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of ServiceCNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of Service
giselly40
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
vu2urc
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI Solutions
Enterprise Knowledge
 

Dernier (20)

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUnderstanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
 
What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
 
CNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of ServiceCNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of Service
 
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
Evaluating the top large language models.pdf
Evaluating the top large language models.pdfEvaluating the top large language models.pdf
Evaluating the top large language models.pdf
 
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
 
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
 
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Script
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed texts
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI Solutions
 
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
 

En vedette

Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsSocial Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie Insights
Kurio // The Social Media Age(ncy)
 

En vedette (20)

Skeleton Culture Code
Skeleton Culture CodeSkeleton Culture Code
Skeleton Culture Code
 
PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024
 
Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)
 
How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024
 
Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsSocial Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie Insights
 
Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024
 
5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary
 
ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd
 
Getting into the tech field. what next
Getting into the tech field. what next Getting into the tech field. what next
Getting into the tech field. what next
 
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentGoogle's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search Intent
 
How to have difficult conversations
How to have difficult conversations How to have difficult conversations
How to have difficult conversations
 
Introduction to Data Science
Introduction to Data ScienceIntroduction to Data Science
Introduction to Data Science
 
Time Management & Productivity - Best Practices
Time Management & Productivity -  Best PracticesTime Management & Productivity -  Best Practices
Time Management & Productivity - Best Practices
 
The six step guide to practical project management
The six step guide to practical project managementThe six step guide to practical project management
The six step guide to practical project management
 
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
 
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
 
12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at Work12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at Work
 
ChatGPT webinar slides
ChatGPT webinar slidesChatGPT webinar slides
ChatGPT webinar slides
 
More than Just Lines on a Map: Best Practices for U.S Bike Routes
More than Just Lines on a Map: Best Practices for U.S Bike RoutesMore than Just Lines on a Map: Best Practices for U.S Bike Routes
More than Just Lines on a Map: Best Practices for U.S Bike Routes
 
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
 

Orientation in the fog: differences between stream processing in edge and cloud

  • 1. Orientierung im Nebel: Unterschiede zwischen Stream Processing in Edge und Cloud
  • 2. • IIoT vs IoT • (Stream) processing at the Edge • How Apache Edgent can help What I’ll talk about
  • 3. • Julian Feinauer • Phd in mathematics • Founder of pragmatic minds GmbH and pragmatic industries GmbH • Apache PLC4X & Apache Edgent (P)PMC member • Twitter: @jfeinauer2 Who am I?
  • 4. Industry 4.0, Digitalization and the Industrial Internet of Things?!
  • 5. • Everybody wants Industrie 4.0 or Digitalization • „Can we do something with Big Data?“ • „We need a Data Lake“ • Can AI solve this for us?“ • This means that the Shopfloor is confrontend with • ... connectivity • ... modern IT infrastructure • ... Cloud technology Current Trend in the Industry
  • 6. • #Neuland for the Industry • Industry is always conservative regarding IT infrastructure • Suppliers try to close the gap but are themselves faced with a completly new situation and new technologies • Very dynamic situation but Industry likes to plan • In many industrial sites we have bad bandwidths But...
  • 7. • „Universal Protocol Adapter“ • Connectivity for PLCs • Bridging the GAP between Shopfloor and IT However...
  • 9. The Industrial Internet of Things PLCs Sensors Actors
  • 10. The Industrial Internet of Things PLCs Sensors Actors Hundreds of internal variables • 10-100 Sensors • Scaling • Data types • 10-100 Actors • Different technologies • Metadata is important
  • 11. The Industrial Internet of Things PLCs Sensors Actors • No discrete Events • Timeseries! • Events have to be extracted based on • ... absolute values (thresholds) • ... (bit-)triggers / value changes • ... derivatives • ... • è Stream Processing at the Edge!
  • 13. (Classic-)Definition: “A data stream can be seen as a continuous and potentially infinite stochastic process in which events occur independently from another”
  • 15. The Industrial Internet of Things PLCs Sensors Actors • No discrete Events • Timeseries! • Events have to be extracted based on • ... absolute values (thresholds) • ... (bit-)triggers / value changes • ... derivatives • ... • è Stream Processing at the Edge!
  • 17. Example: Cycle Detection • A (boolean) marker is true during cycle, false otherwise • We want to detect cycles as events, and record their start / end
  • 19. Example: Cycle Detection • A (boolean) marker is true during cycle, false otherwise • We want to detect cycles as events, and record their start / end • „Oh, we sometimes jitter, so a cycle is only a real cycle if its longer than 10 seconds“
  • 21. Example: Cycle Detection • A (boolean) marker is true during cycle, false otherwise • We want to detect cycles as events, and record their start / end • „Oh, we sometimes jitter, so a cycle is only a real cycle if its longer than 10 seconds“ • „Ah, we forgot, it would be good to also store the pressure at the end of the cycle“
  • 23. Example: Cycle Detection • A (boolean) marker is true during cycle, false otherwise • We want to detect cycles as events, and record their start / end • „Oh, we sometimes jitter, so a cycle is only a real cycle if its longer than 10 seconds“ • „Ah, we forgot, it would be good to also store the pressure at the end of the cycle“ • „In fact, we also need the current, rpm and temperature for the cycle“
  • 25. Example: Error Logging • Exceptions are stored as bit mask • About 2300 bits / exceptions on the machine • Task • Read all bits • See which toggled • Generate Event for Exception on / off
  • 27. Example: Complex Actions • We need production parameters, at defined points • First, wait until bit0 is set, then take the start time • Then, wait until bit1 is set, then take the pressure • Then, wait until bit2 is set, then take the rpm • ...
  • 29. Part II – Which Engine to use?
  • 31. Stream Processing is not Stream Processing Property Cloud Edge Distribution Yes No Scalable Yes No Fast Yes Easy Stateless Yes* No Message Delivery Exactly-Once At least once sufficient
  • 32. From Imperative to Declarative
  • 33. CRUNCH it! • CRUNCH - A framework for declarative industrial data analysis • Not another streaming framework, ABOVE the streaming layer • Created by pragmatic minds GmbH • Open Source sind end of 2018 • Apache 2.0 License • Visit: https://github.com/pragmaticminds/crunch
  • 34. CRUNCH it! • CRUNCH - A framework for declarative industrial data analysis • Not another streaming framework, ABOVE the streaming layer • Created by pragmatic minds GmbH • Open Source sind end of 2018 • Apache 2.0 License • Visit: https://github.com/pragmaticminds/crunch
  • 39. • Incubating project at the Apache Software Foundation • Established 2016 • Edge runtime with • a streaming runtime • real-time analytics capabilities • and cloud communication for constrained devices (i.e., with small footprint), at the edge. Apache Edgent
  • 43. But... • All have their own specific API • Partly open source, but usually very vendor driven • Vendor-Lock in • Entrance barrier is low but this should be a strategic decision • Can become very expensive, long term
  • 44. Edgent to the rescue! • All Open Source (Apache 2.0 Licence) • Cloud Connectors for different providers, no vendor lock-in • Tight integration with other Apache Frameworks, like • Apache PLC4X (incubating) • Apache IoTDB (coming soon) • Complete toolkit to build analytic applications at the edge!
  • 45. What‘s (probably) Next? • More Cloud Connectors! • Tight integration of CRUNCH • Edge Integration Patterns! • Improve Routing / Rule Engine at the Edge („Camel at the Edge“) • Revive the Community
  • 46. Interested? • Visit www.edgent.apache.org • Sign up to our mailing-list: dev@edgent.apache.org • Follow us on twitter: @ApacheEdgent • Start using Edgent • File bugs: https://issues.apache.org/jira/projects/EDGENT • Source code: https://github.com/apache/incubator-edgent • Examples: https://github.com/apache/incubator-edgent-samples • Join our meetup in Frankfurt on Mai 6th in Frankfurt (am Main)
  • 47. Questions? Julian Feinauer T +49 7022 24933 01 M +49 176 2488 3977 E j.feinauer@pragmaticminds.de PRAGMATICMINDS GmbH Zementwerk 1 72622 Nürtingen www.pragmaticminds.de