Webinar with Avnet and ST

1. Copyright © 2019 EdgeImpulse Inc.
Getting started with TinyML
for industrial applications

2. 2
Jan Jongboom
jan@edgeimpulse.com
Hello!
Raphael Apfeldorfer
raphael.apfeldorfer@st.com

3. 3
Typical industrial sensor in 2020
Vibration sensor (up to 1,000 times per second)
Temperature sensor
Water & explosion proof
Can send data >10km using 25 mW power
Processor capable of running >20 million
instructions per second

4. 4
But... what does it actually do?
Once an hour:
• Average motion (RMS)
• Peak motion
• Current temperature

5. 5
99% of sensor data is discarded due to
cost, bandwidth or power constraints.
https://www.mckinsey.com/~/media/McKinsey/Business%20Functions/McKinsey%20Digital/Our%20Insights/
The%20Internet%20of%20Things%20The%20value%20of%20digitizing%20the%20physical%20world/The-
Internet-of-things-Mapping-the-value-beyond-the-hype.ashx

6. 6
Lots of interesting events get lost
Peak

7. 7
Single numbers can be misleading
updown
circle
avg. RMS
3.3650
3.3515

8. 8
On-device intelligence is the only solution
Vibra&on pa+ern
heard that lead to fault
state in a weekTemperature
varies in a way that
I've never seen
before
Machine
oscillates different
than all other
machines in the
factory

9. 9
On-device intelligence is the only solution
Temperature varies in a way
that I've never seen before
(0x1)

10. Machine learning is great at finding
patterns in messy data
(anything you can't reason about in Excel)

11. 11
Machine learning?

12. 12
TinyML
Inspired by "OK Google"
Focus on inferencing, not training
Machine learning model is just a mathematical
function with lots of parameters
Accuracy vs. speed, reducing parameters, hardware-
optimized paths
Targeting battery-powered microcontrollers
Pete Warden
Neil Tan

13. Predictive maintenance
https://www.flickr.com/photos/uwnews/26143149238

14. Audible monitoring

15. Lone worker monitoring
Check new Person Presence Detection example in
FP-AI-VISION1 !
https://www.st.com/en/embedded-software/fp-ai-vision1.html

16. Endpoint
Machine level
Decisions
Distributed AI from Edge to Cloud
Edge
Actions
Data
Gateways
Room or building-level
Decisions
HomeIndustrial
Cloud
Cities or factory-level
decisions
• Fleet monitoring
• Macro Decisions (AI)
• Macro Actuation
Sensor
• Sensing
Smart sensor
• Sensing
• Preprocessing
• Micro classification
Data
MCU
• Real time
• Data privacy
• Micro Decisions (AI)
• Micro Actuation
Actions
Event
detection
Event
detection
Sensor Data
(Option)
Event
MCU / MPU
• Data or Event integration
• Data Privacy
• Meso Decisions (AI)
• Meso Actuation
16

17. GatewaysEndpoint
Edge
Actions
Data
Room or building-level
Decisions
Cloud
Cities or factory-level
decisions
• Fleet monitoring
• Macro Decisions (AI)
• Macro ActuationActions
Event
detection
Connectivity
Security
Sensing
Accelerometer
Gyrometer
Magnetometer
Vibration
PressureAcoustic Temperature
ML Core
Extensive MCU
portfolio
Processing &
Actuating
Machine level
Decisions
STM32MP1
STM32H7
STM32WB
Connectivity
Security
Processing &
Actuating
Event
detection
Sensor Data
(Option)
Event
Distributed AI from Edge to Cloud
17

18. IIS3DWB
Ultra Wide Bandwidth
Accelerometer
ISM330DLC
ISM330DHCX
Wide Bandwidth
Accelerometer + Gyroscope
IIS2DH
Wide Bandwidth, Ultra-low-power
Accelerometer
IIS2MDC
Low-Noise, Low Power
Magnetometer
18
ST portfolio for condition monitoring
Vibration
MP23ABS1TR Analog Differential Microphone
IMP34DT05-A Digital Top Port Microphone
Acoustic
LPS22HH
High Accuracy – Compact Size
Absolute Pressure Sensor
LPS27HHW
LPS33HW
Water Resistant
Absolute Pressure Sensor
STTS22H Digital Temperature Sensor
STLM20 Analog Temperature Sensor
Environmental
High Perf
MCUs
Ultra-low Power
MCUs
Wireless
MCUs
Mainstream
MCUs
MPU MEMS & Sensors

19. Wireless Industrial node
to capture data at industrial grade
19www.st.com/stwin
Industrial-grade sensors
• Industrial scale 9-DoF motion sensors including accelerometer, gyrometer
and an ultra wide-band vibrometer with ultra low noise
• Very high frequency audio and ultrasound microphone
• High precision temperature and environmental monitoring
• Micro SD card for standalone data logging
• BLE connectivity and WiFi expansion board
• USART
STWIN
Process & analyze new
data using trained NNCapture data
9

20. 20
v
From 0
to model
https://www.flickr.com/photos/120586634@N05/14491303478

21. 21
1. Everything starts with raw data
Get data at the highest resolu/on possible - e.g. using serial or directly over WiFi

22. 22
2. Extracting meaningful features
Very dependent on your use case
Raw data can be notoriously hard to deal with
(3s. accelerometer data = 900 data points, 1s. audio data = 16,000 data points)
Use proven DSP algorithms
-1.1200, 0.5300, 10.6300, -0.5200, 1.1600, 13.1600, -0.1600, 1.4200, 13.5900, -0.2400, 1.3200, 10.8500, -0.3700, 0.5100, 7.7800, -0.1900, 0.0500, 8.8400, -0.1900, 0.0500, 8.8400, 0.2400, 0.7300, 11.1900, 0.5200, 1.6500, 12.5200, 0.6400, 1.5000, 10.8200, 0.2900, 0.3300, 7.4000, -0.3100, -0.5600, 8.5900, -0.8800, 0.6600, 11.9200, -0.8800, 0.6600, 11.9200, -0.4500, 1.3900, 11.9100,
-0.0800, 1.6000, 12.7800, -0.1100, 0.3600, 9.1700, -0.0200, 0.0700, 9.8200, 0.0600, 0.9600, 12.0000, 0.2800, 1.7700, 13.2000, 0.2800, 1.7700, 13.2000, 0.2300, 1.5100, 12.6000, 0.2700, 1.0800, 11.4300, 0.0900, 0.9300, 10.9400, 0.1700, 1.2100, 11.0400, 0.4100, 1.8500, 11.5000, 0.3900, 1.9600, 11.4300, 0.3900, 1.9600, 11.4300, 0.2400, 1.4400, 10.7200, 0.0300, 1.1900, 10.3600,
-0.0300, 1.3000, 10.6100, 0.4800, 1.7900, 11.7200, 1.0400, 2.6300, 13.3300, 1.0400, 2.6300, 13.3300, 1.0600, 2.3700, 13.5800, 0.3600, 1.9600, 13.3800, -0.1000, 2.1200, 13.9600, -0.3600, 2.0200, 14.5300, 0.0000, 2.1500, 14.6900, 0.0600, 2.1400, 14.3700, 0.0600, 2.1400, 14.3700, -0.3000, 1.8800, 13.7900, 0.0500, 1.7000, 13.5900, 0.1300, 1.6700, 13.1500, -0.0100, 1.7700, 12.9000,
0.4000, 1.8900, 12.2300, 0.5300, 2.3300, 12.2600, 0.5300, 2.3300, 12.2600, 0.0400, 1.9500, 11.8100, -0.2300, 1.9600, 11.2400, -0.0600, 2.1100, 10.2200, -0.1100, 2.4100, 9.7800, -0.3500, 2.7100, 9.7500, -0.7800, 3.1000, 10.1100, -0.7800, 3.1000, 10.1100, -1.0700, 3.1100, 9.8000, -1.2100, 2.9400, 9.1900, -1.1500, 3.2100, 8.6400, -0.7300, 3.6500, 8.4600, -0.5000, 3.9500, 8.6300,
-0.4300, 3.9100, 8.7400, -0.4300, 3.9100, 8.7400, -0.6400, 3.7800, 8.8700, -1.2000, 3.9200, 8.9300, -1.0800, 4.4400, 8.8100, -0.7800, 4.1900, 8.1200, -0.4400, 4.1000, 7.6400, -0.5400, 4.2000, 7.5600, -0.5400, 4.2000, 7.5600, -1.0700, 4.2600, 7.2700, -1.3000, 4.5100, 7.2300, -1.2600, 4.4600, 6.6900, -1.2800, 4.4100, 6.6000, -1.7000, 4.6800, 7.0800, -2.3400, 5.1100, 7.5900,
-2.3400, 5.1100, 7.5900, -2.8300, 4.8700, 6.8700, -2.9700, 4.7600, 6.2700, -3.2500, 4.6000, 6.1500, -3.4900, 4.5900, 6.2600, -3.3000, 4.9200, 6.3400, -2.7000, 4.9300, 5.8600, -2.7000, 4.9300, 5.8600, -2.9000, 4.5100, 4.9900, -3.5200, 4.3200, 4.9900, -4.1400, 4.2100, 5.7800, -3.7600, 4.1600, 5.7700, -3.0200, 4.2200, 5.4900, -3.0000, 3.8900, 3.9100, -3.0000, 3.8900, 3.9100,
-3.3500, 3.5800, 3.4400, -3.1100, 3.2000, 3.6000, -3.0900, 3.6000, 5.9400, -3.0800, 3.0900, 5.1800, -2.8000, 2.9400, 4.5600, -2.4000, 2.4900, 3.9200, -2.4000, 2.4900, 3.9200, -1.8500, 2.6300, 4.4900, -1.4300, 3.9900, 7.3400, -1.4900, 3.4900, 6.1600, -1.5300, 3.2100, 5.4500, -0.9900, 2.8600, 6.2400, -0.7900, 3.2200, 8.4700, -0.7900, 3.2200, 8.4700, -0.9300, 3.5700, 9.0300,
-1.6600, 2.9600, 7.0700, -1.7600, 2.1000, 6.9000, -1.4600, 2.1100, 9.0100, -1.3000, 2.5400, 10.3000, -1.3000, 2.5400, 10.3000, -1.4500, 2.6500, 9.7800, -1.5300, 1.8900, 8.4100, -1.1400, 1.3000, 9.4400, -0.7500, 1.6100, 10.3600, -0.9400, 1.5300, 9.7800, -1.0700, 1.0100, 9.2700, -1.0700, 1.0100, 9.2700, -0.8600, 1.0200, 10.1100, 0.3900, 1.6800, 11.3200, 1.2900, 1.8500, 11.4700,
1.0700, 1.3200, 11.2500, -0.2100, 1.5800, 12.4300, -1.8100, 1.3500, 12.3300, -1.8100, 1.3500, 12.3300, -2.1800, 1.0400, 11.1600, -1.5400, 0.3000, 10.3100, -0.4700, 0.2700, 11.0900, 0.7900, 1.4100, 12.9800, 1.1600, 1.7100, 12.4700, 0.5800, 0.8700, 9.6300, 0.5800, 0.8700, 9.6300, 0.1300, 0.3100, 9.5600, 0.2800, 0.3600, 10.5600, 0.7400, 0.8500, 11.4200, 0.9300, 1.0200, 11.4300,
0.6700, 0.5600, 10.5300, 0.8500, 0.3500, 9.4100, 0.8500, 0.3500, 9.4100, 1.6600, 1.2800, 10.9200, 2.0500, 0.9900, 9.7000, 2.1300, 0.8800, 10.1900, 2.0500, 0.9100, 11.3300, 1.7700, 1.4100, 12.2700, 1.4800, 1.7600, 12.1000, 1.4800, 1.7600, 12.1000, 0.9400, 1.1300, 10.8500, 0.2000, 0.8000, 10.1200, 0.2600, 1.1600, 10.5800, 0.5100, 1.6500, 10.7800, 0.4600, 1.2000, 9.9900, 0.9100,
0.8400, 9.6400, 0.9100, 0.8400, 9.6400, 1.4500, 0.7400, 10.2500, 2.0200, 1.3000, 11.4500, 1.8100, 1.8700, 12.1300, 1.0500, 1.5300, 12.0200, 0.6200, 0.6700, 11.3100, 0.7100, 0.8500, 12.0000, 0.7100, 0.8500, 12.0000, 0.6400, 1.2200, 13.1400, 1.1300, 2.0400, 14.6200, 0.8300, 2.0200, 15.5100, -0.1400, 1.4800, 15.6500, -0.6300, 1.5900, 16.0500, -1.3100, 1.7100, 16.3900, -1.3100,
1.7100, 16.3900, -1.7300, 1.5800, 16.6300, -1.1500, 1.4400, 16.0300, -0.5300, 1.1700, 15.1000, -0.1800, 0.9900, 14.4600, -0.3300, 1.0100, 13.5100, -0.3300, 1.0100, 13.5100, -0.4400, 0.9100, 12.6700, 0.0400, 1.2300, 12.5400, 0.6900, 2.0500, 13.1600, 0.3100, 1.7700, 12.8600, 0.0300, 1.3800, 11.1100, -0.4400, 1.2200, 9.4900, -0.4400, 1.2200, 9.4900, 0.1100, 1.1400, 7.3100, 0.8500,
2.2500, 8.4600, 0.8600, 3.3700, 11.2200, -0.1100, 2.2800, 8.4400, -1.3800, 1.5300, 7.1700, -1.0600, 1.5400, 6.9500, -1.0600, 1.5400, 6.9500, -0.5200, 2.8300, 8.7100, -0.2100, 2.3500, 8.1800, -0.3400, 2.7000, 8.9200, -0.3000, 2.3100, 8.7500, -0.4800, 1.4700, 7.8700, -0.3600, 0.9400, 6.9700, -0.3600, 0.9400, 6.9700, -0.2300, 1.4700, 7.6100, -0.3300, 2.2300, 8.5000, 0.3000, 1.9200,
7.8600, -0.2300, 1.5700, 6.8700, -1.4900, 1.5600, 6.3700, -2.8200, 1.6200, 7.2000, -2.8200, 1.6200, 7.2000, -3.1600, 1.8800, 7.1500, -2.7600, 2.2900, 6.8500, -2.6000, 2.2200, 6.2600, -2.9000, 1.9900, 5.8900, -3.3800, 2.2200, 6.2600, -3.9000, 2.1700, 6.0300, -3.9000, 2.1700, 6.0300, -3.8600, 2.3800, 5.6600, -3.5300, 2.5200, 5.6700, -3.2400, 2.3700, 5.8200, -3.2800, 2.1800,
5.5200, -3.1500, 2.1800, 5.6500, -3.0900, 2.0700, 5.1600, -3.0900, 2.0700, 5.1600, -2.4300, 2.1000, 5.3800, -2.0200, 2.3600, 6.0800, -2.0000, 2.5200, 6.4500, -2.2400, 2.4500, 6.0000, -2.0500, 1.8400, 4.6500, -1.3800, 1.3000, 4.6400, -1.3800, 1.3000, 4.6400, -1.2800, 1.8600, 6.9400, -1.3000, 2.5600, 9.0300, -1.5400, 2.7600, 8.5000, -1.7700, 1.6400, 6.1400, -1.6800, 1.4200,
7.5900, -1.3200, 2.0800, 9.8300, -1.3200, 2.0800, 9.8300, -0.8200, 2.1600, 10.3900, -0.7800, 1.7300, 9.8300, -1.1300, 1.3400, 9.7100, -1.3600, 1.6800, 10.2400, -1.5200, 1.6000, 9.3200, -1.8700, 1.4900, 9.1900, -1.8700, 1.4900, 9.1900, -1.9300, 1.0600, 9.9500, -1.3100, 0.8100, 10.6900, 0.0200, 2.0400, 11.0600, 0.2700, 2.5800, 9.3900, -0.0500, 2.2800, 7.3200, -0.3000, 0.4400,
7.6300, -0.3000, 0.4400, 7.6300, -1.4600, 1.0800, 12.3700, -1.9600, 1.7500, 15.3800, -0.7100, 2.1500, 14.0700, 0.7400, 1.7800, 10.4700, 0.6800, 0.8900, 9.9500, 0.0400, 1.5200, 12.0800, 0.0400, 1.5200, 12.0800, -0.4900, 1.7900, 12.7500

23. 23
Example of a signal processing pipeline
32,000 => 240

24. 24
Before and after feature extraction

25. Classification
What's happening right now?
Anomaly detection
Is this behavior out of the ordinary?
Forecasting
What will happen in the future?
25
3. Letting the computers figure it out

26. 26
Picking the right algorithm
Classification
Neural network
Anomaly detection
K-means clustering
Forecasting
Regression

27. 27
4. Deploying
Signal processing, neural network and
anomaly detec&on

28. Seamless integration to STM32Cube ecosystem
STM32CubeMX, GUI Builders
Configure & generate code
28
Configure clocks, peripherals
and generate low-level code
Add application code and compile for any
STM32 target with the IDE of your choice
Deploy, debug and monitor
Generate model, optimize with STM32Cube.AI and export as CMSIS-PACK
And commercial IDEs
CMSIS-PACK
Import in CubeMX and benefit from STM32 ecosystem
CMSIS-PACK
running in Edge
Impulse’s Cloud

29. STM32CubeL1
Hardware Abstraction Layer
CMSI
S
STM32CubeF0
Hardware Abstraction Layer
CMSI
S
STM32CubeF2
Hardware Abstraction Layer
CMSI
S
STM32CubeF4
Hardware Abstraction Layer
CMSI
S
STM32CubeF1
Hardware Abstraction Layer
CMSI
S
STM32Cube
Hardware abstraction layer
CMSIS
STM32Cube
Middleware
User code
FirmwareHardware
Making it easy to start development, develop Proof of Concept
STM32
Ecosystem foundations
Configuration Development
Programming Monitoring
Tools
macOS®
And more

30. STM32CubeMX expansion
STM32Cube.AI tool
STM32Cube.AI offers interoperability
with state-of-the-art Deep Learning
design frameworks
Any framework that can export models in
ONNX open format can be imported,
including quantized models
Automatic and fast generation of an
STM32-optimized library
Train NN Model
Convert NN into
optimized code
for MCU
Process & analyze new
data using trained NN
30
On-device validation enable fast
comparison of model accuracy vs
different memory / quantization options
And more

31. STM32Cube.AI main features
31
Dimension
Optimize
Fine Tune
And quickly iterate thanks to on-target validation
STM32Cube.AI is available both as graphical and command line interface

32. Making AI Accessible Now
High Perf
MCUs
Ultra-low Power
MCUs
Wireless
MCUs
Mainstream
MCUs
STM32F0
106 CoreMark
48 MHz
STM32G0
142 CoreMark
64 MHz
STM32F1
177 CoreMark
72 MHz
STM32F3
245 CoreMark
72 MHz
STM32F2
398 CoreMark
120 MHz
STM32F4
608 CoreMark
180 MHz
STM32L0
75 CoreMark
32 MHz
STM32L5
424 CoreMark
110 MHz
STM32L1
93 CoreMark
32 MHz
STM32L4
273 CoreMark
80 MHz
STM32WL
161 CoreMark
48 MHz
STM32L4+
409 CoreMark
120 MHz
STM32G4
550 CoreMark
170 MHz
STM32MP1
4158 CoreMark
650 MHz Cortex -A7
209 MHz Cortex -M4
MPU
Arm® Cortex® core -M0 -M3 -M33 -M4 -M7 dual -A7& -M4-M0+
STM32F7
1082 CoreMark
216 MHz
STM32H7
3224 CoreMark
240 MHz Cortex -M4
480 MHz Cortex -M7
STM32WB
216 CoreMark
64 MHz
Compatible with Deep Learning
STM32Cube.AI ecosystem
Compatible with Machine Learning
Partner ecosystems
Leader in Arm® Cortex®-M 32-bit General Purpose MCU
More than 40,000 customers Over 4 Billion STM32 shipped since 2007 32

33. Making AI Accessible Now
High Perf
MCUs
Ultra-low Power
MCUs
Wireless
MCUs
Mainstream
MCUs
STM32F0
106 CoreMark
48 MHz
STM32G0
142 CoreMark
64 MHz
STM32F1
177 CoreMark
72 MHz
STM32F3
245 CoreMark
72 MHz
STM32F2
398 CoreMark
120 MHz
STM32F4
608 CoreMark
180 MHz
STM32L0
75 CoreMark
32 MHz
STM32L5
424 CoreMark
110 MHz
STM32L1
93 CoreMark
32 MHz
STM32L4
273 CoreMark
80 MHz
STM32WL
161 CoreMark
48 MHz
STM32L4+
409 CoreMark
120 MHz
STM32G4
550 CoreMark
170 MHz
STM32MP1
4158 CoreMark
650 MHz Cortex -A7
209 MHz Cortex -M4
MPU
Arm® Cortex® core -M0 -M3 -M33 -M4 -M7 dual -A7& -M4-M0+
STM32F7
1082 CoreMark
216 MHz
STM32H7
3224 CoreMark
240 MHz Cortex -M4
480 MHz Cortex -M7
STM32WB
216 CoreMark
64 MHz
Compatible with Deep Learning
STM32Cube.AI ecosystem
Compatible with Machine Learning
Partner ecosystems
Leader in Arm® Cortex®-M 32-bit General Purpose MCU
More than 40,000 customers Over 4 Billion STM32 shipped since 2007 33
Vibration
Motion
Environmental
Bio signals
Acoustic
Condition monitoring
Industrial
gateways
Thermal and
RGB cameras

34. Getting started 🚀

35. 35
Get some hardware
ST IoT Discovery Kit
80MHz, 128K RAM, $50
Any smartphone Any other device
With our SDKs

36. 36
Edge Impulse - TinyML as a service
Embedded or edge
compute deployment
options
Test
Edge Device Impulse
Dataset
Acquire valuable
training data securely
Test impulse with
real-time device
data flows
Enrich data and
generate ML process
Real sensors in real time
Open source SDK
Free for developers: edgeimpulse.com

37. End-to-end tutorials on vibra/on, audio, and vision: docs.edgeimpulse.com

38. Demo 🚀

39. https://www.innovationworldcup.com/categories/stwin-challenge/
Join the STWIN CHALLENGE and compete to win
overall prizes worth over $500,000

40. Recap
The ML hype is real
ML + sensors = perfect fit
Start using the remaining 99% of sensor data
edgeimpulse.com / st.com

41. Slides: http://janjongboom.com (will be emailed afterwards)