This document discusses implementing machine learning at the IoT edge using neural networks like GRU. It covers neural network types and cells, training tools, numeric formats for embedded implementations, and references for CMSIS-NN, HDF5, TensorFlow, Python and more. Code examples demonstrate converting models to Q format and optimizing matrix storage for efficient embedded inference.

1. Implementing Machine Learning
At
The IOT Edge
Steve Mylroie
Roietronics

2. Machine Learning
Supervised
Linear Regression
Decision Tree
Unsupervised
Deep Neural Networks
Liner Neural Neural Networks
Convolution Neural Networks
Recurrent Neural Networks
SRRN - Simple Recurrent Neural Networks
LSTM – Long Short Term Memory
GRU - Gated Recurrent Unit

3. GRU NEURAL NETWORK CELL

4. Sigmoid Function
• A sigmoid function is a
mathematical function
having a characteristic "S"-
shaped curve or sigmoid
curve. Often, sigmoid
function refers to the special
case of the logistic function
shown in the first figure and
defined by the formula
S(x)=1/(1+e⁻ˣ)=eˣ/(eˣ+1).

5. Training
•Datasets
•Tools
• Azure Notebooks
• Jupyter Notebooks
• Python 3.6
• Kares
• TensorFlow
•Demo

6. Data Randomization
No Offsets Random 15 Minute Offset per
24 Hour Cycle

7. Implementation
• Tools
• MBED OS5
• CMSIS-NN Package - GRU Example
• HDF5 Extractor
• Float to Q format Converter
• VS Code
• GNU Tools ARM Embedded
• Issues
• GRU Example Problems
• Weight and Bias Value Transfer
• Q format
• Code

8. Numeric Formats
Name bits Value bits Exponent bits Min Value Max Value
Float Half 16 10 5 -1*2**15 1*2**15
Float 32 23 8 -1*2**127 1*2**127
Double 64 52 11 -1*2**1023 1*2**1023
Q0.15 (Q15) 16 15 0 -1+2**-15 1-2**-15
Q0.7 (Q7) 8 7 0 -1+-2**-7 1-2**-7
Bit 15 14 13 12
… 0
Value Sign 1/2 1/4 1/8 … 1/(2^15)
Q15 Format

9. Matrix ordering with Optimization
• Weights are in q7_t and Activations are in q15_t
• Limitation: x4 version requires weight reordering to work
• Here we use only one pointer to read 4 rows in the weight matrix. So if the original q7_t matrix
looks like this:
• | a11 | a12 | a13 | a14 | a15 | a16 | a17 |
• | a21 | a22 | a23 | a24 | a25 | a26 | a27 |
• | a31 | a32 | a33 | a34 | a35 | a36 | a37 |
• | a41 | a42 | a43 | a44 | a45 | a46 | a47 |
• | a51 | a52 | a53 | a54 | a55 | a56 | a57 |
• | a61 | a62 | a63 | a64 | a65 | a66 | a67 |
• We operates on multiple-of-4 rows, so the first four rows becomes
• | a11 | a21 | a12 | a22 | a31 | a41 | a32 | a42 |
• | a13 | a23 | a14 | a24 | a33 | a43 | a34 | a44 |
• | a15 | a25 | a16 | a26 | a35 | a45 | a36 | a46 |

10. Matrix Reordering with Optimzation (Cont)
• The column left over will be in-order. which is: | a17 | a27 | a37 | a47 |
• For the left-over rows, we do 1x1 computation, so the data remains as its
original order.
• So the stored weight matrix looks like this:
• | a11 | a21 | a12 | a22 | a31 | a41 |
• | a32 | a42 | a13 | a23 | a14 | a24 |
• | a33 | a43 | a34 | a44 | a15 | a25 |
• | a16 | a26 | a35 | a45 | a36 | a46 |
• | a17 | a27 | a37 | a47 | a51 | a52 |
• | a53 | a54 | a55 | a56 | a57 | a61 |
• | a62 | a63 | a64 | a65 | a66 | a67 |

11. CMSIS-NN Software Library
• https://arm-
software.github.io/CMSIS_5/NN/html/group__GRUExample.html
• https://arm-
software.github.io/CMSIS_5/DSP/html/group__float__to__x.html
• https://github.com/ARM-software/CMSIS_5
• https://os.mbed.com/blog/entry/uTensor-and-Tensor-Flow-
Announcement/ May 2019

12. References 1
• Under Standing LSTMetwork
http://colah.github.io/posts/2015-08-Understanding-LSTMs/
• cmsis-nn GRU example
https://arm-software.github.io/CMSIS_5/NN/html/group__GRUExample.html
• Azure Notebooks
https://notebooks.azure.com/
• Jupyter Notebooks
https://jupyter.org/
• Keras Documentation
https://keras.io/
• Keras Model Guide
https://keras.io/getting-started/sequential-model-guide/
• Keras Recurrent Layers
https://keras.io/layers/recurrent/
(Scroll down for GRU cell entry)
• TensorFlow
https://www.tensorflow.org/
• Python 3.6 Documentation
https://devdocs.io/python~3.6/

13. References 2
• Heretical Data File 5 (HDF5)
https://portal.hdfgroup.org/display/support/Documentation
• Floating Point to Q Format Converter
https://www.mathworks.com/matlabcentral/fileexchange/61669-decimal-to-fixed-point-q-format-converter
• Mbed OS
https://os.mbed.com/
• CMSIS NN Package
https://arm-software.github.io/CMSIS_5/NN/html/index.html
• CMSIS Packages Down Load
https://github.com/ARM-software/CMSIS_5
• Conversion to CMSIS-NN
https://developer.arm.com/solutions/machine-learning-on-arm/developer-material/how-to-
guides/converting-a-neural-network-for-arm-cortex-m-with-cmsis-nn/single-page
• VSCode
https://code.visualstudio.com/
• Sample Data Sets
https://www.deepsig.io/datasets
• SSD Notebook
https://notebooks.azure.com/smylroie/projects/ssd-demo