This presentation covers a talk on the topic of "AI on the edge". The talk was delivered in the Conference on Artificial Intelligence and Robotics Technology held on Jan 28, 2021 by National Center of Artificial Intelligence Pakistan & working group by Ministry of Science and Technology on AI & Robotics.
1. 1
AI on the Edge
Lifecycle of AI solution to edge computing
Usman Qayyum
Additional Director (AI)
PhD (AI & Robotics)
Post-doc (Self-driving Cars)
mrusmanqayyum@hotmail.com
14. 14
What are we going to discuss now ?
● AI algorithms optimize for hardware
○ Quantization
○ Pruning
● Edge Hardware
○ CPU
○ GPU
○ FPGA
○ ASIC
○ Software Toolkit for Edge
16. 16
● We can use CPU Cache
● No need to read from RAM
● I can run it in a device with smaller memory
17. 17
What can I do to optimize?
COMPUTE LESS
COMPUTE FASTER
MOVE LESS DATA
18. 18
What can I do to optimize?
COMPUTE FASTER
● Increase the speed you compute
○ Increase the clock speed.
● Compute several things at the same time
○ SIMD instructions
○ Multiprocessing
19. 19
What can I do to optimize?
COMPUTE LESS & MOVE LESS DATA
● Decrease the number of parameters
● Decrease the size of the parameters
20. 20
What can I do to optimize?
PRUNING
Removing branches that are not important for our goal
L1 layer regularization
Structured pruning by varying regularizer parameters
Deep compression
21. 21
What can I do to optimize?
QUANTIZE
Reduce the representation of the data values we use
Deep compression
Super Nice explanation of 8bit quantization in TF
27. 27
Sophon Edge
Sophon Edge was released in 2018 and it
features the SoC Sophon BM1880 which
includes a TPU. This SoC can theoretically
deliver up to 2TOPS/s
Price 129$
28. 28
FPGA
FPGA is in the middle point between CPU and ASIC
The good thing of hardware world:
● Low latency
● Low power consumption
The good things from software:
● You can update your hardware
● Cheaper than building in silicon
Quite complex to program
29. 29
FPGA
They are complex to program but...
● Libraries to the rescue!
CHaiDNN Library for Xilinx
OpenVINO for Intel devices
30. 30
Chamaleon96 Cyclone V FPGA
● This board is going to be different
compared to the rest because this
one has a FPGA instead of a
dedicated hardware accelerator
● Price 130$
33. 33
TPU
CPU GPU TPU
● It’s an ASIC
● TPUv1 has only 8bit fixed-point support (ONLY USEFUL FOR TRAINING)
● TPUv2 includes 16bit float (NOW WE CAN TRAIN :D )
34. 34
Google Coral Dev Board
● A Raspberry Pi look like
board but featuring some
sort of TPU that allows
running NNs in the edge
really fast.
● Google Promises real
time inference for NNs
● Price $129
36. 36
Mini TPU
Google says the next:
“Edge TPU enables users to concurrently execute
multiple state-of-the-art AI models per frame, on a
high-resolution video, at 30 frames per second, in a
power-efficient manner.”
IT’S in EARLY ACCESS STAGE :(
Edge TPU
37. 37
Intel Compute Stick 2
USB 3.0 Connection
Able to run object detection at ~10fps
Compatible with Tensorflow, Caffe2 or ONNX
(as long you have the supported layers)
V1 ~60€
V2 ~ 100€ but 8x speed up compared to v1.
38. 38
1.3 TFLOPS FP32 from the GPU - GPU FP16 is 10.4 TFLOPS.
NVIDIA Jetson Xavier
GTX1080ti has ~10TFLOPS at FP32
44. 44
Thanks for your attention.
AI on the Edge
Usman Qayyum
Additional Director (AI)
PhD (AI & Robotics)
Post-doc (Self-driving Cars)
mrusmanqayyum@hotmail.com
