HIPPEROS's at EMVA 2017

Confidential © HIPPEROS 2016
1
Low power high performance real-time computer
vision on a CPU+FPGA hybrid architecture with the
HIPPEROS multicore RTOS
EMVA 2017
2nd European Machine Vision Forum
Vienna, September 7th 2017
Ben Rodriguez, CEO
brodriguez@hipperos.com
www.hipperos.com

2
The Company
• HIPPEROS S.A. founded in January 2014, located at LLN and Brussels (Belgium). Only
company in Belgium specializing in High Performance Embedded Systems (HPES)
solutions (RTOS, middleware & tools)
• Spin-off of PARTS ULB, incubated by WSL, selected by ESA BIC Program with highest
marks ever in Belgium. Member of several Belgian and international clusters and poles
(Infopole, SkyWin, DSP Valley, EMVA, …)
• Very experienced management team, 20+ years of experience with record of industrial
achievements. Advisory board includes senior business executives. The HIPPEROS
team combines 50+ person-years of R&D results, with >100 published papers
• HIPPEROS Academic Partner Program for universities & R&D centers
"Intelligent Autonomy by Computer Vision,
Advanced Robotics and Sensors & Control"
Embedded & Real-Time High Performance Embedded Solutions for next
generation Smart Systems

3
Virtual Reality
Image Processing Everywhere …
Safety Critical Systems
Power Lines Monitor
Logistics &Traffic Control
Avionics
& Space
Drones & Autopilots
Robots & Exoskeletons
Medical
Navigation
& Car Safety

4
 Increasing number of sensors, amount of data, data bandwidth etc.
 Demand for high-performance, low-power and heterogeneous computing
 Build into small embedded devices where size, weight and power matter
 Embedded systems for image processing are highly specialized systems
HOW DO WE PUT A SUPERCOMPUTER INTO AN EMBEDDED BOARD
?
https://commons.wikimedia.org/wiki/File:IBM_Blue_Gene_P_supercomputer.jpg
http://edablog.com/2013/08/08/blackfin-bf609-pvp/
Challenges

5
Challenges
Intensive
Image
Processing
Embedded
Constraints
Time-to-market /
Cost-sensitive
Source: http://www.lnci.org.au

6
The TULIPP Project
HORIZON 2020 Project from 2016 to 2018 with 4.5 M€
funding to address that challenge.
Selected key partners in Europe.
Goal is ecosystem building around a generic platform.

7
 Hard Real-time Image Processing
 High Reliability (Certification)
 High Performance Computing
 Support for Hardware Acceleration
 Low Power Autonomous Devices
 Security
 Reduced Time to Market
 Reduced Development Time & Efforts
 Reduced Development Risks
 Evangelism on Low Power Embedded HPC
 Guidelines for Efficient Development
 Standardization by Generic Platforms
Requirements

8
• Real time imaging application:
– Image size:
• From 512x512 to 3196x3196 pixels
• 16bits/pixel
– Real time:
• From 4 to 100 frame/second
• Zero frames missed !
• Typically 30 to 60 Mega Pixels/second
• Low latency from read out to process / display : << 50 milliseconds
– Algorithms:
• 1st stage : Image correction : “Clean Image”
• 2nd stage : Image Processing : Application oriented
Requirements

9
TULIPP aims to push forward a reference
platform for embedded image processing
applications in order to
– Define implementation rules for
vision-based applications
– Provide guidelines for guaranteed
high performance and low power
– Reduce development time and costs
With the focus on embedded vision-based applications TULIPP will
– Set up an ecosystem
– Work closely with standardization organizations
– Propose new standards derived from the reference platform
General Goals & Structure of TULIPP
Project

10
• The goal is not to develop a fixed embedded computing platform.
• Instead, the TULIPP project aims to be generic and define
– Implementation rules and
– Interfaces
• TULIPP allows for a flexible platform for image processing applications featuring
– Low power consumption
– High, efficient computing performance
– Real-time features and latency
Concepts of TULIPP Reference Platform

11
Components of TULIPP Reference Platform
• The Tulipp project develops an instance of this reference platform consisting of three
layers
– Hardware architecture: a scalable low-power board
– Low-power, real-time operating system and image processing libraries
– Energy-aware tool chain
• Following the implementation rules a developer is able to combine different components
to create a Tulipp compliant platform
– For image processing applications
– Fast development
– Flexible yet optimized solution

12
• Hardware Architecture
– Heterogeneous Systems-on-Chip (SoCs)
• Combinations of CPU, GPU and FPGA
• Dedicated HW accelerators
• Also dedicated real-times cores
– Appropriate hardware system
• Selection of processing elements
• Interconnections of on-chip components
• Interconnections of several SoCs
– Support advanced features
• Switch-off mechanisms
• Dynamic Voltage and
Frequency Scaling (DVFS)
• Dynamic Partial Reconfiguration (DPR)
Hardware Reference Platform

13
• Hardware Architecture
– First developments with platform provided by Sundance 4-core ARM + FPGAs
• Sundance EMC2-DP Carrier
• Trenz SoM with Xilinx Zynq
http://www.sundance.technology/som-cariers/pc104-boards/
Hardware Reference Platform

14
• Requirements for Operating System and Low-Level Libraries
– Hard Real-time Operating System with APIs to
• Support low power at API level at at kernel scheduler level
• Support image processing devices and applications
– Run on instantiated processors
• Support heterogeneous multi-core systems
• Handle hardware resources (FPGAs), allows DPR of the FPGA
• Implementation of communication and synchronization
– Provide for the developer
• Real-time guarantees
• Reliability
• Easy programmability, compliance, certifiable
Real-Time Operating System

15
The HIPPEROS Solution
High Performance Parallel Embedded Real-time Operating
Systems
Multicore
RTOS
Real-Time
Operating
System
= =+ +
Reliability Real-Time Performance
20+ Years of R&D in Kernel Design, IPC, Scheduling, …
 Reliability and Hard Real Time
 Optimized Performance
 Multicore Scalability
 Small Footprint & Low power

16
• High Level Development Toolchain
– A TULIPP compliant platform may have different components from different
vendors
• Expertise required for every vendor specific tool
• Especially for optimized systems featuring high performance, low power
consumption and real-time features
– TULIPP toolchain is a set of Eclipse-based utilities to support the developer
• STHEM - Supporting uTilities for Heterogeneous EMbedded
image processing
• Wraps around, extends and connects existing vendor tools
• Seamless mapping, profiling, analyzing and optimizing an application
Toolchain

17
• Although TULIPP is Generic, some representative Use Cases have been
selected for demonstrating the potential.
– Instantiation of reference platform is use case driven
– Verification of reference platform with image processing applications
• Medical imaging
• Automotive imaging (ADAS)
• Robotic imaging (UAV)
– Different fields of embedded applications, but similar constraints
• Performance
• Power consumption
• Size, volume and cost
– Also real-time constraints
Use Cases

18
https://commons.wikimedia.org/wiki/File:113abcd_Medical_Imaging_Techniques.jpg
• Medical Imaging
– Demand for high performance yet small devices
• Requires processing of large
amount of data
• Mobile imaging equipment replacing
high-end infrastructure devices
– Demand for real-time imaging
• Fast processing of image data
during surgery
• On mobile device
– Tulipp X-Ray use case aims to provide
• Reduction of radiation dose of sensors
• More powerful image processing
• Low power since heat and other
RF emission could disturb sensors
Use Cases

19
Medical imaging use case
• Real-Time X-Ray imaging for surgery
• Reduce radiation dose by 75%
• Add noise removal processing with critical
real-time constraints

20
Medical imaging use case
TDLP
RAW IMAGE
THALES
Processing Unit
CI / ICS
UI
GigE-Vision + Msg
THALES Flat panel detector
Customer system
UI
GigE-Vision + Msg
CI / ICS
Nano Processing Unit
Inside the detector
Based on SoC (credit card size board)
Customer system
THALES Flat panel detector
Before
Tulipp
After
Tulipp

21
• Automotive Driver Assistance System (ADAS)
– Intelligent cars need more and efficient embedded devices
Image processing required for
• Driving safety
• Pedestrian safety
– More active safety systems
• Vehicle, pedestrian and object detection
• Traffic sign and lane recognition
• Night vision and surround view
• Driver monitoring
– Tulipp ADAS use case aims to provide
• Real-time, low latency high-performance
image processing
• Reliability and robustness
https://www.asdreports.com/news-10595/key-players-advanced-driver
-assistance-systems-adas-market-north-america-20152019
http://www.rcs.ei.tum.de/forschung/driver-assistance/
Use Cases

22
• The use case is taken from the many
applications that now enter our cars, doing
emergency braking, lane keeping, etc., with
direction towards autonomous cars
• The chosen image processing algorithm for the
use case is pedestrian detection that typically is
used for emergency braking and driver
assistance systems.
• Pedestrian detection is today mostly made by
either using Viola/Jones classifiers or Deep
Learning
• Viola Jones classifying has been chosen due to
its more challenging memory access patterns
ADAS Use Case

23
Pedestrian
detection
Safety
application
Car
integration
The Use Case
ADAS Use Case

24
• Autonomous Unmanned Aerial Vehicles (UAVs)
– UAVs more common for different applications
• Surveillance, search and rescue,
logistics and research
– On-board real-time processing is
key technology
• Efficient and reliable automatic
collision avoidance needed
• Opposing constraints
– The Tulipp UAV use case aims to provide
• Optimized performance-to-weight and
power-consumption-to-weight figures
• On-board stereo vision & depth estimation
• Real-time and automatic detection of
obstacles for collision avoidance
• Image recognition and identification
https://irevolutions.org/2014/03/24/launching-a-search
-and-rescue-challenge-for-drone-uav-pilots/
Use Cases

25
• VHDL synthesis from C source
• STHEM & HIPPEROS integrated with Xilinx
Vivado
• Transparence of HW/SW for developpers
• Only Software: ~4-6 FPS
• With Hardware acceleration: up to 90 fps
• Roles of HIPPEROS RTOS:
– Hard Real-Time Synchronisation SW + HW Tasks
– Low Power
– FPGA Dynamic Partial Reconfiguration
Achievements so far

26
• A main goal of the TULIPP project is to set up an ecosystem and advisory board
to extend image processing norms according to needs of the industry.
• Everyone is welcome to join and participate actively!
• Information at: http://tulipp.eu/advisory-board-letter-information
• Forum for support: http://support.tulipp.eu
• Contact: coordinator@tulipp.eu
• www.tulipp.eu
Join our Advisory Board

27
Advisory Board Members

28
TULIPP in the news

29
Thank you!
Any Questions ?
Ben Rodriguez, CEO
brodriguez@hipperos.com
www.hipperos.com

30
• Tulipp, “Tulipp: Towards Ubiquitous Low-power Image Processing Platforms – High, efficient and
guaranteed computing performance for image processing applications”, http://www.tulipp.eu, Accessed
on 11.05.2017
• A. Paolillo, O. Desenfans, V. Svoboda, J. Goossens and B. Rodriguez. A new configurable and parallel
embedded real-time micro-kernel for multi-core platforms. In Proceedings of the ECRTS Workshop on
Operating Systems Platforms for Embedded Real-Time applications (ECRTS-OSPERT ’15), July 2015
• Antonio Paolillo, Joël Goossens, Pradeep M. Hettiarachchi and Nathan Fisher. Power Minimization for
Parallel Real-Time Systems with Malleable Jobs and Homogeneous Frequencies. The 20th IEEE
International Conference on Embedded and Real-Time Computing Systems and Applications,
Chongqing, China, August 2014
• Antonio Paolillo, Paul Rodriguez, Nikita Veshchikov, Joël Goossens and Ben Rodriguez. Quantifying
Energy Consumption for Practical Fork-Join Parallelism on an Embedded Real-Time Operating System.
The 24th ACM International Conference on Real-Time Networks and Systems, Brest, France, October
2016
• Kalb, T. et al., “TULIPP: Towards Ubiquitous Low-power Image Processing Platforms”, In Proc. of the
International Conference on Embedded Computer Systems: Architectures, Modeling and Simulation
(SAMOS XV), 2016
• Martin Cornil, Antonio Paolillo, Joël Goossens, Ben Rodriguez. Research and implementation
challenges of RTOS support for heterogeneous computing platforms. HARTS-ULB, Brussels, Belgium,
May 2017
• Tchouchenkov, I., Segor, F., Schoenbein, R., Kollmann, M., Bierhoff, T., Herbold, M., “Detection And
Protection Against Unwanted Small UAVs”, Proceedings of the Eleventh International Conference on
Systems ICONS, 2016
• Ruf, B., Schuchert, T., “Towards real-time change detection in videos based on existing 3D models”,
Proceedings of SPIE, Edinburgh, UK, 2016, vol. 10004, pp. 100041H--100041H-14
Some References

HIPPEROS's at EMVA 2017

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (9)

Similaire à HIPPEROS's at EMVA 2017

Similaire à HIPPEROS's at EMVA 2017 (20)

Plus de Tulipp. Eu

Plus de Tulipp. Eu (14)

Dernier

Dernier (20)

HIPPEROS's at EMVA 2017