Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.


73 vues

Publié le

Low power high performance real-time computer vision on a CPU+FPGA hybrid architecture with the HIPPEROS multicore RTOS

Publié dans : Ingénierie
  • Login to see the comments

  • Soyez le premier à aimer ceci


  1. 1. Confidential © HIPPEROS 2016 1 Low power high performance real-time computer vision on a CPU+FPGA hybrid architecture with the HIPPEROS multicore RTOS EMVA 2017 2nd European Machine Vision Forum Vienna, September 7th 2017 Ben Rodriguez, CEO brodriguez@hipperos.com www.hipperos.com
  2. 2. Confidential © HIPPEROS 2016 2 The Company • HIPPEROS S.A. founded in January 2014, located at LLN and Brussels (Belgium). Only company in Belgium specializing in High Performance Embedded Systems (HPES) solutions (RTOS, middleware & tools) • Spin-off of PARTS ULB, incubated by WSL, selected by ESA BIC Program with highest marks ever in Belgium. Member of several Belgian and international clusters and poles (Infopole, SkyWin, DSP Valley, EMVA, …) • Very experienced management team, 20+ years of experience with record of industrial achievements. Advisory board includes senior business executives. The HIPPEROS team combines 50+ person-years of R&D results, with >100 published papers • HIPPEROS Academic Partner Program for universities & R&D centers "Intelligent Autonomy by Computer Vision, Advanced Robotics and Sensors & Control" Embedded & Real-Time High Performance Embedded Solutions for next generation Smart Systems
  3. 3. Confidential © HIPPEROS 2016 3 Virtual Reality Image Processing Everywhere … Safety Critical Systems Power Lines Monitor Logistics &Traffic Control Avionics & Space Drones & Autopilots Robots & Exoskeletons Medical Navigation & Car Safety
  4. 4. Confidential © HIPPEROS 2016 4  Increasing number of sensors, amount of data, data bandwidth etc.  Demand for high-performance, low-power and heterogeneous computing  Build into small embedded devices where size, weight and power matter  Embedded systems for image processing are highly specialized systems HOW DO WE PUT A SUPERCOMPUTER INTO AN EMBEDDED BOARD ? https://commons.wikimedia.org/wiki/File:IBM_Blue_Gene_P_supercomputer.jpg http://edablog.com/2013/08/08/blackfin-bf609-pvp/ Challenges
  5. 5. Confidential © HIPPEROS 2016 5 Challenges Intensive Image Processing Embedded Constraints Time-to-market / Cost-sensitive Source: http://www.lnci.org.au
  6. 6. Confidential © HIPPEROS 2016 6 The TULIPP Project HORIZON 2020 Project from 2016 to 2018 with 4.5 M€ funding to address that challenge. Selected key partners in Europe. Goal is ecosystem building around a generic platform.
  7. 7. Confidential © HIPPEROS 2016 7  Hard Real-time Image Processing  High Reliability (Certification)  High Performance Computing  Support for Hardware Acceleration  Low Power Autonomous Devices  Security  Reduced Time to Market  Reduced Development Time & Efforts  Reduced Development Risks  Evangelism on Low Power Embedded HPC  Guidelines for Efficient Development  Standardization by Generic Platforms Requirements
  8. 8. Confidential © HIPPEROS 2016 8 • Real time imaging application: – Image size: • From 512x512 to 3196x3196 pixels • 16bits/pixel – Real time: • From 4 to 100 frame/second • Zero frames missed ! • Typically 30 to 60 Mega Pixels/second • Low latency from read out to process / display : << 50 milliseconds – Algorithms: • 1st stage : Image correction : “Clean Image” • 2nd stage : Image Processing : Application oriented Requirements
  9. 9. Confidential © HIPPEROS 2016 9 TULIPP aims to push forward a reference platform for embedded image processing applications in order to – Define implementation rules for vision-based applications – Provide guidelines for guaranteed high performance and low power – Reduce development time and costs With the focus on embedded vision-based applications TULIPP will – Set up an ecosystem – Work closely with standardization organizations – Propose new standards derived from the reference platform General Goals & Structure of TULIPP Project
  10. 10. Confidential © HIPPEROS 2016 10 • The goal is not to develop a fixed embedded computing platform. • Instead, the TULIPP project aims to be generic and define – Implementation rules and – Interfaces • TULIPP allows for a flexible platform for image processing applications featuring – Low power consumption – High, efficient computing performance – Real-time features and latency Concepts of TULIPP Reference Platform
  11. 11. Confidential © HIPPEROS 2016 11 Components of TULIPP Reference Platform • The Tulipp project develops an instance of this reference platform consisting of three layers – Hardware architecture: a scalable low-power board – Low-power, real-time operating system and image processing libraries – Energy-aware tool chain • Following the implementation rules a developer is able to combine different components to create a Tulipp compliant platform – For image processing applications – Fast development – Flexible yet optimized solution
  12. 12. Confidential © HIPPEROS 2016 12 • Hardware Architecture – Heterogeneous Systems-on-Chip (SoCs) • Combinations of CPU, GPU and FPGA • Dedicated HW accelerators • Also dedicated real-times cores – Appropriate hardware system • Selection of processing elements • Interconnections of on-chip components • Interconnections of several SoCs – Support advanced features • Switch-off mechanisms • Dynamic Voltage and Frequency Scaling (DVFS) • Dynamic Partial Reconfiguration (DPR) Hardware Reference Platform
  13. 13. Confidential © HIPPEROS 2016 13 • Hardware Architecture – First developments with platform provided by Sundance 4-core ARM + FPGAs • Sundance EMC2-DP Carrier • Trenz SoM with Xilinx Zynq http://www.sundance.technology/som-cariers/pc104-boards/ Hardware Reference Platform
  14. 14. Confidential © HIPPEROS 2016 14 • Requirements for Operating System and Low-Level Libraries – Hard Real-time Operating System with APIs to • Support low power at API level at at kernel scheduler level • Support image processing devices and applications – Run on instantiated processors • Support heterogeneous multi-core systems • Handle hardware resources (FPGAs), allows DPR of the FPGA • Implementation of communication and synchronization – Provide for the developer • Real-time guarantees • Reliability • Easy programmability, compliance, certifiable Real-Time Operating System
  15. 15. Confidential © HIPPEROS 2016 15 The HIPPEROS Solution High Performance Parallel Embedded Real-time Operating Systems Multicore RTOS Real-Time Operating System = =+ + Reliability Real-Time Performance 20+ Years of R&D in Kernel Design, IPC, Scheduling, …  Reliability and Hard Real Time  Optimized Performance  Multicore Scalability  Small Footprint & Low power
  16. 16. Confidential © HIPPEROS 2016 16 • High Level Development Toolchain – A TULIPP compliant platform may have different components from different vendors • Expertise required for every vendor specific tool • Especially for optimized systems featuring high performance, low power consumption and real-time features – TULIPP toolchain is a set of Eclipse-based utilities to support the developer • STHEM - Supporting uTilities for Heterogeneous EMbedded image processing • Wraps around, extends and connects existing vendor tools • Seamless mapping, profiling, analyzing and optimizing an application Toolchain
  17. 17. Confidential © HIPPEROS 2016 17 • Although TULIPP is Generic, some representative Use Cases have been selected for demonstrating the potential. – Instantiation of reference platform is use case driven – Verification of reference platform with image processing applications • Medical imaging • Automotive imaging (ADAS) • Robotic imaging (UAV) – Different fields of embedded applications, but similar constraints • Performance • Power consumption • Size, volume and cost – Also real-time constraints Use Cases
  18. 18. Confidential © HIPPEROS 2016 18 https://commons.wikimedia.org/wiki/File:113abcd_Medical_Imaging_Techniques.jpg • Medical Imaging – Demand for high performance yet small devices • Requires processing of large amount of data • Mobile imaging equipment replacing high-end infrastructure devices – Demand for real-time imaging • Fast processing of image data during surgery • On mobile device – Tulipp X-Ray use case aims to provide • Reduction of radiation dose of sensors • More powerful image processing • Low power since heat and other RF emission could disturb sensors Use Cases
  19. 19. Confidential © HIPPEROS 2016 19 Medical imaging use case • Real-Time X-Ray imaging for surgery • Reduce radiation dose by 75% • Add noise removal processing with critical real-time constraints
  20. 20. Confidential © HIPPEROS 2016 20 Medical imaging use case TDLP RAW IMAGE THALES Processing Unit CI / ICS UI GigE-Vision + Msg THALES Flat panel detector Customer system UI GigE-Vision + Msg CI / ICS Nano Processing Unit Inside the detector Based on SoC (credit card size board) Customer system THALES Flat panel detector Before Tulipp After Tulipp
  21. 21. Confidential © HIPPEROS 2016 21 • Automotive Driver Assistance System (ADAS) – Intelligent cars need more and efficient embedded devices Image processing required for • Driving safety • Pedestrian safety – More active safety systems • Vehicle, pedestrian and object detection • Traffic sign and lane recognition • Night vision and surround view • Driver monitoring – Tulipp ADAS use case aims to provide • Real-time, low latency high-performance image processing • Reliability and robustness https://www.asdreports.com/news-10595/key-players-advanced-driver -assistance-systems-adas-market-north-america-20152019 http://www.rcs.ei.tum.de/forschung/driver-assistance/ Use Cases
  22. 22. Confidential © HIPPEROS 2016 22 • The use case is taken from the many applications that now enter our cars, doing emergency braking, lane keeping, etc., with direction towards autonomous cars • The chosen image processing algorithm for the use case is pedestrian detection that typically is used for emergency braking and driver assistance systems. • Pedestrian detection is today mostly made by either using Viola/Jones classifiers or Deep Learning • Viola Jones classifying has been chosen due to its more challenging memory access patterns ADAS Use Case
  23. 23. Confidential © HIPPEROS 2016 23 Pedestrian detection Safety application Car integration The Use Case ADAS Use Case
  24. 24. Confidential © HIPPEROS 2016 24 • Autonomous Unmanned Aerial Vehicles (UAVs) – UAVs more common for different applications • Surveillance, search and rescue, logistics and research – On-board real-time processing is key technology • Efficient and reliable automatic collision avoidance needed • Opposing constraints – The Tulipp UAV use case aims to provide • Optimized performance-to-weight and power-consumption-to-weight figures • On-board stereo vision & depth estimation • Real-time and automatic detection of obstacles for collision avoidance • Image recognition and identification https://irevolutions.org/2014/03/24/launching-a-search -and-rescue-challenge-for-drone-uav-pilots/ Use Cases
  25. 25. Confidential © HIPPEROS 2016 25 • VHDL synthesis from C source • STHEM & HIPPEROS integrated with Xilinx Vivado • Transparence of HW/SW for developpers • Only Software: ~4-6 FPS • With Hardware acceleration: up to 90 fps • Roles of HIPPEROS RTOS: – Hard Real-Time Synchronisation SW + HW Tasks – Low Power – FPGA Dynamic Partial Reconfiguration Achievements so far
  26. 26. Confidential © HIPPEROS 2016 26 • A main goal of the TULIPP project is to set up an ecosystem and advisory board to extend image processing norms according to needs of the industry. • Everyone is welcome to join and participate actively! • Information at: http://tulipp.eu/advisory-board-letter-information • Forum for support: http://support.tulipp.eu • Contact: coordinator@tulipp.eu • www.tulipp.eu Join our Advisory Board
  27. 27. Confidential © HIPPEROS 2016 27 Advisory Board Members
  28. 28. Confidential © HIPPEROS 2016 28 TULIPP in the news
  29. 29. Confidential © HIPPEROS 2016 29 Thank you! Any Questions ? Ben Rodriguez, CEO brodriguez@hipperos.com www.hipperos.com
  30. 30. Confidential © HIPPEROS 2016 30 • Tulipp, “Tulipp: Towards Ubiquitous Low-power Image Processing Platforms – High, efficient and guaranteed computing performance for image processing applications”, http://www.tulipp.eu, Accessed on 11.05.2017 • A. Paolillo, O. Desenfans, V. Svoboda, J. Goossens and B. Rodriguez. A new configurable and parallel embedded real-time micro-kernel for multi-core platforms. In Proceedings of the ECRTS Workshop on Operating Systems Platforms for Embedded Real-Time applications (ECRTS-OSPERT ’15), July 2015 • Antonio Paolillo, Joël Goossens, Pradeep M. Hettiarachchi and Nathan Fisher. Power Minimization for Parallel Real-Time Systems with Malleable Jobs and Homogeneous Frequencies. The 20th IEEE International Conference on Embedded and Real-Time Computing Systems and Applications, Chongqing, China, August 2014 • Antonio Paolillo, Paul Rodriguez, Nikita Veshchikov, Joël Goossens and Ben Rodriguez. Quantifying Energy Consumption for Practical Fork-Join Parallelism on an Embedded Real-Time Operating System. The 24th ACM International Conference on Real-Time Networks and Systems, Brest, France, October 2016 • Kalb, T. et al., “TULIPP: Towards Ubiquitous Low-power Image Processing Platforms”, In Proc. of the International Conference on Embedded Computer Systems: Architectures, Modeling and Simulation (SAMOS XV), 2016 • Martin Cornil, Antonio Paolillo, Joël Goossens, Ben Rodriguez. Research and implementation challenges of RTOS support for heterogeneous computing platforms. HARTS-ULB, Brussels, Belgium, May 2017 • Tchouchenkov, I., Segor, F., Schoenbein, R., Kollmann, M., Bierhoff, T., Herbold, M., “Detection And Protection Against Unwanted Small UAVs”, Proceedings of the Eleventh International Conference on Systems ICONS, 2016 • Ruf, B., Schuchert, T., “Towards real-time change detection in videos based on existing 3D models”, Proceedings of SPIE, Edinburgh, UK, 2016, vol. 10004, pp. 100041H--100041H-14 Some References