Presented at Embedded Vision Alliance Summit 2016. Computer vision is central to many modern, cool products and technologies including augmented reality, virtual reality and drones. Thanks to recent advances in system-on-chip and embedded systems design, one can finally implement robust computer vision capabilities for demanding applications on embedded platforms. However, creating such systems is complex and challenging, and requires extensive, deep knowledge and hands-on experience in many areas, such as embedded system architecture, hardware-specific acceleration and memory access patterns. Mistakes in any of these areas can significant delay your project, or even sink it entirely. In this talk, we will explore some of the most common pitfalls of vision product development projects, and present practical ways of avoiding them. We will draw on examples from real-world product development projects.

1. Copyright © 2016 Itseez 1
Real-World Vision Systems Design
Challenges and Techniques
Yury Gorbachev
May 3, 2016

2. Copyright © 2016 Itseez 2
• Computer vision experts, 10 years on market
• Maintainers of OpenCV library
• Own products
• ADAS algorithms suite
• Facence (Face detection, recognition and analysis)
• 3D Scanning (Itseez3D)
• Accelerated CV (Optimized Computer Vision functions)
• Consulting in CV area
• Lot of Proof of Concept work per customer requests
Itseez at Glance

3. Copyright © 2016 Itseez 3
• Specific handling of requirements for vision projects
• Underestimated role of data
• Compute platform selection fallacies
• Hardware assisted vision algorithms
• Planning for customer feedback
• Functionality for easier product maintenance
Topics to Cover

4. Copyright © 2016 Itseez 4
• Requirements for vision products require special handling
• Not always possible to predict data variation upfront
• Specific cases will appear during design or test
• Missed scenarios
• Lighting conditions, blur and glare effects, etc.
• Impact can be pretty substantial
• More scenarios to handle — increase in algorithm complexity
• Additional data could be required
Requirements for Vision Products

5. Copyright © 2016 Itseez 5
• Initial requirement — detect speed limits in Europe
• Standards describe signs pretty well (size, color, etc.)
• Pretty easy to put into requirements
• Some things are not so straightforward to predict though
Example — Speed Limit Sign Detection

6. Copyright © 2016 Itseez 6
• Computer vision heavily depends on data for training and test
• Innovative products require specific data
• Usually not available in public datasets
• Data collection stage is needed prior to algorithm design
• Collect videos/images, preferably with variation and target HW
• Annotate collected data. At first stages done manually
• This is not always well understood by non-vision related customers
• Not always correctly planned within a project
Good Dataset is a Major Part of Solution

7. Copyright © 2016 Itseez 7
• Statistics from road sign dataset used at Itseez ADAS
• Not considering different sign types, weather and lighting conditions
• Annotation will take even more
Example — Speed Limit Signs Dataset

8. Copyright © 2016 Itseez 8
• Plenty of options for compute platform selection
• GPU, DSP and SIMD-enabled SoCs
• Frequently hardware selection is separated from software design
• Could be even pre-selected already before project start
• FACT: Not all compute platforms fit all algorithms
• Separated selection usually results in suboptimal systems
• Higher power consumption & unnecessary complicated algorithms
Premature Compute Platform Selection

9. Copyright © 2016 Itseez 9
• Vectorization is commonly seen as a solution for most of the problems
• Some SoCs sport few vector units with low frequency and few cores
• Obvious choice, perfect fit for vision!
• Not really… Following algorithm will have almost no benefit from SIMD
Example — Suboptimal Hardware Selection
Keypoint
search
External
sensor
Detailed
search
(Cascade)
ROIs Candidates
Candidate
analysis

10. Copyright © 2016 Itseez 10
• Optics and sensors should be selected for a given task
• Color vs. grayscale imagers, FOV vs. distance
• Some tasks are significantly simpler when depth information is available
• E.g., stairway detection for robotics
• Vision algorithm can benefit from other sensor systems
• IMU to consider motion and position (e.g., visual odometry)
• LIDAR like cheap sensors in ADAS to limit search range
• Consider cloud for heavy offline processing if no latency requirements
Missed Opportunities for HW Assisted Vision

11. Copyright © 2016 Itseez 11
• Algorithm working on RGBD is much easier comparing to RGB
• Consider following inputs
Example — Staircase Detection
RGBD image courtesy of: Reza Farid. Region growing planar segmentation for robot action planning

12. Copyright © 2016 Itseez 12
• Prototype algorithms prior to major work and HW selection
• Understand bottlenecks and challenges  update requirements
• Perform HW selection considering algorithm specifics
• Plan for sync between HW and SW during development
Possible Vision Product Design Model
Algorithm
PoC design Prototype
results
Hardware
approach selection
Hardware
Arch.
Software
drop
Algorithm
development
Hardware
drop
HW
Design
Final
product
Original reqs
Updated
reqs

13. Copyright © 2016 Itseez 13
• Vision in products is still pretty new to consumers
• Designers of the product create it with some use cases in mind
• Consumers usually use them differently
• This gap need attention prior to product release
• Target group testing, people outside of CV area
• Leave some time for corrective actions
• Algorithm enhancements and changes
• Documentation updates, tutorial videos, etc.
Customer Acceptance Need Some Work

14. Copyright © 2016 Itseez 14
• Itseez3D application
• Freely available on iPad, needs sensor (Occipital)
• Just walk around a person and scan to get 3D model
• Consumer use cases needed attention
• Tracking from sensor SDK did not support scan interruption
• Own enhanced tracking was implemented
• UI enhancements and video tutorials were done
• Currently about 300 scans per day
Example — Full Body Scanning

15. Copyright © 2016 Itseez 15
• Most vision products require calibration
• Re-calibration might be needed after some time
• Surveillance cameras change position due to wind/shaking
• Focus/visibility loss in optics due to harsh conditions
• Product tampering
• E.g., cover optics intentionally for some time periods
• Always good to be able to detect those things in software
• Blur & sharp edges detection, circular buffer recording
Maintenance Issues in Vision Products

16. Copyright © 2016 Itseez 16
One Video Tells More Than Few Slides
• Two problems here
• Optics needs maintenance (algorithm works though)
• Low sales figures...

17. Copyright © 2016 Itseez 17
• Do not consider requirements as final
• Periodically update with findings, exceptions
• For complicated algorithms perform PoC and iterate planning
• Perform analysis of existing datasets  plan for dataset collection
• Investigate/prototype algorithm prior to hardware selection if possible
• Consider additional sensors as help to vision algorithms
• Ask for customer feedback prior product release
• Place frame source checks in software to detect maintenance issues
Conclusion: Useful Hints

18. Copyright © 2016 Itseez 18
Q&A

19. Copyright © 2016 Itseez 19
• Itseez Web — http://www.itseez.com
• Itseez3D Scanning app — http://www.itseez3d.com
• Contact me: yury.gorbachev@itseez.com
Resources and Contacts