Towards Embedded Computer Vision邁向嵌入式電腦視覺

本著作採用創用CC 「姓名標示」授權條款台灣3.0版

Towards
Embedded
Computer Vision
Wang, Yuan-Kai(王元凱)
Electronic Engineering Department,
Fu Jen Univ. (輔仁大學電機工程系)
Email: ykwang@mails.fju.edu.tw
URL: http://www.ykwang.tw
2010/05/14

王元凱 Towards Embedded Computer Vision p. 2

Contents
1. Embedded Systems
2. Embedded Computer Vision
3. Entertainment Robot (CPU)
4. Embedded Vision Sensor (CPU)
5. Portable Vision Device (DSP)
6. Smart Video Surveillance (FPGA)
7. Summary & Outlook

Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai


1. Embedded Systems



Evolution of Computer
Past Now Future

• Embedded System is a computer that is
• Special-purpose
• Light, Thin, Short, Small
⇒ Limited resources


Embedded Systems
"Without" Sensors

資料來源：資策會MIC ITIS計畫整理


Embedded Systems
"With" Sensors

Wii
Roomba

GPS Exoskeleton
Navigation



Embedded Systems
"With" Image Sensors
DARPA
Augmented
Grand
Reality
Challenge

Surface Intelligent
Computing Robot



2. Embedded
Computer Vision (ECV)



What Is ECV
 Embedded compute vision
 Implements computer vision algorithms
on low-cost, low-power,
constrained hardware
 Constrained hardware
 Low-speed CPU
 Low capacity memory
 No floating-point (FPU)
 Low-resolution image sensor



Embedded Computer Vision
 Embedded System + Camera
+ Computer Vision Algorithm

Image Image Image
Capturing Processing Recognition



Why Smart Camera (1/2)
 Front-end processing
 An example for video surveillance
Classical stationary camera

Smart camera

IOImage Inc.


Why Smart Camera (2/2)
 In-node processing:
Vision sensor network
 Distributed vision system
 Camera networks
 Use multiple cameras
to analyze the scene
 Benefit
 Less problems with
occlusion
 Challenge
 Distributed processing
and reasoning


International Activities (1/2)
 Special conferences
 IEEE Int. Conf. Distributed Smart Cameras
 Special journal issues
 IEEE Journal of Selected Topics in Signal
Processing, vol. 2, no. 4, Aug. 2008
 EURASIP Journal on Embedded Systems,
 Short courses in important CV
conferences
 CVPR07&08: Distributed vision processing in
smart camera networks
 ESC07: Embedded CV and smart cameras
 ICASSP09: Distributed processing in smart
cameras


International Activities (2/2)
 Research projects and Lab.
 Princeton Univ./Georgia Tech.:
Embedded Systems Lab., Wayne Wolf
 Stanford Univ.
Wireless sensor networks Lab.
 UCLA, CMU, MIT
 Delft Univ. of Technology
SmartCam Project
 Graz Univ. of Technology



Three Ways for ECV
 CPU (Central Processing Unit)
 ARM, PowerPC
 DSP (Digital Signal Processor)
 TI, ADI, NXP
 FPGA (Field Programmable Gate Array)
 Altera, Xilinx



Embedded CPU
 Embedded CPU = Low-power CPU
 ARM
 Drawbacks of embedded CPU
for computer vision
 No FPU, usually fixed-point
 Speed: 60MHz ~ 600MHz
 Therefore it is usually developed for
(video) sensor networks



Embedded CPU: MeshEye
 Stanford MeshEye (http://wsnl.stanford.edu/smartcam.html)
 ARM 7 (55MHz), ZigBee node
 3 image sensors "MeshEye:a hybrid-resolution
 30x30 grayscale x 2 smart camera intelligent
in distributed
mote for applications
 640x480 color x 1 surveillance", IPSN-SPOTS, 2007

Object detection



Embedded CPU: CMUCam
 CMU CMUcam3
 ARM7
 ECV applications
 Robotic vision, color tracking,
histogram processing, face detection

(http://www.cmucam.org)


DSP
 DSP is good for signal processing
 SIMD structure for filtering processing
 However, Computer Vision needs
extreme DSP + video port
 Media processors: powerful DSP
 Parallelism: VLIW
 Faster memory, DMA
 Wide data bus
 ECV applications
 Face detection, face recognition, license
plate recognition, vehicle tacking,


DSP : TRICam
 "Visual surveillance on DSP-based
embedded platform," Graz Univ. of
Technology, 2008(Phd. dissertation)
 TI C6414 (600MHz)
 Applications: Adaboost face detection, vehicle
detection, license plate detection



FPGA (1/3)
 Customizable hardware for parallelism
 Reconfigurable computing
 Flexible hardware design by HDL codes on
FPGA.



FPGA (2/3)
 "Hardware, Design and Implementation
Issues on a FPGA-Based Smart Camera,"
ICDSC, 2007



FPGA (3/3)
 ECV application:
object tracking
 Template matching
by MAE
(Maximum
Absolute Error)



Hybrid: MCU+CPLD
 UCLA Cyclops http://research.cens.ucla.edu/projec
ts/2007/Multiscaled_Actuated_Sens
 MCU: Atmega128 ing/Cyclops/

 CPLD: image processing
 ECV app.: Hand gesture recognition



Hybrid: CPU+DSP (1/2)
 "Distributed Embedded Smart Cameras
for Surveillance Applications,"
IEEE Computer, 2006.

Developed for traffic surveillance


Hybrid: CPU+DSP (2/2)



SOC - Xetal
 Philips: Xetal SIMD processor
 InCa
 WiCa
 Xetal3 + 8051
 Stereo sensors (640x480)
 50 GOPS @ 600mWatt
 ZigBee node
 C++ programming
 Used by Stanford Univ.,
Delft University of Technology
 ECV Applications: edge detection, face
detection, hough transform, gesture
recognition, depth estimation, ...


Challenges for ECV
 Algorithm refinement
 Parallel computation
 Function partition, Multi-threading
 Stream processing
 Memory hierarchy optimization
 Hardware design
 Pipeline, SIMD, board design
 Optimized programming skills
 Fixed-point arithmetic
 Memory management
 Intrinsic commands


Our Current Results
 Entertainment robot (CPU)
 Sony AIBO robot with 64-bit RISC CPU
 Application: Game playing,
Face detection/recognition,
Facial expression recognition
 Smart vision sensor: (CPU)
 Self-made sensor with ARM7
 Application : Face detection, Robot
 Portable vision device (DSP)
 Self-made device with Dualcore DSP
 Application: gesture recognition
 Smart video surveillance (FPGA)
 Background subtraction with FPGA
 Application: video surveillance


3. Entertainment Robot

CPU Solution



Two AIBO Models
 Hardware ERS-7
 CPU: 64-bit RISC
576 MHz/192Mhz
 RAM: 64MB/32MB
 Flash: 32MB
 20 motors
 Camera:
 CMOS sensor
 Resolution:
280 × 160 ERS220A
 10 fps


Sensors



Software
 Operating System
 Aperios (Embedded Linux)
 Development tools
 C++
 GCC 3.4.4 on Linux
 Libraries: OpenR, Tekkotsu



Our AIBO Pet
 Rolling Dice
 Face Detection
 Face Recognition
 Facial Expression Recognition



Rolling Dice (1/3)



Rolling Dice (2/3)
 Detect the dice by color detection
 Using Gaussian mixture model (GMM)
and EM algorithm to model colors
p( x | c) =
1
N
1 ( − ( x − mi )T ∑i−1 ( x − mi ))
∑ω
i =1
i
2π ∑i
1/2
e 2

Original Detection
image result

GMM Morphology+CCL


Rolling Dice (3/3)



Face Detection (1/2)



Face Detection (2/2)
 Algorithm:
Adaboost face detection
 Proposed by Viola and Jones in 2001
 Cascaded weak classifiers



Face Recognition (1/3)



 Eigenface approach (PCA)



Facial Expression Recognition 1/3



 3 expressions
 Happy, Surprise, Angry
 Video-based method
 Feature: optical flow,
common vector flow
 Classifier: Hidden Markov model
 Well done in small-resolution images
 140 * 120 (~ QCIF, 176 * 144)



4. Embedded Vision Sensor

CPU Solution



FJUCam
 It is a self-made camera
 Camera module + embedded system
 3S: Small, Smart, Sensing
 Components of the FJUCam
 ARM7 TDMI
 32-bit, 60MHz, 64KB RAM, 128KB
ROM
 CMOS sensor: OV6620 (CIF 50fps)
 CIF(352x288), QCIF, 8-bit RGB



What Is FJUCam (1/3)
• Weight: 35gm
• Power sources: •Size:
• 5V DC current 6 x 4.5 x 5 (cm)
• 5V Cell Battery (W x H x D)
• Power
consumption:
1W

Three Modules
1. Main board, 2. Lens module
3. Storage module


Power Main Board Button/ISP

Serial port
Switch

ARM7
Microcontroller
Frame buffer/FIFO

Front Side
Other interfaces: RS232x2, SPI, I2C, GPIO Back Side 49



Lens Module Storage Module



Software
 Development environment
 C Language
 PC Windows + Cygwin + GCC
 cc3 library (open source developed by
CMU)



Face Detection
 The Adaboost algorithm for face detection
 Proposed by Viola and Jones in 2001
 Cascaded weak classifiers(21 cascades)
 Algorithm refinement
 Reduced to 5 cascades
 Fixed-point arithmetic
 Stream processing for only 64KB memory
utilization

Image
FJUCam Display
Face Detection


Cyclops Robot
 Color tracking



5. Portable Vision Device:
X-EYE
DSP Solution



Goal of X-Eye

55


Components

觸控面板

顯示器移動電源

自製 Camera
USB
外殼 USB 筆電
連接線
Hub
BeagleBoard

微投影機 SD卡 USB-WIFI卡

鍵盤

USB-RS232
讀卡機
Fu Jen University 控制線 Department of Electrical Engineering
滑鼠 Wang, Yuan-Kai


1st Generation Prototype



Photo Mode

Capture Command: capture images

Switch Command: Mode switch


Manage Mode
Original Photos

Next Command

Previous Command

Switch Command 2(to photo)
59


Hardware (1/2)
USER RESET

OMAP3530 Processor
Peripheral I/O •600MHz Cortex-A8
•NEON+VFPv3
•USB Host •16KB/16KB L1
•256KB L2
•JTAG •430MHz C64x+ DSP
•32K/32K L1
•DVI-D video out •48K L1D
•32K L2
•S-Video out •Power VR SGX GPU
•64K on-chip RAM
•SD/MMC+ POP Memory
•256MB LPDDR RAM
•Stereo in/out •256MB NAND flash

•RS-232 serial1

•Alternate power

•USB 2.0 HS OTG
7.6 cm
Fu Jen University
60
Department of Electrical Engineering Wang, Yuan-Kai


Hardware (2/2)

Fu Jen University Department of Electrical Engineering 2010.04.25
Wang, Yuan-Kai
61


Software (1/2)



Software (2/2)
軟體名稱版本功用簡述
Gesture Command
1.0 手勢辨識
Module
OpenCV 1.0 影像處理
FFMpeg 0.5.1 視(音)訊邊解碼
QT 4.6.2 視窗介面
WinXP下安裝Linux
VMWare 6.5.3
工具
Ubuntu (host) 9.04 安裝交叉編譯環境
Ubuntu (client) 9.10 BB上的filesystem
Kernel (client)
Fu Jen University
2.6.29
Department of Electrical Engineering
BB上的kernel
2010.04.25 X-Eye
Wang, Yuan-Kai
63


Algorithm (1/2)
 Gaussian Mixture Model (GMM)
 Model four colors
1
N
1 ( − ( x − mi )T Covi−1 ( x − mi ))
p ( x | c) = ∑ ωi e 2

i =1 2π || Covi ||1/2

 Expectation Maximization (EM)
 Parameter estimation of GMM
E Step M Step
N
1
∑ E(z
N
1
ω p( x j | m , C ) ωit +1 = t +1
) m = ∑ E(z
t t t
)x j
E ( zij ) = Nω t +1
i i i ij i ij
i N j =1 i j =1

∑ ω tp p( x j | mtp , C tp )
=C t +1 1
t +1 ∑
N
E ( zij )[( x j − mit +1 )( x j − mit +1 )T ]
p =1
N ωi
i
j =1

Fu Jen University
64


Algorithm (2/2)
 Color Identification
 c
ˆ =
arg max P( x | c j ), j 1 ~ k
cj

 Performance optimization by Look Up
Table (LUT) for real-time
 Gesture Recognition
 Four gestures: capture, switch, next,
previous,

Fu Jen University
65


6. Smart Video Surveillance

FPGA Solution



Our Experience

Background subtraction, ...
• 2.8 GHz Intel CPU
• Software: C/C++ FPGA
• Frame rate: 10 fps


Background Subtraction
Current
Frame Bk + 1

Background
M k +1 ( x, y ) P k+1 Image Update
Background Image
= Pk +1 ( x, y ) − Bk ( x, y ) -
Bk
M
k + 1
Bk +1 ( x, y )
= αBk ( x, y ) + (1 − α ) Pk +1 ( x, y )
Post Processing

Motion Object Image

Speed up by (1) Circuit design, (2) Paralization


by FPGA (1/3)
 Parallelism: 7-level pipeline
 SIMD with stream processing



by FPGA (2/3)
 Hardware: Altera Cyclone II 2C35
 Design: Verilog HDL with Quartus II Tools

Background New Frame Result
Frame rate
• Background module : 368 fps
• Whole system : 51 fps


by FPGA (3/3)
 Comparisons
 PC: 2.8GHz CPU, C implementation
 FPGA can speed up 500 times
2.8G
51

CPU
FPGA
25M 10

Clock(Hz) FPS


7. Summary and
Future Research Directions



Summary (1/2)
 Embedded CPU is not appropriate for
computer vision
 Although CPU has great flexibility and
programming environment
 Its architecture is interrupt-based
 Designed for I/O-process usage
 Not for data-intensive computing, such as
DSP and image/video processing



Summary (2/2)
 High-performance processor is
necessary for computer vision
 Clock rate is not the crucial point
 But SIMD and algorithm parallelism
do



Questions
 Embedded compute vision
 Low-cost, low-power, Constrained
minimal hardware Resource
 High-Performance Contradiction
computer vision
 Fast speed without cost,
Abundant
power, and hardware Resource
constraints
From contradiction to convergence ?!
Yes by multicore


Challenges (1/2)
 Algorithm decomposition
 Function decomposition
 Partition serial part and parallel part
 Data flow analysis and
data dependency analysis
 Parallelism
 Loop unrolling
 Multithreading
 Pipeline



Challenges (2/2)
 Performance analysis method
 For efficiency improvement
 Implementation efforts
 Choose a good embedded platform for
computer vision
 Software issues
 Hardware issues
 Programming skills
 Multi-threading



Future Research Directions
 Multicore framework
 DSP + CPU : FJUCam2
 Advantage: Using C language
 Challenge: algorithm parallelism
 FPGA + CPU:
 Advantage
 Reconfigurable multicore
 Less Verilog, more C
 Challenge: hardware/software co-design
 GPGPU
 Advantage:
 240~512 cores
 Using C language
 Challenge: algorithm parallelism


FJUCam2 (1/2)
 Next generation FJUCam (MPSoC)
 Adopt multicore technology
 DSP(Media) + CPU
99/6 Image
RAM Resolution
128MB
FJUCam2 VGA

98/2

64KB FJUCam1
CIF
Processor
60MHz 600MHz



FJUCam2 (2/2)
 Algorithms going to be developed
for FJUCam2
 Color tracking
 Gesture recognition
 Face tracking and recognition
 Event detection
 Video summarization
 Sleep monitoring
 Distributed vision processing



Reconfigurable Multi-Core
 FPGA + CPU + Linux
Hardware Software
DE2_70

FPGA
Chip

CMOS
Nios II Processer
Controller

Background
RGB to Y
Subtraction Avalon Bus

RAW
Flash SDRM1 SSRAM DM9000A
to Morphology
Controller Controller Controller Controller
RGB

CMOS CCD
SDRAM0 Flash SDRAM1 SSRAM DM9000A
Capture

Internet

VGA PC



The End

Free for Any Questions



Our Development Boards
Xscale270 Beagle board TI DSP DaVinci 6446

Altera DE2-70 Celoxica RC10+DK



Robotic Vision


王元凱 Towards Embedded Computer Vision p.

本簡報授權聲明
 此簡報內容採用 Creative Commons 「姓名標示 - 非商
業性台灣 3.0 版」授權條款
 歡迎非商業目的的重製、散布或修改本簡報的內容，但
請標明： (1)原作者姓名：王元凱； (2)圖標示：
 簡報中所取用的部份圖形創作乃截取自網際網路，僅供
演講者於自由軟體推廣演講時主張合理使用，請讀者不
得對其再行取用，除非您本身自忖亦符合主張合理使用
之情狀，且自負相關法律責任。


Towards Embedded Computer Vision邁向嵌入式電腦視覺

Recommended

Recommended

More Related Content

What's hot

What's hot (11)

Similar to Towards Embedded Computer Vision邁向嵌入式電腦視覺

Similar to Towards Embedded Computer Vision邁向嵌入式電腦視覺 (20)

More from IEEE International Conference on Intelligent Information Hiding and Multimedia Signal Processing

More from IEEE International Conference on Intelligent Information Hiding and Multimedia Signal Processing (16)

Recently uploaded

Recently uploaded (20)

Towards Embedded Computer Vision邁向嵌入式電腦視覺