As the fields of micro- and nano-technology mature, there is going to be an increased need for industrial tools that enable the assembly and manipulation of micro-parts. The feedback mechanism in a future micro-factory will require computer vision. Within the EU IST MiCRoN project, a computer vision software based on Geometric Hashing and the Bounded Hough Transform to achieve recognition of multiple micro-objects was implemented and successfully demonstrated. In this environment, the micro-objects will be of variable distance to the camera. Novel automated procedures in biology and micro-technology are thus conceivable. This paper presents an approach to estimate the pose of multiple micro-objects with up to four degrees-of-freedom by using focus stacks as models. The paper also presents a formal definition for Geometric Hashing and the Bounded Hough Transform.

1. Object Recognition and Real-Time
Tracking in Microscopy Imaging
Object Recognition and Real-Time Tracking in Microscopy
Imaging
Jan Wedekind
30th Aug – 1st Sep 2006
MMVL http://www.shu.ac.uk/research/meri/mmvl/
MiCRoN http://wwwipr.ira.uka.de/~micron/
Mimas http://sourceforge.net/projects/mimas/
MediaWiki http://vision.eng.shu.ac.uk/mediawiki/
People: J. Wedekind, M. Boissenin, B.P. Amavasai, F. Caparrelli, J. Travis

2. MiCRoN
European Union IST project
Uppsala, Lausanne, St. Ingbert, Athens, Pisa, Barcelona, Karlsruhe
Motivation
Control & GUI
• prototype
soldering/assembly
• cell manipulation
• manipulations inside
vacuum chamber
Project Goals
• Manipulate µm-sized
objects
• Closed-loop control of
Universit¨t Karlsruhe (Germany)
a
robot
http://wwwipr.ira.uka.de/~micron/
• 3D object recognition
http://www.cordis.lu/ist/
and tracking

3. Motivation
MiCRoN robot (i)
Locomotion Platform Gripper Rotor
Scuola Superiore, Uppsala University
Sant’Anna (Italy) (Sweden)
Ecole Polytechnique
F´d´rale de Lausanne
e e
(Switzerland)

4. Motivation
MiCRoN robot (ii)
Power Floor, Syringe
PCB
Task Planning
National Technical
University of Athens
University of Barcelona (Greece)
(Spain)
Fraunhofer Institute, St.
Ingbert (Germany)

5. Motivation
MINIMAN vs. MiCRoN robot
MINIMAN III-2 (5 d.o.f.) MiCRoN (4 d.o.f.)
centre of sphere - end effector = 6.2 cm fits on a 20 cent coin

6. Geometric Hashing
1988, Lamdan & Wolfson

7. Geometric Hashing
Preprocessing & Recognition
Preprocessing Recognition
1988, Lamdan & Wolfson

8. Geometric Hashing
2 Degrees-Of-Freedom (i)
Solder Sphere
• Extract Sobel edges
• Randomly select a feature location

9. Geometric Hashing
2 Degrees-Of-Freedom (ii)
Solder Sphere
0 1
1
 
0 0 1
VM (@ A , m )
1 t1
3
1
m  
0 1
t2  = s − m
1
 
VM (@ A , m )
1
0
2 1 4
0
0
m
  s
1
VM (  , m)
1
0 5 5
1
m ”best match
3
with highest m
VM (u, m)”
u
a single feature-correspondence reveals the object’s pose

10. Geometric Hashing
3 Degrees-Of-Freedom (i)
Syringe Chip
• Extract Sobel edges
• Randomly select two feature locations

11. Geometric Hashing
3 Degrees-Of-Freedom (ii)
Syringe Chip
0 1
s1 , s2 : randomly selected pair of scene features
1
VM (@ A , (m1 , m2 ))
m2
1 m1 , m2 : model feature-pair with highest sum of votes
0 1 0
2
0 1 1
m1
feature tuple coord.−system
1
VM (@ A , (m1 , m2 ))
1
m2  
s2
3 2 1
0
1 object coordinate system
1 VM (  , (m1 , m2 ))
m1 1 s1
x2
m2 α
1 4 5 t
3
x1
4 ’’best match
m1 with highest
VM (u, (m1 , m2 ))” m1 m2 t, α: presumed pose of object
u
two feature-correspondences are revealing the object’s pose

12. Bounded Hough Transform
2001/2004, Greenspan, Shang &
Jasiobedzki

13. Bounded Hough Transform
6 d.o.f. satellite tracking
2001/2004, Greenspan, S. & J.

14. Bounded Hough Transform
2 Degrees-Of-Freedom
Accumulate Votes
t2
t1
model features
scene features
local region

15. Bounded Hough Transform
2 Degrees-Of-Freedom
Accumulate Votes
t2
peak in Hough space
t1
model features
scene features
local region

16. Bounded Hough Transform
3 Degrees-Of-Freedom

17. 4 Degrees-Of-Freedom
Artificial Scene

18. Implementation
Software Architecture
Application Layers

19. Implementation
Microstage with Custom-build Camera

20. Implementation
Graphical User Interface

21. Pushing Sugar

22. Results (i)
video reso- time per stack degrees- recog- time per
lution frame size of- nition- frame
(down- (recogni- freedom rate (tracking)
sampled) tion)
dry run
(load
384×288 0.0081 s - - - -
frames
only)
384×288 0.20 s 7 (x, y, z) 88% 0.020 s
160×120 0.042 s 10 (x, y, z, θ) 87% 0.016 s

23. Results (ii)
video reso- time per stack degrees- recog- time per
lution frame size of- nition- frame
(down- (recogni- freedom rate (tracking)
sampled) tion)
384×288 0.27 s 16 (x, y, z, θ) 88% 0.025 s
384×288 0.072 s 14 (x, y, z, θ) 88% 0.018 s
9 (x, y, z, θ) 35%
192×144 0.32 s 0.022 s
1 (x, y, θ) 45%

24. Conclusion
• Depth estimation based on a single image is possible
• Real-time was achieved
– Real-time recognition possible with low recognition-rate
– Low recognition-rate much more tolerable than low frame-rate
– Real-time tracking solves problem of low recognition-rate
• Focus stack must not be self-similar
• Rough surfaces are rich in features

25. Future Work
Problems and Possible Solutions
Problem Solution
Geometric Hashing scales badly with Use RANSAC with Linear Model
number of objects Hashing
High memory requirements, Use local feature context, use less
parametrisation for more than 2 features, use only salient features
objects is difficult
Sub-sampling decreases accuracy Implement non-uniform partitioning
of Hough-space (adaptive accuracy)
Research Topics
• more than 4 degrees-of-freedom
• develop micro-assembly for industrial application
• develop semi-automated supporting microscope tool for biological
application

26. Homepage
MMVL official website MMVL MediaWiki
www.shu.ac.uk/research/meri/mmvl/ vision.eng.shu.ac.uk/mediawiki/
Open source MiCRoN vision software + test data
Open source Mimas real-time computer vision library