1. A Cheap Chess Robot: Perception
Billy Okal, Oliver Dunkley, Andreas N¨chter
u
Automation and Machine Vision Lab
Jacobs University Bremen
May 13, 2011

2. Introduction Methodology Experiments Conclusion and Future Work
Outline
1 Introduction
Motivation
Previous/Related Work
Research Agenda
2 Methodology
Finding the chessboard
Detecting Changes
Determining Moves
3 Experiments
From scene to 2D view
Detect Move and Query Chess Engine
4 Conclusion and Future Work
Billy Okal, Oliver Dunkley, Andreas N¨chter
u A Cheap Chess Robot: Perception

3. Introduction Methodology Experiments Conclusion and Future Work
Motivation Previous/Related Work Research Agenda
Motivation
What is Robot Chess?
Robot playing a game of chess against another robot or human
opponent.
Billy Okal, Oliver Dunkley, Andreas N¨chter
u A Cheap Chess Robot: Perception

4. Introduction Methodology Experiments Conclusion and Future Work
Motivation Previous/Related Work Research Agenda
Motivation
What is Robot Chess?
Robot playing a game of chess against another robot or human
opponent. We focus on the case of a Robot vs a Human
Billy Okal, Oliver Dunkley, Andreas N¨chter
u A Cheap Chess Robot: Perception

5. Introduction Methodology Experiments Conclusion and Future Work
Motivation Previous/Related Work Research Agenda
Motivation
What is Robot Chess?
Robot playing a game of chess against another robot or human
opponent. We focus on the case of a Robot vs a Human
What is Robot Chess Perception?
Robot Perception generally enables the robot to perceive the
changes in the environment using its sensors and interpret sensory
input to make decisions about its next actions.
Billy Okal, Oliver Dunkley, Andreas N¨chter
u A Cheap Chess Robot: Perception

6. Introduction Methodology Experiments Conclusion and Future Work
Motivation Previous/Related Work Research Agenda
Motivation
What is Robot Chess?
Robot playing a game of chess against another robot or human
opponent. We focus on the case of a Robot vs a Human
What is Robot Chess Perception?
Robot Perception generally enables the robot to perceive the
changes in the environment using its sensors and interpret sensory
input to make decisions about its next actions. In our case, the
sensor is a monocular camera, the environment is a chess
game and actions include moving the chess pieces on the
chessboard.
Billy Okal, Oliver Dunkley, Andreas N¨chter
u A Cheap Chess Robot: Perception

7. Introduction Methodology Experiments Conclusion and Future Work
Motivation Previous/Related Work Research Agenda
Previous/Related Work
We highlight a few interesting/related cases;
Billy Okal, Oliver Dunkley, Andreas N¨chter
u A Cheap Chess Robot: Perception

8. Introduction Methodology Experiments Conclusion and Future Work
Motivation Previous/Related Work Research Agenda
Previous/Related Work
We highlight a few interesting/related cases;
The AAAI small scale manipulation(table top chess) challenge
during AAAI2010 and AAAI2011
Billy Okal, Oliver Dunkley, Andreas N¨chter
u A Cheap Chess Robot: Perception

9. Introduction Methodology Experiments Conclusion and Future Work
Motivation Previous/Related Work Research Agenda
Previous/Related Work
We highlight a few interesting/related cases;
The AAAI small scale manipulation(table top chess) challenge
during AAAI2010 and AAAI2011
Modified Chessboard [http://chiara-robot.org/]
Billy Okal, Oliver Dunkley, Andreas N¨chter
u A Cheap Chess Robot: Perception

10. Introduction Methodology Experiments Conclusion and Future Work
Motivation Previous/Related Work Research Agenda
Previous/Related Work
We highlight a few interesting/related cases;
The AAAI small scale manipulation(table top chess) challenge
during AAAI2010 and AAAI2011
Carefully chosen colors for
pieces and board
Non-standard chessboards
Non standard sizes
Modified Chessboard [http://chiara-robot.org/]
Billy Okal, Oliver Dunkley, Andreas N¨chter
u A Cheap Chess Robot: Perception

11. Introduction Methodology Experiments Conclusion and Future Work
Motivation Previous/Related Work Research Agenda
Previous/Related Work
We highlight a few interesting/related cases;
The AAAI small scale manipulation(table top chess) challenge
during AAAI2010 and AAAI2011
Carefully chosen colors for
pieces and board
Non-standard chessboards
Non standard sizes
Modified Chessboard [http://chiara-robot.org/]
Systems that involved mounting the camera directly above the
chessboard as in [Urting:2003] and [Groen:1992]
Billy Okal, Oliver Dunkley, Andreas N¨chter
u A Cheap Chess Robot: Perception

12. Introduction Methodology Experiments Conclusion and Future Work
Motivation Previous/Related Work Research Agenda
Previous/Related Work
We highlight a few interesting/related cases;
The AAAI small scale manipulation(table top chess) challenge
during AAAI2010 and AAAI2011
Carefully chosen colors for
pieces and board
Non-standard chessboards
Non standard sizes
Modified Chessboard [http://chiara-robot.org/]
Systems that involved mounting the camera directly above the
chessboard as in [Urting:2003] and [Groen:1992]
A fews humanoids like REEM-A(30 dof, stereo-cam) by Pal
Robotics. etc
Billy Okal, Oliver Dunkley, Andreas N¨chter
u A Cheap Chess Robot: Perception

13. Introduction Methodology Experiments Conclusion and Future Work
Motivation Previous/Related Work Research Agenda
Research Agenda
Common Feature of Previous Work
Billy Okal, Oliver Dunkley, Andreas N¨chter
u A Cheap Chess Robot: Perception

14. Introduction Methodology Experiments Conclusion and Future Work
Motivation Previous/Related Work Research Agenda
Research Agenda
Common Feature of Previous Work
Constraints imposed on the chessboard and pieces to simplify the
problem.
Billy Okal, Oliver Dunkley, Andreas N¨chter
u A Cheap Chess Robot: Perception

15. Introduction Methodology Experiments Conclusion and Future Work
Motivation Previous/Related Work Research Agenda
Research Agenda
Common Feature of Previous Work
Constraints imposed on the chessboard and pieces to simplify the
problem.
Which of these constraints can we relax/get rid of?
Billy Okal, Oliver Dunkley, Andreas N¨chter
u A Cheap Chess Robot: Perception

16. Introduction Methodology Experiments Conclusion and Future Work
Motivation Previous/Related Work Research Agenda
Research Agenda
Common Feature of Previous Work
Constraints imposed on the chessboard and pieces to simplify the
problem.
Which of these constraints can we relax/get rid of?
Our Attempt
Allow for camera to move relative to chessboard (including
moving out of the scene)
Use a un-modified chessboard
Use standard chess piece set (Staunton standard tournament
pieces No 5 - from Amazon)
All the above using cheap camera (a USB webcam) and basic
hardware (4 dof + 1 for gripper).
Billy Okal, Oliver Dunkley, Andreas N¨chter
u A Cheap Chess Robot: Perception

17. Introduction Methodology Experiments Conclusion and Future Work
Finding the chessboard Detecting Changes Determining Moves
Methodology
Finding the chessboard in the scene
Detecting Changes on the chessboard (moves)
Decoding the changes to identify user moves
Generating new moves and transform these into free space
poses for planning
Billy Okal, Oliver Dunkley, Andreas N¨chter
u A Cheap Chess Robot: Perception

18. Introduction Methodology Experiments Conclusion and Future Work
Finding the chessboard Detecting Changes Determining Moves
Finding the chessboard in the scene
Algorithm 1: Detect Board in the Scene
Data: Raw Scene Image (img) - 800x600
Result: 4 Corners of the chessboard
begin
img edges ←− EdgeDetect(img );
all lines ←− LineDetect(img edges);
MergeCloseLines(all lines);
bounding lines ←− FindBoundingLines(all lines);
corners ←− ComputerIntersections(bounding lines);
return corners;
end
Billy Okal, Oliver Dunkley, Andreas N¨chter
u A Cheap Chess Robot: Perception

19. Introduction Methodology Experiments Conclusion and Future Work
Finding the chessboard Detecting Changes Determining Moves
Finding the chessboard in the scene
Algorithm 2: Detect Board in the Scene
Data: Raw Scene Image (img) - 800x600
Result: 4 Corners of the chessboard
begin
img edges ←− EdgeDetect(img );
all lines ←− LineDetect(img edges);
MergeCloseLines(all lines);
bounding lines ←− FindBoundingLines(all lines);
corners ←− ComputerIntersections(bounding lines);
return corners;
end
Billy Okal, Oliver Dunkley, Andreas N¨chter
u A Cheap Chess Robot: Perception

20. Introduction Methodology Experiments Conclusion and Future Work
Finding the chessboard Detecting Changes Determining Moves
Finding the chessboard in the scene
Algorithm 3: Detect Board in the Scene
Data: Raw Scene Image (img) - 800x600
Result: 4 Corners of the chessboard
begin
img edges ←− EdgeDetect(img );
all lines ←− LineDetect(img edges);
MergeCloseLines(all lines);
bounding lines ←− FindBoundingLines(all lines);
corners ←− ComputerIntersections(bounding lines);
return corners;
end
Billy Okal, Oliver Dunkley, Andreas N¨chter
u A Cheap Chess Robot: Perception

21. Introduction Methodology Experiments Conclusion and Future Work
Finding the chessboard Detecting Changes Determining Moves
Project the chessboard to a 2D view
Homography/Projective Transformation
Given a 2D point p = (x, y ) on the image plane, find a matrix H
such that the point p = (x , y ) given by p = Hp lies on a 2D
plane P 2 of choice.
In homogenous coordinates we have;
a b
p = (a, b, c), s.t x = , y= and c = 0 (1)
c c
a b
p = (a , b , c ), s.t x = , y = and c = 0 (2)
c c
    
a h1 h2 h3 a
 b  =  h4 h5 h6   b  (3)
c h7 h8 h9 c
[Hartley:2000]
Billy Okal, Oliver Dunkley, Andreas N¨chter
u A Cheap Chess Robot: Perception

22. Introduction Methodology Experiments Conclusion and Future Work
Finding the chessboard Detecting Changes Determining Moves
Project the chessboard to a 2D view
We plugin all the correspondences for the four corners and arrive at
the following system;
    
a1 b1 1 0 0 0 −a1 a1 −b1 a1 h1 a1

 a2 b2 1 0 0 0 −a2 a2 −b2 a2 
 h2  
  a2 


 a3 b3 1 0 0 0 −a3 a3 −b3 a3 
 h3  
  a3 


 a4 b4 1 0 0 0 −a4 a4 −b4 a4 
 h4  
= a4 
 (4)

 0 0 0 a1 b1 1 −a1 b1 −b1 b1 
 h5  
  b1 


 0 0 0 a2 b2 1 −a2 b2 −b2 b2 
 h6  
  b2 

 0 0 0 a3 b3 1 −a3 b3 −b3 b3  h7   b3 
0 0 0 a4 b4 1 −a4 b4 −b4 b4 h8 b4
Which then can be solved using a number of well established
algorithms.
Billy Okal, Oliver Dunkley, Andreas N¨chter
u A Cheap Chess Robot: Perception

23. Introduction Methodology Experiments Conclusion and Future Work
Finding the chessboard Detecting Changes Determining Moves
Project the chessboard to a 2D view
We plugin all the correspondences for the four corners and arrive at
the following system;
    
a1 b1 1 0 0 0 −a1 a1 −b1 a1 h1 a1

 a2 b2 1 0 0 0 −a2 a2 −b2 a2 
 h2  
  a2 


 a3 b3 1 0 0 0 −a3 a3 −b3 a3 
 h3  
  a3 


 a4 b4 1 0 0 0 −a4 a4 −b4 a4 
 h4  
= a4 
 (4)

 0 0 0 a1 b1 1 −a1 b1 −b1 b1 
 h5  
  b1 


 0 0 0 a2 b2 1 −a2 b2 −b2 b2 
 h6  
  b2 

 0 0 0 a3 b3 1 −a3 b3 −b3 b3  h7   b3 
0 0 0 a4 b4 1 −a4 b4 −b4 b4 h8 b4
Which then can be solved using a number of well established
algorithms. In our implementation we make use of a OpenCV
methods cv::getPerspectiveTransform and
cv::warpPerspective.
[Hartley:2000]
Billy Okal, Oliver Dunkley, Andreas N¨chter
u A Cheap Chess Robot: Perception

24. Introduction Methodology Experiments Conclusion and Future Work
Finding the chessboard Detecting Changes Determining Moves
Detecting Changes on the chessboard(moves)
Locate blobs/change spots
1 Difference image (Given two images
f (x, y ) at time t1 and f (x , y ) at
time t2 , difference is
g (x, y ) = |f − f |, [Farahat:2003]
2 Remove noise by erosion and dilation
3 Distance Transform (label pixels with
distance to background)
4 Find the pixel whose label is largest
5 Flood fill the pixel above
6 Repeat steps 3,4,5 until threshold
Map change spots to board changes.
Use image dimensions and know chess
square sizes to place changes on board
squares. Generate change strings.
Billy Okal, Oliver Dunkley, Andreas N¨chter
u A Cheap Chess Robot: Perception

25. Introduction Methodology Experiments Conclusion and Future Work
Finding the chessboard Detecting Changes Determining Moves
Detecting Changes on the chessboard(moves)
Locate blobs/change spots
1 Difference image (Given two images
f (x, y ) at time t1 and f (x , y ) at
time t2 , difference is
g (x, y ) = |f − f |, [Farahat:2003]
2 Remove noise by erosion and dilation
3 Distance Transform (label pixels with
distance to background)
4 Find the pixel whose label is largest
5 Flood fill the pixel above
6 Repeat steps 3,4,5 until threshold
Map change spots to board changes.
Use image dimensions and know chess
square sizes to place changes on board
squares. Generate change strings.
Billy Okal, Oliver Dunkley, Andreas N¨chter
u A Cheap Chess Robot: Perception

26. Introduction Methodology Experiments Conclusion and Future Work
Finding the chessboard Detecting Changes Determining Moves
Detecting Changes on the chessboard(moves)
Locate blobs/change spots
1 Difference image (Given two images
f (x, y ) at time t1 and f (x , y ) at
time t2 , difference is
g (x, y ) = |f − f |, [Farahat:2003]
2 Remove noise by erosion and dilation
3 Distance Transform (label pixels with
distance to background)
4 Find the pixel whose label is largest
5 Flood fill the pixel above
6 Repeat steps 3,4,5 until threshold
Map change spots to board changes.
Use image dimensions and know chess
square sizes to place changes on board
squares. Generate change strings.
Billy Okal, Oliver Dunkley, Andreas N¨chter
u A Cheap Chess Robot: Perception

27. Introduction Methodology Experiments Conclusion and Future Work
Finding the chessboard Detecting Changes Determining Moves
Determining chess moves from the changes
Convert chessboard changes into moves
Compare every pair of changes say a2b3 against an 8 × 8 state
matrix and return a move string
Repeat over the whole change string
Billy Okal, Oliver Dunkley, Andreas N¨chter
u A Cheap Chess Robot: Perception

28. Introduction Methodology Experiments Conclusion and Future Work
Finding the chessboard Detecting Changes Determining Moves
Determining chess moves from the changes
Convert chessboard changes into moves
Compare every pair of changes say a2b3 against an 8 × 8 state
matrix and return a move string
Repeat over the whole change string
Query chess engine for moves based on detected moves
Using standard chess engine communication protocol (UCI -
Universal Chess Interface).
Communication handles via ROS messages
Billy Okal, Oliver Dunkley, Andreas N¨chter
u A Cheap Chess Robot: Perception

29. Introduction Methodology Experiments Conclusion and Future Work
Finding the chessboard Detecting Changes Determining Moves
Determining chess moves from the changes
Convert chessboard changes into moves
Compare every pair of changes say a2b3 against an 8 × 8 state
matrix and return a move string
Repeat over the whole change string
Query chess engine for moves based on detected moves
Using standard chess engine communication protocol (UCI -
Universal Chess Interface).
Communication handles via ROS messages
Convert suggested moves into poses in free space
Poses are defines as a pair of Position p = (x, y , z)T and
Orientation q = (x, y , z, w )T to allow for design freedom in
planning.
Using chessboard size, we interpret a move string say a2b3
into respective pose say p = (24, 14, 2)T q = (0, 0, 0, 1)T
Billy Okal, Oliver Dunkley, Andreas N¨chter
u A Cheap Chess Robot: Perception

30. Introduction Methodology Experiments Conclusion and Future Work scene to 2D view Detect Move and Query Chess Engine
From
Find Board and project it to 2D View
[[Movie]]
Billy Okal, Oliver Dunkley, Andreas N¨chter
u A Cheap Chess Robot: Perception

31. Introduction Methodology Experiments Conclusion and Future Work scene to 2D view Detect Move and Query Chess Engine
From
Determine Move and query Chess Engine
Billy Okal, Oliver Dunkley, Andreas N¨chter
u A Cheap Chess Robot: Perception

32. Introduction Methodology Experiments Conclusion and Future Work
Conclusion and Future Work
Conclusions
We have demonstrated that the constraint of using a modified
chessboard and/or pieces can be relaxed without compromising
performance
We demonstrated on very minimalistic hardware, use of better
hardware should produce even better performance
Camera need not be fixed
Billy Okal, Oliver Dunkley, Andreas N¨chter
u A Cheap Chess Robot: Perception

33. Introduction Methodology Experiments Conclusion and Future Work
Conclusion and Future Work
Conclusions
We have demonstrated that the constraint of using a modified
chessboard and/or pieces can be relaxed without compromising
performance
We demonstrated on very minimalistic hardware, use of better
hardware should produce even better performance
Camera need not be fixed
Future Work
Investigate how the task could be solved multiple cameras
(stereo vision)
Adding other types of sensors (e.g. Kinect)
Add object recognition modules
Improve the blob detection module
Billy Okal, Oliver Dunkley, Andreas N¨chter
u A Cheap Chess Robot: Perception

34. Introduction Methodology Experiments Conclusion and Future Work
The End
Thank you for your attention
Questions/Comments?
Special Thanks
DAAD/PROFIN,
Automation Group Members
For further information check
B.Okal and O.Dunkley, ”A Cheap Chess Robot: Planning and Perception”,
Bachelor Thesis 2011.
Software Tools
http://www.ros.org
http://opencv.willowgarage.com
http://openrave.programmingvision.com
Billy Okal, Oliver Dunkley, Andreas N¨chter
u A Cheap Chess Robot: Perception