Pen-Based Gesture Recognition

Pen-Based Gesture Recognition

Beat Signer
Institute for Information Systems, ETH Zurich
8092 Zurich, Switzerland

http://vub.academia.edu/BeatSigner

USI Seminar, May 30, 2007

Outline
 Motivation
 Digital Pen and Paper Functionality
 Gesture recognition algorithms
 Evaluation of algorithms
 iGesture framework

USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 2

Pen-Based User Interfaces

Light Pen Graphics Tablet

PDAs and Tablet PCs Digital Pen and Paper

Palm Graffiti Gestures


Microsoft Application Gestures
scratch-out erase content double-circle paste

triangle insert left-semicircle undo

square action item right-semicircle redo

star action item caret past/insert

check check-off inverted-caret insert

curlicue cut chevron-left application-specific

double-curlicue copy chevron-right application-specific

circle application-specific arrow-up application-specific


Customised Gestures


Gesture Design Guidelines
 Different gestures should not look too similar
 better recognition results
 Gestures should be user friendly
 intuitivity and usability
 Gesture set size
 Reuse of gestures
 same semantics for different applications
 application-specific gestures


Digital Pen and Paper
 Developed by Anoto
 Different pen manufacturers
 Logitech, Maxel, Nokia, ...

 Pattern space
 60 million km2


Digital Ink Processing
 Parse proprietary pen formats
 position, timestamp, force, tilt, …
 Stroke detection
 pen up / down events
 time difference between succeeding points
 distance between succeeding points, …
 Interpolation / Filtering


Note Representation
<note>
<trace>
<point>
<x>59.664</x><y>56.364</y>
<timestamp>1103123178464</timestamp>
<force>224.0</force>
</point>
<point>
<x>59.671</x><y>56.378</y>
<timestamp>1103123178484</timestamp>
<force>227.0</force>
</point>
...
<trace>
...
</note>


Ink Markup Language (InkML), W3C
 XML language for representing digital ink
 Pen movement data
 <trace> and <traceFormat> elements
 x,y coordinates, force, angles
 Device-specific information
 resolution, sample rate, …


Ink Markup Language (InkML)
<ink>
<trace>10 0, 9 14, 8 28, 7 42, 6 56, 6 70, 8 84, 8 98,
8 112, 9 126, 10 140, 13 154, 14 168, 17 182, 18 188
</trace>
<trace>130 155, 144 159, 158 160, 170 154, 179 143,
179 129, 166 125, 152 128, 140 136, 131 149, 126 163
</trace>
...
<inkSource manufacturer="magicomm" model="G303">
<sampleRate uniform="true" value="70"/>
<srcProperty name="weight" value="100" units="g"/>
</inkSource>
...
</ink>


Gesture Recognition Algorithms
 Offline recognition algorithms
 static image
 Online recognition algorithms
 spatio-temporal representation
 Recognition methods
 statistical classification, neural networks, …
 Supported gesture types
 single-stroke or multi-stroke


SiGeR Algorithm
 Simple Gesture Recogniser (Scott Swigart)
 Gestures described by regular expressions
 N, NE, E, SE, S, SW, W and NW
W

(E)+(N)+(W)+(S)+

S N
(NE|E|SE)+(NW|N|NE)+
(SW|W|NW)+(SE|S|SW)+
E


SiGeR Algorithm ...
 Use additional statistical information
Description = Directions [";" Constraints];
Directions = Direction ["," Directions];
Direction = "N"|"NE"|"E"|"SE"|"S"|"SW"|"W"|"NW";
Constraints = Constraint ["AND" Constraints];
Constraint = Operand Operator Operand;
Operator = "EQ"|"NEQ"|"GT"|"GTE"|"LT"|"LTE";
Operand = "N"|"NE"|"E"|"SE"|"S"|"SW"|"W"|"NW"|
"DIAGONAL"|"STRAIGHT"|"PROXIMITY"|"STOPPOINTS";


SiGeR Algorithm ...
W

 Examples
E,N,W,S;STRAIGHT GT 0.8 S N
AND PROXIMITY LT 0.2

E

W,SW,SE,NE,N,S;
PROXIMITY GT 0.7
AND STRAIGHT GT 0.3


Rubine Algorithm, 1991
 Statistical classification algorithm for single
stroke gestures (training / classification)
 A gesture G is represented as vector of P
sample points
G  s0 ,... sP1, with si  xi , yi , ti 

 Feature vector f extracted from G
f   f1 ,... f F 

Rubine Features

( x2  x0 )
f1  cos  
( x2  x0 ) 2  ( y2  y0 ) 2
( y 2  y0 )
f 2  sin  
( x2  x0 ) 2  ( y2  y0 ) 2
f 3  ( xmax  xmin ) 2  ( ymax  ymin ) 2
ymax  ymin
f 4  arctan
xmax  xmin
f5  xP 1  x0 2   yP 1  y0 2
( xP 1  x0 )
f 6  cos  
f5


Rubine Features …

f 7  sin  
 yP 1  y0 
f5
Let xi  xi 1  xi yi  yi 1  yi
P2
f 8   xi  yi
2 2

i 0

xi yi 1  xi 1yi
Let  i  arctan
xi xi 1  xi yi 1
P2 P 2 P2
f9  i f10    i f11    i
2

i 1 i 1 i i

x  y
P 2 2 2
Let ti  ti 1  ti f12  max i 2 i
i 0 ti
f13  t P 1  t0

Rubine Training / Classification
 Training phase
gesture samples
for class c
Optimal
Classifier
wc  wc 0 ,..., wcF 
ˆ ˆ ˆ

 Recognition / classification phase
F
vcˆ  wcˆ 0   wcˆi f i
i 1


SiGrid Algorithm
 Signature algorithm
000000 000001 000101 000100 100100 100101 100001 100000

000010 000011 000111 000110 100110 100111 100011 10010

001010 001011 001111 001110 101110 101111 101011 101010

001010 001001 001101 001100 101100 101101 101001 101000
Example
011000 011011 011111 011110 111100 111101 111001 111000 8 x 8 Grid
011010 011011 0111111 011110 111110 111111 111011 111010

010010 010011 010111 010110 110110 110111 110011 110010

010000 010001 010101 010100 110100 110101 110001 110000


Evaluation Grafitti Numbers
E-Rubine Rubine SiGrid SiGeR

Correct 147 134 132 133

Error 3 15 15 4

Reject 0 1 3 13

Number of gesture classes: 10
Training: 15 examples for each gesture class (collected by 1 person)
Test Samples: 150 (collected by 3 different persons)


Evaluation

TP TP : true positive
Precision 
TP  FP FP : false positive
FN : false negative
TP
Recall 
TP  FN

2  Precison  Recall
F  Measure 
Precision  Recall


Evaluation Grafitti Numbers
E-Rubine Rubine SiGrid SiGeR

Correct 147 134 132 133

Error 3 15 15 4

Reject 0 1 3 13

Precision 0.980 0.899 0.898 0.970

Recall 1.000 0.993 0.978 0.911

F-Measure 0.990 0.944 0.936 0.940

Test Samples: 150 (collected by 3 different persons)


Evaluation Grafitti Letters (1)
E-Rubine Rubine SiGrid

Correct 334 280 273

Error 52 107 114

Reject 4 3 3

Precision 0.865 0.724 0.705

Recall 0.988 0.989 0.989

F-Measure 0.923 0.836 0.824

Test Samples: 390 (collected by 3 persons)


Evaluation Grafitti Letters (2)

Correct 342 305 297

Error 18 48 66

Reject 3 10 0

Precision 0.950 0.864 0.818

Recall 0.991 0.968 1.000

F-Measure 0.970 0.913 0.900

Training: 4x4 examples for each gesture class (collected by 4 persons)
Test Samples: 363 (collected by 4 persons)


Evaluation MS Application Gestures

Correct 196 178 145

Error 4 19 32

Reject 0 3 23

Precision 0.980 0.904 0.819

Recall 1.000 0.983 0.863

F-Measure 0.990 0.942 0.840

Test Samples: 200 (collected by 1 person)


Multi-Stroke Gestures


Evaluation Multi-Stroke Gestures

Correct 75 72 73

Error 0 3 2

Reject 0 0 0

Precision 1.000 0.960 0.973

Recall 1.000 1.000 1.000

F-Measure 1.000 0.980 0.986

Test Samples: 75 (collected by the same person)


iGesture Framework
 Exercise
 download iGesture
 analyse the design of the
framework
 investigate the existing
algorithms
 design and test your own
algorithm
 build a small application
based on your recogniser


iGesture Architecture Overview

Management Evaluation
Console Tools

Recogniser

Common Data Structures


Gesture Representation


Recognition Algorithm


Persistence


iGesture Test Bench Tab


iGesture Admin Tab


iGesture Test Data Tab


Capture of Samples


Evaluation Tools


Recogniser API
Configuration configuration =
XMLTool.importConfiguration(new File(RUBINE));
GestureSet gestureSet =
XMLTool.importGestureSet(new File(RUBINE_SET));
configuration.addGestureSet(gestureSet);
recogniser = new Recogniser(configuration);
...
ResultSet result = recogniser.recognise(note);


Conclusions
 Digital pen and paper technology
 Gesture recognition algorithms
 SiGeR, Rubine (E-Rubine), SiGrid
 Evaluation
 precision, recall and F-measure
 General iGesture recognition framework


References
 Beat Signer, Ueli Kurmann and Moira C. Norrie,
iGesture: A General Gesture Recognition
Framework, Proc. of ICDAR 2007, 9th Intl.
Conference on Document Analysis and
Recognition, Curitiba, Brazil, September 2007


References ...
 Beat Signer, Moira C. Norrie and Ueli Kurmann,
iGesture: A Java Framework for the
Development and Deployment of Stroke-Based
Online Gesture Recognition Algorithms,
Technical Report ETH Zurich, TR561,
September 2007


References ...
 Dean Rubine, Specifying Gestures by Example,
Proc. of ACM SIGGRAPH '91, 18th Intl.
Conference on Computer Graphics and
Interactive Techniques
 iGesture Open Source Framework
 http://www.igesture.org


Pen-Based Gesture Recognition

Recommandé

Recommandé

Contenu connexe

En vedette

En vedette (9)

Plus de Beat Signer

Plus de Beat Signer (20)

Dernier

Dernier (20)

Pen-Based Gesture Recognition