DIY: Computer Vision with GWT. #GWTcreate 2015 - Munich Francesca Tosi & Alberto Mancini

1. DIY: computer vision
with GWT
Francesca Tosi Alberto Mancini
francesca@jooink.com alberto@jooink.com

2. Alberto
Software Developer &
Linux Sysadmin
C/C++ ( since 80’s :/ )
GWT ( since 1.6 )
Javascript & Java
alberto@jooink.com
+AlbertoMancini

3. Francesca
Web and mobile developer,
software engineer and
architect
C/C++, CFD, HPC
GWT since 2009
“with a passion for clean code and
fine tuned details”
francesca@jooink.com
@francescatosi
+FrancescaTosi

4. Devfest’s CodeProject
A couple of years(**)
ago we started working on a
pet project trying to explore how mixing modern
WebAPIs and an effective programming approach(*)
can enable us to develop complex client-side
applications running in a browser.
(*)
read: GWT, i.e. a typed language compiled into js
(**)
for the 2012’s GDG DevFest season

5. Anybody knows
Browsers are a good
“execution environnement”
for evoluted web-pages ….

6. Browsers are
enough ?

7. I Mean ...
C’mon we are in 2015:
browsers are certainly
more than enough
for any kind of application !
GoogleDocs, Inbox, Plus, Scala vm in js, JsDOSBox…

8. BETting
The right tool(*)
is enough to let us to
develop
- complex
- compute-intensive
browser-based applications ?
(*)
GWT of course :)

9. The RIGHT TOOL
Should (at least) let us:
● master the (quite tricky) web platform
● code safely … in a type-safe language
(but provide native-like flexibility and performance)
● reuse existing code
… and of course produce a fast and reliable application

10. NO,no way
Our forecast was

11. Target
We were working on Augmented Reality and we had an
expertize on it so we started trying to realize an
AR webApp in order to show that this kind of
applications is way too-complex for the performance
of a browser.

12. Augmented Reality (browser based)
Marker
Based
Augmented
Reality

13. Marker Based AR
Essentially composed of 3 steps
● Marker detection and identification
(finding the marker in a picture)
● Computation of the marker pose
(det. six degrees of freedom w.r.t camera, f.i.)
● Syntetic objects collocation
(object space to camera space transf)

14. Marker Based AR

15. NyARToolkit, do not reinvent the wheel
NyARToolKit
is an ARToolKit
built with
100% pure Java.
[great
performance,
featurefull, GPL]
http://nyatla.jp/nyartoolkit/wp/

16. GWT-NyARToolkit
… given an image we get a 3x4 (3x3 + a
translation vector actually) matrix that
identifies the marker in the camera space
nyar.update(i_sensor); ← ‘push’ an image
found = nyar.isExistMarker(marker_id); ← query if a marker is detected
mm = nyar.getMarkerMatrix(marker_id); ← compute the collocation matrix
https://code.google.com/p/gwt-nyartoolkit/
http://jooink.blogspot.com

17. Browser Based AR
<video/>
<canvas/>
Ny
HTTP
?
?

18. GWT … glue

19. Acquiring Video (and eventually Audio)

20. WebRTC

21. Acquiring Video (and eventually Audio)
Acquisition of Audio/Video stream via simple js api.

22. Why WebRTC ?

23. Why WebRTC ?
Plugins for IE
e.g.
Temasys provides ad
ad hoc crafted
replacement of
adapter.js

24. Audio/Video
Acqusition Audio/Video stream via simple js apis.
navigator.getUserMedia(constraints,successCallBack,errorCallBack)

25. WebRTC with GWT
GWT (Elemental at the beginning)
At the beginning elemental, but right now (using
jnsi and now jsinterop) we are able to do all the
operation without dependencies GWT.
https://code.google.com/p/elemental-getusermedia-demo
https://github.com/jooink/gwt-getusermedia

26. Browser Based AR
<video/>
the video stream
can (has to be) be played in a <video/> tag
“native” (jsni)
video.src = URL.createObjectURL(stream);

27. Browser Based AR
<canvas/>
<video/>
canvasCtx2D.drawImage(video, ...);

28. Browser Based AR
<canvas/>
<video/>
canvasCtx2D.drawImage(video, ...);
canvasCtx2D.getImageData(sx, sy, sw, sh)

29. Browser Based AR
<video/>
Ny
<canvas/>

30. Browser Based AR
<video/>
<canvas/>
Ny
3D models to be shown on the marker
after the recognition

31. Browser Based AR
<video/>
<canvas/>
Ny
HTTP

32. WebGL
We had to reinvent the wheel :(
three.js, BabylonJS
parallax (GWT !)
awesome shading rendering and lighting
tend to mix webgl-interaction with 3D graphics

33. WebGL, gwt-webgl
<video/>
Ny
<canvas/>
https://code.google.com/p/gwt-webgl
based on Elemental with some modifications
damn thin wrapper around webgl standard

34. Tiny3D
● 3D geometry (4x4 & affine matrices)
● Lambertian and Phong lighting
● Polygons, Polyhedra, normals etc..
Wavefront OBJ import

35. Browser Based AR
<video/>
<canvas/>
Ny
HTTP

36. picshare
We lost the BET
http://picshare.jooink.com

37. Browsers are enough!

38. GWT is more than enough!

39. mGWT
GWT
Widgets &
more
for mobile

40. FirefoxOS
Web
developers
preferred
mobile OS?
B2G

42. Browser Based AR
<video/>
<canvas/>
Ny
HTTP

43. Browser Based AR
<video/>
<canvas/>
Ny
HTTP

44. Computer Vision
Computer Vision
As told by Alberto, after a while we needed to have
some more complex features on this tool for image
recognition, in order to have more information on
the image we are working with.

45. Computer Vision
Computer Vision
There exists a lot of CV libs (complete, tested and
fine tuned) for example:
- OpenCV
- VLFeat
- VLX
- …
C/C++

46. Computer Vision
Computer Vision
There exists a lot of CV libs (complete, tested and
fine tuned) for example:
- OpenCV
- VLFeat
- VLX
- …
- BoofCV
C/C++
Java

47. Computer Vision
Computer Vision by BoofCV
There is a great library for Computer Vision
interely written in Pure Java: BoofCV → we decided
to adapt BoofCV for GWT.

48. BoofCV
BoofCV is a complete & complex project
BoofCV is an opensource Java library for computer
vision & robotics applications.
BoofCV is organized into several packages:
image processing, features, geometric vision, calibration,
recognition, visualize and IO.

49. BoofCV
Written by Peter Abeles
www.boofcv.org
Link to the source code:
https://github.com/lessthanoptimal/BoofCV

50. BoofCV
Adapting an entire big & complex
library is not is a funny game.
BoofCV depends on other Java libs, so we must adapt
all the dependencies

51. BoofCV
- DDogLeg: Java numerical lib for
non-linear optimization, robust
model-fitting, polynomial root
finding, sorting and more;
- GeoRegression: open source java
geometry lib for scientific
computing;
- EJML: dense matrix linear
algebra library written in Java

52. BoofCV
- GWT-DDogLeg:
https://github.com/jooink/gwt-ddogleg
http://gwtddogleg.appspot.com/
- GWT-GeoRegression:
https://github.com/jooink/gwt-georegression
http://gwtgeoregression.appspot.com/
- GWT-EJML:
https://github.com/jooink/gwt-ejml
http://gwtejml.appspot.com/

53. EJML
- EJML:
dense matrix linear algebra library written in
Java
https://github.com/lessthanoptimal/ejml

54. GWT-EJML
GWT-EJML
is an adapter-project around EJML that let us to use
the original library:
- provides .gwt.xml files
- supersources many classes
- contains running examples

55. GWT-EJML
GWT-EJML
published on github
https://github.com/jooink/gwt-ejml

56. GWT-EJML
org.EJML.gwt.xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE module PUBLIC "-//Google Inc.//DTD Google Web Toolkit 2.6.1.
rc1//EN" "http://google-web-toolkit.googlecode.com/svn/tags/2.6.1.
rc1/distro-source/core/src/gwt-module.dtd">
<module>
<source path="ejml"/>
</module>

57. GWT-EJML
.gwt.xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE module PUBLIC "-//Google Inc.//DTD Google Web Toolkit 2.7.0
//EN"
"http://gwtproject.org/doctype/2.7.0/gwt-module.dtd">
<module rename-to='gwtejml'>
<inherits name='org.EJML'/>
...
<super-source path="jre" />
...
</module>

58. GWT-EJML
The Strategy
we decided to use the SuperSourcing in order to
change/adapt the class of the basic lib and leave
the ejml project unchanged
→ the maintainability is easier

59. GWT-EJML
SuperSourcing
We added the super source path in the gwt.xml config
file
<super-source path="jre" />
and we created the file to modify in that directory
(using the right path of the file in the main
project).

60. GWT-EJML
.gwt.xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE module PUBLIC "-//Google Inc.//DTD Google Web Toolkit 2.7.0
//EN"
"http://gwtproject.org/doctype/2.7.0/gwt-module.dtd">
<module rename-to='gwtejml'>
<inherits name='org.EJML'/>
...
<super-source path="jre" />
...
</module>

61. GWT-EJML
Supersourced parts of JRE
java.io.ByteArrayOutputStream
public class ByteArrayOutputStream extends OutputStream {
public ByteArrayOutputStream() { … }
public void write(int arg0) { … }
}

62. GWT-EJML
Supersourced parts of EJML
org.ejml.ops.MatrixIO;
public class MatrixIO { … }
org.ejml.alg.dense.decomposition
.eig.symm.SymmetricQREigenHelper
org.ejml.alg.dense.decomposition
.svd.implicitqr.SvdImplicitQrAlgorithm
java.io.
PrintStream
printf()

63. GWT-EJML
Supersourced parts of EJML
java.io.PrintStream does not have printf() in the
gwt emulated jre
We could not supersource the PrintStream because
already exist on emulated jre and we cannot super-
sourcing just a method.

64. GWT-EJML
Link
https://github.com/jooink/gwt-ejml
http://gwtejml.appspot.com/

65. DDogLeg
- DDOGLEG:
Java numerical lib for non-linear optimization,
robust model-fitting, polynomial root finding,
sorting and more;
https://github.com/lessthanoptimal/ddogleg

66. GWT-DDogLeg
.gwt.xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE module PUBLIC "-//Google Inc.//DTD Google Web Toolkit 2.7.0
//EN"
"http://gwtproject.org/doctype/2.7.0/gwt-module.dtd">
<module rename-to='gwtddogleg'>
<inherits name='org.DDOGLEG'/>
<inherits name='com.jooink.gwtejml.gwtEJML_lib'/>
...
<super-source path="jre" />
...
</module>

67. GWT-DDogLeg
Supersourced parts of DDOGLEG:
java.lang.OutOfMemoryError
Unchecked Exception …

68. GWT-DDogLeg
Supersourced parts of DDOGLEG:
org.ddogleg.struct.FastQueue
org.ddogleg.struct.FastQueueList
→ Reflection (not supported by gwt-jre)

69. GWT-DDogLeg
Link
https://github.com/jooink/gwt-ddogleg
http://gwtddogleg.appspot.com/

70. GEOREGRESSION
- GEOREGRESSION:
open source java geometry lib for scientific
computing;
https://github.com/lessthanoptimal/georegression

71. GWT-GEOREGRESSION
- GWT-GEOREGRESSION:
minor modifications

72. GWT-GEOREGRESSION
.gwt.xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE module PUBLIC "-//Google Inc.//DTD Google Web Toolkit 2.7.0
//EN" "http://gwtproject.org/doctype/2.7.0/gwt-module.dtd">
<module rename-to='gwtgeoregression'>
<inherits name='georegression.GEOREGRESSION'/>
<inherits name='com.jooink.gwtejml.gwtEJML_lib'/>
<inherits name='com.jooink.gwtddogleg.gwtDDogLeg_lib'/>
...
</module>

73. GWT-GEOREGRESSION
Link
https://github.com/jooink/gwt-georegression
http://gwtgeoregression.appspot.com/

74. BoofCV
- BOOFCV:
fast computer vision library written entirely in
Java;
https://github.com/lessthanoptimal/boofcv

75. .gwt.xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE module PUBLIC "-//Google Inc.//DTD Google Web Toolkit 2.7.0
//EN" "http://gwtproject.org/doctype/2.7.0/gwt-module.dtd">
<module rename-to='gwtboofcv'>
<inherits name='boofcv.BOOFCV'/>
<inherits name='com.jooink.gwtejml.gwtEJML_lib'/>
<inherits name='com.jooink.gwtddogleg.gwtDDogLeg_lib'/>
<inherits name='com.jooink.gwtGeoRegression.gwtGeoRegression_lib'/>
...
<super-source path="jre" />
...
</module>

76. BoofCV
GWT-BOOFCV:
● Reflection
● Image managment

77. BoofCV
GWT-BOOFCV:
● Reflection:
○ used mostly to handle Arrays of primitive
types
○ probably unneeded upstream
working with Peter to understand if can be
avoided through, for instance, a few of
factories.
● Image managment

78. BoofCV
Reflection:
java.lang.reflect.Array
Array.newInstance( … )
Array.getLength( … )
...

79. BoofCV
GWT-BOOFCV:
● Reflection
● Image managment

80. BoofCV
Image managment:
of course we had to work-out the different point of
view about Images:
Browser → (<img/> <video/>)
BoofCV → array of values
→ we used <canvas/> (also in this case);
canvas.getContext2d().getImageData( … );

81. BoofCV
EXAMPLES:
- Interest Point detection
- Interest point association

82. BoofCV
EXAMPLE 1
Interest Point detection:
ImageFloat32
DetectDescribePoint<ImageFloat32,
SurfFeature>detDesc.detect(boofImage);

83. BoofCV
- Interest point
detection
http://static.jooink.com/experiments/boofcv-gwt/interest-points/

84. BoofCV
- Interest point
association

85. Questions? Contact us :
Francesca Tosi
R&D at Jooink Team
francesca@jooink.com
Alberto Mancini
Dev at Jooink Team
alberto@jooink.com
Thanks !
Please rate this presentation http://gwtcreate.com/agenda