Alberto Mancini - Francesca Tosi WebRTC ... GWT & in-browser computation Talk @ Open Source Day 2013

1. WebRTC
... GWT & in-browser computation
Alberto Mancini - alberto@jooink.com
Francesca Tosi - francesca@jooink.com

2. WebRTC
Plug-in free real-time communication …
WebRTC is a free, open project that enables web browsers
with Real-Time Communications (RTC) capabilities via
simple Javascript APIs.
source: webrtc.org

3. WebRTC
Just another JS API
or
WebRTC is a new front in the long
war for an open and
unencumbered web
Brendan Eich (Mozilla CTO and inventor of JavaScript)
?

4. WebRTC
Just another JS API
or
WebRTC is a new front in the long
war for an open and
unencumbered web
Brendan Eich (Mozilla CTO and inventor of JavaScript)
?

5. Once upon a time … browsers
(js-)applications
were sandboxed and
kept far away from
the hosting
computer’s
resources.

6. Once upon a time … browsers
(js-)applications
were sandboxed and
kept far away from
the hosting
computer’s
resources.

7. Browser Piercing (aka HTML5)
a Javascript Application cannot
➔ access the filesystem
➔ open f/d socket connections
➔ use graphics accelerator
➔ span multiple threads

8. Browser Piercing (aka HTML5)
a Javascript Application cannot
➔ access the filesystem → and then FileAPI
➔ open f/d socket connections → and then WebSockets
➔ use graphics accelerator → and then WebGL
➔ span multiple threads → and then WebWorkers

9. Browser Piercing
a Javascript Application cannot
➔ acquire audio and video
➔ communicate P2P

10. Browser Piercing
a Javascript Application cannot
➔ acquire audio and video
➔ communicate P2P

11. WebRTC … secure enough?
“… WebRTC has encryption baked right into
it; there's actually no way to send
unencrypted media using a WebRTC
implementation … both Chrome and Firefox
implement.”

12. WebRTC … secure enough?
But ...
“If the developers fail to carefully consider the
security implications of their choices, the
safeguards mandated by the specification will
not be enough to guarantee the security of
their WebRTC-based applications and the
users.”

13. WebRTC … secure enough?
Do not ...
“... it would be very easy to inadvertently click
on something that gave camera or microphone
control to someone I don't know and don't care
to know.”
Courtney Sato - nerd queen @ConstellationRG

14. WebRTC … secure enough?
See also:
WebRTC: APIs, Protocols and Security
Considerations - Part 1 / Part 2
WebRTC Security and Confidentiality
Security Considerations for WebRTC
WebRTC Security Architecture

15. WebRTC across platforms
● Chrome (mobile too)
● Firefox (mobile too)
● Opera
● Native Java and Obj-C

16. WebRTC JS-API
Some code
…

17. WebRTC JS-API
Acquiring video and audio
navigator.getUserMedia(constraints,
successCallback,
errorCallback);

18. WebRTC JS-API
Acquiring video and audio
navigator.getUserMedia(constraints,
successCallback,
errorCallback);

19. WebRTC JS-API
Constraints
● Controls the contents of the MediaStream
● Media type, resolution, frame rate
var constraints = {video: true};
video: {
mandatory: { ... },
optional [{... }]
}

20. WebRTC JS-API
Constraints
● Controls the contents of the MediaStream
● Media type, resolution, frame rate
var constraints = {video: true};
video: {
mandatory: { chromeMediaSource: ‘screen’ },
optional [{... }]
}

21. WebRTC JS-API
Acquiring video and audio
navigator.getUserMedia(constraints,
successCallback,
errorCallback);

22. WebRTC JS-API
errorCallback
function errorCallback(error) {
console.log("error: ", error);
}
;-)

23. WebRTC JS-API
Acquiring video and audio
navigator.getUserMedia(constraints,
successCallback,
errorCallback);

24. WebRTC JS-API
successCallback
function successCallback(stream) {
...
}
stream: MediaStream; a flux of audio- or video-related
data.

25. WebRTC JS-API
successCallback
function successCallback(stream) {
var video = ...
video.src =
window.URL.createObjectURL(stream);
}
(W3C File API)

26. WebRTC JS-API
full sample
var constraints = {video: true};
function successCallback(stream) {
var video = document.querySelector("video");
video.src = window.URL.createObjectURL(stream);
}
function errorCallback(error) {
console.log("navigator.getUserMedia error: ", error);
}
navigator.getUserMedia(constraints, successCallback, errorCallback);

27. WebRTC JS-API
Live Demo - 1
http://goo.gl/7X91Ie

28. WebRTC JS-API
RTCPeerConnection
→
getUserMedia
+
RTCPeerConnection
←

29. WebRTC JS-API
RTCDataChannel
Bidirectional communication of arbitrary data
var position = {x: 3.0, y: -2.0};
...
var message = {txt: ‘...’, timestamp: ...};

30. RTCPeerConnection sample
pc = new RTCPeerConnection(null);
pc.onaddstream = gotRemoteStream;
pc.addStream(localStream);
pc.createOffer(gotOffer);
function gotOffer(desc) {
pc.setLocalDescription(desc);
sendOffer(desc);
}
function gotAnswer(desc) {
pc.setRemoteDescription(desc);
}
function gotRemoteStream(e) {
attachMediaStream(remoteVideo, e.stream);
}

31. RTCPeerConnection
Live Demo - 2
https://apprtc.appspot.com

32. RTCPeerConnection

33. RTCPeerConnection

34. RTCPeerConnection

35. STUN

36. TURN

37. WebRTC made easy
...
● Video chat:
○ SimpleWebRTC
○ easyRTC
○ webRTC.io
● Peer-to-peer data:
○ PeerJS
○ Sharefest

38. Our target

39. Our target
● getUserMedia: gives us real time stream from the camera
● we can draw the <video/> into a <canvas/>
● from the canvas we can extract pixel data getImageData()
and then elaborate.

40. Computing in the browser
goals
- Almost native performance
- Manage complex algorithms and applications
on the browser

41. Our target
cam WebRTC <video/>
canvas ImageData
UInt8ArrayGWT(NyARToolkit)mv matrix
WebGL OBJ Model
Loader3D Shaders
WebGL
video
effects
Video

42. Computing in the browser
constraints
- reuse existing code
- cross-browser (as much as possible)

43. Computing in the browser
our approach
- javascript as a target language (GWT)
- hand written optimized pieces of code

44. GWT
GWT Web Toolkit
- java to javascript (optimizing) compiler
- emulated jre
- widget set
- develop in java and execute in javascript

45. JooinK’s WebRTC sample
Sample
http://picshare.jooink.com
Demo …

46. PicShare
Tecnologies
● WebGL for 3D graphics
● WebRTC for MediaSteams
● ImgUR for pictures store
● AppEngine to store indices
● GWT as the glue

47. FOSS
gwt-nyartoolkit.googlecode.com
gwt-webgl.googlecode.com
The strategy: compile a java-lib with GWT into javascript

48. What’s next
Challanges
● real-time on mobile apps
● real world recognition: computer vision
(BoofCV)

49. What’s next
Strategy
● use TypedArrays everywhere
● offload computation to the graphic
accelerator through WebGL
● help the JIT compiler/optimizer with asm.js

50. References
● http://io13webrtc.appspot.com/#1
(where we got inspiration and most of the pictures)
● http://www.youtube.com/watch?v=p2HzZkd2A40
● http://www.webrtc.org (spec)

51. References
● http://www.jooink.com
● http://jooink.blogspot.com
● http://www.mokabyte.it

52. That’s all folks!!!
Francesca Tosi
francesca@jooink.com
Alberto Mancini
alberto@jooink.com

53. Markers - use at picshare.jooink.com