An introduction to Stream processing pipelines with a focus on MediaStream pipelines. Then a deep dive into the Video/Canvas pipeline and how that can be used to process streams of image frames in real-time. Find out more about why ArrayBuffers are the key innovation driving this evolution and how WebRTC will make this pervasive.

1. https://buildAR.com - image credit
It's alive!
MediaStream processing pipelines
on the Augmented Web Platform

2. Who's making these outlandish claims?
Rob Manson @nambor
CEO of MOB-labs the creators of buildAR.com
Chair of the W3C's Augmented Web Community Group
Invited Expert with the ISO, W3C & the Khronos Group
Author of “Getting started with WebRTC” - (Packt Publishing)
https://buildAR.com

3. What is the Augmented Web?
https://buildAR.com

4. It's what comes after HTML5
https://buildAR.com

5. The web is evolving...
1992-2000
WWW: Page based websites
key innovation: HREF
2000-2013
Web 2.0: AJAX driven social services
key innovation: XHR
2013-...
AWE: Sensor driven real-time web of things
key innovation: ArrayBuffer
http://mob-labs.com : image credit

6. https://buildAR.com

7. https://buildAR.com
JSARToolkit demo

8. https://buildAR.com
FAST algorithm feature detection demo

9. Test your device at: theAWEsomeWEB.com
https://buildAR.com

10. So what the hell is a
Stream processing pipeline?
https://buildAR.com

11. https://buildAR.com
Stream processing pipelines
1. Get Stream
2. Connect to a Stream pipeline
A way to connect a Stream to an ArrayBuffer
3. Get ArrayBuffer
5. Populate a scene Array with Array Buffer Views
6. Move through Views to process data
7. Output events and metadata
8. Update UI and/or send requests

12. Then what is a
MediaStream processing pipeline?
https://buildAR.com

13. https://buildAR.com
MediaStream processing pipelines
1. Get MediaStream
MediaStream from getUserMedia or RTCPeerConnection
2. Connect to a MediaStream pipeline
Canvas/Video, Image Capture or Recording pipelines
3. Get ArrayBuffer
new ArrayBuffer()
5. Populate a scene Array with Array Buffer Views
var scene = [new Uint8ClampedArray(buffer), ...]
6. Move through Views to process data
for each (var view in scene) { …process(view)... }
7. Output events and metadata
return results
8. Update UI and/or send requests
Wave hands inspriationally here!

14. https://buildAR.com
This is where the APIs are still evolving
1. Get MediaStream
MediaStream from getUserMedia or RTCPeerConnection
2. Connect to a MediaStream pipeline
Canvas/Video, Image Capture or Recording pipelines
3. Get ArrayBuffer
new ArrayBuffer()
6. Move through Views to process data
for each (var view in scene) { …process(view)... }
7. Output events and metadata
return results
8. Update UI and/or send requests
Wave hands inspriationally here!
5. Populate a scene Array with Array Buffer Views
var scene = [new Uint8ClampedArray(buffer), ...]

15. https://buildAR.com
This is where the Augmented Web comes to life
1. Get MediaStream
MediaStream from getUserMedia or RTCPeerConnection
2. Connect to a MediaStream pipeline
Canvas/Video, Image Capture or Recording pipelines
3. Get ArrayBuffer
new ArrayBuffer()
6. Move through Views to process data
for each (var view in scene) { …process(view)... }
7. Output events and metadata
return results
8. Update UI and/or send requests
Wave hands inspriationally here!
5. Populate a scene Array with Array Buffer Views
var scene = [new Uint8ClampedArray(buffer), ...]

16. https://buildAR.com
This is where the fun is!
1. Get MediaStream
MediaStream from getUserMedia or RTCPeerConnection
2. Connect to a MediaStream pipeline
Canvas/Video, Image Capture or Recording pipelines
3. Get ArrayBuffer
new ArrayBuffer()
6. Move through Views to process data
for each (var view in scene) { …process(view)... }
7. Output events and metadata
return results
8. Update UI and/or send requests
Wave hands inspriationally here!
5. Populate a scene Array with Array Buffer Views
var scene = [new Uint8ClampedArray(buffer), ...]

17. #$!%@#%@#$@$@!!!!!
https://buildAR.com

18. https://buildAR.com
Browser's have always had a “binary barrier”
The Video element
stream -> <video>

19. https://buildAR.com
Now we've broken through the “binary barrier”
The Video/Canvas pipeline
stream -> <video> -> <canvas> -> image data -> array buffer -> process -> output

20. https://buildAR.com
Now we've broken through the “binary barrier”
The Video/Canvas pipeline
stream -> <video> -> <canvas> -> image data -> array buffer -> process -> output
The Mediastream Image Capture API pipeline
stream -> track -> image capture -> image data -> array buffer -> process -> output

21. https://buildAR.com
Now we've broken through the “binary barrier”
The Video/Canvas pipeline
stream -> <video> -> <canvas> -> image data -> array buffer -> process -> output
The Mediastream Image Capture API pipeline
stream -> track -> image capture -> image data -> array buffer -> process -> output
The MediaStream Recording API pipeline
stream -> recorder -> blob -> file reader -> array buffer -> process -> output

22. https://buildAR.com
Now we've broken through the “binary barrier”
The Video/Canvas pipeline
stream -> <video> -> <canvas> -> image data -> array buffer -> process -> output
The Mediastream Image Capture API pipeline
stream -> track -> image capture -> image data -> array buffer -> process -> output
The MediaStream Recording API pipeline
stream -> recorder -> blob -> file reader -> array buffer -> process -> output
Plus other non-MediaStream pipelines:
????? -> Web Sockets/XHR/File/JS -> array buffer -> process -> output

23. Lets take a closer look at the
Video/Canvas MediaStream pipeline
https://buildAR.com

24. https://buildAR.com
The Video/Canvas MediaStream pipeline
Access the camera (or some other stream) source
a. getUserMedia()
Setup <video> element in the DOM
a. declaratively then via getElementById or similar
b. createElement(“video”) (no need to appendChild())
Pipe camera stream into <video>
a. video.src = stream
Setup <canvas> element in the DOM
a. declaratively then via getElementById or similar
b. createElement(“canvas”) then appendChild()
Get 2D drawing context
a. canvas.getContext('2d');
Draw <video> frame onto <canvas>
a. canvas.drawImage(video, top, left, width, height);
Get RGBA Uint8ClampedArray of the pixels
a. context.getImageData(top, left, width, height).data.buffer;
Burn CPU (not GPU) cycles
a. fore each(var obj in view) { … }
NOTE: Integrate other streams & sensor data here
Render results
a. using HTML/JS/CSS
b. using another <canvas> and drawImage()
c. using WebGL
d. a combination of all
1. Get MediaStream
MediaStream from getUserMedia or RTCPeerConnection
2. Connect to a MediaStream pipeline
Canvas/Video, Image Capture or Recording pipelines
3. Get ArrayBuffer
new ArrayBuffer()
5. Populate a scene with Array Buffer Views
var scene = [new Uint8ClampedArray(buffer), ...]
6. Move through Views to process data
for each (var view in scene) { …process(view)... }
7. Output events and metadata
return results
8. Update UI and/or send requests
Wave hands inspriationally here!
Follow along on github: github.com/buildar/getting_started_with_webrtc

25. https://buildAR.com
The Video/Canvas MediaStream pipeline
Follow along on github: github.com/buildar/getting_started_with_webrtc
Access the camera (or some other stream) source
a. getUserMedia()
Setup <video> element in the DOM
a. declaratively then via getElementById or similar
b. createElement(“video”) (no need to appendChild())
Pipe camera stream into <video>
a. video.src = stream
Setup <canvas> element in the DOM
a. declaratively then via getElementById or similar
b. createElement(“canvas”) then appendChild()
Get 2D drawing context
a. canvas.getContext('2d');
Draw <video> frame onto <canvas>
a. canvas.drawImage(video, top, left, width, height);
Get RGBA Uint8ClampedArray of the pixels
a. context.getImageData(top, left, width, height).data.buffer;
Burn CPU (not GPU) cycles
a. fore each(var obj in view) { … }
NOTE: Integrate other streams & sensor data here
Render results
a. using HTML/JS/CSS
b. using another <canvas> and drawImage()
c. using WebGL
d. a combination of all
setInterval()
RequestAnimationFrame()
setTimeout()

26. https://buildAR.com
What's in the near future?
Integrating WebRTC and Visual Search
Using WebGL/GLSL to utilise GPU parallelism
Khronos Group's OpenVX
Khronos Group's Camera Working Group
Lots more demos to share! 8)

27. https://buildAR.com
What's in the near future?
Integrating WebRTC and Visual Search
Using WebGL/GLSL to utilise GPU parallelism
Khronos Group's OpenVX
Khronos Group's Camera Working Group
Lots more demos to share! 8)

28. https://buildAR.com
What's in the near future?
Integrating WebRTC and Visual Search
Using WebGL/GLSL to utilise GPU parallelism
Khronos Group's OpenVX
Khronos Group's Camera Working Group
Lots more demos to share! 8)

29. https://buildAR.com
Join the W3C's
Augmented Web Community Group
http://w3.org/community/ar