State of the Art of WebRTC testing, First Presentation of KITE.

1. Real-Time Communication
Testing Evolution
with WebRTC
Dr Alex Gouaillard
CoSMo Software Consulting
IMTC, W3C, IETF

2. RTC Interoperability: Web Apps
Chromecast

3. • W3C Standardization process (features supported or not)
• Stand-alone tests
• Single browser (one at a time)
(JS API) Compliance and Usual Testing
Test the Web Forward

4. What about WebRTC 1.0?
Why so low test coverage?
• Some parts of specifications were just written
• Tests not needed early on in standardization process
• Many tests need P2P communication / interop testing
Media devices APIs WebRTC 1.0 APIs
Test coverage as of November 2016 (W3C TPAC)

5. Ex. of RTC feature not Testable before:
Network and Protocol Testing: ICE
Client A Client B
STUN server
What is my public
IP address?
Your public IP address
is x.x.x.x:y

6. Compliance and Usual Testing vs.
Interoperability Testing
Compliance & Usual
Testing
Interoperability
Testing
• Interaction automation
• Stand-alone tests
• Single browser
• Interaction automation
• Two synchronized instances
of the same test
• 2+ browsers run in parallel
 New testing tools needed

7. Browsers Interaction Automation:
WebDriver protocol
ChromeDriver
geckodriver
Microsoft WebDriver
SafariDriver

8. Browsers as a Service:
hosted selenium nodes

9. In-Progress: WebDriver needs update
for WebRTC Security & others
• Permission prompt
– Geolocation
– Camera
– Microphone
– Bluetooth
– …
• Fake Media / Mock Capturer
• ICE Candidate Filtering
• Unsecure origin

10. Wrap-Up: WebDriver for RTC Tests
• Interaction
• Multiple browsers in parallel
• Synchronization of instances
of same test between browsers

11. Lots of tools out there
addressing a subset of the problem
WebRTC-test
Jitsi-Hammer
Kurento Testing Framework
Jattack

12. Ok, existing tools are limited but …
How bad can it be, really?
Rough Goal: Test all possible pairs of
[Browser] x [Browser Revision]
x
[OS] x [OS Revision]

13. WhatweWANTtotest
496
© Dr. Alex Gouaillard @ Citrix Systems, 2016. Document provided under CC BY-NC 4.0

14. Whatwecouldtesttoday 115
23%
© Dr. Alex Gouaillard @ Citrix Systems, 2016. Document provided under CC BY-NC 4.0
381
77%
1 OS
2 Oses
(Not tested)

15. © Dr. Alex Gouaillard @ Citrix Systems, 2016. Document provided under CC BY-NC 4.0
6
5%
WhatweACTUALLYtesttoday
21
18%
88
77%
Tested
WIP
Not tested

16. Ok, it’s REALLY bad …
Get visibility: list and evaluate

17. Evaluation of Interop Testing Tools
Dimension 1: Tests Triggers and frequency
6 months
Weekly (2d)
Daily / Nightly (4h)
Commit-based and fast (20mn)
Experimental (manually triggered)

18. Evaluation of Interop Testing Tools
Dimension 2: Type of SE nodes I want to use
Self-Hosted, Pre-configured VM
Hosted, as-a-service (e.g. BrowserStack)
Self-Hosted, Ephemeral VM
Hosted, Ephemeral VM (e.g. Travis)
Local / physical machines
[Apple]
[Only stable browsers]
[Extensions and plug-ins]
[Modified browsers]
[no manual settings]
[Extensions and plug-ins]
[Apple]
[limited OS set]

19. Evaluation of Interop Testing Tools
Dimension 3: OS I want to support

20. Evaluation of Interop Testing Tools
Dimension 4: Which Browsers?
Easy, kind of tested already, great WebDriver support
No Apple VM on non-apple Hardware
Webdriver is more recent and less mature than others
WebRTC? What’s your flavor~~ ?
Permission prompt? Fake Media?
36+% of Enterprise market (Windows 7)
which plugin?

21. Evaluation of Interop Testing Tools
Dimension 5: How early do I want to test?
=> How fast do you want your fix in
Ex: chrome 58  chrome 61
Stable
Windows Insider
Windows Insider

22. Evaluation of Interop Testing Tools
Own cloud
On-premises
Dimension 6: Freedom, Convenience and Price
Hosted

23. Trigger
Type of SE node
# OSes
Browsers
Browser channel
Hosted/on-prem
testRTC
Hosted

24. Trigger
Type of SE node
# OSes
Browsers
Browser channel
Hosted/on-prem
Travis CI
Hosted

25. Trigger
Type of SE node
# OSes
Browsers
Browser channel
Hosted/on-prem
BrowserStack
Hosted

26. Trigger
Type of SE node
# OSes
Browsers
Browser channel
Hosted/on-prem
Sauce Labs
Hosted

27. Ok, I see what’s missing for each.
What’s the right design now?

28. KITE: Karoshi Interoperability
Testing Engine
Because for most of us, manually testing webrtc
interop felt like “death by overwork”

29. KITE: Karoshi Interoperability Testing Engine
Automating RTC P2P tests
• Generic (any app, any back-end)
• Reusable (any process, trigger, …)
• Easy to maintain (no death  )
Stable / Beta / Dev / Canary
+
Own cloudOn-premises Hosted

30. KITE
Trigger
Type of SE node
# OSes
Browsers
Browser channel
Hosted/on-prem
KITE
Hosted
+
On-premises

31. KITE Design
Easy maintenance by divide and conquer
• Selenium Grid
• Test engine
• Test
• Dashboard

32. KITE Design
Conductor
Dispatcher
Test Runner Test Runner
Test Runner
Config => BrowserObj
Compute list< N-Tuple< BrowserObj > >
• { Test, Callback }
• list< N-Tuple< BO > >
handle concurrency
if resource: 1 N-tuple => 1 test runner =>1 thread
1 2 … N
• { Test, Callback }
• 1 N-Tuple< BO >
Instruments N BO
Run Test
Callback( results )
BrowserObj available
• Cr/FF - Mac - Rem-BB
• BrX - OSX - Rem-BaaS (SauceLabs)
• BrY - OSY - Rem-VM (AWS)
Result
Dashboard
• { N, Test, Callback }
• list< configs >

33. Configuration File{
"name": "local selenium example",
"callback": "http://localhost:8080/kiteweb/datacenter",
"remotes": [
{
"type": "local",
"remoteAddress": "http://localhost:4444/wd/hub"
}
],
"tests": [
{
"name": "IceConnectionTest",
"tupleSize": 2,
"testImpl": "org.webrtc.kite.IceConnectionTest"
}
],
"browsers": [
{
"browserName": "firefox"
},
{
"browserName": "chrome"
}
]
}

34. Configuration File{
"name": "local selenium example",
"callback": "http://localhost:8080/kiteweb/datacenter",
"remotes": [
{
"type": "local",
"remoteAddress": "http://localhost:4444/wd/hub"
}
],
"tests": [
{
"name": "IceConnectionTest",
"tupleSize": 2,
"testImpl": "org.webrtc.kite.IceConnectionTest"
}
],
"browsers": [
{
"browserName": "firefox"
},
{
"browserName": "chrome"
}
]
}
Location of dashboard
to send tests results
"callback": "http://localhost:8080/kiteweb/datacenter",

35. Configuration File{
"name": "local selenium example",
"callback": "http://localhost:8080/kiteweb/datacenter",
"remotes": [
{
"type": "local",
"remoteAddress": "http://localhost:4444/wd/hub"
}
],
"tests": [
{
"name": "IceConnectionTest",
"tupleSize": 2,
"testImpl": "org.webrtc.kite.IceConnectionTest"
}
],
"browsers": [
{
"browserName": "firefox"
},
{
"browserName": "chrome"
}
]
}
Registration of
Selenium server
"remotes": [
{
"type": "local",
"remoteAddress": "http://localhost:4444/wd/hub"
}
],

36. Configuration File{
"name": "local selenium example",
"callback": "http://localhost:8080/kiteweb/datacenter",
"remotes": [
{
"type": "local",
"remoteAddress": "http://localhost:4444/wd/hub"
}
],
"tests": [
{
"name": "IceConnectionTest",
"tupleSize": 2,
"testImpl": "org.webrtc.kite.IceConnectionTest"
}
],
"browsers": [
{
"browserName": "firefox"
},
{
"browserName": "chrome"
}
]
}
List of tests
"tests": [
{
"name": "IceConnectionTest",
"tupleSize": 2,
"testImpl": "org.webrtc.kite.IceConnectionTest"
}
],

37. Configuration File{
"name": "local selenium example",
"callback": "http://localhost:8080/kiteweb/datacenter",
"remotes": [
{
"type": "local",
"remoteAddress": "http://localhost:4444/wd/hub"
}
],
"tests": [
{
"name": "IceConnectionTest",
"tupleSize": 2,
"testImpl": "org.webrtc.kite.IceConnectionTest"
}
],
"browsers": [
{
"browserName": "firefox"
},
{
"browserName": "chrome"
}
]
}
List of browsers to
run the tests
"browsers": [
{
"browserName": "firefox"
},
{
"browserName": "chrome"
}
]

41. KITE Dashboard

42. KITE Dashboard

43. That’s very nice but it feels like
something is missing …

44. Global RTC Interoperability
Chromecast

45. Global RTC Interoperability:
MOBILE (Browsers)?
Chromecast

46. Global RTC Interoperability
Beyond Desktop Browsers
Appium
• Mobile testing framework
• For native, hybrid and mobile web apps

47. Global RTC Interoperability:
NATIVE, IoT and embedded
Chromecast
IoT
Electron

48. Global RTC Interoperability
Beyond Desktop Browsers
Electron
• Open source framework
• To build cross platform desktop applications
(HTML, CSS, JS)
• Combines Chromium and node.js into a single
runtime

49. Global RTC Interoperability:
Beyond 1-1
Jitsi
• Open source multiplatform
• Videoconferencing
• Instant messaging

50. Global RTC Interoperability:Beyond 1-1
Jitsi

51. Global RTC Interoperability
Make it easy: Grid Manager
• Set up a grid through a web app
• Set up Selenium hubs and nodes
• Download and set up
– Browsers
– Browsers WebDrivers
• Backup everything for follow-up uses
• Tear-down and clean-up after tests

52. Thank you!