All thingsrtc finding hidden issues with ml

Finding Hidden Call Quality Issues with
Machine Learning
chad hart
13 June 2019
1
Deliver Better User Experiences

Callstats.io: The WebRTC Monitoring Leader
• Founded in 2014 by IETF and W3C authors
• Customers across vertical industries
• Over 1B collected WebRTC datapoints each month
• Integrated into Amazon Connect
2
NXTEC CORP

callstats.io Voice Quality Solutions
3
WebRTC Monitoring
• Network status
• Service metrics
AI-Driven Troubleshooting
• Anomaly detection
• Root cause analysis
• Notifications
Active Network Testing
• Connectivity verification
• Performance metrics

Data Collection and Monitoring Architecture
4

Why do we need ML?
Dealing with Big Data
6
Photo credit: Flickr User Gavin Bell (CC BY-NC-ND 2.0)
https://www.flickr.com/photos/gavinbell/535261899/

Challenge: getStats provides a lot of data
Example:
• Go to
https://webrtc.github.io/samples/src/content/peerconnection/pc1/
• Start the sample
• Type:
7
pc1.getStats().then(stats=>stats.forEach(stat=>console.log(stat)))

• [{"id":"RTCCertificate_70:47:DC:96:5C:24:A1:C1:F0:A2:89:39:FC:B1:D9:69:66:EB:4A:F6:12:F0:68:43:1B:F9:D4:ED:D6:E2:D1:FE","timestamp":1560111536209.789,"type":"certificate","fingerprint":"70:47:DC:96:5C:24:A1:C1:F0:A2:89:39:FC:B1:D9:69:66:EB:4A:F6:12:F0:68:43:1B:F9:D4:ED:D6:E2:D1:FE","fingerprintAlgorithm":"sha-
256","base64Certificate":"MIIBFjCBvaADAgECAgkAhuXyX8OJm0EwCgYIKoZIzj0EAwIwETEPMA0GA1UEAwwGV2ViUlRDMB4XDTE5MDYwODIwMTU0MFoXDTE5MDcwOTIwMTU0MFowETEPMA0GA1UEAwwGV2ViUlRDMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEQ6ZyQYSC7xZHSUhq6mxCWa+/VwRdO0U1GuMAUYjYSkAmgzfJ0mW TZbZWJ12V7LlimkOZtdXCv2UPSh3sDOHu/DAKBggqhkjOPQQDAgNIADBFAiBDBGAgqa2dxlewHn5661OgMK6xfanL13l9aLoyuvdCgAIhAOmal8xD0S0ZLRKO/WDOb3
RRLAqxCcn0Jmur/OPdoEDl"},{"id":"RTCCertificate_D1:3F:71:1A:FE:32:78:3D:AA:93:06:F3:E6:AF:22:E9:5A:BC:B7:D2:AD:D6:A4:49:FD:9F:FB:00:49:89:4C:5A","timestamp":1560111536209.789,"type":"certificate","fingerprint":"D1:3F:71:1A:FE:32:78:3D:AA:93:06:F3:E6:AF:22:E9:5A:BC:B7:D2:AD:D6:A4:49:FD:9F:FB:00:49:89:4C:5A","fingerprintAlgorithm":"sha-
256","base64Certificate":"MIIBFzCBvaADAgECAgkA7svx50DzS6swCgYIKoZIzj0EAwIwETEPMA0GA1UEAwwGV2ViUlRDMB4XDTE5MDYwODIwMTU0MFoXDTE5MDcwOTIwMTU0MFowETEPMA0GA1UEAwwGV2ViUlRDMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEg3VCpwhtcNE3rEaBH8GTrWpQTcbnPGWEz0i2XAYE/A8pATEPoXb7mPojyEy8LxkmPSjkwrEodppNzA7X8Yjv2TAKBggqhkjOPQQDAgNJADBGAiEA3MWsKxnwNWssNhZ+mSGhZca63A8tvFIdRyf+CBOzNYwCIQD1JTiEdHntgBQ9/ZcaRVbQK
MBU1X6+xYmtnwjJ+x2umw=="},{"id":"RTCCodec_0_Inbound_0","timestamp":1560111536209.789,"type":"codec","payloadType":0,"mimeType":"audio/PCMU","clockRate":8000},{"id":"RTCCodec_0_Inbound_103","timestamp":1560111536209.789,"type":"codec","payloadType":103,"mimeType":"audio/ISAC","clockRate":16000},{"id":"RTCCodec_0_Inbound_104","timestamp":1560111536209.789,"type":"codec","payloadType":104,"mimeType":"audio/ISAC","clockRate":32000},{"id":"RTCCodec_0_Inbound_105","t
imestamp":1560111536209.789,"type":"codec","payloadType":105,"mimeType":"audio/CN","clockRate":16000},{"id":"RTCCodec_0_Inbound_106","timestamp":1560111536209.789,"type":"codec","payloadType":106,"mimeType":"audio/CN","clockRate":32000},{"id":"RTCCodec_0_Inbound_110","timestamp":1560111536209.789,"type":"codec","payloadType":110,"mimeType":"audio/telephone-
event","clockRate":48000},{"id":"RTCCodec_0_Inbound_111","timestamp":1560111536209.789,"type":"codec","payloadType":111,"mimeType":"audio/opus","clockRate":48000},{"id":"RTCCodec_0_Inbound_112","timestamp":1560111536209.789,"type":"codec","payloadType":112,"mimeType":"audio/telephone-event","clockRate":32000},{"id":"RTCCodec_0_Inbound_113","timestamp":1560111536209.789,"type":"codec","payloadType":113,"mimeType":"audio/telephone-
event","clockRate":16000},{"id":"RTCCodec_0_Inbound_126","timestamp":1560111536209.789,"type":"codec","payloadType":126,"mimeType":"audio/telephone-
event","clockRate":8000},{"id":"RTCCodec_0_Inbound_13","timestamp":1560111536209.789,"type":"codec","payloadType":13,"mimeType":"audio/CN","clockRate":8000},{"id":"RTCCodec_0_Inbound_8","timestamp":1560111536209.789,"type":"codec","payloadType":8,"mimeType":"audio/PCMA","clockRate":8000},{"id":"RTCCodec_0_Inbound_9","timestamp":1560111536209.789,"type":"codec","payloadType":9,"mimeType":"audio/G722","clockRate":8000},{"id":"RTCCodec_0_Outbound_0","timestamp":1560111
536209.789,"type":"codec","payloadType":0,"mimeType":"audio/PCMU","clockRate":8000},{"id":"RTCCodec_0_Outbound_103","timestamp":1560111536209.789,"type":"codec","payloadType":103,"mimeType":"audio/ISAC","clockRate":16000},{"id":"RTCCodec_0_Outbound_104","timestamp":1560111536209.789,"type":"codec","payloadType":104,"mimeType":"audio/ISAC","clockRate":32000},{"id":"RTCCodec_0_Outbound_105","timestamp":1560111536209.789,"type":"codec","payloadType":105,"mimeType":"a
udio/CN","clockRate":16000},{"id":"RTCCodec_0_Outbound_106","timestamp":1560111536209.789,"type":"codec","payloadType":106,"mimeType":"audio/CN","clockRate":32000},{"id":"RTCCodec_0_Outbound_110","timestamp":1560111536209.789,"type":"codec","payloadType":110,"mimeType":"audio/telephone-
event","clockRate":48000},{"id":"RTCCodec_0_Outbound_111","timestamp":1560111536209.789,"type":"codec","payloadType":111,"mimeType":"audio/opus","clockRate":48000},{"id":"RTCCodec_0_Outbound_112","timestamp":1560111536209.789,"type":"codec","payloadType":112,"mimeType":"audio/telephone-event","clockRate":32000},{"id":"RTCCodec_0_Outbound_113","timestamp":1560111536209.789,"type":"codec","payloadType":113,"mimeType":"audio/telephone-
event","clockRate":16000},{"id":"RTCCodec_0_Outbound_126","timestamp":1560111536209.789,"type":"codec","payloadType":126,"mimeType":"audio/telephone-
event","clockRate":8000},{"id":"RTCCodec_0_Outbound_13","timestamp":1560111536209.789,"type":"codec","payloadType":13,"mimeType":"audio/CN","clockRate":8000},{"id":"RTCCodec_0_Outbound_8","timestamp":1560111536209.789,"type":"codec","payloadType":8,"mimeType":"audio/PCMA","clockRate":8000},{"id":"RTCCodec_0_Outbound_9","timestamp":1560111536209.789,"type":"codec","payloadType":9,"mimeType":"audio/G722","clockRate":8000},{"id":"RTCCodec_1_Inbound_100","timestamp":15
60111536209.789,"type":"codec","payloadType":100,"mimeType":"video/VP9","clockRate":90000},{"id":"RTCCodec_1_Inbound_101","timestamp":1560111536209.789,"type":"codec","payloadType":101,"mimeType":"video/rtx","clockRate":90000},{"id":"RTCCodec_1_Inbound_102","timestamp":1560111536209.789,"type":"codec","payloadType":102,"mimeType":"video/H264","clockRate":90000},{"id":"RTCCodec_1_Inbound_107","timestamp":1560111536209.789,"type":"codec","payloadType":107,"mimeType":"vi
deo/rtx","clockRate":90000},{"id":"RTCCodec_1_Inbound_108","timestamp":1560111536209.789,"type":"codec","payloadType":108,"mimeType":"video/H264","clockRate":90000},{"id":"RTCCodec_1_Inbound_109","timestamp":1560111536209.789,"type":"codec","payloadType":109,"mimeType":"video/rtx","clockRate":90000},{"id":"RTCCodec_1_Inbound_114","timestamp":1560111536209.789,"type":"codec","payloadType":114,"mimeType":"video/red","clockRate":90000},{"id":"RTCCodec_1_Inbound_115","times
tamp":1560111536209.789,"type":"codec","payloadType":115,"mimeType":"video/rtx","clockRate":90000},{"id":"RTCCodec_1_Inbound_116","timestamp":1560111536209.789,"type":"codec","payloadType":116,"mimeType":"video/ulpfec","clockRate":90000},{"id":"RTCCodec_1_Inbound_119","timestamp":1560111536209.789,"type":"codec","payloadType":119,"mimeType":"video/rtx","clockRate":90000},{"id":"RTCCodec_1_Inbound_120","timestamp":1560111536209.789,"type":"codec","payloadType":120,"mimeT
ype":"video/rtx","clockRate":90000},{"id":"RTCCodec_1_Inbound_121","timestamp":1560111536209.789,"type":"codec","payloadType":121,"mimeType":"video/rtx","clockRate":90000},{"id":"RTCCodec_1_Inbound_122","timestamp":1560111536209.789,"type":"codec","payloadType":122,"mimeType":"video/rtx","clockRate":90000},{"id":"RTCCodec_1_Inbound_123","timestamp":1560111536209.789,"type":"codec","payloadType":123,"mimeType":"video/H264","clockRate":90000},{"id":"RTCCodec_1_Inbound_124",
"timestamp":1560111536209.789,"type":"codec","payloadType":124,"mimeType":"video/H264","clockRate":90000},{"id":"RTCCodec_1_Inbound_125","timestamp":1560111536209.789,"type":"codec","payloadType":125,"mimeType":"video/H264","clockRate":90000},{"id":"RTCCodec_1_Inbound_127","timestamp":1560111536209.789,"type":"codec","payloadType":127,"mimeType":"video/H264","clockRate":90000},{"id":"RTCCodec_1_Inbound_96","timestamp":1560111536209.789,"type":"codec","payloadType":9
6,"mimeType":"video/VP8","clockRate":90000},{"id":"RTCCodec_1_Inbound_97","timestamp":1560111536209.789,"type":"codec","payloadType":97,"mimeType":"video/rtx","clockRate":90000},{"id":"RTCCodec_1_Inbound_98","timestamp":1560111536209.789,"type":"codec","payloadType":98,"mimeType":"video/VP9","clockRate":90000},{"id":"RTCCodec_1_Inbound_99","timestamp":1560111536209.789,"type":"codec","payloadType":99,"mimeType":"video/rtx","clockRate":90000},{"id":"RTCCodec_1_Outbound_
100","timestamp":1560111536209.789,"type":"codec","payloadType":100,"mimeType":"video/VP9","clockRate":90000},{"id":"RTCCodec_1_Outbound_101","timestamp":1560111536209.789,"type":"codec","payloadType":101,"mimeType":"video/rtx","clockRate":90000},{"id":"RTCCodec_1_Outbound_102","timestamp":1560111536209.789,"type":"codec","payloadType":102,"mimeType":"video/H264","clockRate":90000},{"id":"RTCCodec_1_Outbound_107","timestamp":1560111536209.789,"type":"codec","payload
Type":107,"mimeType":"video/rtx","clockRate":90000},{"id":"RTCCodec_1_Outbound_108","timestamp":1560111536209.789,"type":"codec","payloadType":108,"mimeType":"video/H264","clockRate":90000},{"id":"RTCCodec_1_Outbound_109","timestamp":1560111536209.789,"type":"codec","payloadType":109,"mimeType":"video/rtx","clockRate":90000},{"id":"RTCCodec_1_Outbound_114","timestamp":1560111536209.789,"type":"codec","payloadType":114,"mimeType":"video/red","clockRate":90000},{"id":"RTC
Codec_1_Outbound_115","timestamp":1560111536209.789,"type":"codec","payloadType":115,"mimeType":"video/rtx","clockRate":90000},{"id":"RTCCodec_1_Outbound_116","timestamp":1560111536209.789,"type":"codec","payloadType":116,"mimeType":"video/ulpfec","clockRate":90000},{"id":"RTCCodec_1_Outbound_119","timestamp":1560111536209.789,"type":"codec","payloadType":119,"mimeType":"video/rtx","clockRate":90000},{"id":"RTCCodec_1_Outbound_120","timestamp":1560111536209.789,"typ
e":"codec","payloadType":120,"mimeType":"video/rtx","clockRate":90000},{"id":"RTCCodec_1_Outbound_121","timestamp":1560111536209.789,"type":"codec","payloadType":121,"mimeType":"video/rtx","clockRate":90000},{"id":"RTCCodec_1_Outbound_122","timestamp":1560111536209.789,"type":"codec","payloadType":122,"mimeType":"video/rtx","clockRate":90000},{"id":"RTCCodec_1_Outbound_123","timestamp":1560111536209.789,"type":"codec","payloadType":123,"mimeType":"video/H264","clockRate"
:90000},{"id":"RTCCodec_1_Outbound_124","timestamp":1560111536209.789,"type":"codec","payloadType":124,"mimeType":"video/H264","clockRate":90000},{"id":"RTCCodec_1_Outbound_125","timestamp":1560111536209.789,"type":"codec","payloadType":125,"mimeType":"video/H264","clockRate":90000},{"id":"RTCCodec_1_Outbound_127","timestamp":1560111536209.789,"type":"codec","payloadType":127,"mimeType":"video/H264","clockRate":90000},{"id":"RTCCodec_1_Outbound_96","timestamp":15
60111536209.789,"type":"codec","payloadType":96,"mimeType":"video/VP8","clockRate":90000},{"id":"RTCCodec_1_Outbound_97","timestamp":1560111536209.789,"type":"codec","payloadType":97,"mimeType":"video/rtx","clockRate":90000},{"id":"RTCCodec_1_Outbound_98","timestamp":1560111536209.789,"type":"codec","payloadType":98,"mimeType":"video/VP9","clockRate":90000},{"id":"RTCCodec_1_Outbound_99","timestamp":1560111536209.789,"type":"codec","payloadType":99,"mimeType":"video/r
tx","clockRate":90000},{"id":"RTCIceCandidatePair_eDlSn4+Z_o4HO2DpI","timestamp":1560111536209.789,"type":"candidate-
pair","transportId":"RTCTransport_0_1","localCandidateId":"RTCIceCandidate_eDlSn4+Z","remoteCandidateId":"RTCIceCandidate_o4HO2DpI","state":"succeeded","priority":9115038255631187000,"nominated":true,"writable":true,"bytesSent":43425072,"bytesReceived":196122,"totalRoundTripTime":0.031,"currentRoundTripTime":0.001,"availableOutgoingBitrate":3539445,"requestsReceived":82,"requestsSent":1,"responsesReceived":82,"responsesSent":82,"consentRequestsSent":81},{"id":"RTCIceCandidate_
eDlSn4+Z","timestamp":1560111536209.789,"type":"local-candidate","transportId":"RTCTransport_0_1","isRemote":false,"networkType":"wifi","ip":"10.10.100.49","port":63479,"protocol":"udp","candidateType":"host","priority":2122260223,"deleted":false},{"id":"RTCIceCandidate_o4HO2DpI","timestamp":1560111536209.789,"type":"remote-
candidate","transportId":"RTCTransport_0_1","isRemote":true,"ip":"10.10.100.49","port":52235,"protocol":"udp","candidateType":"host","priority":2122260223,"deleted":false},{"id":"RTCMediaStreamTrack_sender_3","timestamp":1560111536209.789,"type":"track","trackIdentifier":"93c3b5d1-5ad2-4d6f-b4ef-
9026d140cd12","remoteSource":false,"ended":false,"detached":false,"kind":"audio","audioLevel":0,"totalAudioEnergy":0,"totalSamplesDuration":0},{"id":"RTCMediaStreamTrack_sender_4","timestamp":1560111536209.789,"type":"track","trackIdentifier":"a6b75d65-225e-4a17-8566-
5ebfde288572","remoteSource":false,"ended":false,"detached":false,"kind":"video","frameWidth":640,"frameHeight":480,"framesSent":5883,"hugeFramesSent":3},{"id":"RTCMediaStream_OIL9zTwVhYSpGstFDdnOlBGaKGbLz16BJnlb","timestamp":1560111536209.789,"type":"stream","streamIdentifier":"OIL9zTwVhYSpGstFDdnOlBGaKGbLz16BJnlb","trackIds":["RTCMediaStreamTrack_sender_3","RTCMediaStreamTrack_sender_4"]},{"id":"RTCOutboundRTPAudioStream_3291843167","timestamp":156011153
6209.789,"type":"outbound-rtp","ssrc":3291843167,"isRemote":false,"mediaType":"audio","kind":"audio","trackId":"RTCMediaStreamTrack_sender_3","transportId":"RTCTransport_0_1","codecId":"RTCCodec_0_Outbound_111","packetsSent":9748,"retransmittedPacketsSent":0,"bytesSent":856558,"retransmittedBytesSent":0},{"id":"RTCOutboundRTPVideoStream_3763384231","timestamp":1560111536209.789,"type":"outbound-
rtp","ssrc":3763384231,"isRemote":false,"mediaType":"video","kind":"video","trackId":"RTCMediaStreamTrack_sender_4","transportId":"RTCTransport_0_1","codecId":"RTCCodec_1_Outbound_96","firCount":0,"pliCount":0,"nackCount":0,"qpSum":31335,"packetsSent":39380,"retransmittedPacketsSent":0,"bytesSent":42009407,"retransmittedBytesSent":0,"framesEncoded":5883,"totalEncodeTime":24.969},{"id":"RTCPeerConnection","timestamp":1560111536209.789,"type":"peer-
connection","dataChannelsOpened":0,"dataChannelsClosed":0},{"id":"RTCTransport_0_1","timestamp":1560111536209.789,"type":"transport","bytesSent":43425072,"bytesReceived":196122,"dtlsState":"connected","selectedCandidatePairId":"RTCIceCandidatePair_eDlSn4+Z_o4HO2DpI","localCertificateId":"RTCCertificate_70:47:DC:96:5C:24:A1:C1:F0:A2:89:39:FC:B1:D9:69:66:EB:4A:F6:12:F0:68:43:1B:F9:D4:ED:D6:E2:D1:FE","remoteCertificateId":"RTCCertificate_D1:3F:71:1A:FE:32:78:3D:AA:93:06:F3:E6:
AF:22:E9:5A:BC:B7:D2:AD:D6:A4:49:FD:9F:FB:00:49:89:4C:5A"},{"id":"RTCCertificate_70:47:DC:96:5C:24:A1:C1:F0:A2:89:39:FC:B1:D9:69:66:EB:4A:F6:12:F0:68:43:1B:F9:D4:ED:D6:E2:D1:FE","timestamp":1560111849871.613,"type":"certificate","fingerprint":"70:47:DC:96:5C:24:A1:C1:F0:A2:89:39:FC:B1:D9:69:66:EB:4A:F6:12:F0:68:43:1B:F9:D4:ED:D6:E2:D1:FE","fingerprintAlgorithm":"sha-
pair","transportId":"RTCTransport_0_1","localCandidateId":"RTCIceCandidate_eDlSn4+Z","remoteCandidateId":"RTCIceCandidate_o4HO2DpI","state":"succeeded","priority":9115038255631187000,"nominated":true,"writable":true,"bytesSent":114210539,"bytesReceived":509086,"totalRoundTripTime":0.085,"currentRoundTripTime":0.001,"availableOutgoingBitrate":3580817,"requestsReceived":207,"requestsSent":1,"responsesReceived":207,"responsesSent":207,"consentRequestsSent":206},{"id":"RTCIceCandi
date_eDlSn4+Z","timestamp":1560111849871.613,"type":"local-candidate","transportId":"RTCTransport_0_1","isRemote":false,"networkType":"wifi","ip":"10.10.100.49","port":63479,"protocol":"udp","candidateType":"host","priority":2122260223,"deleted":false},{"id":"RTCIceCandidate_o4HO2DpI","timestamp":1560111849871.613,"type":"remote-
connection","dataChannelsOpened":0,"dataChannelsClosed":0},{"id":"RTCTransport_0_1","timestamp":1560111849871.613,"type":"transport","bytesSent":114210539,"bytesReceived":509086,"dtlsState":"connected","selectedCandidatePairId":"RTCIceCandidatePair_eDlSn4+Z_o4HO2DpI","localCertificateId":"RTCCertificate_70:47:DC:96:5C:24:A1:C1:F0:A2:89:39:FC:B1:D9:69:66:EB:4A:F6:12:F0:68:43:1B:F9:D4:ED:D6:E2:D1:FE","remoteCertificateId":"RTCCertificate_D1:3F:71:1A:FE:32:78:3D:AA:93:06:F3:E
6:AF:22:E9:5A:BC:B7:D2:AD:D6:A4:49:FD:9F:FB:00:49:89:4C:5A"},{"id":"RTCCertificate_70:47:DC:96:5C:24:A1:C1:F0:A2:89:39:FC:B1:D9:69:66:EB:4A:F6:12:F0:68:43:1B:F9:D4:ED:D6:E2:D1:FE","timestamp":1560111929491.604,"type":"certificate","fingerprint":"70:47:DC:96:5C:24:A1:C1:F0:A2:89:39:FC:B1:D9:69:66:EB:4A:F6:12:F0:68:43:1B:F9:D4:ED:D6:E2:D1:FE","fingerprintAlgorithm":"sha-
pair","transportId":"RTCTransport_0_1","localCandidateId":"RTCIceCandidate_eDlSn4+Z","remoteCandidateId":"RTCIceCandidate_o4HO2DpI","state":"succeeded","priority":9115038255631187000,"nominated":true,"writable":true,"bytesSent":132172362,"bytesReceived":588712,"totalRoundTripTime":0.097,"currentRoundTripTime":0,"availableOutgoingBitrate":3580817,"requestsReceived":239,"requestsSent":1,"responsesReceived":239,"responsesSent":239,"consentRequestsSent":238},{"id":"RTCIceCandidate
_eDlSn4+Z","timestamp":1560111929491.604,"type":"local-candidate","transportId":"RTCTransport_0_1","isRemote":false,"networkType":"wifi","ip":"10.10.100.49","port":63479,"protocol":"udp","candidateType":"host","priority":2122260223,"deleted":false},{"id":"RTCIceCandidate_o4HO2DpI","timestamp":1560111929491.604,"type":"remote-
connection","dataChannelsOpened":0,"dataChannelsClosed":0},{"id":"RTCTransport_0_1","timestamp":1560111929491.604,"type":"transport","bytesSent":132172362,"bytesReceived":588712,"dtlsState":"connected","selectedCandidatePairId":"RTCIceCandidatePair_eDlSn4+Z_o4HO2DpI","localCertificateId":"RTCCertificate_70:47:DC:96:5C:24:A1:C1:F0:A2:89:39:FC:B1:D9:69:66:EB:4A:F6:12:F0:68:43:1B:F9:D4:ED:D6:E2:D1:FE","remoteCertificateId":"RTCCertificate_D1:3F:71:1A:FE:32:78:3D:AA:93:06:F3:E
6:AF:22:E9:5A:BC:B7:D2:AD:D6:A4:49:FD:9F:FB:00:49:89:4C:5A"}]

Several data dimensions
Transport Endpoint Platform Infrastructure
ISP name /
AS number
Device type
Topology
(P2P / SFU / MCU)
User network:
End-user location , Network
Type
Operating system Server locations
Media engine:
Audio and video frame rate
and size variations
Browser / RTC stack Server network
Packetization:
Round Trip Time,
Jitter, Packet loss
CPU type App Version

Challenge: dealing with lots of data
appVer
buildVer
buildName
osName
osVer
totalMeanAudioRtt
totalMeanVideoRtt
countVideoRtt
countAudioRtt
totalInboundAudioPacketsLost
totalInboundVideoPacketsLost
totalOutboundAudioPacketsLost
totalOutboundVideoPacketsLost
totalInboundAudioPackets
totalInboundVideoPackets
totalOutboundAudioPackets
totalOutboundVideoPackets
totalMeanInboundVideoThroughput
countInboundVideoThroughput
totalMeanInboundAudioThroughput
countInboundAudioThroughput
totalMeanOutboundAudioThroughput
countOutboundAudioThroughput
totalMeanOutboundVideoThroughput
Many Metrics

appVer
buildVer
buildName
osName
osVer
totalMeanAudioRtt
totalMeanVideoRtt
countVideoRtt
countAudioRtt
Many Metrics Measures×
• Average
• Mean
• Percentile
• Skew
• Kurtosis

appVer
buildVer
buildName
osName
osVer
totalMeanAudioRtt
totalMeanVideoRtt
countVideoRtt
countAudioRtt
Many Metrics Measures× × n samples
• Average
• Mean
• Percentile
• Skew
• Kurtosis
• Sample rate
• Number of participants
• Call duration

appVer
buildVer
buildName
osName
osVer
totalMeanAudioRtt
totalMeanVideoRtt
countVideoRtt
countAudioRtt
Many Metrics Measures× × n samples
• Average
• Mean
• Percentile
• Skew
• Kurtosis
• Sample rate
• Number of participants
• Call duration
× m customers
• Customer segments

Why do we need ML?
Reducing costs
1
Image source: http://xingwu.me/2014/12/10/My-AWS-Account-Got-Compromised/

Challenge: figuring out what matters
Customer
Value
Operating
Complexity

Why do we need ML?
Saving users from analytics overload
1

Typical troubleshooting approach
17
Generate
hypothesis
Look for
anomalies
Filter
Evaluate
sub-
segment

Traditional troubleshooting challenges
18
Generate
hypothesis
Look for
anomalies
Filter
Evaluate
sub-
segment
Experience
required to
generate good
hypotheses
Not all anomalies
are obvious
Many filters to
define
Many sub-
segments to
review

Ideal solution:
the system should tell you where to look
19

Finding Hidden Call Quality Issues with
Machine Learning
20
Deliver Better User Experiences

Machine Learning approaches
21
Image source: https://towardsdatascience.com/deep-learning-for-image-classification-why-its-challenging-where-we-ve-been-and-what-
s-next-93b56948fcef

Machine Learning approaches
22
Image source: https://towardsdatascience.com/deep-learning-for-image-classification-why-its-challenging-where-we-ve-been-and-what-
s-next-93b56948fcef

Methodology
23
Collect
data
Check the
data
Clean the
data
Feature
reduction
Clustering Labeling

Clean your data
• Missing values
• Duplicates
• Outliers
• Balancing
• Normalization
26

Dimension reduction
27
Feature
elimination
Feature
selection
Feature
extraction
Reducing unnecessary complexity

Feature selection
Correlation matrix to determine significance
28
meanAudioRtt meanVideoRtt Local Latitude
Local
Longitude
Server
Latitude
Server
Longitude
meanAudioRtt 100% 42% -24% 59% 0% 0%
meanVideoRtt 42% 100% -15% 29% 0% 0%
Local Latitude -24% -15% 100% -23% 0% 0%
Local Longitude 59% 29% -23% 100% 0% 0%
Server Latitude 0% 0% 0% 0% 100% -100%
Server Longitude 0% 0% 0% 0% -100% 100%

Round Trip Time (RTT)
29
Round-trip time is the time it takes for a packet to travel through an IP network,
from a sending endpoint to a receiving endpoint and back, not including the time
to process the packet at its destination. Many factors affect RTT, like propagation
delay, processing delay, queuing delay, and transmission delay.
Sender ReceiverNetwork
𝑡 𝑠𝑡𝑎𝑟𝑡
𝑡 𝑎𝑐𝑘
𝑅𝑇𝑇 = 𝑡 𝑎𝑐𝑘 − 𝑡 𝑠𝑡𝑎𝑟𝑡 − 𝑡 𝑠𝑙𝑟
𝑡 𝑠𝑙𝑟

Feature reduction
Principal Component Analysis
30
http://setosa.io/ev/principal-component-analysis/
• Find feature-combinations that have the most variation
• Creates a new set of dimensions to maximize these variations
• Remove the feature-combinations that don’t show much variation

Clustering with Gaussian Mixture Model
(GMM)
31

Optimizing the model
32
Overfit model
Fit model
Minimize the number of
variables needed in your
model
Minimize the number of
clusters
Maintain predictiveness on
new datasets

Determining the optimal cluster size with
Bayesian Inference Criterion (BIC)
33Number of Clusters
BICScore
Number of PCA
dimensions

Methodology: Iterative in practice
34
Collect
data
Check the
data
Clean the
data
Dimension
reduction
Clustering Labeling

Example of a customer handling
Millions of calls a day
3

RTT distribution for this service is
36
Range
Proportion of the
data (%)
Low 0 < RTT < 150 68%
Medium 150 < RTT < 250 19%
Large 250 < RTT < 700 12%
Very large RTT > 700 1%

RTT distribution for this service is
37
Area for
exploration
Range
Proportion of the
data (%)
Low 0 < RTT < 150 68%
Medium 150 < RTT < 250 19%
Large 250 < RTT < 700 12%
Very large RTT > 700 1%

Cluster 0: 16% of high RTTs
• Network conditions are good
• Local users in a specific country
• Distance can be the main cause for large
RTTs

• Bad network conditions
• Large jitter, Large fraction loss
• Users all over the world
• Roughly large distance
• Network congestion and distance can be
the main cause of large RTTs
42

• Good Network condition
• Local users in Asia
• Main cause of large RTT can be large
distance

Analysis summary
44
32% of calls
have high
RTT's
20% due to
congestion
2 geographic
problem
areas
Check server infrastructure & network
Add/move servers

Real solution:
the system tells you where to look
45

All thingsrtc finding hidden issues with ml

All thingsrtc finding hidden issues with ml

Recommandé

Recommandé

Contenu connexe

Similaire à All thingsrtc finding hidden issues with ml

Similaire à All thingsrtc finding hidden issues with ml (20)

Plus de callstats.io

Plus de callstats.io (16)

Dernier

Dernier (20)

All thingsrtc finding hidden issues with ml