The document describes how WebRTC technology can be used to enable real-time video calls between doctors and patients. It outlines a solution built using Twilio, IBM Watson, and other Bluemix services. The solution allows a patient to initiate a video call with their doctor, with the patient's audio being transcribed in real-time using Watson Speech to Text. The transcription is stored and analyzed for personality insights. It is also sent back to the doctor. The solution was built in 3 days, but faced some challenges around ordering of calls, browser compatibility, scaling websockets, and dropping connections to Watson.

1. WebRTC Invades the Doctor’s Office

2. Jeff Sloyer
Developer Advocate
IBM Bluemix
@jsloyer

3. WebRTC Invades the Doctor’s Office

4. s://releaseblueprints.ibm.com/download/attachments/34868360/Green64.png?version=1&modificationDate=1392659234814
RUN APPS
YOUR WAY
What is Bluemix?
CATALOG OF
SERVICES / APIs
FLEXIBLE
TOOLING

5. s://releaseblueprints.ibm.com/download/attachments/34868360/Green64.png?version=1&modificationDate=1392659234814
RUN APPS
YOUR WAY
What is Bluemix?
CATALOG OF
SERVICES / APIs
FLEXIBLE
TOOLING
Instant Runtimes
Containers
VMs
Partner
IBM
Your Own
Open Source
Or Integrate Your Own
Use Ours

7. Web Page
Patient
1. Click to call
Doctor

8. Web Page
Patient
1. Click to call
Doctor
Twilio Video Chat
2. Initiate
WebRTC call

9. Web Page
Patient
1. Click to call
Doctor
Twilio Video Chat
2. Initiate
WebRTC call
Doctor
3. Doctor
comes online
for
appointment

10. Web Page
Patient
1. Click to call
Doctor
Twilio Video Chat
2. Initiate
WebRTC call
Doctor
3. Doctor
comes online
for
appointment
4. Call
established

11. Web Page
Patient
1. Click to call
Doctor
Twilio Video Chat
2. Initiate
WebRTC call
Doctor
3. Doctor
comes online
for
appointment
4. Call
established
5. Doctor and
patient video
chat

12. Web Page
Patient
1. Click to call
Doctor
Twilio Video Chat
2. Initiate
WebRTC call
Doctor
3. Doctor
comes online
for
appointment
4. Call
established
Node.Js app in Bluemix
.js6. Patient audio stream
sent over web sockets
to node backend
5. Doctor and
patient video
chat

13. Web Page
Patient
1. Click to call
Doctor
Twilio Video Chat
2. Initiate
WebRTC call
Doctor
3. Doctor
comes online
for
appointment
4. Call
established
Node.Js app in Bluemix
.js6. Patient audio stream
sent over web sockets
to node backend
Watson Speech
to Text
7. Patient audio stream
sent over web sockets
to IBM Watson
5. Doctor and
patient video
chat

14. Web Page
Patient
1. Click to call
Doctor
Twilio Video Chat
2. Initiate
WebRTC call
Doctor
3. Doctor
comes online
for
appointment
4. Call
established
Node.Js app in Bluemix
.js6. Patient audio stream
sent over web sockets
to node backend
Watson Speech
to Text
7. Patient audio stream
sent over web sockets
to IBM Watson
8. Audio transcribed in
real time
5. Doctor and
patient video
chat

15. Web Page
Patient
1. Click to call
Doctor
Twilio Video Chat
2. Initiate
WebRTC call
Doctor
3. Doctor
comes online
for
appointment
4. Call
established
Node.Js app in Bluemix
.js6. Patient audio stream
sent over web sockets
to node backend
Watson Speech
to Text
7. Patient audio stream
sent over web sockets
to IBM Watson
8. Audio transcribed in
real time
Cloudant NoSQL DB
9. Transcribed audio is
stored
5. Doctor and
patient video
chat

16. Web Page
Patient
1. Click to call
Doctor
Twilio Video Chat
2. Initiate
WebRTC call
Doctor
3. Doctor
comes online
for
appointment
4. Call
established
Node.Js app in Bluemix
.js6. Patient audio stream
sent over web sockets
to node backend
Watson Speech
to Text
7. Patient audio stream
sent over web sockets
to IBM Watson
8. Audio transcribed in
real time
Cloudant NoSQL DB
9. Transcribed audio is
stored
5. Doctor and
patient video
chat
10. Determine personality of
the patent
Watson Personality
Insights

17. Web Page
Patient
1. Click to call
Doctor
Twilio Video Chat
2. Initiate
WebRTC call
Doctor
3. Doctor
comes online
for
appointment
4. Call
established
Node.Js app in Bluemix
.js6. Patient audio stream
sent over web sockets
to node backend
Watson Speech
to Text
7. Patient audio stream
sent over web sockets
to IBM Watson
8. Audio transcribed in
real time
Cloudant NoSQL DB
9. Transcribed audio is
stored
5. Doctor and
patient video
chat
11. Transcribed audio is
sent back to the doctor
in real-time
10. Determine personality of
the patent
Watson Personality
Insights

18. Web Page
Patient
1. Click to call
Doctor
Twilio Video Chat
2. Initiate
WebRTC call
Doctor
3. Doctor
comes online
for
appointment
4. Call
established
Node.Js app in Bluemix
.js6. Patient audio stream
sent over web sockets
to node backend
Watson Speech
to Text
7. Patient audio stream
sent over web sockets
to IBM Watson
8. Audio transcribed in
real time
Cloudant NoSQL DB
9. Transcribed audio is
stored
5. Doctor and
patient video
chat
11. Transcribed audio is
sent back to the doctor
in real-time
10. Determine personality of
the patent
Watson Personality
Insights

19. Web Page
Patient
1. Click to call
Doctor
Twilio Video Chat
2. Initiate
WebRTC call
Doctor
3. Doctor
comes online
for
appointment
4. Call
established
Node.Js app in Bluemix
.js6. Patient audio stream
sent over web sockets
to node backend
Watson Speech
to Text
7. Patient audio stream
sent over web sockets
to IBM Watson
8. Audio transcribed in
real time
Cloudant NoSQL DB
9. Transcribed audio is
stored
5. Doctor and
patient video
chat
11. Transcribed audio is
sent back to the doctor
in real-time
10. Determine personality of
the patent
Watson Personality
Insights

20. I
Image Source: Shutterstock
WebRTC - iOS and the browser

21. I
Twilio Javascript SDK
Image Source: Shutterstock
WebRTC - iOS and the browser
Twilio iOS SDK

22. Image Source: Shutterstock

23. Image Source: Shutterstock
Built a solution in
3 days…

24. Image Source: Shutterstock

25. Image Source: Shutterstock
Code time!

26. Record the local stream
function startRecording(myStream) {
blah = myStream;
audioContext = new AudioContext();
var gain = audioContext.createGain();
var audioInput = audioContext.createMediaStreamSource(myStream);
audioInput.connect(gain);
microphone = audioContext.createScriptProcessor(bufferSize, inputChannels,
outputChannels);
microphone.onaudioprocess = _onaudioprocess.bind(this);
gain.connect(microphone);
microphone.connect(audioContext.destination);

27. Open a socket to our Node.Js app
mySocket = io.connect();
mySocket.on('connect', function() {
console.log('socket.onconnect()');
connected = true;
onstart();
});
mySocket.on('message', function(msg){
console.log(msg);
//console.log('demo.onresult()');
showResult(msg, "local");
});
}

28. Convert the audio to PCM
function _onaudioprocess(data) {
// Check the data to see if we're just getting 0s
// (the user isn't saying anything)
var chan = data.inputBuffer.getChannelData(0);
onAudio(_exportDataBuffer(new Float32Array(chan)));
}

29. Emit a web socket message
function onAudio(data) {
if (mySocket.connected) {
mySocket.emit('message', {audio: data, rate:
microphone.sampleRate});
}
};

30. Initiate a websocket connection from Node to
IBM Watsonsocket.on('message', function(data) {
if (!session.open) {
session.open = true;
var payload = {
session_id: session.session_id,
cookie_session: session.cookie_session,
content_type: 'audio/l16; rate=' + (data.rate || 48000),
continuous: true,
interim_results: true
};
//open a connection to ibm watson
session.req = speechToText.recognizeLive(payload, observe_results(socket, true));
//listen for result
speechToText.observeResult(payload, observe_results(socket, false));
} else if (data.disconnect) {
session.req.end();
} else {
session.req.write(data.audio);
}
});

31. Send transcribed text from Watson to web client
var observe_results = function(socket, recognize_end) {
var session = sessions[socket.id];
return function(err, chunk) {
if (err) {
console.log(log(socket.id), 'error:', err);
socket.emit('onerror', {
error: err
});
session.req.end();
socket.disconnect();
} else {
var transcript = (chunk && chunk.results && chunk.results.length > 0);
if (transcript && !recognize_end) {
socket.emit('message', chunk);
}
if (recognize_end) {
socket.disconnect();
}
}
};
};

32. Receive the transcribed text from our Node app
function showResult(data, streamLocation) {
var textElement = $("#collapse-" + patientId + " .transcript ." + streamLocation + ".text");
textElement.show();
textElement.parent().show();
//if there are transcripts
if (data.results && data.results.length > 0) {
//if is a partial transcripts
if (data.results.length === 1 ) {
var paragraph = textElement.children().last(),
text = data.results[0].alternatives[0].transcript || '';
//Capitalize first word
text = text.charAt(0).toUpperCase() + text.substring(1);
// if final results, append a new paragraph
if (data.results[0].final){
text = text.trim() + '.';
$('<p></p>').appendTo(textElement);
}
paragraph.text(text);
}
}
}

33. Image Source: Shutterstock

34. Lessons Learned…
Image Source: Shutterstock

35. Image Source: Shutterstock

36. Order for starting a
call is buggy
Image Source: Shutterstock

37. Image Source: Shutterstock

38. Issues with remote
streams support in
Chrome and Firefox
Image Source: Shutterstock

39. Image Source: Shutterstock

40. Multiple websocket
audio streams are
hard
Image Source: Shutterstock

42. Getting the audio
format right…

44. Personality of the
patient could not be
determined till after
the call

46. Scaling
websockets…

47. Image Source: Shutterstock

48. Spying on the media
stream
Image Source: Shutterstock

49. Image Source: Shutterstock

50. Websocket
connections
dropped to Watson
Image Source: Shutterstock

51. Image Source: My cat (Moose)

52. Questions?
Image Source: My cat (Moose)