This presentation contains a tutorial devoted to showing how Java developers can create rich multimedia applications with Kurento. Java developers will find natural Kurento development model, which is based on standard Java EE technologies and is inspired on the WWW Servlet model.
If you have ever developed a Web application, you may be familiar with this scheme. At the browser, HTML and JavaScript code is in charge of user interaction and generates HTTP requests to the server. This code is usually programmed with the help of APIs such as jQuery, DOM, XHR or others. Upon reception, HTTP requests are processed by some kind of server side technology (e. g.. PHP, Java, Ruby, etc.) using service APIs providing features such as DB access, communications, transactions, XML parsing, and others. As a result, an HTTP response is issued and sent back to the client. Following this scheme, both server and client side APIs are just capabilities simplifying developer work and providing abstractions for programming faster and more efficiently.
Kurento technologies adapt to the Web development model so that, from a programmer perspective, Kurento can be seen just as an additional set of APIs. Developers does not need to learn novel programming schemes and can reuse all their knowledge and previous background on WWW application development. When you need multimedia, just use Kurento APIs. For the rest, use your preferred APIs or reuse previous code. Kurento APIs have been designed for simplicity and Web developers will find them familiar and intuitive. Most of the low level details related to codecs, formats, protocols, profiles and containers are abstracted by the framework. Programmers just concentrate on specifying the sequence of processing steps that they want to execute on the media flows.
Handwritten Text Recognition for manuscripts and early printed texts
Developing rich multimedia applications with Kurento: a tutorial for Java Developers
1. www.kurento.org
Real-time Multimedia Stream Processing
Developing rich multimedia applications with KurentoDeveloping rich multimedia applications
with Kurento: a tutorial for Java
developers
lulop@kurento.org
http://www.kurento.org
2. www.kurento.org
Multimedia infrastructures for the
Future Internet
Media is
here
Media got
there
Media got
there
Analyze
Transform
Store
Transport
Enrich
Augment
Adapt
Sensors
Context
Events
Media is
here
2
5. www.kurento.org
•Interoperable media exchange (multiplatform/multiprotocol)
•WebRTC, RTP, HTTP (video tag), etc.
•Process media (Computer vision, augmented reality, media indexing, etc.)
•Media and metadata recording and recovery
•Transform and adapt media (H.264, H.263, VP8, Ogg, and others)
•Media routing and mixing
•Etc.
Multimedia infrastructure
• REST API
• JavaScript API
• Java API
APIs
• LGPL 2.1
Is distributed through a flexible FOSS license
What’s Kurento
5
6. www.kurento.org
Kurento Media Server (KMS): the
nucleus of Kurento
• KMS is a middleware for media streams
– Receives the stream
– Process the stream
– Issues the stream
Send
Receive
Analyze
Augment
Enrich
Transform
Transcode
Record
Process
Replicate
Media
Source
Media
Sink
KMS
6
7. www.kurento.org
The Media API: The API for accessing
KMS capabilities
Send
Receive
Analyze
Augment
Enrich
Transform
Transcode
Record
Process
Replicate
Media
Source
Media
Sink
KMS
Java
Media API
JavaScript
Media API
REST
API
Applications define the processing
of streams getting through KMS
7
8. www.kurento.org
Media API: Media Elements and Media
Pipelines
SinkSRC
Sink
SRC
SRCSink
Sink
Media Element
• Provides a specific media
functionality
› Send/receive media
› Process media
› Transform media
• Exchange media through
› Sources
› Sinks
Media pipeline
• Chain of media elements
implementing the desired media
logic.
• The Media API provides the
capability of creating media
pipelines by joining media
elements of the toolbox
Media Element
Sink
SRC
8
9. www.kurento.org
Media API: trivial example (Java)
MediaPipeline mp = contentSession.getMediaPipelineFactory().create();
PlayerEndpoint playerEndpoint = mp.newPlayerEndpoint(file:///myFile.webm).build();
HttpGetEndpoint httpEndpoint = mp.newHttpGetEndpoint().terminateOnEOS().build();
playerEndpoint.connect(httpEndpoint);
httpEndpoint.getUrl(); //URL where the media is made available
Media Pipeline (inside KMS)
HttpGetEndpoint
Media from
file or URI
HTTP media
streaming
Sink
SRC
PlayerEndpoint
9
10. www.kurento.org
Is that enough? Think about the WWW
development model
Process WWW request
- DDBB access
- Authentication
- XML processing
- Etc.
HTTP request:
I want this resource
HTTP response:
The resource
• Intuition behind the WWW model
– Client asks what it wants
• GET /this_resource HTTP/1.1
– Server side APIs execute the associated processing
You develop this using …
• Servlet
• JSP
• ASP
• PHP
• Ruby on Rails
• CGI
• Etc.
10
11. www.kurento.org
We need an equivalent model: the role
of the Signaling Plane
• We need “something” with the capability of negotiating what,
when and how media is exchanged
• A protocol for “saying”
– I want “this media” …
• Identification of the media to exchange
– File in hard-drive, IP camera, user, etc.
– in “this way” …
• Identification of the processing of media
– Augmented, analyzed, etc.
– with “this format” …
• Quality
– Codec, screen-size, frame-rate, etc.
– at “this moment”
• Stream control
– Play, stop, start, pause, etc.
11
12. www.kurento.org
Don’t get it? think about WWW
development again …
Process WWW request
- DDBB access
- Authentication
- XML processing
- Etc.
HTTP request:
I want this resource
HTTP response:
The resource
Process media request
- Media API
- DDBB access
- Authentication
- XML processing
- Etc.
Signaling request:
I want this media
Signaling response:
The media is here
This is how you
create WWW
applications
(Servlets, ASP, PHP,
Rails, etc.)
This is how you create
Kurento enabled
applications:
Multimedia RTC is just
another feature of your
application
You, as a
programmer,
create this
12
13. www.kurento.org
Abstracting the signaling plane: The
Content Handler
KMS
SinkSRC
Sink
SRC
Sink
SRCSink
Media API
REST API (Open API protocol) The Content Handler
Equivalent to a Servlet/ASP/PHP script
Exposes APIs for specifying:
- when receiving “this request”…
- … execute “this logic”
Developer can use the media API
Code building the media pipeline and
executing the application logic the
developer wants
13
14. www.kurento.org
Kurento Application Server: The
container of Handlers
• Is an extension of Java EE technologies.
• Compatible with all Java EE Servlet
containers
• Hold Handlers
– Your specific application code
• Receives signaling requests:
– I want “this media” in this way …
• Dispatches request to the appropriate
handler
– @annotations based mapping
• Generate an answer showing “where
and how to find the media”
– URL where media is located
– Negotiated SDP
Java EE compatible container
HTTP
Servlet
SIP
Servlet
Web
service
s
Kurento
REST API
Specific handler implementations
Media API DD.BB.
Kurento Application Server (KAS)
Other java
APIs.
14
15. www.kurento.org
Kurento Architecture: putting it all
together
Kurento Media Server (KMS)
Receive
Video
Augmented
Reality
Send
Video
Computer
Vision
Video Playing
and Recording
Java EE compatible container
HTTP
Servlet
SIP
Servlet
Web
service
s
Kurento
REST API
Specific handler implementations
Signaling and
WWW traffic
Media
Media
Signaling and
WWW traffic
Media API DD.BB.
Kurento Application Server (KAS)
Other java
APIs.
15
16. www.kurento.org
Application execution flow
Client
Code
Application
Server (KAS)
Media
Server (KMS)
I want this media in this way …
(JSON-RPC request)
Commands requesting
the creation of a pipeline
What you want is here …
(JSON-RPC answer)
Media
negotiation
phase
Media
exchange
phase
1
2
Specific application
logic at the
server-side
(Content Handler)
Media
pipeline
creation
Media exchange between client and server
16
17. www.kurento.org
Content Handler: trivial example
//Specify the type of service provided by this handler: Player, Recorder, WebRTC, RTP, etc.
@HttpPlayerService(path = "/player”) //Mapping of handler specified in path
public class MyPlayerHandler extends HttpPlayerHandler {
@Override
public void onContentRequest(HttpPlayerSession contentSession) {
//Thie client wants the media this handler provides
//Create the pipeline for providing the media
}
@Override
public void onContentStarted(HttpPlayerSession contentSession) {
//Media started flowing, you can execute additional actions
}
@Override
Public void onSessionTerminated(HttpPlayerSession contentSenssion){
//Media exchange termianted, you can collect your resources
}
17
18. www.kurento.org
Let’s develop with Kurento
• What you need
– A Kurento instance
• You can install your own Kurento instance
• You can launch a Kurento instance at the FI-LAB (FI-WARE project)
– http://lab.fi-ware.org
• Getting help
– Kurento web site
• http://www.kurento.org
– Kurento mailing list
• https://groups.google.com/forum/#!forum/kurento
– Twitter
• @kurentoms
18
19. www.kurento.org
Kurento Hello World: Playing a file
with an HTML5 client
Media Pipeline
HttpGetEndpoint
Media from
file or URI
HTTP media
streaming
Sink
SRC
PlayerEndpoint
Media API
REST API (Open API protocol)
Handler codeI want
“this media”
Media is
“at this URL”
19
20. www.kurento.org
Playing a file with an HTML5 client:
Handler code@HttpPlayerService(path = "/player”)
public class MyPlayerHandler extends HttpPlayerHandler {
@Override
public void onContentRequest(HttpPlayerSession contentSession) throws Exception {
MediaPipeline mp = contentSession.getMediaPipelineFactory().create();
contentSession.releaseOnTerminate(mp);
PlayerEndpoint playerEndpoint = mp.newPlayerEndpoint(
"http://media.w3.org/2010/05/sintel/trailer.webm").build();
contentSession.setAttribute("player", playerEndpoint);
HttpGetEndpoint httpEndpoint = mp.newHttpGetEndpoint().terminateOnEOS().build();
playerEndpoint.connect(httpEndpoint);
contentSession.start(httpEndpoint);
}
@Override
public void onContentStarted(HttpPlayerSession contentSession) {
PlayerEndpoint playerEndpoint = (PlayerEndpoint) contentSession.getAttribute("player");
playerEndpoint.play();
}
}
Source: https://github.com/Kurento/kurento-media-framework/blob/develop/kmf-samples/kmf-
tutorial/src/main/java/com/kurento/tutorial/MyPlayerHandler.java
20
21. www.kurento.org
Playing a file with an HTML5 client:
client code<!DOCTYPE html>
<html>
<head>
<script src="./js/kws-content-api.js"></script>
<script>
var conn;
function start() {
var options = {
remoteVideoTag : "remoteVideo"
};
conn = new kwsContentApi.KwsContentPlayer("./player", options);
}
function terminate() {
conn.terminate();
}
</script>
</head>
<body>
<button onclick="start();">Start</button>
<button onclick="terminate();">Terminate</button>
<br />
<video id="remoteVideo" autoplay></video>
</body>
</html>
Source: https://github.com/Kurento/kurento-media-framework/blob/develop/kmf-samples/kmf-
tutorial/src/main/webapp/player.html
21
22. www.kurento.org
Playing a file with an HTML5 client:
See the example working
https://www.youtube.com/watch?v=n5BQlhYgGSo
22
23. www.kurento.org
Media Pipeline
Adding Computer Vision
HttpGetEndpoint
Media
from
file or
URI
HTTP media
streaming
Sink
SRC
PlayerEndpoint
SRC
Sink
JackVaderFilter
Media API
REST API (Open API protocol)
Handler codeI want
“this media”
Media is
“at this URL”
23