Developing applications with Kurento
•
4 j'aime•4,480 vues
This presentation was used in the context of a FI-WARE webminar for introducing Kurento. Kurento is a framework for building multimedia and streaming applications based on predefined blocks. Send and receive median through RTP, WebRTC, HTTP and RTSP. Use processing for making face detection, plate recognition or object tracking. Use augmented reality, group communications or media mixing and blending among others. During the webminar, we used Kurento APIs for showing how to create media applications for videoconferencing or video streaming in a simple and seamless manner. We also demonstrated how these applications can be enriched with Kurento's advanced processing capabilities.
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à Developing applications with Kurento
Similaire à Developing applications with Kurento (20)
Dernier
Dernier (20)
Developing applications with Kurento
- 1. Open APIs for Open Minds Real-time Multimedia Stream Processing Developing rich multimedia applications with Kurento
- 2. Mul$media infrastructures for the Future Internet 2 Media is here Media got there Media got there Analyze Transform Store Transport Enrich Augment Adapt Sensors Context Events Media is here
- 3. Enrich, augment, adapt, analyze, transform, store: what’s the problem? 3 Complexity
- 4. Future Internet Mul$media Infrastructure Simple Development APIs The FI-‐WARE Stream-‐oriented Generic Enabler Implementa$on 4
- 5. 5 • Interoperable media exchange (mul$plaNorm/mul$protocol) • WebRTC, RTP, HTTP (video tag), etc. • Process media (Computer vision, augmented reality, media indexing, etc.) • Media and metadata recording and recovery • Transform and adapt media (H.264, H.263, VP8, Ogg, and others) • Media rou$ng and mixing • Etc. Provides mul$media capabili$es to the FI-‐WARE infrastructure • REST API • JavaScript API • Java API Exposes those capabili$es through a simple to use APIs • LGPL 2.1 Is distributed through a flexible FOSS license Kurento
- 6. Kurento Media Server (KMS): the nucleus of Kurento 6 • KMS is a middleware for media streams – Receives the stream – Process the stream – Issues the stream Send Receive Analyze Augment Enrich Transform Transcode Record Process Replicate Media Source Media Sink KMS
- 7. The Media API: The API for accessing KMS capabili$es 7 Send Receive Analyze Augment Enrich Transform Transcode Record Process Replicate Media Source Media Sink KMS Java Media API JavaScript Media API Other languages Applica$ons define the processing of streams geang through KMS
- 8. Media API: Media Elements and Media Pipelines 8 Sink%SRC% Sink% SRC% SRC%Sink% Sink% § Media Element • Provides a specific media functionality › Send/receive media › Process media › Transform media • Exchange media through › Sources › Sinks § Media pipeline • Chain of media elements implementing the desired media logic. • The Media API provides the capability of creating media pipelines by joining media elements of the toolbox Media Element Sink SRC
- 9. Developers create applica$ons just connec$ng Media Elements 9 Protocols( &(Codecs( Computer(( Vision( Augmented(( Reality( Mul9sensory(( Mul9media( Media(( Repository( IPTV( Integra9on( Group( Communica9ons( Sink% H>pEndpoint( Sink%SRC% RtpEndpoint( SRC$ DataChannelEndpoint( Sink%SRC% WebRtcEndpoint( Sink% SRC% BarCodeReader( Sink% SRC% PointerTracker( Sink% SRC% FaceDetector( Sink% SRC% FaceOverlay( Sink% SRC% ChromaFilter( Sink% SRC% HeartRateBlender( Sink% RecorderEndpoint( SRC$ PlayerEndpoint( Sink% CdnUpload( Sink%IPTVConnector( Mixer( Sink%SRC% Sink% SRC% Sink% SRC%Sink% Sink% Applica$on 2 Sink% SRC% Sink% SRC%Sink% Sink%SRC% Sink% Applica$on 3 Sink%SRC% SRC%Sink% SRC% Sink% Sink%SRC% Applica$on 1 Toolbox of media elements
- 10. Media API: trivial example (Java) MediaPipeline mp = contentSession.getMediaPipelineFactory().create(); PlayerEndpoint playerEndpoint = mp.newPlayerEndpoint(file:///myFile.webm).build(); H2pGetEndpoint hepEndpoint = mp.newHepGetEndpoint().terminateOnEOS().build(); playerEndpoint.connect(hepEndpoint); hepEndpoint.getUrl(); //URL where the media is made available 10 Media Pipeline HepGetEndpoint Media from file or URI HTTP media streaming Sink SRC PlayerEndpoint
- 11. Is that enough? Think about the WWW development model 11 Process WWW request -‐ DDBB access -‐ Authen$ca$on -‐ XML processing -‐ Etc. HTTP request: I want this resource HTTP response: The resource • Intui$on behind the WWW model – Client asks what it wants – Server side APIs execute the associated processing
- 12. We need an equivalent model: the role of the Signaling Plane • The API must provide nego$a$on capabili$es – I want “this media” … • Iden$fica$on of the media to exchange – File in hard-‐drive, IP camera, user, etc. – in “this way” … • Iden$fica$on of the processing of media – Augmented, analyzed, etc. – with “this format” … • Quality – Codec, screen-‐size, frame-‐rate, etc. – at “this moment” • Stream control – Play, stop, start, pause, etc. 12
- 13. Don’t get it? think about WWW development again … 13 Process WWW request -‐ DDBB access -‐ Authen$ca$on -‐ XML processing -‐ Etc. HTTP request: I want this resource HTTP response: The resource Process media request -‐ Media API -‐ DDBB access -‐ Authen$ca$on -‐ XML processing -‐ Etc. Signaling request: I want this media Signaling response: The media is here Intui$on behind tradi$onal WWW Applica$ons (Servlets, ASP, PHP, Rails, etc.) Intui$on behind Kurento development APIs: Mul$media RTC is just another feature of your applica$on
- 14. Dealing with the signaling: The Content Handler 14 KMS Sink SRC Sink SRC Sink SRC Sink Media API REST API (Open API protocol) The Content Handler Equivalent to a Servlet/ASP/PHP script -‐ When receiving “this request”… -‐ execute “this logic” Developer can use the media API Code building the media pipeline and execu$ng the applica$on logic the developer wants
- 15. Kurento Architecture 15 Kurento Media Server (KMS) Receive Video Augmented Reality Send Video Computer Vision Video Playing and Recording Java EE compaJble container HTTP Servlet SIP Servlet Web services Kurento REST API Specific handler implementa$ons Signaling and WWW traffic Media Media Signaling and WWW traffic Media API DD.BB. Kurento ApplicaJon Server (KAS) Other java APIs.
- 16. Applica$on execu$on flow Client Code Applica$on Server (KAS) Media Server (KMS) I want this media in this way … (JSON) Commands reques$ng the crea$on of a pipeline What you want is here … (JSON) Media negoJaJon phase Media exchange phase 1 2 Specific applica$on logic at the server-‐side (Content Handler) Media pipeline creaJon Media exchange between client and server
- 17. Content Handler: trivial example @H2pPlayerService(path = "/player”) public class MyPlayerHandler extends HepPlayerHandler { @Override public void onContentRequest(HepPlayerSession contentSession) { //Create the pipeline for providing media through HTTP } @Override public void onContentStarted(HepPlayerSession contentSession) { //Media started flowing, you can execute addi$onal ac$ons } @Override Public void onSessionTerminated(HepPlayerSession contentSenssion){ //Media exchange termianted, you can collect your resources }
- 18. Let’s develop with Kurento • What you need – A Kurento instance • You can install your own Kurento instance • You can launch a Kurento instance at the FI-‐LAB – hep://lab.fi-‐ware.org • Geang help – FI-‐WARE catalog entry • hep://catalogue.fi-‐ware.org/enablers/stream-‐oriented-‐kurento – Installa$on guide • heps://forge.fi-‐ware.org/plugins/mediawiki/wiki/fiware/index.php/ StreamOriented_-‐_Installa$on_and_Administra$on_Guide – Developer guide • heps://forge.fi-‐ware.org/plugins/mediawiki/wiki/fiware/index.php/ StreamOriented_-‐_User_and_Programmers_Guide – Kurento web site • hep://www.kurento.org 18
- 19. Kurento Hello World: Playing a file 19 Media Pipeline HepGetEndpoint Media from file or URI HTTP media streaming Sink SRC PlayerEndpoint Media API REST API (Open API protocol) Create the pipeline connec$ng: HepGetEndpoint PlayerEndpoint I want “this media”
- 20. Playing a file: Handler code @HepPlayerService(path = "/player”) public class MyPlayerHandler extends HepPlayerHandler { @Override public void onContentRequest(HepPlayerSession contentSession) throws Excep$on { MediaPipeline mp = contentSession.getMediaPipelineFactory().create(); contentSession.releaseOnTerminate(mp); PlayerEndpoint playerEndpoint = mp.newPlayerEndpoint( "h2p://media.w3.org/2010/05/sintel/trailer.webm").build(); contentSession.setAeribute("player", playerEndpoint); H2pGetEndpoint h2pEndpoint = mp.newH2pGetEndpoint().terminateOnEOS().build(); playerEndpoint.connect(h2pEndpoint); contentSession.start(hepEndpoint); } @Override public void onContentStarted(HepPlayerSession contentSession) { PlayerEndpoint playerEndpoint = (PlayerEndpoint) contentSession.getAeribute("player"); playerEndpoint.play(); } } 20 Source: heps://github.com/Kurento/kmf-‐tutorial/blob/master/src/main/java/com/kurento/tutorial/MyPlayerHandler.java
- 21. Playing a file: client code 21 <!DOCTYPE html> <html> <head> <script src="./js/kws-‐content-‐api.js"></script> <script> var conn; func$on start() { var op$ons = { remoteVideoTag : "remoteVideo" }; conn = new kwsContentApi.KwsContentPlayer("./player", op$ons); } func$on terminate() { conn.terminate(); } </script> </head> <body> <bueon onclick="start();">Start</bueon> <bueon onclick="terminate();">Terminate</bueon> <br /> <video id="remoteVideo" autoplay></video> </body> </html> Source: heps://github.com/Kurento/kmf-‐tutorial/blob/master/src/main/webapp/player.html
- 22. Media Pipeline Adding Computer Vision 22 HepGetEndpoint Media from file or URI HTTP media streaming Sink SRC PlayerEndpoint SRC Sink JackVaderFilter Media API REST API (Open API protocol) Create the pipeline connec$ng: HepGetEndpoint, Filter and PlayerEndpoint I want “this media”
- 23. Adding Computer Vision: Handler code 23 @HepPlayerService(path = "/playerWithFilter”) public class MyPlayerHandler extends HepPlayerHandler { @Override public void onContentRequest(HepPlayerSession contentSession) throws Excep$on { MediaPipeline mp = contentSession.getMediaPipelineFactory().create(); contentSession.releaseOnTerminate(mp); PlayerEndpoint playerEndpoint = mp.newPlayerEndpoint( "h2p://media.w3.org/2010/05/sintel/trailer.webm").build(); contentSession.setAeribute("player", playerEndpoint); JackVaderFilter filter = mp.newJackVaderFilter().build(); H2pGetEndpoint hepEndpoint = mp.newHepGetEndpoint().terminateOnEOS().build(); filter.connect(hepEndpoint); playerEndpoint.connect(filter); contentSession.start(hepEndpoint); } @Override public void onContentStarted(HepPlayerSession contentSession) { PlayerEndpoint playerEndpoint = (PlayerEndpoint) contentSession.getAeribute("player"); playerEndpoint.play(); } } Source: heps://github.com/Kurento/kmf-‐tutorial/blob/master/src/main/java/com/kurento/tutorial/MyPlayerWithFilter.java
- 24. Adding Computer Vision: Client code 24 <!DOCTYPE html> <html> <head> <script src="./js/kws-‐content-‐api.js"></script> <script> var conn; func$on start() { var op$ons = { remoteVideoTag : "remoteVideo" }; conn = new kwsContentApi.KwsContentPlayer("./playerWithFilter", op$ons); } func$on terminate() { conn.terminate(); } </script> </head> <body> <bueon onclick="start();">Start</bueon> <bueon onclick="terminate();">Terminate</bueon> <br /> <video id="remoteVideo" autoplay></video> </body> </html> Source: heps://github.com/Kurento/kmf-‐tutorial/blob/master/src/main/webapp/playerFilter.html
- 25. Adding Computer Vision: Client code 25 <!DOCTYPE html> <html> <head> <script src="./js/kws-‐content-‐api.js"></script> <script> var conn; func$on start() { var op$ons = { remoteVideoTag : "remoteVideo" }; conn = new kwsContentApi.KwsContentPlayer("./playerWithFilter", op$ons); } func$on terminate() { conn.terminate(); } </script> </head> <body> <bueon onclick="start();">Start</bueon> <bueon onclick="terminate();">Terminate</bueon> <br /> <video id="remoteVideo" autoplay></video> </body> </html> Source: heps://github.com/Kurento/kmf-‐tutorial/blob/master/src/main/webapp/playerFilter.html
- 26. 26 Source: heps://github.com/Kurento/kmf-‐tutorial/blob/master/src/main/webapp/playerFilter.html Thanks ! • hep://fi-‐ppp.eu • hep://fi-‐ware.org • hep://ww.kurento.org • Follow @Fiware on Twieer ! • Follow @Kurentoms on Twieer !