fosdem 2016 - creating rich webrtc applications with kurento
TRANSCRIPT
![Page 2: FOSDEM 2016 - Creating rich WebRTC Applications with Kurento](https://reader035.vdocument.in/reader035/viewer/2022070515/5878339f1a28abef5d8b712d/html5/thumbnails/2.jpg)
http://www.kurento.org - [email protected]
Who I amAssociate Professor
Escuela Técnica Superior de Ingenieros de TelecomunicaciónUniversidad Rey Juan Carlos (south Madrid)
Researcher in the area of RTC
• http://www.nubomedia.eu• http://www.fiware.org
FOSS enthusiast
• Kurento project lead: http://www.kurento.org • NUBOMEDIA Community lead: http://www.nubomedia.eu
![Page 3: FOSDEM 2016 - Creating rich WebRTC Applications with Kurento](https://reader035.vdocument.in/reader035/viewer/2022070515/5878339f1a28abef5d8b712d/html5/thumbnails/3.jpg)
http://www.kurento.org - [email protected]
Real-Time media Communications
Capture Encode Cipher Transport Transport Decipher Decode RenderNetwork
The media plane
The signaling plane
I wan to call you, do you accept?
Yes, do it in this way
![Page 4: FOSDEM 2016 - Creating rich WebRTC Applications with Kurento](https://reader035.vdocument.in/reader035/viewer/2022070515/5878339f1a28abef5d8b712d/html5/thumbnails/4.jpg)
http://www.kurento.org - [email protected]
WebRTC to the rescue.
Before WebRTCFirst wave of
WebRTC technologies
Begin End• APIs• Standards• FOSS
Development experience
when working with real-time media
Common WebRTC application (p2p communications)
WebRTC video stream
![Page 5: FOSDEM 2016 - Creating rich WebRTC Applications with Kurento](https://reader035.vdocument.in/reader035/viewer/2022070515/5878339f1a28abef5d8b712d/html5/thumbnails/5.jpg)
http://www.kurento.org - [email protected]
WebRTC infrastructures
Peer-to-Peer WebRTC Application (without media infrastructure)
WebRTC video stream
WebRTC Application based on media infrastructuremedia infrastructure
![Page 6: FOSDEM 2016 - Creating rich WebRTC Applications with Kurento](https://reader035.vdocument.in/reader035/viewer/2022070515/5878339f1a28abef5d8b712d/html5/thumbnails/6.jpg)
http://www.kurento.org - [email protected]
WWW VS (Web)RTC applications
Control
Multimedia Application logic
(developers’ code)
Multimedia Capabilities
RecordingTranscoding
RoutingMixing
AnalyzingEtc.
MediaTraffic
RTC Media APIs
Multimedia Clients
Events
Application Signaling
Control
WWW Application logic
(developers’ code)
Database Capabilities
Data storageData recoveryData querying
Data processingEtc.
DD.BB. API
WWW Clients
Events
Application Signaling
![Page 7: FOSDEM 2016 - Creating rich WebRTC Applications with Kurento](https://reader035.vdocument.in/reader035/viewer/2022070515/5878339f1a28abef5d8b712d/html5/thumbnails/7.jpg)
http://www.kurento.org - [email protected]
![Page 8: FOSDEM 2016 - Creating rich WebRTC Applications with Kurento](https://reader035.vdocument.in/reader035/viewer/2022070515/5878339f1a28abef5d8b712d/html5/thumbnails/8.jpg)
http://www.kurento.org - [email protected]
Control
Multimedia Application logic
(developers’ code)
Multimedia Capabilities
RecordingTranscoding
RoutingMixing
AnalyzingEtc.
RTC Media APIs
Events
![Page 9: FOSDEM 2016 - Creating rich WebRTC Applications with Kurento](https://reader035.vdocument.in/reader035/viewer/2022070515/5878339f1a28abef5d8b712d/html5/thumbnails/9.jpg)
http://www.kurento.org - [email protected]
Kurento: a WebRTC infrastructure an its APIs
Control
Multimedia Application logic
(developers’ code)
RecordingTranscoding
RoutingMixing
AnalyzingAdapting
MediaTraffic
RTC Media APIs
Multimedia Clients
Events
Application Signaling
Computervision
Augmentedreality
BlendingEtc.
Multimedia Capabilities
Kurento Media Server(KMS)
Kurento Client APIKurento Room APIKurento Tree API
![Page 11: FOSDEM 2016 - Creating rich WebRTC Applications with Kurento](https://reader035.vdocument.in/reader035/viewer/2022070515/5878339f1a28abef5d8b712d/html5/thumbnails/11.jpg)
http://www.kurento.org - [email protected]
The Kurento FOSS Community
Community support WebRTC worldwide reference
International awards More than 300 companies
Very active mailing list
First result in Google
![Page 12: FOSDEM 2016 - Creating rich WebRTC Applications with Kurento](https://reader035.vdocument.in/reader035/viewer/2022070515/5878339f1a28abef5d8b712d/html5/thumbnails/12.jpg)
http://www.kurento.org - [email protected]
The Kurento FOSS project• http://www.kurento.org
– Main web site
• https://www.twitter.com/kurentoms – Main social channel
• https://groups.google.com/forum/#!forum/ kurento– Main mailing list
• https://github.com/kurento – Main repository
• https://www.youtube.com/channel/UCFtGhWYqahVlzMgGNtEmKug– Kurento Youtube channel
![Page 13: FOSDEM 2016 - Creating rich WebRTC Applications with Kurento](https://reader035.vdocument.in/reader035/viewer/2022070515/5878339f1a28abef5d8b712d/html5/thumbnails/13.jpg)
http://www.kurento.org - [email protected]
Developing with KMS:The Kurento Client API
Media Element• Provides a specific media
functionality
› Send/receive media
› Process media
› Transform media• Exchange media through› Sources
› Sinks
Media pipeline• Chain of media elements
implementing the desired media logic.
• The Media API provides the capability of creating media pipelines by joining media elements of the toolbox
Media Element
Sink
SRC
![Page 14: FOSDEM 2016 - Creating rich WebRTC Applications with Kurento](https://reader035.vdocument.in/reader035/viewer/2022070515/5878339f1a28abef5d8b712d/html5/thumbnails/14.jpg)
http://www.kurento.org - [email protected]
A modular API
• Modular in the sense of modularity*– Isolation
• Internal states of a module don’t affect the rest– Abstracion
• Internal states are hidden behind an interface– Composability
• Interface must enable module recombination and assembly– Reusability
• If a module has it, you can use it– Extensibility
• If a module does not have it, you can add it.
* Baldwin, Carliss Young, and Kim B. Clark. Design rules: The power of modularity. Vol. 1. MIT press, 2000
![Page 16: FOSDEM 2016 - Creating rich WebRTC Applications with Kurento](https://reader035.vdocument.in/reader035/viewer/2022070515/5878339f1a28abef5d8b712d/html5/thumbnails/16.jpg)
http://www.kurento.org - [email protected]
The Kurento Client API in one word
sourceElement.connect(sinkElement)
connect
![Page 18: FOSDEM 2016 - Creating rich WebRTC Applications with Kurento](https://reader035.vdocument.in/reader035/viewer/2022070515/5878339f1a28abef5d8b712d/html5/thumbnails/18.jpg)
http://www.kurento.org - [email protected]
//MediaPipeline is a holder of media elementsMediaPipeline pipeline = kurento.createMediaPipeline();
//Create your media elementsWebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(pipeline).build();
//Connect your media elementswebRtcEndpoint.connect(webRtcEndpoint);
Control
Multimedia Application logic
(developers’ code)
RecordingTranscoding
RoutingMixing
AnalyzingAdapting
Kurento Media APIs
Multimedia Clients
Events
Application Signaling
Computervision
Augmentedreality
BlendingEtc.
Multimedia Capabilities
//Where to render the local and remote streamsvar options = {
localVideo : videoInput, remoteVideo : videoOutput, onicecandidate : onIceCandidate }
//Start media capture and communicationswebRtcPeer = new kurentoUtils.WebRtcPeer.WebRtcPeerSendrecv(
options, function(error) { if (error) return console.error(error); webRtcPeer.generateOffer(onOffer); });
Application Server Code (Java)
Client Server Code
Application source for Spring (Java EE)Application source for Node.js (JavaScript)
![Page 19: FOSDEM 2016 - Creating rich WebRTC Applications with Kurento](https://reader035.vdocument.in/reader035/viewer/2022070515/5878339f1a28abef5d8b712d/html5/thumbnails/19.jpg)
http://www.kurento.org - [email protected]
Recording
Sink
SRC
Sink
//MediaPipeline is a holder of media elementsMediaPipeline pipeline = kurento.createMediaPipeline();
//Create your media elementsWebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(pipeline).build(); RecorderEndpoint recorderEndpoint = new RecorderEndpoint.Builder(pipeline).build(“file:///myfolder/myfile.mp4”);
//Connect your media elementswebRtcEndpoint.connect(webRtcEndpoint);webRtcEndpoint.connect(recorderEndpoint);
Application source for Spring (Java EE)Application source for Node.js (JavaScript)
![Page 20: FOSDEM 2016 - Creating rich WebRTC Applications with Kurento](https://reader035.vdocument.in/reader035/viewer/2022070515/5878339f1a28abef5d8b712d/html5/thumbnails/20.jpg)
http://www.kurento.org - [email protected]
Playing
Sink
SRC SR
C
//MediaPipeline is a holder of media elementsMediaPipeline pipeline = kurento.createMediaPipeline();
//Create your media elementsWebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(pipeline).build(); PlayerEndpoint playerEndpoint = new PlayerEndpoint.Buildr(pipeline).build(“file:///myfolder/myfile.mp4”); //RSTP and HTTP URIs also supported
//Connect your media elementsplayerEndpoint.connect(webRtcEndpoint);
Application source for Spring (Java EE)Application source for Node.js (JavaScript)
![Page 21: FOSDEM 2016 - Creating rich WebRTC Applications with Kurento](https://reader035.vdocument.in/reader035/viewer/2022070515/5878339f1a28abef5d8b712d/html5/thumbnails/21.jpg)
http://www.kurento.org - [email protected]
Interoperating
Media Pipeline
WebRTC audiostreaming
Sink
SRC
SRC
Sink
Media Pipeline
Sink
SRC
SRC
Sink
WebRTC videostreaming
RTP audiostreaming
RTP videostreaming
Transparent transcoding!!
![Page 22: FOSDEM 2016 - Creating rich WebRTC Applications with Kurento](https://reader035.vdocument.in/reader035/viewer/2022070515/5878339f1a28abef5d8b712d/html5/thumbnails/22.jpg)
http://www.kurento.org - [email protected]
Some Media processing modules• Augmented reality
– MarkerArModule– MarkerlessArModule
• Computer vision– FaceDetector– NoseDetector– EyeDetector– CrowdDetector– PointerDetector– MotionDetector– VirtualFence– Etc.
Sink
SRC
Sink
SRC
Sink
SRC
Sink
SRC
Sink
SRC
![Page 23: FOSDEM 2016 - Creating rich WebRTC Applications with Kurento](https://reader035.vdocument.in/reader035/viewer/2022070515/5878339f1a28abef5d8b712d/html5/thumbnails/23.jpg)
http://www.kurento.org - [email protected]
Using media processing modules
Sink
SRC
Sink
SRC
CrowdEventsSR
C
Sink
SRC
CrowdDetector
RtspPlayer
Sink
SRC
See demo
See demo
![Page 24: FOSDEM 2016 - Creating rich WebRTC Applications with Kurento](https://reader035.vdocument.in/reader035/viewer/2022070515/5878339f1a28abef5d8b712d/html5/thumbnails/24.jpg)
http://www.kurento.org - [email protected]
Implementing group communications: just connect
Media Pipeline
WebRTCstreaming
Sink
SRC
WebRTCstreaming
SRC
Sink
SRC
Sink
WebRTCstreaming
WebRtcEndpoint 1
WebRtcEndpoint 3
WebRtcEndpoint 2
SRC
Sink
WebRTCstreaming
WebRtcEndpoint 4
User 1 User 2
User 3
User 4
![Page 25: FOSDEM 2016 - Creating rich WebRTC Applications with Kurento](https://reader035.vdocument.in/reader035/viewer/2022070515/5878339f1a28abef5d8b712d/html5/thumbnails/25.jpg)
http://www.kurento.org - [email protected]
Group communications: RTP topologies
Sink
SRC
SRC Sink
SinkSRC
SRC
Sink
• Media Mixing Mixer– Composite
• Media Switching Mixer– Connect primitive– Natural topology
• SFU– WebRtcSfu
• Simulcast
![Page 26: FOSDEM 2016 - Creating rich WebRTC Applications with Kurento](https://reader035.vdocument.in/reader035/viewer/2022070515/5878339f1a28abef5d8b712d/html5/thumbnails/26.jpg)
http://www.kurento.org - [email protected]
Creating your own modules
KurentoIDL Compiler
JSON
Module IDL Description
Java Java-Script
cpphpp
Media API code Toolchain
Auto-generated code
ModuleImplementation
cmake
Interface
Developer
(1)
(2)
(3)
![Page 27: FOSDEM 2016 - Creating rich WebRTC Applications with Kurento](https://reader035.vdocument.in/reader035/viewer/2022070515/5878339f1a28abef5d8b712d/html5/thumbnails/27.jpg)
http://www.kurento.org - [email protected]
Going up the API stack: the room API
createRoomdeleteRoom
joinRoomleaveRoom
publisMediaunpublishMedia
onParticipantJoinedonParticipantLeft
onMediaPublishedonMediaUnpublished
onMessage
See demo
![Page 28: FOSDEM 2016 - Creating rich WebRTC Applications with Kurento](https://reader035.vdocument.in/reader035/viewer/2022070515/5878339f1a28abef5d8b712d/html5/thumbnails/28.jpg)
http://www.kurento.org - [email protected]
Going up the API stack: the tree API
createTreedeleteTree
joinAsPresenterjoinAsViewer
leave
![Page 30: FOSDEM 2016 - Creating rich WebRTC Applications with Kurento](https://reader035.vdocument.in/reader035/viewer/2022070515/5878339f1a28abef5d8b712d/html5/thumbnails/30.jpg)
http://www.kurento.org - [email protected]
Where to start
• http://www.kurento.org/documentation – Installation and administration guide– Java tutorials for Spring and Java EE developers– JavaScript tutorials for Node.js developers– JSON-RPC protocol documentation
• https://github.com/kurento – Main repository