mp 1: audio/ video recorder and player cs414: multimedia system instructor: klara nahrstedt february...
TRANSCRIPT
![Page 1: MP 1: Audio/ Video Recorder and Player CS414: Multimedia System Instructor: Klara Nahrstedt February 7 th, 2012](https://reader034.vdocument.in/reader034/viewer/2022050908/56649d985503460f94a8365f/html5/thumbnails/1.jpg)
MP 1: Audio/ Video Recorder and Player
CS414: Multimedia SystemInstructor: Klara Nahrstedt
February 7th, 2012
![Page 2: MP 1: Audio/ Video Recorder and Player CS414: Multimedia System Instructor: Klara Nahrstedt February 7 th, 2012](https://reader034.vdocument.in/reader034/viewer/2022050908/56649d985503460f94a8365f/html5/thumbnails/2.jpg)
Learning Goals• Capturing a video and audio using a webcam
• Storing the audio and video data in PC using compression
• Playing a video from the stored file with fast forward, rewind, pause and play functionalities
• Playing an audio from the stored file
• Comparing the media frames across different compression
![Page 3: MP 1: Audio/ Video Recorder and Player CS414: Multimedia System Instructor: Klara Nahrstedt February 7 th, 2012](https://reader034.vdocument.in/reader034/viewer/2022050908/56649d985503460f94a8365f/html5/thumbnails/3.jpg)
Covered Aspects of Multimedia
Image/VideoCapture
MediaServerStorage
Transmission
CompressionProcessing
Audio/VideoPresentationPlaybackAudio/Video
Perception/ Playback
Audio InformationRepresentation
Transmission
AudioCapture
A/V Playback
Image/Video InformationRepresentation
![Page 4: MP 1: Audio/ Video Recorder and Player CS414: Multimedia System Instructor: Klara Nahrstedt February 7 th, 2012](https://reader034.vdocument.in/reader034/viewer/2022050908/56649d985503460f94a8365f/html5/thumbnails/4.jpg)
System ArchitectureImage/VideoCapture
CompressionProcessing
Audio InformationRepresentation
AudioCapture
Image/Video InformationRepresentation
Audio/Video Perception/ Playback
![Page 5: MP 1: Audio/ Video Recorder and Player CS414: Multimedia System Instructor: Klara Nahrstedt February 7 th, 2012](https://reader034.vdocument.in/reader034/viewer/2022050908/56649d985503460f94a8365f/html5/thumbnails/5.jpg)
Capture
System Modules and Data Flow: Recording
Filter Encoder Store
Player Thread
Webcam
ThreadThread
Encode StoreMuxer… … … …
• You need to create separate threads• Muxer is optional. You may need it to run the video file using standard player
Frame rate, resolution
type
Encode… …
![Page 6: MP 1: Audio/ Video Recorder and Player CS414: Multimedia System Instructor: Klara Nahrstedt February 7 th, 2012](https://reader034.vdocument.in/reader034/viewer/2022050908/56649d985503460f94a8365f/html5/thumbnails/6.jpg)
System Modules and Data Flow: Playback
Capture Decode Player
Media File
Capture DecodeDemuxer… … … …
• Display the video on your GUI and place the audio data to the sound card• Demuxer is optional. You may need it run standard video files from Internet
type
Rate control
Decode … …
![Page 7: MP 1: Audio/ Video Recorder and Player CS414: Multimedia System Instructor: Klara Nahrstedt February 7 th, 2012](https://reader034.vdocument.in/reader034/viewer/2022050908/56649d985503460f94a8365f/html5/thumbnails/7.jpg)
System Modules and Data Flow: Monitoring in Recording
Capture Filter Encode Store
Player Thread
Monitoring Point Monitoring Component
Webcam
Thread
Thread
Thread
• Store the monitoring output to a file• [Optional] You may draw chart showing [time vs. compression ratio (r)],
[time vs. compression time (t)] and [time vs. frame size (s)] [bonus point]– X-axis: Time in 5 second interval– Y-axis: Average values (of r, s or t) in last 5 seconds
![Page 8: MP 1: Audio/ Video Recorder and Player CS414: Multimedia System Instructor: Klara Nahrstedt February 7 th, 2012](https://reader034.vdocument.in/reader034/viewer/2022050908/56649d985503460f94a8365f/html5/thumbnails/8.jpg)
System Modules and Data Flow: Monitoring in Playback
Capture Decode Player
Monitoring ComponentThread
Media File
ThreadThread
Monitoring Point
• Store the monitoring output to a file• [Optional] You may draw plot showing [time vs. decompression time (d)]
[bonus point] [Hint: Try JFreeChart library]– X-axis: Time in 5 second interval– Y-axis: Average values (of d) in last 5 seconds
![Page 9: MP 1: Audio/ Video Recorder and Player CS414: Multimedia System Instructor: Klara Nahrstedt February 7 th, 2012](https://reader034.vdocument.in/reader034/viewer/2022050908/56649d985503460f94a8365f/html5/thumbnails/9.jpg)
gstreamer Architecture
• Goal: create a multimedia application using gstreamer• gstreamer uses 3rd party plugins for processing (over 150)
gstreamer Core Framework
Multimedia Application
Encoder Decoder Muxer Sink sourceDemuxer
3rd Party plugins
![Page 10: MP 1: Audio/ Video Recorder and Player CS414: Multimedia System Instructor: Klara Nahrstedt February 7 th, 2012](https://reader034.vdocument.in/reader034/viewer/2022050908/56649d985503460f94a8365f/html5/thumbnails/10.jpg)
How does it work (1) ?
• You need to create element for each system modules• Elements are equipped with Pads: source and sink pads
Sink Source
gstreamer elements
• The number of source and sink pad varies depending on the element type
• You can add/ create additional sink or source pad
![Page 11: MP 1: Audio/ Video Recorder and Player CS414: Multimedia System Instructor: Klara Nahrstedt February 7 th, 2012](https://reader034.vdocument.in/reader034/viewer/2022050908/56649d985503460f94a8365f/html5/thumbnails/11.jpg)
Creating gstreamer elements• Java code example to create an element
– Elements element_name = ElementFactory.make (“plugin name”, “your defined name”)
• Java code example to set element property– element_name.set(“property”, “value”)
Capturesource
Filter Sink Src filter
Encode Sink Srcencoder
MuxerSink
Srcmuxer
Store Sinksink
Src
Sink
![Page 12: MP 1: Audio/ Video Recorder and Player CS414: Multimedia System Instructor: Klara Nahrstedt February 7 th, 2012](https://reader034.vdocument.in/reader034/viewer/2022050908/56649d985503460f94a8365f/html5/thumbnails/12.jpg)
How does it work (2) ? • You need to link the elements
– Linking define whom is receiving and sending data to whom– This is similar to linking the system modules
Capture Filter
Src Sink Src
Media Source Media Filter
![Page 13: MP 1: Audio/ Video Recorder and Player CS414: Multimedia System Instructor: Klara Nahrstedt February 7 th, 2012](https://reader034.vdocument.in/reader034/viewer/2022050908/56649d985503460f94a8365f/html5/thumbnails/13.jpg)
gstreamer-java pipeline
• Java example code for creating, and linking a pipeline– Pipeline pipe= new Pipeline(“test”);– pipe.addMany(source, filter,encoder, muxer,sink);– Element.linkMany(source, filter,encoder, muxer,sink);– pipe.play();
Capture Filter Encode Store
Webcam
Src Sink Src Sink Src
Media Source Media Filter Media Encoder
Sink Src
Media Muxer
Sink
Media Sink
![Page 14: MP 1: Audio/ Video Recorder and Player CS414: Multimedia System Instructor: Klara Nahrstedt February 7 th, 2012](https://reader034.vdocument.in/reader034/viewer/2022050908/56649d985503460f94a8365f/html5/thumbnails/14.jpg)
How to create multiple pipelining• Several pipelines from the same source or towards same sink
– Example: recording and playback at the same time
Capture Filter Encoder Store
Player Thread
Webcam
ThreadThread
Sink
Src Sink
Src
Sink Src
Media SourceMedia Filter
Media Encoder
Sink Src
Media Muxer
Sink
Media Sink
Media SinkSrc
![Page 15: MP 1: Audio/ Video Recorder and Player CS414: Multimedia System Instructor: Klara Nahrstedt February 7 th, 2012](https://reader034.vdocument.in/reader034/viewer/2022050908/56649d985503460f94a8365f/html5/thumbnails/15.jpg)
How to create multiple pipelining• You can create additional pads• You can use tee elements• Use queue elements after tee
Sink
Src
Sink Srcsource
encoder
Sink Src
muxer
Sink
Media Sink
SinkSink Src
filter
Sink
Src
tee
Src queue2
queue1
Sink Src
Sink Src
![Page 16: MP 1: Audio/ Video Recorder and Player CS414: Multimedia System Instructor: Klara Nahrstedt February 7 th, 2012](https://reader034.vdocument.in/reader034/viewer/2022050908/56649d985503460f94a8365f/html5/thumbnails/16.jpg)
How to redirect frames to application• How to deliver a frame to your application• Use appsink element
Src Sink Src Sink Src
Media Source Media Filter Media Encoder
Sink Src
AppSink
• Java example code for getting frames from AppSink– AppSink appsink = (AppSink) ElementFactory.make(“appsink”,
null);– appsink.set(“emit-signal”, true);– appsink.setSync(false);
Emit Signals
![Page 17: MP 1: Audio/ Video Recorder and Player CS414: Multimedia System Instructor: Klara Nahrstedt February 7 th, 2012](https://reader034.vdocument.in/reader034/viewer/2022050908/56649d985503460f94a8365f/html5/thumbnails/17.jpg)
Some gstreamer plugins
• Video webcam source: v4l2src • Audio webcam source: alsasrc • Video or Audio file source: filesrc• Video Encoder: ffenc_mpeg4 (mpeg4), jpegenc (mjpeg), …• Video Decoder: ffdec_mpeg4 (mpeg4), jpegdec (mjpeg), …• Audio Encoder: vorbisenc, alawenc, mulawenc• Audio Decoder: vorbisdec, alawdec, mulawdec• Muxer [optional]: avimux (avi), matroskamux(mkv), … • Demuxer [optional]: avidemux (avi), • Audio sink: alsasink
![Page 18: MP 1: Audio/ Video Recorder and Player CS414: Multimedia System Instructor: Klara Nahrstedt February 7 th, 2012](https://reader034.vdocument.in/reader034/viewer/2022050908/56649d985503460f94a8365f/html5/thumbnails/18.jpg)
Evaluations
Features Points Properties
Video Recording 15 Simultaneous Record and Playback
Video Compression 10 mjpeg, mpeg4
Audio Recording 10
Audio Compression 10 pcm, vorbis/ mp3
Video Playback 15 FF, RW, S, P Functionalities
Audio Playback 10
Graphical Interface 10 User-friendly
System Monitor 10 Monitor at each 5 sec interval
Report Writing 10 User manual and Development manual
Monitor Visualizer 10 Hint: you may try JFreeChart
Camera pan and tilt 10
Required Points: 100, Optional Points: 20
Points are will be considered based on live demo and interview performance
![Page 19: MP 1: Audio/ Video Recorder and Player CS414: Multimedia System Instructor: Klara Nahrstedt February 7 th, 2012](https://reader034.vdocument.in/reader034/viewer/2022050908/56649d985503460f94a8365f/html5/thumbnails/19.jpg)
Environment and Equipment
• Choose what ever language you like, Gstreamer is compatible with most popular languages
C/C++ JAVAhttp://docs.gstreamer.com/display/GstSDK/Tutorials
https://code.google.com/p/gstreamer-java/
Up-to-date detailed tutorials, step-by-step runnable sample codes
Brief introduction, few samples
Requires external GUI libraries: GTK+(used in the tutorial site), Qt (very powerful)
Built-in GUI libraries
Not supported by EWS (installation of external lib)
Supported by EWS
![Page 20: MP 1: Audio/ Video Recorder and Player CS414: Multimedia System Instructor: Klara Nahrstedt February 7 th, 2012](https://reader034.vdocument.in/reader034/viewer/2022050908/56649d985503460f94a8365f/html5/thumbnails/20.jpg)
Environment and Equipment
• If you choose to use EWS..– group directories– /team/cs414/G# (# is your group number)
• If you choose to use your own machines..– Windows/Mac– Android/iOS
• Feel free to use your own camera/mic, or you can borrow two logitech cameras (with mic embedded) from Engineering IT (Barb Leisner)