Multimedia System Overview and Data Flow Analysis
Explore the multimedia system architecture and data flow for audio/video recording, playback, and monitoring. Covering aspects such as capturing, storing, compression, and playback functionalities. Learn about system modules, threads, encoding, decoding, and monitoring components.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
MP 1: Audio/ Video Recorder and Player CS414: Multimedia System Instructor: Klara Nahrstedt February 7th, 2012
Learning Goals Capturing a video and audio using a webcam Storing the audio and video data in PC using compression Playing a video from the stored file with fast forward, rewind, pause and play functionalities Playing an audio from the stored file Comparing the media frames across different compression
Covered Aspects of Multimedia Audio/Video Presentation Playback Image/Video Capture Audio/Video Perception/ Playback Image/Video Information Representation Transmission Transmission Compression Processing Audio Capture Media Server Storage Audio Information Representation A/V Playback
System Architecture Image/Video Capture Audio/Video Perception/ Playback Image/Video Information Representation Compression Processing Audio Capture Audio Information Representation
System Modules and Data Flow: Recording Frame rate, resolution type Thread Thread Capture Filter Encoder Store Webcam Player Thread Encode Muxer Store Encode You need to create separate threads Muxer is optional. You may need it to run the video file using standard player
System Modules and Data Flow: Playback Media File type Capture Decode Player Rate control Decode Capture Demuxer Decode Display the video on your GUI and place the audio data to the sound card Demuxer is optional. You may need it run standard video files from Internet
System Modules and Data Flow: Monitoring in Recording Thread Monitoring Component Monitoring Point Thread Thread Capture Filter Encode Store Webcam Player Thread Store the monitoring output to a file [Optional] You may draw chart showing [time vs. compression ratio (r)], [time vs. compression time (t)] and [time vs. frame size (s)] [bonus point] X-axis: Time in 5 second interval Y-axis: Average values (of r, s or t) in last 5 seconds
System Modules and Data Flow: Monitoring in Playback Monitoring Point Thread Media File Monitoring Component Thread Thread Capture Decode Player Store the monitoring output to a file [Optional] You may draw plot showing [time vs. decompression time (d)] [bonus point] [Hint: Try JFreeChart library] X-axis: Time in 5 second interval Y-axis: Average values (of d) in last 5 seconds
gstreamer Architecture Multimedia Application gstreamer Core Framework Sink source Encoder Decoder Muxer Demuxer 3rd Party plugins Goal: create a multimedia application using gstreamer gstreamer uses 3rd party plugins for processing (over 150)
How does it work (1) ? You need to create element for each system modules Elements are equipped with Pads: source and sink pads Source Sink gstreamer elements The number of source and sink pad varies depending on the element type You can add/ create additional sink or source pad
Creating gstreamer elements Java code example to create an element Elements element_name = ElementFactory.make ( plugin name , your defined name ) Java code example to set element property element_name.set( property , value ) source Src Capture filter Filter Src Sink encoder Encode Src Sink muxer Sink Muxer Src Sink sink Store Sink
How does it work (2) ? You need to link the elements Linking define whom is receiving and sending data to whom This is similar to linking the system modules Capture Filter Src Sink Src Media Source Media Filter
gstreamer-java pipeline Webcam Capture Filter Encode Store Media Encoder Media Muxer Media Filter Media Source Media Sink Src Src Sink Src Src Sink Sink Sink Java example code for creating, and linking a pipeline Pipeline pipe= new Pipeline( test ); pipe.addMany(source, filter,encoder, muxer,sink); Element.linkMany(source, filter,encoder, muxer,sink); pipe.play();
How to create multiple pipelining Several pipelines from the same source or towards same sink Example: recording and playback at the same time Thread Thread Capture Filter Encoder Store Webcam Player Thread Media Encoder Media Muxer Media Sink Media Filter Src Sink Src Sink Sink Media Source Src Src Sink Media Sink Src Sink
How to create multiple pipelining You can create additional pads You can use tee elements Use queue elements after tee muxer Media Sink Sink Src Sink queue1 encoder tee Src Src Sink Sink filter source Src Src Sink Sink Src Src queue2 Sink Src Sink Sink
How to redirect frames to application How to deliver a frame to your application Use appsink element Emit Signals Media Encoder AppSink Media Filter Media Source Src Src Src Src Sink Sink Sink Java example code for getting frames from AppSink AppSink appsink = (AppSink) ElementFactory.make( appsink , null); appsink.set( emit-signal , true); appsink.setSync(false);
Some gstreamer plugins Video webcam source: v4l2src Audio webcam source: alsasrc Video or Audio file source: filesrc Video Encoder: ffenc_mpeg4 (mpeg4), jpegenc (mjpeg), Video Decoder: ffdec_mpeg4 (mpeg4), jpegdec (mjpeg), Audio Encoder: vorbisenc, alawenc, mulawenc Audio Decoder: vorbisdec, alawdec, mulawdec Muxer [optional]: avimux (avi), matroskamux(mkv), Demuxer [optional]: avidemux (avi), Audio sink: alsasink
Evaluations Required Points: 100, Optional Points: 20 Features Points Properties Video Recording 15 Simultaneous Record and Playback Video Compression 10 mjpeg, mpeg4 Audio Recording 10 Audio Compression 10 pcm, vorbis/ mp3 Video Playback 15 FF, RW, S, P Functionalities Audio Playback 10 Graphical Interface 10 User-friendly System Monitor 10 Monitor at each 5 sec interval Report Writing 10 User manual and Development manual Monitor Visualizer 10 Hint: you may try JFreeChart Camera pan and tilt 10 Points are will be considered based on live demo and interview performance
Environment and Equipment Choose what ever language you like, Gstreamer is compatible with most popular languages C/C++ JAVA http://docs.gstreamer.com/display/G stSDK/Tutorials Up-to-date detailed tutorials, step- by-step runnable sample codes Requires external GUI libraries: GTK+(used in the tutorial site), Qt (very powerful) https://code.google.com/p/gstreame r-java/ Brief introduction, few samples Built-in GUI libraries Not supported by EWS (installation of external lib) Supported by EWS
Environment and Equipment If you choose to use EWS.. group directories /team/cs414/G#(# is your group number) If you choose to use your own machines.. Windows/Mac Android/iOS Feel free to use your own camera/mic, or you can borrow two logitech cameras (with mic embedded) from Engineering IT (Barb Leisner)
