Multimedia System Overview and Data Flow Analysis
MP 1: Audio/ Video Recorder
and Player
CS414: Multimedia System
Instructor: Klara Nahrstedt
February 7
th
, 2012
Learning Goals
•
Capturing a video and audio using a webcam
•
Storing the audio and video data in PC using compression
•
Playing a video from the stored file with fast forward, rewind, pause and
play functionalities
•
Playing an audio from the stored file
•
Comparing the media frames across different compression
Covered Aspects of Multimedia
I
m
a
g
e
/
V
i
d
e
o
C
a
p
t
u
r
e
M
e
d
i
a
S
e
r
v
e
r
S
t
o
r
a
g
e
T
r
a
n
s
m
i
s
s
i
o
n
C
o
m
p
r
e
s
s
i
o
n
P
r
o
c
e
s
s
i
n
g
A
u
d
i
o
/
V
i
d
e
o
P
r
e
s
e
n
t
a
t
i
o
n
P
l
a
y
b
a
c
k
A
u
d
i
o
/
V
i
d
e
o
P
e
r
c
e
p
t
i
o
n
/
P
l
a
y
b
a
c
k
A
u
d
i
o
I
n
f
o
r
m
a
t
i
o
n
R
e
p
r
e
s
e
n
t
a
t
i
o
n
T
r
a
n
s
m
i
s
s
i
o
n
A
u
d
i
o
C
a
p
t
u
r
e
A
/
V
P
l
a
y
b
a
c
k
I
m
a
g
e
/
V
i
d
e
o
I
n
f
o
r
m
a
t
i
o
n
R
e
p
r
e
s
e
n
t
a
t
i
o
n
System Architecture
I
m
a
g
e
/
V
i
d
e
o
C
a
p
t
u
r
e
C
o
m
p
r
e
s
s
i
o
n
P
r
o
c
e
s
s
i
n
g
A
u
d
i
o
I
n
f
o
r
m
a
t
i
o
n
R
e
p
r
e
s
e
n
t
a
t
i
o
n
A
u
d
i
o
C
a
p
t
u
r
e
I
m
a
g
e
/
V
i
d
e
o
I
n
f
o
r
m
a
t
i
o
n
R
e
p
r
e
s
e
n
t
a
t
i
o
n
A
u
d
i
o
/
V
i
d
e
o
P
e
r
c
e
p
t
i
o
n
/
P
l
a
y
b
a
c
k
Capture
System Modules and Data Flow:
Recording
Filter
Encoder
Store
Player
Thread
Webcam
Thread
Thread
•
You need to create separate threads
•
Muxer is optional. You may need it to run the video file using standard player
System Modules and Data Flow:
Playback
Capture
Decode
Player
Media File
•
Display the video on your GUI and place the audio data to the sound card
•
Demuxer is optional. You may need it run standard video files from Internet
Rate control
System Modules and Data Flow:
Monitoring in Recording
Capture
Filter
Encode
Store
Player
Thread
Monitoring Point
Monitoring Component
Webcam
Thread
Thread
Thread
•
Store the monitoring output to a file
•
[Optional] You may draw chart showing [
time vs. compression ratio (r)
],
[
time vs. compression time (t)
] and [
time vs. frame size (s)
] [
bonus point
]
–
X-axis: Time in 5 second interval
–
Y-axis: Average values (of r, s or t) in last 5 seconds
System Modules and Data Flow:
Monitoring in Playback
Capture
Decode
Player
Monitoring Component
Thread
Media File
Thread
Thread
Monitoring Point
•
Store the monitoring output to a file
•
[Optional] You may draw plot showing [
time vs. decompression time (d)
]
[
bonus point
] [
Hint: Try JFreeChart library
]
–
X-axis: Time in 5 second interval
–
Y-axis: Average values (of d) in last 5 seconds
gstreamer
Architecture
•
Goal: create a multimedia application using
gstreamer
•
gstreamer
uses 3
rd
party plugins for processing (over 150)
gstreamer
Core Framework
Multimedia Application
Encoder
Decoder
Muxer
Sink
source
Demuxer
3
rd
Party plugins
How does it work (1) ?
•
You need to create
element
for each system modules
•
Elements are equipped with Pads: source and sink pads
Sink
Source
gstreamer elements
•
The number of source and sink pad varies depending on
the element type
•
You can add/ create additional sink or source pad
Creating gstreamer elements
•
Java code example to create an element
–
Elements element_name = ElementFactory.make (“plugin name”,
“your defined name”)
•
Java code example to set element property
–
element_name.set(“property”, “value”)
Capture
source
Filter
filter
Encode
encoder
Muxer
muxer
Store
sink
Src
Sink
How does it work (2) ?
•
You need to link the
elements
–
Linking define whom is receiving and sending data to whom
–
This is similar to linking the system modules
Capture
Filter
Src
Media Source
Media Filter
gstreamer-java pipeline
•
Java example code for creating, and linking a pipeline
–
Pipeline pipe= new Pipeline(“test”);
–
pipe.addMany(source, filter,encoder, muxer,sink);
–
Element.linkMany(source, filter,encoder, muxer,sink);
–
pipe.play();
Capture
Filter
Encode
Store
Webcam
Media Source
Media Filter
Media Encoder
Media Muxer
Media Sink
How to create multiple pipelining
•
Several pipelines from the same source or towards same sink
–
Example: recording and playback at the same time
How to create multiple pipelining
•
You can create additional pads
•
You can use
tee elements
•
Use
queue elements
after tee
source
encoder
muxer
Media Sink
Sink
filter
tee
Src
queue2
queue1
How to redirect frames to application
•
How to deliver a frame to your application
•
Use
appsink element
Media Source
Media Filter
Media Encoder
AppSink
•
Java example code for getting frames from AppSink
–
AppSink appsink = (AppSink)
ElementFactory.make(“appsink”, null);
–
appsink.set(“emit-signal”, true);
–
appsink.setSync(false);
Emit Signals
Some
gstreamer
plugins
•
Video webcam source:
v4l2src
•
Audio webcam source:
alsasrc
•
Video or Audio file source:
filesrc
•
Video Encoder:
ffenc_mpeg4 (mpeg4), jpegenc (mjpeg), …
•
Video Decoder:
ffdec_mpeg4 (mpeg4), jpegdec (mjpeg), …
•
Audio Encoder
: vorbisenc,
alawenc, mulawenc
•
Audio Decoder:
vorbisdec,
alawdec, mulawdec
•
Muxer [optional]:
avimux (avi), matroskamux(mkv), …
•
Demuxer [optional]
: avidemux (avi),
•
Audio sink
: alsasink
Evaluations
Required Points: 100, Optional Points: 20
Points are will be considered based on live demo and interview performance
Environment and Equipment
•
Choose what ever language you like, Gstreamer is
compatible with most popular languages
Environment and Equipment
•
If you choose to use EWS..
–
group directories
–
/team/cs414/G#
(# is your group number)
•
If you choose to use your own machines..
–
Windows/Mac
–
Android/iOS
•
Feel free to use your own camera/mic, or you can
borrow two logitech cameras (with mic embedded)
from Engineering IT (Barb Leisner)
Explore the multimedia system architecture and data flow for audio/video recording, playback, and monitoring. Covering aspects such as capturing, storing, compression, and playback functionalities. Learn about system modules, threads, encoding, decoding, and monitoring components.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
MP 1: Audio/ Video Recorder and Player CS414: Multimedia System Instructor: Klara Nahrstedt February 7th, 2012
Learning Goals Capturing a video and audio using a webcam Storing the audio and video data in PC using compression Playing a video from the stored file with fast forward, rewind, pause and play functionalities Playing an audio from the stored file Comparing the media frames across different compression
Covered Aspects of Multimedia Audio/Video Presentation Playback Image/Video Capture Audio/Video Perception/ Playback Image/Video Information Representation Transmission Transmission Compression Processing Audio Capture Media Server Storage Audio Information Representation A/V Playback
System Architecture Image/Video Capture Audio/Video Perception/ Playback Image/Video Information Representation Compression Processing Audio Capture Audio Information Representation
System Modules and Data Flow: Recording Frame rate, resolution type Thread Thread Capture Filter Encoder Store Webcam Player Thread Encode Muxer Store Encode You need to create separate threads Muxer is optional. You may need it to run the video file using standard player
System Modules and Data Flow: Playback Media File type Capture Decode Player Rate control Decode Capture Demuxer Decode Display the video on your GUI and place the audio data to the sound card Demuxer is optional. You may need it run standard video files from Internet
System Modules and Data Flow: Monitoring in Recording Thread Monitoring Component Monitoring Point Thread Thread Capture Filter Encode Store Webcam Player Thread Store the monitoring output to a file [Optional] You may draw chart showing [time vs. compression ratio (r)], [time vs. compression time (t)] and [time vs. frame size (s)] [bonus point] X-axis: Time in 5 second interval Y-axis: Average values (of r, s or t) in last 5 seconds
System Modules and Data Flow: Monitoring in Playback Monitoring Point Thread Media File Monitoring Component Thread Thread Capture Decode Player Store the monitoring output to a file [Optional] You may draw plot showing [time vs. decompression time (d)] [bonus point] [Hint: Try JFreeChart library] X-axis: Time in 5 second interval Y-axis: Average values (of d) in last 5 seconds
gstreamer Architecture Multimedia Application gstreamer Core Framework Sink source Encoder Decoder Muxer Demuxer 3rd Party plugins Goal: create a multimedia application using gstreamer gstreamer uses 3rd party plugins for processing (over 150)
How does it work (1) ? You need to create element for each system modules Elements are equipped with Pads: source and sink pads Source Sink gstreamer elements The number of source and sink pad varies depending on the element type You can add/ create additional sink or source pad
Creating gstreamer elements Java code example to create an element Elements element_name = ElementFactory.make ( plugin name , your defined name ) Java code example to set element property element_name.set( property , value ) source Src Capture filter Filter Src Sink encoder Encode Src Sink muxer Sink Muxer Src Sink sink Store Sink
How does it work (2) ? You need to link the elements Linking define whom is receiving and sending data to whom This is similar to linking the system modules Capture Filter Src Sink Src Media Source Media Filter
gstreamer-java pipeline Webcam Capture Filter Encode Store Media Encoder Media Muxer Media Filter Media Source Media Sink Src Src Sink Src Src Sink Sink Sink Java example code for creating, and linking a pipeline Pipeline pipe= new Pipeline( test ); pipe.addMany(source, filter,encoder, muxer,sink); Element.linkMany(source, filter,encoder, muxer,sink); pipe.play();
How to create multiple pipelining Several pipelines from the same source or towards same sink Example: recording and playback at the same time Thread Thread Capture Filter Encoder Store Webcam Player Thread Media Encoder Media Muxer Media Sink Media Filter Src Sink Src Sink Sink Media Source Src Src Sink Media Sink Src Sink
How to create multiple pipelining You can create additional pads You can use tee elements Use queue elements after tee muxer Media Sink Sink Src Sink queue1 encoder tee Src Src Sink Sink filter source Src Src Sink Sink Src Src queue2 Sink Src Sink Sink
How to redirect frames to application How to deliver a frame to your application Use appsink element Emit Signals Media Encoder AppSink Media Filter Media Source Src Src Src Src Sink Sink Sink Java example code for getting frames from AppSink AppSink appsink = (AppSink) ElementFactory.make( appsink , null); appsink.set( emit-signal , true); appsink.setSync(false);
Some gstreamer plugins Video webcam source: v4l2src Audio webcam source: alsasrc Video or Audio file source: filesrc Video Encoder: ffenc_mpeg4 (mpeg4), jpegenc (mjpeg), Video Decoder: ffdec_mpeg4 (mpeg4), jpegdec (mjpeg), Audio Encoder: vorbisenc, alawenc, mulawenc Audio Decoder: vorbisdec, alawdec, mulawdec Muxer [optional]: avimux (avi), matroskamux(mkv), Demuxer [optional]: avidemux (avi), Audio sink: alsasink
Evaluations Required Points: 100, Optional Points: 20 Features Points Properties Video Recording 15 Simultaneous Record and Playback Video Compression 10 mjpeg, mpeg4 Audio Recording 10 Audio Compression 10 pcm, vorbis/ mp3 Video Playback 15 FF, RW, S, P Functionalities Audio Playback 10 Graphical Interface 10 User-friendly System Monitor 10 Monitor at each 5 sec interval Report Writing 10 User manual and Development manual Monitor Visualizer 10 Hint: you may try JFreeChart Camera pan and tilt 10 Points are will be considered based on live demo and interview performance
Environment and Equipment Choose what ever language you like, Gstreamer is compatible with most popular languages C/C++ JAVA http://docs.gstreamer.com/display/G stSDK/Tutorials Up-to-date detailed tutorials, step- by-step runnable sample codes Requires external GUI libraries: GTK+(used in the tutorial site), Qt (very powerful) https://code.google.com/p/gstreame r-java/ Brief introduction, few samples Built-in GUI libraries Not supported by EWS (installation of external lib) Supported by EWS
Environment and Equipment If you choose to use EWS.. group directories /team/cs414/G#(# is your group number) If you choose to use your own machines.. Windows/Mac Android/iOS Feel free to use your own camera/mic, or you can borrow two logitech cameras (with mic embedded) from Engineering IT (Barb Leisner)
