Introducing GStreamer
From CInLUG
Conteúdo[] |
Introduction
One of the most heated debates (and that started one of ) is "what is the best media player": Xine, VLC, MPlayer, Totem or {insert your favorite}? In the fore mentioned discussion, Jesus (Sanchez) ask me to explain why I think GStreamer is so cool, and then we came to this article.
it's a framework to help creation of media streaming aplications. Media streaming? Framework? It was not about media players, there isn't some confusion here? Yes, there is, and one of the purposes for this article is to state well the differences. For example, many people criticize Totem for its bad voice-image sync in the videos, or that it can't play WMV files, but the problem was in GStreamer, not in Totem. He could be criticized for his functionalities was not in the taste of the user X or Y, but not for the sync or abscense of codecs.
In The Land of Monolithic Media Players
So, let's see the media players. The next picture shows a diagram of three video players very well known: Xine, MPlayer e VLC.
What we see here is clear, and not promising for multimedia in Linux, each one implements the same functionalities in his own manner, efforts are duplicated, there is no cooperation, and the "cool features for the user" can only be implemented when all the infrastructure starts to work. And to make things worse, with each one using his own plugin infrastructure, codecs ported to one will not work with other. Not without weeping and gnashing of teeth. And this examples deal only with video players, if we include audio players, video and audio editors, the hopes of Linux as an attractive platform to both home users and multimedia professionals, goes down the toilet, passing through the arena of the portable devices - where the competition with M$ and other proprietary software companies its more fierce. Not that the mentioned application aren't good, contrary to this, they are excelent, some of them have libraries to creation of other apps. But these libraries are very specific to decode media, and if I want to create an editor? We will need to use another API: more time, more duplicated efforts, and this way it goes on. doing
GStreamer
Repeating: GStreamer is a framework for development of multimedia streaming application. We'll talk about the "multimedia stream" in another moment, for now look at the next picture presents a view that, even superficial, will give us an initial idea of where GStreamer is positioned in the great scheme of the things.
GStreamer provides an abstraction layer over components to manipulate any kind of format to which exists an installed plugin. The applications don't need to worry in implementing decoders, mixers, sync nor codecs, they can concentrate only in the "cool features for the user", what means better apps and happier people. :)
GStreamer is also multiplatform, and was seen in x86 Linux, PPC, ARM, x86 Solaris and SPARC, MacOSX, Microsoft Windows and IBM OS/400.
See a complete list of features in:
In The User Land: "I Don't Care, Just Open My Files!"
About the example in the beginning, the problem of sync in Totem, what leads people to install totem-xine (Totem with Xine libraries), lasted until version 0.8 of GStreamer, and was due to threads sync bugs, these and many others were fixed with the release of 0.10 series. After this release many multimedia applications started to use GStreamer in their infrastructure.
Some applications:
- Amarok: music player with many features.
- Bonfire: cds/dvds burner (tested and approved! ;D).
- Istanbul: records videos of your desktop (useful to create tutorials and presentations)
- Jokosher: multitrack musical editor, made with Python.
- Listen: another feature rich music player.
- Pitivi: non-linear video editor.
- Thoggen: tool to create DVD backups.
Besides these, GStreamer is used in the and in the new platform from Palm
Remember that GStreamer is a project from , and was designed to be as neutral as possible, and could be used by any Desktop (even running in Windows), with licenses choosen to avoid legal problems, very common when you are dealing with multimedia proprietary codecs, and to give the maximum of freedom to developers. GStreamer is under LGPL, and some of its plugins under a BSD license.
Before we dive (but not too much) in the GStreamer architecture, here are instructions about how to install all codecs in a Debian/Ubuntu system. For other distributions look at your repositories, to those who like compilation, the source is here:
It's very simple: go on Synaptic (or apt-get, or aptitude) and install every package that begins with gstreamer0.10, except the ones terminated in -dbg, these are for debugging, if you want the documentation (it's excelent), can install the ones terminated with -doc. And don't forget to add the Universe and Multiverse repositories in Ubuntu, and Non-Free on Debian.
Due to the quantity of plugins and of doubts about licensing aroused by some distributions, the GStreamer plugins were divided in:
- gst-plugins-base: stable set of plugins providing some basic functionalities.
- gst-plugins-good: good quality, correct functioning and LGPL license.
- gst-plugins-ugly: good quality and correct functioning, but your licenses impose problems to distributors.
- gst-plugins-bad: not very good, some are not maintained (volunteers?).
Note that the licensing problems of some codecs, like mp3 and avi, are the motive for the various distributions can't come with support for these formats. And this doesn't hinder the user to download them for himself, not counting as distribution.
Elementar Fundamentals of GStreamer
As I promised, let's talk about this media stream. Think about the passage of time in a music that you are listening, in a movie that you are watching or even in your life - all these are streams, like a river. In GStreamer all begins with a source of the media stream, and that can be anything: a file, audio from a mic, a streaming from internet, etc. The stream originated in this source is received by a sink: other file, a screen, earphones. Every stream needs a source and an ending point. A diagram will help the understanding:
Elements
In the GStreamer nomenclature every component, except the stream itself, is called Element, and must have ate least a source, a sink, or both, or many of each one. Elements can be in four states:
- Null: standard state.
- Ready: buffers are allocated and files are opened, but the stream is waiting.
- Paused: the stream is open, but its "movement" is frozen.
- Playing: same as Paused, but the data is flowing.
Filters
In his way to his destiny the stream may encounter some elements which receive the stream in its sink, do some manipulation, and deliver the result in his source. These elements are called filters and normally various of them are necessary.
Pads
Sources and sinks are called pads, and they are like "doors" that the stream pass through. Normally they have very specific data manipulation capabilities, and can restrict the data types that can pass through. If you want to process an mp3 stream will need the right elements (which already come with suitable pads) to deal with the data. The pads also offer information about his capabilities to elements that can connect with them. The following picture show specific source pads to audio and video of a demuxer (channel separator).
There are three kinds of pads:
- Dynamic: appear and desappear as need arises and end; if, for example, the demux from the former example was for an OGG stream, it will show the pads in accordance with the content: audio, video or both.
- Request: pads created on demand.
- Ghost pads: boo!
Bin
It's simply an element container. In the left side of the Bin in the following illustration, there is a sink pad connected to the sink of "element1". This sink pad is called ghost pad, and is a kind of symbolic link to the sink in "element1". Using ghost pads and the fact that the Bins can receive instructions to change state (Playing, Pause, etc.), we can deal with them like normal elements and integrade them in higher level streams, abstracting complicated details.
A special kind of Bin is called Pipeline, and its the higher level Bin that contains all the elements and other Bins of the... pipeline. Hmmm, this is ugly. I try to say: if we could call a complete system a Pipeline, I say that "assemble a Pipeline" to transform WAV to OGG, OGG to MP3, to extract audio from a DivX, etc.
Bus
The manual said that this is a system to transfer messages between the pipeline threads. The application developer doesn't need to create the Bus, which is created by the pipeline, neither worry about threads, just define a handler to the messages arriving through the pipeline. Works like some kind of callback, but polling is also possible.
Buffers and Events
This one is short. Buffers keep the stream data on the pipeline, Events contain control information.
And these are the components of a Pipeline, and now we know their names. Those who deals with hardware development (hello Engineering folks!) must have noted the similarities with the terms and components used in the development of electronics systems.
Plugins
What are these? Are they codecs? Or elements? Are codecs elements? More or less. If I want to exorcise an WMV video to OGG, I will need the right elements to understand both formats, so we can say that elements are codecs. Plugin is the way that elements are"packed", they are blocks of code that can be loaded as an shared object, that the Egypt slaves, ops, Windows users call DLL. A plugin could make available one or more elements and encapsulate all information about these.
Pitfdll
This is a special plugin. In case that there is not a free/open source codec to the format you are wanting to play/open/convert, is able to load binary DLL (DirectShow) and QTX (QuickTime). More information and . I do this in 0.8 and I doesn't even remember if it works, I think so. I don't advise the use of Windows binary codecs, except as last solution.
Some Practice, For Mercy
All this information is necessary not only to understand the GStreaming work, but alto to create applications that use all this fire power. Now we are able to create pipelines! We could write a program that uses GStreamer and build the pipeline with objects, or assemble it with a graphical tool like gst-editor, or use the gst-launch. This tool is installed together with GStreamer, and it serves to build pipelines in the command line, so you can write scripts to, for example, convert all your collection of MOV videos to OGG/Theora. The elements are connected through exclamation ("!") signs, that do the same role of the traditional ("|") pipes from Unix shell.
The examples were extracted from the man pages of the gst-launch, the exact name will depend in the version of GStreamer. We will use gst-launch-0.10. If you installed all the plugins everything must be ok.
Play a mp3 file
$ gst-launch-0.10 filesrc location="weezer - hit me baby one more time (britney cover).mp3" ! mad ! audioconvert ! audioresample ! osssink
Leaving the connection as PAUSED ...
The connection is doing PREROLL ...
The connection has done PREROLL ...
Leaving the connection in PLAYING ...
New clock: audioclock0
After the music ends, the following messages will appear:
Obtained element EOS "pipeline0".
Execution ended after 213340055000 ns.
Leaving the connection as PAUSED ...
Leaving the connection in READY ...
Leaving the connection in NULL ...
FREEING the connection ...
Pipeline diagram:
As shown in the diagram, the media stream begins in the filesrc element, which brings the content of the file to the pipeline; it's an element from the Ugly plugins, and decodes mp3. audioconvertaudioresample are elements from the basic plugins, their names are sugestive, e to run away from explain I say that "more details can be seen "; I also tested without audioresample and worked fine (gst-launch-0.10 filesrc location="weezer - hit me baby one more time (britney cover).mp3" ! mad ! audioconvert ! osssink), but without audioconvert, no deal, including an error message: and
- ATENTION
- erroneous conection: unable to link mad0 to osssink0
The plugin package documentation says:
- Audioconvert
- convert buffers of raw audio between various possible formats. It supports conversion from integer to float, signal and endianness.
And osssink sends the sound to the speakers.
Play an Ogg file
Go to and let's listen to Neil Armstrong dizendo saying "a small step for a man, a great step for cinlug". Serious! See for yourself:
$ gst-launch-0.10 filesrc location="Frase_de_Neil_Armstrong.ogg" ! oggdemux ! vorbisdec ! audioconvert ! audioresample ! osssink
It's like the former, except that in place of mad, we put oggdemux and vorbisdec. Ogg it's just a container for encoded data that may come in the form of audio, video or text (subtitles), so the need for a demux to separate the channel; vorbisdec decodes the stream coming from oggdemux and send it to the speakers (osssink).
Convert a file from mp3 to Ogg/Vorbis
$ gst-launch-0.10 filesrc location="weezer - hit me baby one more time (britney cover).mp3" ! mad ! audioconvert ! vorbisenc ! oggmux ! filesink location="weezer-hitmebaby.ogg"
To test:
$ gst-launch-0.10 filesrc location="weezer-hitmebaby.ogg" ! oggdemux ! vorbisdec ! audioconvert ! audioresample ! osssink
Playing an Ogg/Theora Video Channel and Vorbis Audio
Get the video (with a Creative Commons license) "A New Computer" in (fast link: ). In the following pipeline, we identify the ogg demux with the label "d" (name=d), and we say that a sequence of elements connects to it using the notation "d." (see the dot after d). Before you get confused with this mess, behold the diagram.
$ gst-launch-0.10 filesrc location="A_New_Computer--small.ogg" ! oggdemux name=d d. ! queue ! theoradec ! ffmpegcolorspace ! ximagesink d. ! queue ! vorbisdec ! audioconvert ! audioresample ! osssink
The element queue forces the use of separate threads for the audio and video channels, ffmpegcolorspace convert videos of a colorspace to another, and ximagesink it's a standard video consumer for the X Server.
Do It Yourself
Right now you could be thinking "how could I know what components to use for my media?". Don't worry, but don't stay waiting either! To make your very own pipeline to filter your midia streams in the way you want, you must know the available elements. As was said before, the elements come in packages of plugins, e these have a documentation where each element is explained, in a way that you can choose which better fits your needs. See the online documentation:
- GStreamer Core Plugins
- GStreamer Base Plugins
- GStreamer Good Plugins
- GStreamer Bad Plugins
- GStreamer Ugly Plugins
- GStreamer FFMPeg Plugins
Gst-editor
There is a graphical tool for creation and use of pipelines, one just need to include elements and link them together with the mouse.
The picture show a pipeline well fit to play DVD, send his audio to computer sound output, and for a file.
This version of gst-editor was written in C, and uses GStreamer 0.8. One of Google's "Summer of Code" sponsored projects is to create a new version of gst-editor using GStreamer 0.10. This version will be written in Python with PyGTK, and will show its components in a Cairo based Canvas, called . (Downloaded the demo; very interesting.) The GSTEditor developer's blog is here:
Conclusion
I hope this have worth the read and that you understand the idea behind GStreamer and how he can make a solid basis for multimedia application development. And now you can convert and play your media files.
For more of gst-launch possibilities, see:
- gst-launch man page
The official site is one of the best sources of information about GStreamer, from where I highlight:
- GStreamer Application Development Manual
- GStreamer Plugin Writer's Guide
In the next article, programming with Python and GStreamer.
| Autor:
|
:
