A renegotiate event for GStreamer

Currently we have a problem at GStreamer that we can only make an element renegotiate using pad buffer allocs. Check the function documentation for understanding how it works.

By using pad buffer allocs, one element can ask downstream if it wants a new caps, but it can't tell upstream to pick a new caps. This would help in dynamic pipelines and applications that do element hot-swapping, which might happen on camerabin(2).

So I started a first attempt at creating a new upstream event to make the pipeline (or a part of it) do a new caps negotiation, trying to pick optimal caps.

I had 2 basic use cases in mind:

• videotestsrc ! capsfilter name=cf ! fakesink
• videotestsrc ! capsfilter caps="" ! ffmpegcolorspace ! videoscale ! capsfilter name=cf ! fakesink

In both cases, the capsfilter named 'cf' would change its caps property periodically, making the pipeline renegotiate to pick a new compatible and optimal caps.

The resulting patches were really simple and I only modified basetransform and basesrc (other than adding the new event to core). Keep in mind that I'm still experimenting and we should search for regressions that this might cause. Next I'd like to go for a scenario with elements with multiple src/sink pads (demuxer/tee/selector).

The patches are on a gstreamer branch here and the test cases were added on a branch on my -base clone.

gst-opencv design choices

While continuing wrapping new OpenCV functions into GstElements yesterday, I faced an interesting design choice on the mappings of OpenCV functions' parameters to GstElement's properties.

Take a look at cvSmooth docs. You can see that it has a type parameter, followed by param1, param2, param3 and param4 that have different semantics if different type is used. The question is how to expose those in the 'cvsmooth' GstElement?

I could think of 3 different choices here:

1) Go straightforward and use the same API as OpenCV

As a result, we should have an element with the properties named after the OpenCV parameters:

"cvsmooth type=blur param1=5 param2=3 param3=0.0 param4=0.0"

This results in a very not intuitive API, but we keep it aligned with OpenCV's, making it easy to people that already know one API to use the other one. The element docs would mostly point to OpenCV's docs. Resulting code is simple and easy to maintain.

2) Have multiple elements: cvsmoothblur, cvsmoothgaussian, cvsmooth...

We could have each smooth algorithm (type) into a separate element and have its properties reflect the semantics of this type. For example, we would have cvsmoothblur, cvsmoothmedian and one for each type. The properties of each one would named accordingly to its semantics, instead of some paramX.

This provides a nice API but might increase the number of elements for every function that has this type or a similar parameter. I don't know how common this is. This might be a good solution if there are a few of those. A downside is that switching the type has to use hot-swapping but I don't think this is a common use case.

3) Expose properties for each semantics and use them only if their type is selected.

We still keep it to one element, but we add one property for each semantic a parameter can assume. Those would only be used it we have its corresponding type is selected.

For example: param3 might be the "gaussian standard deviation" or the "color sigma" if type is gaussian or bilateral respectively. We add those 2 properties (standard-deviation and color-sigma) that are only going to be used if their types are selected.

This makes those lines possible:

"cvsmooth type=gaussian standard-deviation=5.0" or

"cvsmooth type=bilateral color-sigma=1.0"

Code is a little messier than options above.

Given those options, I really don't like option 3. I'm considering 1 or 2. From a quick look at some pages of OpenCV's transformations API I could see that this is not very common, and when it happens, only one parameter has a 'variable semantic', looks like I picked the trickiest one as my example.

So, which option would you chose?

GstCollectPads2 branch

This week I decided to grab the GstCollectPads2 patch (from bug #415754) and start a branch in my freedesktop repository for porting muxers to it. So far we've got:

• oggmux
• avimux
• matroskamux (patch by Mark)
• asfmux
• qtmux

If you always wanted GStreamer to be able to mux subtitles into your movies, now is the time to provide specs/samples/patches for it. And installing from this branch (I try to keep them up-to-date with current git master) and test it a lot in you favorite applications would help to find regressions from the porting process.

Suggestions to improve GstCollectPads2 or other use cases are also welcome.

asfmux is now at gstreamer-plugins-bad

I just pushed asfmux plugin into gstreamer-plugins-bad. So, forget the old repository, the recent code will now be at -bad.

And it now has mp3, wmv3 and wma3 support!

Live streaming of ASF content

Just pushed a couple of changes to ASF muxer plugin that enable streaming live content of ASF media. Basically it contains some fixes to timestamping and a new boolean property to asfmux called "is-live". When enabled, asfmux won't attempt to push indexes at the end of the file (as this has no sense in live streams) and also won't try to seek back to the headers to rewrite some values that it couldn't predict when started the file.

Here is the simplest example of its use in gst-launch commands:

The sender:

• gst-launch-0.10 -ve videotestsrc ! ffenc_wmv2 ! asfmux name=m is-live=true ! rtpasfpay ! udpsink host=127.0.0.1 port=3333 audiotestsrc ! ffenc_wmav2 ! m.
• gst-launch-0.10 udpsrc port=3333 ! "put the caps here" ! rtpasfdepay ! decodebin2 name=d ! queue ! ffmpegcolorspace ! autovideosink d. ! queue ! audioconvert ! autoaudiosink

Remember to replace the caps after udpsrc with the caps of the srcpad of the rtpasfpay element in the first pipeline. If you have any problems using it, please report!

The asf plugin can be found here.

ASF: we need a parser

While developing rtpasfpay I found out that I can't just do (in gst-launch syntax):

somesrc ! asfmux ! rtpasfpay ! udpsink

because the ASF headers need to be updated after the stream ends and those values are needed by rtpasfpay.

Additionally, I can't get an ASF from a file and feed it to rtpasfpay because the buffers would be unaligned with the ASF objects and packets, requiring rtpasfpay to parse the data before doing anything related to RTP. I'd say we need a parser.

The idea is to do all the parsing at the parser (obvious) and only some checks of conformance at the rtp payloader. As I had already started developing the payloader, I took it as farther as I could (without using a parser) and its current state is now at the repository. It is still unfinished (read: doesn't work).

The parser

Now I'm starting to develop the asfparse element. ASF files have 3 main parts: first there are headers, then the data (where the packets are) and lastly a series of indexes objects. The parser would, sequentially:

1. Group all the headers into a single buffer and push it
2. Push the "Data object" header (everything but the packets)
3. Put each packet into a separate buffer and push them
4. Put each of the indexes object into a separate buffer and push them
5. EOS

The parser should work both in pull or push mode, I'll start with push mode.