mpeg4 trouble

Expanding qtmux to support mpeg4 was going all well, everything was almost ready! Then the ES Descriptor appeared. For .mov files it was only used for AAC description atoms. For mpeg4 it has to be used for every format, and:

1. This ES Descriptor has a DecoderConfigDescriptor that has 'objecttype' and 'streamtype' integer fields that would identify the stream format and I can find no info on how to map the formats to those values (other than AAC).
2. The DecoderConfigDescriptor has some 'bitrate' and 'decoding buffer size' fields which aren't usually provided by GStreamer, so I'm just filling that up with zeros.

If there is someone out there that knows how to map formats to those fields or any idea on how GStreamer could provide bitrate (perhaps tag events?) I'd really appreciate some help. I've been stuck for almost a week.

Also I've started to test the mpeg4 files in a PS3, too bad it doesn't give meaningful error messages.

AAC and MP3 at qtmux

I know I just posted, but I'd like to make a separate post for this.

Jan Schmidt(thaytan) sent me a patch that got AAC and MP3 to work! Thanks!

And he also sent me an optimization patch!

GSoC summary and thanks

Now that Summer of Code ended, I'd like to summarize what has been done around qtmux and some random thoughts around it.

Firstly, I'd like to thank the guys from GStreamer for letting me participate with them, specially to Win (my mentor) and Christian(AFAIK, the org leader for GSoC). Also to those that answered my daily questions at IRC (mainly Mike, David and Tim). This was a great experience, learned lots of stuff about muxing, GStreamer and quicktime/mp4 file formats. Thanks also to the guys from pitivi for reporting problems and their effort for already adding qtmux to it.

It feels great to see the community actively using, testing, reporting bugs and requesting features at this early stage of qtmux, this is really motivating! Also, it will be moving soon to GStreamer repos. That means more developers working on it, consequently better quality code and more supported formats.

Looking at the application proposal, most of the formats promised are already working, but qtmux still lacks the flavor selection. That's why the next step is adding isomedia ('isom') output format, some work is already commited, but it is still at an early experimental state.

Lastly, another thanks to you all (GStreamer and Google), I really enjoyed participating and I hope that qtmux is already usable and meeting the expectations.

A lot of help

The last week has been unusually productive due to 3 people helping me with qtmux (well, everyday I get help from people at #gstreamer, but those 3 were directly helping).

In a non chronological order:

• Edward Hervey (bilboed at IRC) did some changes to faac plugin and now we can have qtmux muxing mov files that contain AAC which play at totem (gstreamer) and mplayer. (Quicktime Player still can't play those).
• Chris Cooksey is a possible future user of qtmux (one that really understands from the mov container) and while testing qtmux spotted 7 or 8 problems in the resulting files and reported me, so we fixed them (most related to timescale, dates and durations in the file).
• David Schleef (ds at IRC) sent me a patch for adding Dirac support into qtmux (it works!).
  

Thanks to you 3.

finally h264!

Just for letting you know: qtmux now supports h264 (and also a-law and mu-law)

Yay! \o/

Update on qtmux

I've been through the semester final exams the last weeks (my university timeline is all messed up) and time was short, so I couldn't update here with news on qtmux, but the semester in now finally over, so here are the new features:

• raw formats (both audio and video) and mp3 are working correctly in all players I tested (totem, vlc, mplayer and quicktime)
• support for h263 has been added and also works on the players above
• support for aac and amr has been added and it still doesn't work at quicktime, but works in totem and mplayer (vlc went nuts on my machine, and I still haven't got it to work again)
• ilst atom metadata support (used by itunes and qtdemux) has been added

I couldn't get aac and amr working, after some days searching for specs and testing applications that could generate qt files with amr or aac with no success, I noticed too much time was being lost and started working on other features.
None of the applications I tested could generate a qt file (with amr or aac) that quicktime would actually play, and I couldn't find any useful spec. I also tried inspecting bytes in some files that played successfully, but just looking at them wasn't enough, the different bytes are easy to find, but not easy to understand.

I said that I would add flavor selection in the previous post, but I was advised to work on some widely used formats first, so I did it.

GSoC is comming to an end in the next days, but work won't, qtmux has still a lot to be worked on and I'll keep it up.

Good news!

Qtmux has just generated the first mov files that have audio and video that plays synchronously. MPlayer and GStreamer were able to successfully play it, VLC couldn't (not sure why, yet). Still have to test it in another players. But this is a huge step.

Now, I'll probably be working on adding flavour selection (qt, mp4 or 3gpp) using a property in the element (maybe using caps?).

Raw video into qtmux

Raw video formats were added to qtmux: 'rgb' and 'yuv' (only one 'yuv' format, but easily extensible to others). The current state is that it is possible to package video 'xor' audio into a mov file, and they are played fine.

What's keeping me busy now is to add support for synchronized audio and video. Some work has already been done, but it is still not as expected. I've re-read the spec trying to identify what info am I missing, still no clue, but I'll keep looking :)

New audio formats for qtmux

Support for 8-bit and 16-bit pcm audio have just been added to qtmux.

I'm stepping now into video data. As it requires less knowledge about profiles and codecs, I'll start with raw rgb video. And then try to insert one format for compressed data (h263 or h264).

Inserting 8bit and 16bit pcm audio made me found a possible bug in qtdemux, already reported in bugzilla: #542410

The first one

I should announce that the first mp4 file has been created using qtmux (at least, the first one capable of playing nicely). It is still only able of handling mp3 audio. But that means that the underlying mp4 library reached some stable level and now the work focus should be into the qtmux element itself.

Next steps:

1. Add another audio format support (probably raw audio)
2. Add video support (h263? Suggestions?)
   
3. Add multiplexing support (finally!)

I thought I was going to reach this goal before, but my graduation required some extra hours in the last weeks and mp4 atoms' fields given me some headache on understanding its semantics, mainly in how to define what media type (and parameters) for a track. 'qtdemux' has been a great help, it outputs some really helpful debug messages. Thanks for the guys who developed it! Great job!

Code is here: http://gforge.embedded.ufcg.edu.br/scm/?group_id=122

Refactoring

I should have posted this before, but the code for GSoC is already hosted online at http://gforge.embedded.ufcg.edu.br/scm/?group_id=122 (there's info about checkout there, and also you can browse the code online)

After writing a lot of repetitive code for atoms serialization I decided to stop and rethink it. So, creating a function for serializing each type of property an atom can have made the code much shorter and better to read. This refactoring proved even more useful, when I found out that mp4 files had to be big endian, I did the changes to support it in no time, as I only had one spot to change.

Other than that, I've been using qtdump for validating my generated files, but sometimes it only reports "unsupported file type", and I need more than that. So, I'm also adding more verbosity to qtdump (and also qtinfo). I'm trying to make it able to tell why it is not a valid mp4 file.

As always, comments and critics on my code are very welcome.

GSoC First Lines of Code

Since last week, I started coding the first lines of the qtmux (yes, that will be the name of the new plugin/element).
The muxer is still at an early 'draft state', but what concerns me more at this point is creating the MPEG4 Atoms hierarchy to be used by the muxer. This hierarchy will last during the whole project and it could cause some headaches later.

I thought of two main options:

• Create a gobject class hierarchy: it is better organized and we could have some common interfaces, but it would be probably too much of a overhead
• Create a 'struct hierarchy': composing structs is pretty easy and straightforward to do. Having common structs at the top place might also provide some common methods, but probably difficult to maintain

Structs seems better to me at this point, I'm for the KISS (Keep It Simple, Stupid) principle. And I also don't expect to be changing those atoms often (after my code gets stable, at least). But, of course, I'm accepting comments and sugestions and critics.

Other than that, I'll be starting a repository to put the code in the following days, so that interested people might take a look at it.

Let the games begin

GSoC has just started this year (I've been accepted :D) and I'll use this space to provide information about my ongoing work. My project is to create and develop the quicktimemp4mux plugin for GStreamer.

For the past weeks, I've been studying specs and code on mpeg4. Right now, I'm starting to think on how to validate the resulting files my plugin will be able to generate. Some options that have already been suggested to me:

• use qtdump to inspect the files
  
• use a GStreamer pipeline to play the files

Other than that, I'm also wondering what could be my first goal at the plugin implementation, and I think that being able to create a mpeg4 file that contains only one audio track (possibly mp3) would be a nice start.

That's it, I hope that GSoC will be fun and, mostly, that I manage to finish a good quality plugin so that everybody gets happy :D