Support for 8-bit and 16-bit pcm audio have just been added to qtmux.
I'm stepping now into video data. As it requires less knowledge about profiles and codecs, I'll start with raw rgb video. And then try to insert one format for compressed data (h263 or h264).
Inserting 8bit and 16bit pcm audio made me found a possible bug in qtdemux, already reported in bugzilla: #542410