Thursday, May 6, 2010

Hacking in gst-opencv

It has been years since I last used OpenCV. We (me and friends working on a lab at the university) used it to process images on batches or to process frames live from a webcam. Things could have been much easier if I knew GStreamer back then. Said so, I decided to take a look at gst-opencv to see what we already can do with it.

There are a few features wrapped as elements at this moment and they work quite well, but it could have a much larger feature set and it seems no one has been recently working on this. Given those and having a little spare time these days, I decided to start hacking on gst-opencv and trying to put it together with the other modules. I'd prefer to have a gst-opencv module, but adding it as a new plugin into gst-plugins-bad is also an option. What do you think?

Current features

[Edited: It seems the videos can only be seen directly on the post at blogspot]

Some nice stuff can already be done with the current elements. Let me show some.

I recorded this video outside some minutes ago:

We can use edgedetect on it and see its edges:
Command: gst-launch uridecodebin uri=youruri ! queue ! ffmpegcolorspace ! edgedetect ! ffmpegcolorspace ! theoraenc ! oggmux ! filesink location=result.ogg

Or we can segment it with pyramidsegment and have a nice effect (some people would enjoy this in PiTiVi?) or use it in machine vision stuff?
Command: gst-launch uridecodebin uri=youruri ! queue ! ffmpegcolorspace ! pyramidsegment ! ffmpegcolorspace ! theoraenc ! oggmux ! filesink location=result.ogg

OpenCV already ships some face detection profiles for you (at Ubuntu, it goes into /usr/share/opencv/haarcascades/), so you can use them with facedetect element, or train your own classifiers to use with it. I stood with the default and tried on some pictures, here are 2 of them:

I think it works pretty well :)
You can disable the circles and just get messages with the faces' positions and do whatever you want with them.

Other than those, there's also 'textwrite', 'templatematch' and 'faceblur' elements.

Current work

I've been working on a simple base class that will make it easier to map simple 1 to 1 OpenCV functions into elements and providing some common properties (like ROI and COI) and GstBuffer-IplImage conversion. This will help covering more functions and should be enough to get me acquainted again to the API, after it I can go for the fancier stuff.

For example, take cvSmooth function, we should only have to write code to map its parameters into properties and a simplified chain function that already works on IplImages instead of GstBuffers.


gst-opencv's main repository is at github, I have my personal branches here. From time to time I ping Elleo to upgrade at github, but I hope we can get this upstream in the next weeks.


Josh said...

Excellent! I am looking forward to using the base classes. I would encourage you to release early and often. :)

Josh said...

I'm particularly interested in all the helper functions I imagine you'll be creating, such as converting back and forth between GStreamer caps and OpenCV types. The first element I'd like to write is colorspace conversion.

Malte Nuhn said...

I'm just wondering if you happened to be able to compile gst-opencv on a mac? I would appreciate any hints and experiences you have on a mac.


Imagelife said...

I read couple of your blogs. I liked your work.Thanks for sharing.
I am using Opencv for Color tracking. on ARM-Linux platform. My real time color tracking using opencv working fine on ARM-Linux platform. But there is delay in output. When I just stream Logitech C910 HD webcam with gstreamer, its output with no delay.But I don't know how to integrate this gstreamer with opencv for color tracking. I hope you can help me out. My ARM-Linux platform is Beagleboard Xm running angstrom/ubuntu.
Please guide me. share your ideas and thoughts.