Interview with Edward Hervey about the PiTiVI video editor

This is the fourth in a series of interviews about open source multimedia, the previous interviews were about Jokosher, Totem and
Empathy. For this interview we talk with Edward Hervey who is the maintainer of the PiTiVI video editor. Edward will talk to us about the current status of the PiTiVi video editor and their plans going forward.

So to start us of could you tell us a little about who you are and what open source projects you are working on?

My name's Edward Hervey I'm a half-french, half-scot living in Barcelona and am one of the founding partners of Collabora Multimedia. I have been using Linux and Free Software since around 1995 but only started contributing in 2003 when I finally had enough programming skills to start working on PiTiVi. For several years I concentrated mostly on the GStreamer multimedia framework and plugins in order to get them to a decent level for video-editing and am still actively doing that.

I've also taken up maintainership over the years of the python bindings for GStreamer, the GNonLin plugins (core dependencies of PiTiVi which does the non-linear editing heavy-lifting), as well as the FFMpeg plugins for GStreamer.

How would you describe the goals of PiTiVI in comparison with mainstream video editing applications such as iMovie, Avid or Final Cut Pro?

There are many goals for PiTiVi, but I'd say the fundamental goal from which all other goals derive is to be a video editor framework without any limitations (unlike all other editors that have got very specific limitations to what they can do or support).

Getting rid of the limitations of formats, devices, filters,... we can support is brought to us through the use of the GStreamer multimedia framework. All other editors have hardcoded this, whereas we can for example be proud in being the only Free editor not tied to any patent-encumbered libraries ! This also means that if a company makes a proprietary GStreamer plugin for a specific format/codec/device ... you also get it for 'free' in PiTiVi without having to do anything. This is the one limitation barrier we solved thanks to our design choices.

We also don't want to have a video editor whose functionnalities are static, that's why we chose to (re)write PiTiVi in python, and make the whole design very modular. The UI is not tied to the core backend, in fact you can even use it without a UI. The way the core backend is designed, you could also write different ways to represent the Timeline, some might be better suited depending on your level of expertise or what you would like to concentrate on most (which depend on where you're located in a production workflow).

The end goal is for PiTiVi to bring the basic editing tools without putting in any restrictions. That, unfortunately, comes at a cost : having to make sure all bugs/issues/limitations are fixed/removed at the proper level instead of adding hacks over hacks. And that takes time...

PiTiVI is written in Python, I guess the first question a lot of people will ask themselves when hearing that is if the use of Python doesn't cause performance issues for a multimedia application,especially something as resource heavy as a video editor?

Ah ah ah :) It's simple logic. What is the most cpu-intensive part of video editing ? Multimedia-processing (whether it's for viewing, applying effects, rendering, ...). That is done entirely by GStreamer and its plugins (Written in C, running in separate threads, used by PiTiVi through the python bindings).

Then what else takes up the rest of the CPU ? The UI. That is handled by GTK+/X/Cairo (Written in C, used by PiTiVi through the python bindings).

So what remains is the logic hooking up everything together and bringing in the notions specific to NLE. And we want this to be as fast and flexible to write as possible, therefore a language like Python comes in very handy. Also Python is fun to write :)

For those wondering, when playing back a Timeline in PiTiVi, the only thing the python code is doing is ... displaying the current position being played. Same thing when rendering. All the rest is entirely handled by GStreamer, its plugins and GTK+.

There is currently a lot of work happening in GStreamer to support the MXF container format. Most people would probably never have heard of this format, why is it important?

Since the beginning of computer-assisted editing (also called Non-Linear editing), every software was using different formats/codecs to work on. What profesionals and consumers were using were also different, including for the cameras/vcr/capture devices. Add to that the fact that most formats weren't documented. That means a different format per software, per camera, per output, .... a total mess.

Several of the key playors in the profesional market, members of the SMPTE (Society of Motion Picture and Television Engineers), got their act together and wrote the specifications for a new container format that would be fully documented, allow for metadata, be recordable straight on camera, easy to stream while also being fast to seek in ,... It also uses the same building blocks as AAF (Advanced Authoring Format) which is an open file format for editing projects used by Avid, Final Cut Pro and many other NLE out there.

With MXF support being added to GStreamer, we can now use in PiTiVi the same formats as the professional NLE out there ! Avoiding conversions from one format to another is a key point when it comes to integrating PiTiVi in existing production workflows.

Which multimedia formats do PiTiVI currently support?

Anything the GStreamer plugins installed on your system can support, although some formats are inherently better for fast video editing, like those offering accurate seeking, I-frame only codecs, etc...

What is the state of PiTiVI and what roadmap do you have for it going forward?

Over the past 6 months we have done an in-depth refactoring of PiTiVi, based on the analysis I did last summer of the past 5 years of feedback we got on PiTiVi. Apart from enabling us to add new features much moreeasily, we have already implemented much needed features, like multi-layer editing, thumbnailing, trimming, ripple/roll editing, generic file format support, etc...

We are looking to do a first preview release of the new pitivi now in May. We'll then be gradually adding more features in the User Interface, such as transitions, effects, titling, slip/slide editing and more file format supports (allowing you for example to import projects from other editors).

For those not to familiar with the terms, what do you mean with multi-layer editing, thumbnailing, trimming and ripple/roll editing?

multi-layer editing: The possibility of being able to sort your videos not only through time, but also through several parallel layers.

thumbnailing : Having thumbnails displayed for each source used in the timeline, allowing you to quickly see what's being used in the timeline at any given point without having to seek.

trimming: Changing the beginning or end position of a source.

Ripple editing : shorten or lengthen an in- or out-point of a single piece of media, while moving all the media to the right of that clip up or down the timeline, to adjust for the new size of the altered clip.

Roll editing : Same as ripple, except the total running time of the group of clips remains unchanged.

Who are the current main contributors to PiTiVI apart from yourself?

The two main contributors apart from myself are Brandon Lewis, who started working during the Summer of Code 2007 and works mainly on the user-interface, and Alessandro Decina who started working in October mainly assisting me in the lower levels of PiTiVi and improving GStreamer. Both are Collabora Multimedia employees since last fall, working 100% of the time on PiTiVi. For those wondering about the business reasons for putting so much resources into PiTiVi at Collabora Multimedia, well first of all we do consulting work for customers in the content creation industries who want to use the PiTiVI editor for specific tasks in their production workflow and secondly we have customers who want to use the infrastructure we created for PiTiVI to set up automatic video manipulation tasks in a UI-less fashion. For example automatic cropping, rescaling and transcoding.

What future developments in GStreamer are you looking for in terms of enabling new features in PiTiVI?

Even though we have working editing features with the current GStreamer core and plugins, quite a number of features are being worked on to improve even more the editing experience.

One much needed feature is to implement the support of GstIndex in more plugins. This will allow non-editing-friendly files to be scanned when imported and have a file index created, allowing faster seeking in those files thereafter. This will also allow applications like PiTiVi to be able to have a better knowledge of the layout of the files, like the location of keyframes so that we can know ahead of time where's the most efficient location to seek to, or where we can do lossless cuts.

A never-ending work to be done in GStreamer is to wrap more and more libraries as plugins. People often mistake GStreamer as a codec library, which is completely wrong, GStreamer's a framework, allowing application writers to not have to worry about the processing parts, and processing library writers to not have to worry about the applications using them. Some libraries we should definitely get wrapped as GStreamer plugins are gavl and the overwhelming amount of excellent AVISynth filters (which will be done this summer as part of Summer of Code). All of these will definitely improve the user experience for PiTiVi.

Apart from that, we are going to carry on improving the memory usage and cpu usage of GStreamer core and the various plugins. And we're looking into making better usage of various hardware devices like graphic cards through VDPAU, VA-API and the gst-plugins-gl plugins. These will enable GStreamer applications to offload a lot of processing onto hardware devices, leaving more cpu time available for generic processing.

With Jokosher and PiTiVI both being multimedia editing applications, using GStreamer and written in Python, the natural question is if there is a lot of cooperation between the two projects?

Unfortunately not as much as can be expected. We've been helping each other a lot for all the underlying parts (GStreamer, python bindings, GNonLin plugins), but when it comes to the user interface and core logic we have different use cases and different code bases. It would be great to have tighter integration, allowing you to use Jokosher to edit the audio parts of a video clip, and PiTiVi to do the video parts.

What are the major technical challenges you are facing in terms of writing a video editor for linux/unix systems? How ready is the linux infrastructure for instance in terms of hosting such an application? And what is your opinion on the state of multimedia on linux/unix in general?

Open-source multimedia has come a *LONG* way in the past 10 years. For those who don't remember... we had to use different players depending on the formats back then and you could barely capture video from DV cameras. But the problem is that everybody is doing their own software in their own corner. There's a total lack of cohesion between all the various multimedia applications/libraries/device-support on linux which is IMHO the reason why we're not yet the reference platform for multimedia creation. Unlike on other plaftorms (win/mac), we don't have to rewrite our own libraries for certain multimedia/device support, but we can re-use existing ones... why aren't we there ? GStreamer tries to bring everything together and solve most of those issues, but is still getting kicked back by some software developers for some crazy unknown reasons. The day we solve that issue... we'll be the killer desktop platform just as we managed it for embedded devices and servers.

Which other multimedia applications do use yourself?

Totem, Rhythmbox, audacity (ouch, doesn't use GStreamer). I also sometimes use avisynth/virtualdub (win32 applications) through WINE, but that might change soon with a Google SoC project to wrap AVISynth filters as GStreamer plugins.

Well thank you Edward for taking the time to do this interview, best of luck going forward!

To learn more about PiTiVi visit pitivi.org or visit #pitivi on irc.freenode.net.

Interview done by Christian F.K. Schaller.