Phonon

So, there's much interest in the Phonon + Quasar code - as I had hoped

To reproduce the result on your computer you need to get current phonon, current phonon-xine (with libxine >= 1.1.12) and current quasar from SVN. For those that don't know where Quasar lives, you can find it here: http://websvn.kde.org/trunk/playground/base/quasar/. To compile quasar you need to set the KDEDIR environment variable to the prefix where you installed phonon and then run qmake and make in the source dir. Then go to the examples/phonon subdir and run

./phonon /path/to/video.ogg

Have fun detecting edges in your videos ...

I've got it working a long time ago, but since this Akademy it is finally available in KDE trunk: Phonon now can give the raw video frames (as RGB, YV12 or YUY2) to an application and Quasar gained a node to upload those frames as textures and uses Shaders to convert YUV formats to RGB and then do arbitrary effects with them. The cool part about that, obviously, is that the data intensive calculations are happening on the graphics card (you need OpenGL 2 for this to work). To demonstrate the power of this I wrote a simple example that connects the PhononInputNode to an EdgeDetection node and then to a RenderNode. I did a screencapture of the result, but that looks a lot worse than the real performance, which is totally smooth and, of course, doesn't show much CPU usage.

You can also see me adjusting the threshold value of the edge detection filter with a keyboard shortcut.

Here's the code:

Things to note:

• the PhononInputNode is both a Phonon::MediaNode and a Quasar::Node
• the Quasar API is not final, especially wrt. PhononInputNode

After a month of no KDE coding I'm back to do a few things in KDE again - though I really want to (and need to) do more for University if I ever want to get done with it...

Anyway, what I wanted to show is how to do simple animations in the UI that should help the user to understand how the different UI elements play together. I have one such case with the Phonon KCM where I now tried the following animation to clear things up a bit:

more_ui_feedback.ogg

The way I achieved this effect was to add a special Animation widget that sits above the other widgets and is not added to any QLayout. Reimplement the paintEvent to draw the animation frames and use a QTimeLine member to call repaint(). It needs two pixmaps: one of the old content of the QListView at the right and one with the new content. The paintEvent then draws the old content over the QListView so that it seems the widget contents have not changed yet and draws the new content depending on the frame number of QTimeLine (with 50% opacity). QTimeLine::finished() is connected to the deleteLater slot which removes the animation widget.

Here's the code:

Today I checked my mails and saw a few Phonon-related commits. I always read those commit messages with priority... and I was a bit surprised. I knew this day would come. But I didn't know when and I didn't know whether it's going to be GPL+commercial or LGPL. And I found the commit messages all said "[License: LGPL]". *dance*

Now what does that license mean? It means everybody can contribute without having to sign a copyright assignment. That just makes it so much easier to handle contributions!

Now I know what code I'm going to read and play with the next days.

Read the rest here and here

After playing with the mixer API a bit I feel confident to be able to go forward a propose a design.

class diagram

I'm happy for people to look over it. Don't think about implementation details but whether this API would be easy to understand, use and still powerful enough for all our desktop uses.

After a phonon/phonon-xine debug session (where I got stuck at KIO playback showing inexplicable memory access violations inside xine-lib) I had to do something creative again. So I tackled the mixer.

First I started with trying out ALSA's softvol plugin, and whether that would have less latency than the software volume control inside xine-lib. As it's close to impossible to measure I have to rely on my senses and that's telling me there's no audible difference. Ok, so for AudioOutput's volume property I could stay with xine-lib's offering, but seeing softvol work for the first time that got me started on exploring the ALSA mixer.

When an application uses an ALSA "device" that contains the softvol plugin then this will add a new "integer control" to the mixer. KMix and also alsamixer do not show the new control without a restart, and the control does not disappear again when the application is done and no app uses the softvol anymore.

From there I went to look at kmix (trunk)'s abilities wrt. custom layout definitions and whether it would be possible to group KDE made softvol controls in a sensible way. This was not getting where I wanted without touching the code so I finally started libphononmixer.

When you want to use libasound (ALSA's user space lib) to fulfill a task it is, IMHO, best to not go from the doxygen generated documentation (which partially on their website is empty anyway) but open an editor in the libasound code and look at the header and source files. As the doxygen comments often don't tell much more than the function and parameter names already do the code is really the best to go with. And now I have a Mixer class that can be constructed from a Phonon::AudioOutputDevice object, a QString or an enum value. This installs a QSocketNotifier on the poll_descriptors libasound gives me and reads out all the events when the filedescriptor is ready for reading. With this I get all change notifications of a given mixer, even when a softvol control is added.

From there I'll go and make a nice high-level mixer API that lists all available mixers, gives notifications when a mixer appears/disappears, lists all the elements of a mixer, makes those elements give value change notifications and so on. I'll look into the possibility of removing softvol controls from the mixer, and whether it's possible to identify their origin and group them correctly. Once this is all ready it'll be the perfect base for a Plasma Data Engine and then a Plasma Applet for controlling your sound.

Personal Note: I'm close to being done with my sickness and probably today I'll go back home (where I'm alone because my wife is far away in Spain ). I stayed with my parents for the time I was sick. It's great to have them, not only as fallback solution, but in those cases it becomes so much more obvious just how much a great family is worth!

It's been a month already since I was at the Trolltech office in Oslo, but ever since I wanted to write about it...

Trolltech invited me to be able to discuss the design of Phonon or, more in general, what their needs are for a multimedia API that they want to provide with or in addition to Qt. The first three days (Wed.-Fri.) we've been whiteboarding the whole day, which was quite exhausting but OTOH exactly what I always wished to be able to do when I was designing Phonon in the beginning. The outcome was that we agreed that we want to have an API that provides all the flexibility of a graph based approach while simplifying the concepts as good as possible so that non-experts have no trouble using the API for simple tasks without getting any details wrong.

Before

This is where we needed a small change in the Phonon frontend API to make this possible. Before August Phonon allowed "graphs" of the form:

Path changes

Now the AudioPath and VideoPath objects are replaced by a value based class called Path, which is hidden for simple cases: vs. or in code:

I wanted to keep the Path concept to make it easier to support the most common "graph reconnection": to insert/remove effects. With the common graph you'd have to disconnect media and output, connect media to effect, and effect to output and also somehow tell the objects that they should not lose any signal data while they're still disconnected and that playback needs to continue seemlessly after the effect insertion. With the path object it's a simple

Mixing

Another change is that is not allowed anymore. The implicit mixing of audio data is removed. (For video it was never allowed anyway). The plan is to add a special class that handles mixing after the 4.0 release. Implementation note: This is close to impossible to implement using libxine.

What's the gain?

Perhaps you've already seen it, but why is this now more flexible than what Phonon provided before? The answer is that you can now define arbitrary graphs like . In a later version we also want to have an object that can "contain" multiple MediaObjects in order to be able to control them in sync. This is basically the concept of a media graph containing multiple source nodes. But if you have only one source, which is the case for most apps, the MediaObject defines the graph implicitly. This hides the graph concept for most developers using Phonon, which IMO helps to make Phonon easier to use.

Trolltech

Now what's Trolltech's role with regard to Phonon? They want to provide multimedia API with Qt 4.4. And they decided to set their bets on Phonon. This means they will work (with me and the rest of the community that is interested) on continuing the design of the frontend API to support use cases like capture, transcoding, VoIP apps, Video chat, ... And they will work on backends for Windows, MacOS X and Linux (including embedded). KDE will ship the xine backend with 4.0, but that one has several problems:

• libxine is GPL-only, so they cannot tell commercial customers to simply use Phonon on Linux for multimedia
• libxine does not support mixing of audio streams, and I tried to separate the clock and ticket system in libxine to make it more flexible, but I feel like this is a dead end. libxine is designed for simple playback tasks and that's it.
• no capture or encoding with libxine

On another note, 2h per workday, while I'm working on Phonon, are now sponsored by Trolltech. This additional money is very welcome, especially now that we'll have to pay the rent for a flat in Heidelberg and another one in Sevilla.

NMM Release 1.0.0 "April"

I'll be taking a closer look at this release the next weeks and will be more actively looking into the NMM backend again.

Two weeks have passed since I was in Norway, but I wanted to let you all know about it...

Trolltech sponsored and organized a meeting in their offices in Oslo to review the Solid and Phonon APIs and to get Trolltech on board to look into support for other platforms. It was a week of very fruitful discussions and hacking.

The typical day looked like this:
Get up at 7-8 (or later if you couldn't finish hacking the night before ) and go for the best breakfast in town. Then walk to the office (just a few minutes walk) and start hacking until the office fills with all the right people that want to do Phonon/Solid API reviews. We met in the room labeled "Learning" where we were seated in comfortable bean bags or sofas looking at the header files projected onto the whiteboard. And then everybody was free to ask questions:

• Why is that function there? Why not make it private?
• Why is this called AbstractSomething if the user of the API is not meant to reimplement the class?
• Is this class necessary?
• Can we have a better name for "..."? Are there similar methods in Qt so that we can keep the naming consistent?
• What applications will use this? Only one or two?

Anyway this kind of API review was really good for Phonon and Solid and more APIs could benefit from such work. Too bad that we don't have the time to do it for all of kdelibs

The most invasive change in Phonon was the removal of all MediaProducer classes except MediaObject in favor of an additional class MediaSource. So instead of deciding whether you need a MediaObject, MediaQueue, ByteStream or AvCapture object you now take a MediaObject and call setCurrentSource(MediaSource). MediaSource is just a simple class with implicit constructors so you'll be able to write:

I am about to get rid of the core/ui split of the Phonon libraries. As you can see, a GUI application that only uses functionality from phononcore does not really benefit from the split:

objdump -T libphononcore.so|grep "\.text"|wc -l
617
objdump -T libphononui.so|grep "\.text"|wc -l
127

You might argue about the non-GUI apps that want to use Phonon. But seriously, what apps are those?

If you want to, you can still try to convince me that the split adds value. But you have to make some really good arguments to justify the added complexity on the frontend and backend sides and the added exported symbols that are necessary for it.

The last days I worked on reducing the dependencies of libphononcore. So far it linked to libkio + libsolid. KIO+kdeui were used for the MediaObject KIO fallback, KMessageBox, KIcon and KNotification. I've moved all those (except the KMessageBox stuff - it uses KMessage now) into interfaces that are implemented in libphononui now and libphononcore will make use of if the application links against phononui. There's no need to call any method in phononui to make it work, it's completely automatic. But that also means you need phononui for KIO fallback (which is not 100% intuitive). Perhaps KIO fallback should be a plugin, but I think that's overkill.

Now my changes are awaiting the last KMimeType cleanups, so that I can use KMimeType in phononcore from kdecore (it was in libkio before).

Also I worked on making the "unit" tests to be compiled as two different sets: one that force the fake backend and can be run automatically on "make test". And another set of the tests that should be run manually: to be used by backend developers. For the latter I was contemplating a little GUI that runs all the tests (and where it makes sense, runs the tests multiple times) and gives a nice overview what fails reliably, what fails sporadically and what works reliably + all the debug logs that go with it. This is very useful since backends that use threads sometimes work and sometimes don't. Also it might depend on timing issues with KIO or any other data source: in short there's great potential for race conditions. So stress testing the backend makes a lot of sense.

For example I found one serious bug in the xine backend that would happen about 1% of the time only by stress testing ByteStream. I.e. every time the address of the ByteStream object contained a '#'. You don't want to know the details...

Christian Schaller just blogged about it: Fluendo will join the work on the Phonon-GStreamer backend which Tim Beaulen started a few weeks ago. This is great news as that will surely bring the best GStreamer backend possible to KDE4.

I'm very sad that I have to keep from working on KDE/Phonon too much these days. But I really have to get done with university at some point. And working on KDE doesn't help my studies.

On another note my laptop is out of service for now as its harddisk is broken and Samsung's guarantee becomes void when I change the hdd myself. So I'm waiting for them to pick it up, exchange the hdd and send it back... no KDE compiling for now (which helps studying for Uni ).

One of the things I've been working on a lot lately is to make KIO work seamlessly with Phonon. The idea is that you can throw any URL at a MediaObject and it will work even if the mediaframework used by the backend doesn't support the given protocol. A protocol that isn't supported by Xine, for example, is sftp://. MediaObject handles those URLs by first trying whether the backend will play the URL without further help, if that doesn't work it asks the backend to create a ByteStream object which provides an interface for streaming the media data using QByteArrays. It then sets up a KIO::FileJob (the new one that supports seeking) or KIO::TransferJob if the protocol doesn't support seeking and connects the data and flow control signals accordingly.

One of the Phonon test programs playing a file accessed over sftp from the computer of my wife.

With the announcement of the Phonon website (and for many it also was an announcement of the whole project) I received a lot of reactions. Far more than I had expected!

While there are still some people asking why KDE 4 will not "standardize" on GStreamer, most understood my reasoning though and are looking forward (together with me) to Phonon + KDE 4.

Yesterday linux.com published an article about Phonon. The author of the article contacted me right after the announcement to produce, what I think is the best independent text about Phonon so far.

On Friday and Saturday I was able to participate in presenting KDE on LinuxTag in Wiesbaden and on Saturday morning Michael Repplinger from NMM and I held a one hour presentation on Phonon and NMM and demoed the audio/video playback capabilities of the Phonon-NMM backend. It was also the first time for me to see the network capabilities of the backend in action: The Phonon-NMM control application was able to route the audio and video signals to another computer on the network such that both computers played perfectly in sync.