KDE

So, there's much interest in the Phonon + Quasar code - as I had hoped

To reproduce the result on your computer you need to get current phonon, current phonon-xine (with libxine >= 1.1.12) and current quasar from SVN. For those that don't know where Quasar lives, you can find it here: http://websvn.kde.org/trunk/playground/base/quasar/. To compile quasar you need to set the KDEDIR environment variable to the prefix where you installed phonon and then run qmake and make in the source dir. Then go to the examples/phonon subdir and run

./phonon /path/to/video.ogg

Have fun detecting edges in your videos ...

I've got it working a long time ago, but since this Akademy it is finally available in KDE trunk: Phonon now can give the raw video frames (as RGB, YV12 or YUY2) to an application and Quasar gained a node to upload those frames as textures and uses Shaders to convert YUV formats to RGB and then do arbitrary effects with them. The cool part about that, obviously, is that the data intensive calculations are happening on the graphics card (you need OpenGL 2 for this to work). To demonstrate the power of this I wrote a simple example that connects the PhononInputNode to an EdgeDetection node and then to a RenderNode. I did a screencapture of the result, but that looks a lot worse than the real performance, which is totally smooth and, of course, doesn't show much CPU usage.

You can also see me adjusting the threshold value of the edge detection filter with a keyboard shortcut.

Here's the code:

Things to note:

• the PhononInputNode is both a Phonon::MediaNode and a Quasar::Node
• the Quasar API is not final, especially wrt. PhononInputNode

As and there will surely be a few more people traveling the same way, I wanted to ask if there's interest in traveling together - or to share a car ride. I've already talked to Martin Gräßlin who lives in Mannheim (I'm in Heidelberg, in case you didn't know already), and we'd probably take a train together. Neither of us owns a car, but if you do and drive by Mannheim/Heidelberg (A5) and want company and people sharing the high fuel cost with you, let me know. If not, you still might want to try to meet up in the train... in that case also let me know.

For the train I'd probably take the ICE 610 (12:35 Mannheim - Köln) and ICE 14 (14:44 Köln - Bruxelles-Nord) on Fr, 8th, of course. But sharing a car ride would probably be cheaper.

Now back to exam preparations...

After a month of no KDE coding I'm back to do a few things in KDE again - though I really want to (and need to) do more for University if I ever want to get done with it...

Anyway, what I wanted to show is how to do simple animations in the UI that should help the user to understand how the different UI elements play together. I have one such case with the Phonon KCM where I now tried the following animation to clear things up a bit:

more_ui_feedback.ogg

The way I achieved this effect was to add a special Animation widget that sits above the other widgets and is not added to any QLayout. Reimplement the paintEvent to draw the animation frames and use a QTimeLine member to call repaint(). It needs two pixmaps: one of the old content of the QListView at the right and one with the new content. The paintEvent then draws the old content over the QListView so that it seems the widget contents have not changed yet and draws the new content depending on the frame number of QTimeLine (with 50% opacity). QTimeLine::finished() is connected to the deleteLater slot which removes the animation widget.

Here's the code:

Today I checked my mails and saw a few Phonon-related commits. I always read those commit messages with priority... and I was a bit surprised. I knew this day would come. But I didn't know when and I didn't know whether it's going to be GPL+commercial or LGPL. And I found the commit messages all said "[License: LGPL]". *dance*

Now what does that license mean? It means everybody can contribute without having to sign a copyright assignment. That just makes it so much easier to handle contributions!

Now I know what code I'm going to read and play with the next days.

Read the rest here and here

Yesterday on IRC I was reminded of the nice abstractions Qt has with QPaintDevice. (There are many useful paint devices, but most important for this discourse are QWidget, QPixmap and QImage.) So there are several things Qt4 allows you to do:

• QPainter::setRedirected
• QPixmap::grabWidget
• QWidget::render

Inspired by the video from Mirco Müller I wanted to see what could be done with those functions. I started with setRedirected but couldn't make it work immediately. So I did what the docs said and tried QPixmap::grabWidget. And without much effort I was able to draw a widget someplace else. But then I wanted to transform the image which needs a conversion to QImage. Expecting that Qt might optimize a bit if it renders directly into a QImage instead of a QPixmap and then converts to QImage I used QWidget::render to render a widget directly onto a QImage. Piece of cake. Now I had a QImage, transformed and painted into a different widget. Last missing piece is to update the image if the original widget changes (e.g. mouse or keyboard events). QObject::installEventFilter makes it easy to listen for paint events on the other widget and update accordingly. And to make it a fully featured class I added the ability to look for child widget paint events and made it repaint only as little as needed.

The result is this code:

Video

Today I wanted to see whether this eye-candy can be put into KDE at some place and I have not succeeded. Perhaps somebody can make a mockup of how this could be used without making you feel "something is wrong".

Anyway more pictures:

Notice that all this is just fooling around with what Qt provides. This is neither useful nor going into KDE 4.0

For the interested people, the implementation of WidgetMirror looks like this:

After playing with the mixer API a bit I feel confident to be able to go forward a propose a design.

class diagram

I'm happy for people to look over it. Don't think about implementation details but whether this API would be easy to understand, use and still powerful enough for all our desktop uses.

After a phonon/phonon-xine debug session (where I got stuck at KIO playback showing inexplicable memory access violations inside xine-lib) I had to do something creative again. So I tackled the mixer.

First I started with trying out ALSA's softvol plugin, and whether that would have less latency than the software volume control inside xine-lib. As it's close to impossible to measure I have to rely on my senses and that's telling me there's no audible difference. Ok, so for AudioOutput's volume property I could stay with xine-lib's offering, but seeing softvol work for the first time that got me started on exploring the ALSA mixer.

When an application uses an ALSA "device" that contains the softvol plugin then this will add a new "integer control" to the mixer. KMix and also alsamixer do not show the new control without a restart, and the control does not disappear again when the application is done and no app uses the softvol anymore.

From there I went to look at kmix (trunk)'s abilities wrt. custom layout definitions and whether it would be possible to group KDE made softvol controls in a sensible way. This was not getting where I wanted without touching the code so I finally started libphononmixer.

When you want to use libasound (ALSA's user space lib) to fulfill a task it is, IMHO, best to not go from the doxygen generated documentation (which partially on their website is empty anyway) but open an editor in the libasound code and look at the header and source files. As the doxygen comments often don't tell much more than the function and parameter names already do the code is really the best to go with. And now I have a Mixer class that can be constructed from a Phonon::AudioOutputDevice object, a QString or an enum value. This installs a QSocketNotifier on the poll_descriptors libasound gives me and reads out all the events when the filedescriptor is ready for reading. With this I get all change notifications of a given mixer, even when a softvol control is added.

From there I'll go and make a nice high-level mixer API that lists all available mixers, gives notifications when a mixer appears/disappears, lists all the elements of a mixer, makes those elements give value change notifications and so on. I'll look into the possibility of removing softvol controls from the mixer, and whether it's possible to identify their origin and group them correctly. Once this is all ready it'll be the perfect base for a Plasma Data Engine and then a Plasma Applet for controlling your sound.

Personal Note: I'm close to being done with my sickness and probably today I'll go back home (where I'm alone because my wife is far away in Spain ). I stayed with my parents for the time I was sick. It's great to have them, not only as fallback solution, but in those cases it becomes so much more obvious just how much a great family is worth!

It's been a month already since I was at the Trolltech office in Oslo, but ever since I wanted to write about it...

Trolltech invited me to be able to discuss the design of Phonon or, more in general, what their needs are for a multimedia API that they want to provide with or in addition to Qt. The first three days (Wed.-Fri.) we've been whiteboarding the whole day, which was quite exhausting but OTOH exactly what I always wished to be able to do when I was designing Phonon in the beginning. The outcome was that we agreed that we want to have an API that provides all the flexibility of a graph based approach while simplifying the concepts as good as possible so that non-experts have no trouble using the API for simple tasks without getting any details wrong.

Before

This is where we needed a small change in the Phonon frontend API to make this possible. Before August Phonon allowed "graphs" of the form:

Path changes

Now the AudioPath and VideoPath objects are replaced by a value based class called Path, which is hidden for simple cases: vs. or in code:

I wanted to keep the Path concept to make it easier to support the most common "graph reconnection": to insert/remove effects. With the common graph you'd have to disconnect media and output, connect media to effect, and effect to output and also somehow tell the objects that they should not lose any signal data while they're still disconnected and that playback needs to continue seemlessly after the effect insertion. With the path object it's a simple

Mixing

Another change is that is not allowed anymore. The implicit mixing of audio data is removed. (For video it was never allowed anyway). The plan is to add a special class that handles mixing after the 4.0 release. Implementation note: This is close to impossible to implement using libxine.

What's the gain?

Perhaps you've already seen it, but why is this now more flexible than what Phonon provided before? The answer is that you can now define arbitrary graphs like . In a later version we also want to have an object that can "contain" multiple MediaObjects in order to be able to control them in sync. This is basically the concept of a media graph containing multiple source nodes. But if you have only one source, which is the case for most apps, the MediaObject defines the graph implicitly. This hides the graph concept for most developers using Phonon, which IMO helps to make Phonon easier to use.

Trolltech

Now what's Trolltech's role with regard to Phonon? They want to provide multimedia API with Qt 4.4. And they decided to set their bets on Phonon. This means they will work (with me and the rest of the community that is interested) on continuing the design of the frontend API to support use cases like capture, transcoding, VoIP apps, Video chat, ... And they will work on backends for Windows, MacOS X and Linux (including embedded). KDE will ship the xine backend with 4.0, but that one has several problems:

• libxine is GPL-only, so they cannot tell commercial customers to simply use Phonon on Linux for multimedia
• libxine does not support mixing of audio streams, and I tried to separate the clock and ticket system in libxine to make it more flexible, but I feel like this is a dead end. libxine is designed for simple playback tasks and that's it.
• no capture or encoding with libxine

On another note, 2h per workday, while I'm working on Phonon, are now sponsored by Trolltech. This additional money is very welcome, especially now that we'll have to pay the rent for a flat in Heidelberg and another one in Sevilla.

NMM Release 1.0.0 "April"

I'll be taking a closer look at this release the next weeks and will be more actively looking into the NMM backend again.

Today I installed PowerTOP to check out how energy efficient my system is working. And one app that showed bad numbers was knotify. knotify itself isn't doing anything wrong, except that it's using aRts...

So I looked into the aRts sources and very soon found the QTimer that was firing every 50ms == 20 wakeups per second. The reason for this timer was to handle notifications from "the bottom of the event loop". But really, a timer was overkill as calling NotificationManager::the()->run() on demand would be enough. So I un-inlined the NotificationManager::send() method (which is almost exclusively used internally) to tell the IOManager of pending notifications. But IOManager is an abstract class and I needed a new virtual function. To keep BC I reused an unrelated virtual function for my purpose (magic values in the arguments tell the implementation what to do), and now knotify does not show up in the list of power hungry applications in PowerTOP anymore.

Please try the patch and tell me whether it works for you.

Two weeks have passed since I was in Norway, but I wanted to let you all know about it...

Trolltech sponsored and organized a meeting in their offices in Oslo to review the Solid and Phonon APIs and to get Trolltech on board to look into support for other platforms. It was a week of very fruitful discussions and hacking.

The typical day looked like this:
Get up at 7-8 (or later if you couldn't finish hacking the night before ) and go for the best breakfast in town. Then walk to the office (just a few minutes walk) and start hacking until the office fills with all the right people that want to do Phonon/Solid API reviews. We met in the room labeled "Learning" where we were seated in comfortable bean bags or sofas looking at the header files projected onto the whiteboard. And then everybody was free to ask questions:

• Why is that function there? Why not make it private?
• Why is this called AbstractSomething if the user of the API is not meant to reimplement the class?
• Is this class necessary?
• Can we have a better name for "..."? Are there similar methods in Qt so that we can keep the naming consistent?
• What applications will use this? Only one or two?

Anyway this kind of API review was really good for Phonon and Solid and more APIs could benefit from such work. Too bad that we don't have the time to do it for all of kdelibs

The most invasive change in Phonon was the removal of all MediaProducer classes except MediaObject in favor of an additional class MediaSource. So instead of deciding whether you need a MediaObject, MediaQueue, ByteStream or AvCapture object you now take a MediaObject and call setCurrentSource(MediaSource). MediaSource is just a simple class with implicit constructors so you'll be able to write:

I am about to get rid of the core/ui split of the Phonon libraries. As you can see, a GUI application that only uses functionality from phononcore does not really benefit from the split:

objdump -T libphononcore.so|grep "\.text"|wc -l
617
objdump -T libphononui.so|grep "\.text"|wc -l
127

You might argue about the non-GUI apps that want to use Phonon. But seriously, what apps are those?

If you want to, you can still try to convince me that the split adds value. But you have to make some really good arguments to justify the added complexity on the frontend and backend sides and the added exported symbols that are necessary for it.