Continue reading »]]> You’ve seen the news: the first release candidate for Qt 5.0 has just been released. .And if you haven’t, you can go download it from http://qt-project.org/downloads. I’d like to first of all congratulate everyone involved in getting it out, with a special nod to the release team. Thanks for all the work!

But I’d like to talk about what will exactly happen in the next couple of weeks from now until the 5.0 final.

If you’re familiar with previous Qt releases, you may have noticed that our release candidates weren’t really release candidates. In particular, the release plan called for exactly one release candidate and we knew the time between that release and the final. We also knew that the release candidate wasn’t really a candidate because there were still changes we needed to make. Finally, after those changes were in — and some of which were quite significant — we released the final directly, without a second release candidate.

No more. This time, we’re doing it the right way. More importantly, it’s all done in the proper, Open Source and Openly Governed way.

The Qt 5.0 Release Team is composed of people from many different companies, not just one, testing many different platforms. We’ve released the alpha, the two betas and this release candidate as a group. Sure, there have been growth pains, especially in the Beta 1 release, but those have mostly been ironed out now.

The last mistake we fixed was one that I had unfortunately caused: in order to support the Tier specification for the 5.0 release, I had required that packages produced by the Qt Project be tested for 48 hours before they could be released. The idea was to make sure that everyone got the chance to participate in the release process and especially the opportunity to find and fix bugs on their platforms before the release went out. And that’s what we were doing for the Release Candidate.

Then it dawned on me: that IS the release candidate model. In other words, we were doing release candidate release candidates!

After a discussion on Monday’s release team meeting, we agreed to drop that indirection and just do regular release candidates. Here’s what we’ll do on releases from now on:

1. Release team prepares a package set;
2. Release team does a sanity check on the packages:
   • did the build succeed?
   • do the packages include the latest / correct commits?
   • do the installers work?
   • etc.
3. If the sanity check went ok, the release team releases this package set as a new release candidate;
4. For a week, we’ll collect bug reports and other issues, as well as fixes;
5. Then a subjective decision needs to be made: do we have outstanding showstopper issues? Were there any important changes that require wide testing?
   • If so, go back to the first step and let’s do a new Release Candidate;
   • If not, repeat the packaging and sanity-checking steps but for the Final.

We released RC1 today and I can bet we will find issues that require intrusive fixes. That means you should expect an RC2 package in a week or a week and a half. Hopefully, that RC2 should be the last we need, though.

So, please, get RC1 now, test it and let us know. There’s usually a lot of helpful people in the #qt-labs IRC channel on Freenode, especially during European daytime. And any issues you find that could potentially be showstoppers, file them in our bug tracking system at http://bugreports.qt-project.org.

Let’s try and get the final out by the end of the year.

Continue reading »]]> Lars Knoll, Qt Project’s Chief Maintainer, blogs to let us know that Qt 5.0 beta 1 is out. Hurray! Go forth and download.

You can dowload the source code and binaries from the official Qt Project CDN. The tarballs for building on Unix systems like Linux are in the split_sources subdir — Linux distribution packagers will want to use them. Once those packages exist for distributions, they will be listed in the Qt 5 unofficial builds page.

This is the first Qt 5 release to also include installable binaries. Windows and Mac users have had them for ages in Qt 4, and Linux users have enjoyed them in the past in the form of the Qt SDK binary installers. As far as I can remember, this is the first pure Qt library release to contain Linux installers.

But, as the nature of the beast goes, those installers are known to work only on Ubuntu distributions. The main reason for that is because Qt 5 depends on the ICU libraries, whose developers went to the “OpenSSL School of Releasing” (along with the Boost developers) and haven’t learned yet to make binary-compatible releases. Sorry about that. If you don’t have the build capacity to compile the sources yourself, you may want to wait until packaged, binary builds show up for your distribution (in the form of RPMs and DEBs).

The goal of the beta, as I explained in a previous blog post release is to gather feedback on the implementation and to get bug reports. From this point on, the Qt 5 API is “soft-frozen”, meaning that it will not change incompatibly any more, except to fix major issues that we encounter or we’re told about in the form of feedback. If that happens, we’ll make sure to make a note of it in the release notes.

That means that Qt 5.0 beta1 is a suitable starting point for porting applications and writing new code. Your work will not be wasted. But you might run into bugs, so please report them to us, in the Qt Project Task Tracker. We’re also very interested in bugs related to packaging, building, the installers, documentation, etc. Just be sure to look first at the Known Issues page before reporting anything.

Continue reading »]]> Last week, I wrote three blogs about the situation with starting child processes on Unix and being notified of their exit. I raised several problems with the current implementation, which I have tried to solve and I have now a proposal for. If you haven’t yet, you should take some time to read the previous three blogs:

• Part 1: Launching processes on Unix;
• Part 2: http://www.macieira.org/blog/2012/07/forkfd-part-2-finding-out-that-a-child-process-exited-on-unix/;
• Part 3: QProcess’s requirements and current solution;

The road so far

I explained in the first blog how one launches processes on Unix, by way of the fork and execve system calls, and the problem associated with file descriptors being inherited without closing in the child processes. I also showed how Linux has solved this problem. Since no other Unix system has yet done the same, they are excluded from going forward. They need to be brought into the 2010s first and leave the 1970s behind.

In the second blog, I went over the contortions required to be notified that a child process has exited, which uses the SIGCHLD signal. Designed in the early Unix times, signal handlers have conceptual problems with two modern requirements: libraries and multi-threading. And in the third blog, I explained the requirements that QProcess presents and how Qt has tried so far to solve those problems.

Unfortunately, there are two issues that can’t be solved. One is the race condition involved in the installation of the SIGCHLD signal handler, and the other is its uninstallation when the library is being unloaded. With the current API, unless I missed something, it’s not possible to do this cleanly. That leads me to the conclusion that signal handlers should really have been left in the 1970s.

The solution I propose

The solution I’d like to see implemented requires another change to the Linux kernel. Attentive readers may have guessed what I want by the title of the blog: I want a new system call named forkfd. Similar to additions to Linux like the signalfd, timerfd_create and eventfd, this would be a function that opens a new file descriptor.

Its man page would be something like the following:

NAME

forkfd - create a child process and a file descriptor for being notified of its exit

SYNOPSIS

int forkfd(int flags, pid_t *pid);

DESCRIPTION

forkfd() creates a file descriptor that can be used to be notified of when a child process exits. This file descriptor can be monitored using select(2), poll(2) or similar mechanisms.

The flags parameter can contain the following values ORed to change the behaviour of forkfd():

FFD_NONBLOCK

Set the O_NONBLOCK file status flag on the new open file descriptor. Using this flag saves extra calls to fnctl(2) to achieve the same result.

FFD_CLOEXEC

Set the close-on-exec (FD_CLOEXEC) flag on the new file descriptor. This flag applies to the parent process side of the fork and new processes created after that. The child process created by forkfd() does not have this file descriptor open.

The file descriptor returned by forkfd() supports the following operations:

read(2)

When the child process exits, then the buffer supplied to read(2) is used to return information about the status of the child in the form of one siginfo_t structure. The buffer must be at least sizeof(siginfo_t) bytes. The return value of read(2) is the total number of bytes read.

poll(2), select(2) (and similar)

The file descriptor is readable (the select(2) readfds argument; the poll(2) POLLIN flag) if the child has exited or signalled via SIGCHLD.

close(2)

When the file descriptor is no longer required it should be closed.

RETURN VALUE

On success, in the parent process forkfd() returns a new forkfd file descriptor and sets the PID of the child process to *pid; in the child process, it returns FFD_CHILD_PROCESS and sets *pid to zero. On error, -1 is returned and errno is set to indicate the error, with no process being created.

This solution has the following benefits:

• No signal handler installation or uninstallation is necessary, which avoids both outstanding unfixable issues;
• If no signal handler is needed, there is no need to start a thread for managing the child process status;
• Notification is sent via a read notification on a file descriptor, which all event-driven applications know how to handle, plus it matches the requirements that QProcess has in its own waitFor functions;
• The child process is automatically reaped by the read() call, which avoids the need to call wait or waitpid.

Implementing it in userland

I tried to implement the above function in userland, in pure C using only POSIX calls. The idea was that this code could be used in many different libraries to solve their process-management problems. I came up with three implementations:

Using pthreads

Source code: header, source

The first attempt was a direct rewrite of the QProcess solution in C, using only POSIX calls. The code has a global pthread_mutex_t that protects a doubly-linked list of currently-running processes. It installs the SIGCHLD handler under a mutex lock, creates a pipe, and forks. In the child side of the fork, it closes the pipe and returns the magic constants. In the parent side, it adds the writing end of the pipe and the PID to the list, and returns the reading end.

Since I wrote this code before I realised the fatal flaw with SA_SIGINFO, this code is still using it. It writes the siginfo_t structure received in the signal handler to the process manager, by way of a private pipe. That one, in turn, will read the PID from the structure and proceed to write the structure again to the user, via the writing end of the pipe that was saved in the forkfd() call.

This code is fixable, by making the signal handler write one byte to the pipe (or use eventfd) and have the process manager thread loop over the currently-known child processes, calling waitpid on each and synthesising siginfo_t for the user.

Lock-free

Source code: same header, source code

The problem with the first implementation, besides relying on SA_SIGINFO, is that it requires pthreads and mutexes. I wanted something lock-free, so I started writing that. I wrote a lock-free structure to replace the doubly-linked list of PID and writing pipe pairs, based on previous experience with Qt’s lock-free timer ID allocator. I haven’t done exhaustive testing on it, but it’s simple enough that it’s probably correct (pending further reviews).

This solution is, unfortunately, still based on SA_SIGINFO (why? because I hadn’t realised it was a problem by then; I only did so when writing the blog). The way it works is that the signal handler will read from this structure and figure out, based on the PID that from siginfo_t, what the file descriptor to write is. The signal handler also does the necessary waitpid call to reap the child process.

Unfortunately, this solution doesn’t work, since it introduces several race conditions. One is an extremely rare situation, in which the library with this code is being unloaded in one thread while the signal handler is still running in another. More importantly, though, there’s a race condition between the time of the fork and the addition of the PID to the list of children. This condition didn’t happen before because of the mutex: the process manager thread would not read from the list until the forkfd function released the lock, after adding the child process.

This implementation is still salvageable, though. First, it needs to stop relying on SA_SIGINFO, which means it must iterate over all the known children inside the signal handler, doing waitpid calls on each. Second, with the absence of a lock, it must prevent the child process from exiting before its PID and pipe are added to the list. That can be done by adding an extra, blocking pipe between the parent and child process: the child process tries to read() from it, suspending itself, until the parent process releases it by writing something.

Adding a spin lock

Source code: same header, source code

The solution I ended up writing to the race conditions of the previous implementation was to add a spin lock (why this and not the pipe lock I described above? Because it hadn’t occurred to me until just now). It’s a step back from the fully lock-free solution, but not all the way back to the pthreads implementation. For one thing, it doesn’t start a thread for the process management. For another, since it implements the spinlock on its own, it can lock inside the signal handler (note that pthread_mutex_lock is not a permitted function inside one).

I just had to be careful about one thing: before locking the spin lock, the calling thread must block SIGCHLD using pthread_sigmask. If it didn’t do that, the signal handler could be called asynchronously in the same thread as the one where the spin lock is locked, producing a deadlock.

Choosing a solution

To be honest, none of the three solutions are the ideal ones. If I had to choose between one of them, I’d go for the lock-free one for personal reasons, but the spin-lock one might have fewer bugs in the threading code.

But that’s not what I want. What I really want is that forkfd be implemented in the kernel, so that no signal handler is involved, eliminating the unsolvable problems that those introduce.

If there are any kernel hackers listening in, do you think there’s a chance?

Continue reading »]]> In the previous posts onmy series of blogs about starting and managing sub-processes on Unix, I talked about how it’s implemented and how the current solutions have limitations. On this post, I’ll show how QtCore has solved the problem (to the extent that it can be solved) and what requirements a new solution must fulfill.

Links to the previous posts:

• Part 1: Launching processes on Unix
• Part 2: Finding out that a child process exited on Unix

QProcess’s API

The QProcess class is a very powerful and flexible class. Its API in Qt 4 and Qt 5 is the same, an evolution from the Qt 3 API. Like other Qt I/O classes, QProcess’s API is entirely asynchronous, with functions to make it work under synchronous circumstances.

For those not familiar with Qt’s API design, Qt classes have named callbacks called “signals” (that have nothing to do Unix signals, except the name) that identify a particular change or event that has happened. QProcess has four important signals:

• started
• readyRead (inherited from QIODevice)
• bytesWritten (inherited too)
• finished

Each of these represent one activity or state change in the sub-process. Those signals are emitted by the QProcess object from inside the main application’s event loop, allowing the application code to be entirely asynchronous and simply react to the changes as needed.

In addition, each of those four signals is paired with a synchronous function whose name is waitFor followed by the name of the signal. As the name indicates, the function blocks until either a timeout expires or until that particular signal is emitted. The waitFor functions the application code to be synchronous (blocking) where it needs to.

The first of those four signals, started, is quite simple. It may seem weird at first glance to have a signal indicating that the process has started. After all, shouldn’t the start function return an error condition instead? In fact, it exists because we didn’t want to put any requirements in the OS scheduler: the parent process could continue executing for some time before the scheduler decided to let the child execute and, eventually, execve.

I’m not going to go into the details of how to determine whether the sub-process successfully execve'd. It’s not relevant to our story: in the first blog, I explained how starting a process is a done deal and works just fine.

In turn, the fourth of those signals, the finished signal, is the interesting one for us. The other two are relevant, though, for one particular reason: the parent process does not know what the child process will do next. The child process could write something to its stdout, it could consume its stdin, or it could exit, crash, core dump, or otherwise disappear. That means we need to monitor at all times up to three pipes (stdin, stdout and stderr) as well as whatever mechanism we’re using to be notified that the process has exited. That’s the first of our requirements.

The requirements

The first requirement, as I’ve just explained, is that the main application event loop be able to monitor up to three pipes and the child process’s exit mechanism (whichever we make it). Not only that, the three waitFor functions that pair with runtime signals — that is, waitForReadyRead, waitForBytesWritten, and waitForFinished — need to do the same. In both cases, the requirement boils down to the same: whatever the exit notification mechanism is, it needs to be accessible from a select or poll or it needs to interrupt such a call.

The second requirement is that this mechanism scale for multiple child processes simultaneously. One event loop needs to be able to monitor multiple pipes and the exit notification from multiple processes. Though by the design of the API, this requirement does not apply to the waitFor functions.

The third requirement is that this needs to work in a multithreaded environment. That is, multiple event loops or multiple waitFor functions might be running simultaneously.

And finally, the fourth requirement is that, if we write a UNIX signal handler, we obey the requirements for signal handlers.

The Qt solution

Let’s build it step by step. As I’ve shown in the previous blogs, the current state of the art solution is to install a signal handler. Yes, it has an unfixed and non-workaroundable race condition during the installation, and it has an unfixable problem with uninstallation of the handler. Those issues are out of our control, though of course the point of this series of blogs is to try and solve them, or propose a solution that avoids them completely.

The current solution is also something that exists today and has existed for over 8 years. That means it does not use Linux’s signalfd solution. I’ll explore that possibility later, when we discuss how to improve the current solution.

The signal handler that Qt installs is very similar to the code block I had in part two: it “does something”, then it chains to the previous handler. What is that “something”?

When our SIGCHLD handler is called, we don’t know which child process has exited because we didn’t set the SA_SIGINFO flag (again, more on that later). Since we don’t have the PID of the child process, the only way of getting it is by doing a wait or waitpid call. But as I discussed in the first part, we can’t do a “wait for any child” in Qt, because it could interfere with the operations of other libraries like GLib. Since we can’t be told which child has exited, the only solution we have is to check all of our children to see if any has exited.

Enter the fourth requirement: we can only use functions that are explicitly listed in the list of POSIX signal-safe functions. That excludes locking any mutex: remember that the signal handler could be called in any thread, including a thread that has a particular mutex currently locked.

The Qt solution is to write a single byte on a pipe that ends in a process manager. That way, we exit from the signal handler and can continue operating from the a regular context, where we can lock mutexes and get a list of all children currently managed by QProcess. What happens next is that Qt launches a storm of waitpid calls, one for each child process, with the WNOHANG flag set. We’re hoping that this will yield one child having exited, but it could be more than one and it could also be zero (e.g., a child process that was managed by GLib).

But wait, there’s the third requirement to consider: all the user threads, including the main thread, could be blocked doing something or other. Therefore, this process manager needs to be run in either a thread that is dedicated, is not blocked, or is currently in one of the QProcess::waitFor functions. The Qt solution is the first one: there’s a dedicated thread. This solution makes sense if you consider that, otherwise, all Qt-based event loops would need to add the reading end of the SIGCHLD handler’s pipe to their select or poll list, in which case all of those threads might be woken up by the activity, even if only one can do the work.

If this process manager finds out a child process that has exited, it needs to notify the thread that will emit the finished signal. It does that by writing to a pipe, whose reading end is in the source list for select or poll.

Actually, I embellished the Qt solution a little here. The process manager doesn’t do the waitpid call. It simply writes a byte to all QProcess’s pipes and then lets each QProcess object call waitpid. There’s an obvious improvement opportunity here — though of course, that requires that the process manager communicate what the exit status of the child process was.

Improving by using SA_SIGINFO

Turns out that this is exactly what set me upon this path. A few days ago, I was talking to some colleagues on our internal IRC channel when the subject of signals and SA_SIGINFO came up. It occurred to me that the OS does tell us which child process exited, as part of an extra parameter to the signal handler.

With that information in hand, we don’t have to call waitpid on each and every one of our child processes to figure out which one has exited. In fact, we know from the start which one it is, even if it’s not one of ours. I actually wrote a patch to do this and you can see it in Qt’s code review tool (it’s not approved yet at the time of this writing).

I modified the existing code by writing not a simple byte from the SIGCHLD handler to the process manager, but the 4 bytes of the pid_t type containing the PID of the child that exited. Then the process manager simply needs to search for that PID and notify the exact QProcess object whose child exited.

The improvement is clear: as a response to a SIGCHLD being delivered, the application executes exactly one waitpid call.

What’s the problem with this? Well, two of them. First, what happens if the actual signal handler for SIGCHLD was not ours, but something like I wrote on the second blog:

static struct sigaction old_sigaction;
static void sigchld_handler(int signum)
{
    /* my code goes here */
 
    if (old_sigaction.sa_handler != SIG_IGN
            && old_sigaction.sa_handler != SIG_DFL)
        old_sigaction.sa_handler(signum);
}

Do you see what will happen when our signal handler tries to dereference the second parameter, of type siginfo_t *? Crash.

The other problem is more serious: what happens if a second child process exits while our signal handler is running? POSIX requires that this second signal be queued, so our handler is executed again. And what happens if a third child exits too? Well… in that case we’re toast: the third SIGCHLD simply gets dropped.

That’s pretty much a showstopper. Coupled with the comments I’ve received in the review tool and on IRC, which pointed me to an issue with the pipe buffer being full causing either an unhandled EWOULDBLOCK error or a possible deadlock, it indicates to me that I must abandon this solution.

Improving by using signalfd

Since we’re talking about Linux only since the beginning of the second blog, we could use Linux’s signalfd mechanism and avoid a signal handler altogether. Here are some implementation possibilities and the problems with them:

1. We could have one distinct signalfd on SIGCHLD per thread, but I don’t think it’s determined which of the signalfd get woken up by the delivery of the signal. If the answer isn’t “all of them”, this won’t work for us and I’m sure the answer isn’t that.
2. We could have the same signalfd in all threads, but then we run into the problem of “which select gets woken up” by the activity. What’s more, we don’t know if GLib doesn’t have its own signalfd installed, causing the problems from #1 above.
3. We could have a dedicated “process manager” thread that is the only one listening for the signal. However, this solution is hardly different from the time-tried signal handler. Moreover, it also falls short in the multiple-library-criterion as #2.

Add to that the fact that a signalfd is only delivered if all threads in the process have blocked the signal. If even one of them has it unblocked, the signal will be delivered to a signal handler in that thread, bypassing the signalfd completely. We can’t be sure that the user hasn’t created a thread and cleared the signal blocking mask.

No matter what we do, any solution in userspace will require a signal handler and will run into the unsolved problems from the second blog. In the next issue, I’ll explore what solutions I’m proposing, both in userspace and in kernel space.

Continue reading »]]> On my previous blog, I said that the solutions we’ve got implemented on Linux are a good start, but not the full solution. We can start a child process properly, but we still can’t properly find out when it exited.

Linux or nothing

First of all, let me get one thing straight: from this point on, I’ll only be thinking about Linux. If you’re running anything else, I don’t care about you. But you may continue reading anyway.

I have a couple of reasons for doing that, one of them being that it’s the easiest to get the new API I’m proposing accepted. It’s probably just as easy to get the other open source BSDs to do the same, but not so much on the commercial Unixes, which would involve a long lead time, product management, NDAs, etc.

But the most important reason is that any of those other Unixes are years behind Linux on the state of the art. If your OS hasn’t done its homework for the past 4 years and introduced the API I mentioned in the previous blog, then I don’t care about that OS.

More to the point: the problem I am trying to solve is related to multithreading and the race conditions that are involved in such a scenario. Without those APIs, there’s no possibility of thread-safety anyway.

How to be notified of a child process exiting

There are two ways of being notified that a child process has exited: a blocking (synchronous) and a non-blocking (asynchronous) method. The blocking one is fairly simple: a call to waitpid without the WNOHANG flag (i.e., “do hang”). On Linux, the waitpid call is backed by the kernel system call of the same name, so we know the kernel is doing the right thing.

The problem with the blocking API is that, as it turns out, it’s blocking. It’s unsuitable to be run in the same thread that is handling user interaction and painting in a GUI application. If you want to use it, you need to start a thread for it. To make matters worse, to implement this in a generic-purpose library like Qt or Glib, you’d need to start one thread per child process.

The waitpid function can be used to wait in one of three conditions: one specific child process given by its PID, one specific process group given its PGID, or all child processes. The process group idea looks interesting at first, since each library could create one such group and move all the child processes it cares about into it. However, process groups have other purposes and side-effects, including the fact that the child process can change its session ID and process group, which exclude them from a generic solution. And clearly waiting for any and all child processes is not acceptable for a generic library, for it cannot know whether there are processes started outside of its control, like when both Qt and Glib are being used.

Problem two: chaining signal handlers

That leaves us with the asynchronous method of being notified of a child process exiting, which is done via POSIX signals. More specifically, by the delivery of the SIGCHLD (also spelt SIGCLD). And here we run into a series of problems that aren’t solved today.

The first of them is that there’s no thread-safe way of installing a signal handler in a generic library. The system call signal can be used to install a signal handler. This function is fine if the handler is installed by the application developer, like the case of handling SIGINT (the signal that Ctrl+C sends) or SIGTERM and performing some clean-ups.

But it’s not acceptable for a generic library. Again let’s take the case of an application using both Qt and Glib, either directly or indirectly. Since both libraries need to install a signal handler, it stands to reason that one handler must somehow call the other to let it do its work. That means signal is out.

Fortunately, there’s sigaction and this system call not only installs a new handler, it also returns the old handler too, so one handler can call the other. There’s a multithreading problem here, but let’s look into that later. The code to install the handler would look something like this:

static struct sigaction old_sigaction;
static void sigchld_handler(int signum)
{
    /* my code goes here */
 
    if (old_sigaction.sa_handler != SIG_IGN
            && old_sigaction.sa_handler != SIG_DFL)
        old_sigaction.sa_handler(signum);
}

Installation can be achieved with this API, but how about uninstallation? That’s where the unsolved problem lies. The first solution that comes to mind, a gross one and the simplest possible, is to ignore uninstallation and simply decree that the handler will remain there until the process exits completely.

That brute-force solution breaks down the moment we introduce unloading of libraries. Now, you may remember my saying that library and plugin unloading are a bad idea and should be avoided, because they create a whole number of problems. Yes, that’s true, and this is one of them. So I do recommend that you avoid unloading libraries and plugins in your applications. However, we’re looking for a generic solution here, so we must take unloading into account.

During the library unloading process, the signal handler must be uninstalled. The way to uninstall it is to install something else. But what? The options that come to mind are:

1. install the default signal handler, SIG_DFL; or
2. ignore the signal, by installing SIG_IGN; or
3. install the previous signal handler, the one we saved from the sigaction call.

Unless you’re trying to be funny or you see some other problem that I don’t, you’ll suggest the third option, right? Therefore, we uninstall our handler like this:

    sigaction(SIGCHLD, NULL, old_sigaction);

Can you see the problem?

What happens if our handler is not the currently-installed handler? That could happen if the de-initialisation order is different from the initialisation one. As a concrete example, imagine an application that uses Qt, so QtCore got loaded at process start and will not be unloaded until the process exit. Then the application does this, in order:

1. it loads a plugin that uses Glib;
2. the plugin uses the g_spawn_async function, causing Glib to install its SIGCHLD handler;
3. the application uses QProcess, causing Qt to install its SIGCHLD handler;
4. the Glib-using plugin is unloaded and Glib tries to uninstall its handler.

At this point, Glib will uninstall Qt’s handler too, rendering QProcess unusable. The current code in qprocess_unix.cpp tries to work around this problem by doing:

    struct sigaction currentAction;
    ::sigaction(SIGCHLD, 0, &currentAction);
    if (currentAction.sa_handler == qt_sa_sigchld_handler) {
        ::sigaction(SIGCHLD, &qt_sa_old_sigchld_handler, 0);
    }

Let’s ignore for a moment the fact that the above code is neither thread-safe nor async signal-safe. Another thread could be trying to install a handler at the same time, and a SIGCHLD could be delivered in-between the two calls to sigaction. Let’s ignore it because we have a bigger problem: if Qt’s handler isn’t the topmost handler installed, Qt is forced to leave its handler installed. And if QtCore is about to be unloaded, there’s a very big chance that the next SIGCHLD delivery will crash the application!

Problem three: thread-safety in sigaction

This problem exists not because of the API, but because of the implementation. I am assuming that the kernel side of sigaction is correctly implemented and it will do its proper locks in case two threads of the same process try to install signal handlers for the same signal at the same time. Let’s also assume that the kernel does not allow a signal to be delivered to the process while it is modifying its own structures of the signal handlers.

The problem exists in the userland because glibc’s struct sigaction is different from the kernel’s ABI. That forces glibc to allocate a local (stack-based) structure so its address can be passed to the system call. Upon return, it needs to copy the contents into our old_sigaction variable.

Do you see the problem? There’s a race condition there.

A signal could be delivered to the process after the kernel returned to user-space, but before the glibc code could copy the contents to our variable. That means our newly-installed signal handler could be called before the old_sigaction was filled in. That would mean the old handler would not get called as it should be. And chances are that the signal being delivered wasn’t meant to our handler anyway — after all, you’d have problems in your code if you could receive your own SIGCHLD before your handler were ready.

In reality, it is actually worse than the above description: since the struct sigaction structure is not filled in atomically, our signal handler could see a partially-filled structure. And in any case, since glibc’s code allocates it on the stack and does not pre-fill it with zeroes before the system call, if the handler is called with the chain link not completely filled, there’s a good chance that the chain call will end up in a garbage address.

Propsed solution for problem three

The solution for this problem is simple: glibc must not memcpy. That means the userspace struct sigaction must be equal to the kernel’s. And therein lies another problem: to change that structure now, we’d have to break the C library ABI. It can be done with ELF versioning, but it does not remove an ABI change, which could turn up again if there were code that shared struct sigaction across libraries.

However, there’s no solution that I can see for problem two and it’s a serious issue. On the next blog, I’ll explore the solution that Qt has used for a few years and the problems with it.

Continue reading »]]> Have you ever tried to launch a sub-process on Unix? POSIX.1 has several APIs for doing that, including: fork+execve and posix_spawn. Starting a child process is not difficult, but ensuring that they behave properly and you get notified when the child dies, that is difficult.

First, posix_spawn

I want to concentrate only on fork(2)+execve(2), so let’s talk about the new API first. The posix_spawn API was first introduced in POSIX.1-2001, derived from the earlier “1d” specification. If your Unix system is not POSIX.1-2001-compliant, you won’t have it. And even if you have a 2001- or 2008-compliant system, the specification still says today:

APPLICATION USAGE

These functions are part of the Spawn option and need not be provided on all implementations.

So if you want to write cross-platform code, you’re not going to depend on it. (Unless you need to use it for systems that don’t have fork(), like QNX Neutrino).

In any case, for my interest here — Linux — two other factors come into play:

1. The kernel has no posix_spawn API, so the C library would need to implement it in userspace, using fork and execve anyway.
2. Glibc does not implement it today, it simply returns ENOSYS.

With that in mind, let’s focus on the traditional API.

fork and execve

The traditional API for launching a process on Unix systems is to first fork your process and then replace it with another. This two-step process is extremely flexible and has allowed for many uses and abuses over the years. For example, before we had proper thread support, forking and communicating with the child forks was a common way of operating. In fact, even today the extremely popular Apache web server continues to offer a module that handles requests on the time-proven non-threaded fork-based implementation.

When the call to fork succeeds, execution will continue in two different processes: the parent and the child. The parent process receives the child process’s identifier (the PID) for later use, like kill(2) or to distinguish between notifications from multiple children.

Usually, the child process will perform some clean up and preparation before later calling execve. The POSIX API offers several variations of in the exec family, but they all boil down to execve: the path to the executable is absolute, the arguments are in a vector and the environment to be passed down is known.

As I said in the introduction, so far so good. This is easy, flexible and proven by time. Yet it has some problems we’ll explore.

First problem: inheriting file descriptors

The POSIX specification for execve declares:

File descriptors open in the calling process image shall remain open in the new process image, except for those whose close-on-exec flag FD_CLOEXEC is set.

This allows the child process to inherit the standard streams of the C library — stdin, stdout and stderr. When you launch a process from a shell, for example, the process will inherit the connection to the terminal so you’ll get the output on your screen.

This feature is also what allows parent and child process to communicate and for redirections to happen. When you type in the terminal something like:

$ process > output.log

What the shell is doing is making sure that the stdout stream is connected to the file output.log before it calls execve.

Yet the major flaw in this API is that FD_CLOEXEC flag is not the default. That means at every point in that you call a function that opens a file descriptor, you must remember to also make the file descriptor close-on-exec if you don’t want to leak it to the child process.

It was a major flaw in the 1970s when this API was designed, but not catastrophic. With proper care, one could make it work. And if a particular function was going to close the file descriptor anyway before any chance of forking, it did not have to bother.

It became showstopper at the end of the 1990s when we got threads. That means that even careful code that sets the flag immediately upon opening the file descriptor, like the following, is not safe:

    int fd = open("/dev/null", O_WRONLY);
    fcntl(fd, F_SETFD, FD_CLOEXEC);

Why it isn’t safe? Because another thread could call fork in-between the opening of the file descriptor and the setting of the FD_CLOEXEC attribute. That means that, despite the care made in ensuring that the file descriptor doesn’t leak, it can still leak.

First solution: add new APIs

Recently, through the efforts of the former glibc maintainer Ulrich Drepper, we’ve got a few new system calls or modifications to the existing ones on Linux and on glibc that solve the problem above. All of the system calls on the Linux kernel that can create a file descriptor take an extra parameter that indicates whether the FD_CLOEXEC should be set upon creation. The above source code becomes on a relatively recent glibc:

    int fd = open("/dev/null", O_WRONLY | O_CLOEXEC);

And similarly for a call to socket:

    int server_fd = socket(AF_INET, SOCK_STREAM | SOCK_CLOEXEC, IPPROTO_TCP);

For the system calls that created file descriptors but had no way of passing extra flags, a new system call was created, such as:

    int pipe_fd[2];
    pipe2(pipe_fd, O_CLOEXEC);
    dup3(pipe_fd[0], STDIN_FILENO, O_CLOEXEC);
    accept4(server_fd, &addr, &addrlen, SOCK_CLOEXEC);

This also allows us to pass O_NONBLOCK or SOCK_NONBLOCK and save us another pair of system calls to set the flag on.

The solution that Ulrich Drepper and the kernel community came up with is elegant and solves the race condition problem. I also made Qt use those system calls automatically a couple of releases ago and contributed a patch to Glib to do the same.

That part of the problem is solved, on Linux at least, and using a modern glibc or eglibc. Still, it’s not enough, as I’ll explore on my next blog.

Continue reading »]]> Early in the Qt 5 development cycle, we had made the decision to deprecate QPointer and replace it with the more modern QWeakPointer. That decision is now reversed, so please continue using QPointer where you were using them. Moreover, don’t use QWeakPointer except in conjunction with QSharedPointer.

To understand the reason behind this back and forth, we need to go back a little in history.

Understanding QPointer

Almost 3 years ago, I wrote a blog about the smart pointer classes in Qt. In that blog, I talked about how QPointer had some broken semantics and how it should be deprecated. I even mentioned that it was slow.

The reason it is slow in Qt 4 is its implementation. It’s a direct descendant of the Qt 3 QGuardedPtr and Whenever you created a QGuardedPtr for a given object, Qt would simply add the pointer and the guard to a global hashing table. In Qt 4, the implementation is the same, except that now it required locking a global mutex. (For those who don’t remember Qt 3, QObject could not be used outside the main thread there).

That means that creating a QPointer in Qt 4 requires locking a global mutex and inserting an item into a global hash, which may involve rehashing depending on how many items it holds. And when that QObject was destroyed, it would need to iterate over the hash and notify all the watchers that the object had died.

Enter QWeakPointer

With Qt 4.5, I introduced QSharedPointer and QWeakPointer, which are a lot more efficient. The way that those two communicate is by way of a “reference-counted reference-counter”. That is, the private data common to those two classes contains basically two reference counters: the strong and the weak one. The strong counter counts the lifetime of the pointer, where a value of 1 or higher indicates that the object is still alive, whereas a zero indicates that it’s deleted. The weak counter controls the lifetime of the private data itself.

In other words, the strong counter counts how many QSharedPointer objects are attached to this particular pointer, whereas the weak counter counts both QSharedPointer and QWeakPointer instances. So you see how the last QSharedPointer instance being destroyed causes the pointer to be deleted too.

With Qt 4.6, I added a feature that allowed one to use QWeakPointer to track QObject instances directly, without going through QSharedPointer. When trying to figure out how to optimise QPointer, I realised that the “reference-counted reference counter” is the best solution — actually, for tracking without QSharedPointer, a “reference-counted boolean” would be enough, but since we didn’t have QAtomicBool, we regular reference counter would do just fine.

The solution is simple: if you create a the first instance of the guard, it allocates this private block and sets the strong counter to indicate that the object is alive. When the object is deleted, it simply sets the strong counter to zero. The use of the weak counter allows us to share the ownership of this private, so it’s quite fast.

And in Qt 5…

When we started working on Qt 5, it was clear that the previous implementation of QPointer had to go. We don’t want to keep old cruft for another 3-6 years, or however long it’s going to take until we reach Qt 6. But we also promised to maintain source compatibility as much as possible with Qt 4, so we couldn’t just remove the class.

Instead, last November, one developer simply rewrote QPointer on top of QWeakPointer and deprecated it. By keeping the old API and coupling it with the new implementation, we suddenly had a fast QPointer. But the deprecation stayed, because we thought we had “too many smart pointer classes” (see the title of my blog).

In March, when we started to try and clean up the Qt code base of our own deprecated classes, we realised that QPointer is used just about everywhere. Since we had a properly fast implementation, there was actually no good reason to keep generating compiler warnings about the use of that class. That led to QPointer being un-deprecated.

Finally, a month and a half ago, the other shoe dropped. The argument of “too many classes” is not valid if it’s trumping over having proper API and classes that work correctly. What we realised is that overloading QWeakPointer‘s purpose was making its API worse instead, leading to doubts about when to use some of its member functions. Therefore, we deprecated instead QWeakPointer use without QSharedPointer.

Conclusion

That means you should continue using QPointer in your code. Unfortunately, if you’re still targetting Qt 4, it means your code will not be as fast as it could be, but at least it will be clean.

It also means the QWeakPointer::data() member is deprecated and you should not use it.

Qt’s smart pointer classes are neatly grouped now as follows:

Continue reading »]]> Yesterday, one of my contributions to Qt was merged which finally adds better support for optimised raster painting on Windows, with SSE2 and AVX instructions. This feature has long been present on the Unix systems, but it was somewhat lacking on Windows.

If you’ve read my past blogs, you know I often talk about and work on Single Instruction Multiple Data (SIMD) improvements. The idea is quite simple: if you have a lot of identical operations to do and your source data is independent from one another, you can execute all of those operations in parallel, improving the throughput (processors are optimised for loading chunks of memory of a certain size, so if we only use small quantities, we’ve wasted resources). In the past, I’ve mostly worked on SIMD for string operations, like comparison, searching, and conversion to and from Latin-1. That’s sometimes unrewarding because strings are quite small, so we don’t get the full gain of the improved throughput.

But you might not know that SIMD in Qt actually started in the QtGui library, in the raster drawing code. There, the data sizes are often in the order from several kilobytes to multiple megabytes — a tiny 16×16 icon has 256 pixels, each of which is 4 bytes wide, which adds up to 1 kB; you reach 1 MB at 512×512. As you might gather, even copying such data blocks is a somewhat expensive operation. So it’s no wonder that the more common ones of compositing, alpha blending, etc., needed optimisation. And I cannot claim credit for doing them, those were done by very talented hackers working at Trolltech back in the day.

My history with the drawables started about 6 months ago, during the last romjul, when I realised that the optimisations applied to the raster painting code could use some love. Back then, we were still mixing MMX code into the painting code, even when we reported we were using SSE. In fact, when Qt said it was enabling SSE (not SSE2), it was actually just using some new instructions that came with SSE, but on the old MMX technology registers. My first action in that area for Qt 5.0 was to finally remove support for the old MMX-era optimisations, all of which only increased the code size in a Qt build but weren’t used anywhere. The next-level of optimisations (SSE2 and above) overrode the older ones — remember that all 64-bit capable processors have SSE2 support.

Another thing I noticed back then was that we weren’t using the full extent of the optimisations possible. With GCC, we were forced to pass some extra compiler options so GCC would allow us to use some intrinsic functions to execute SSE2 and SSSE3, but that was not the case for the Microsoft compiler. In addition, the Windows configuration did not try to use the intrinsics to verify if they were really available, it simply checked for the presence of the header that usually declares them. What’s more, those checks had not been updated for the SSSE3 optimisations that were done in 2010 in cooperation with Intel, which meant that those optimisations were disabled on Windows.

On Unix, right after removing the old MMX-era code, I proceeded to a very quick and easy gain: add AVX support, the new generation of SIMD instructions from Intel. It was easy because I barely had to write code: if you compile SSE2-era code with GCC’s -msse2avx option (which is automatically enabled by -mavx), it will generate the code using the new AVX instructions. The advantage lies in the fact that the AVX instructions use a new coding mechanism (called the VEX prefix) which specifies an additional register, allowing the compiler to use fewer instructions to accomplish the same goal. Using the expanded 256-bit registers will have to wait for AVX2, coming next year.

Except that even this easy improvement had never come to Windows either. Until now.

To enable Windows support, I had to update the way that the configuration detected the capabilities of the compiler, which is what took most of my time: dealing with building on Windows and with the binary configure.exe is not exactly my forte. Now, like on Unix, the Windows configuration will ask the compiler to try and compile some code. The checks are now shared with Unix, so we have the full range of checks available: SSE2, SSE3, SSSE3, SSE4.1, SSE4.2, AVX, and AVX2. Previously, the only one that remained after I removed the MMX-era checks was SSE2.

Another update I made was to tell the Microsoft compiler to improve code generated. Since it did not require special compiler options to enable its support for SSE2, no one had thought until now to pass it the /arch:SSE2 option. Like on Unix, now we pass this option to the compiler whenever we’re compiling code that uses SSE2 anyway, making the compiler use the extended instruction set for generic code, not just what we wrote with intrisincs. From there, adding support for /arch:AVX was trivial: if you have Microsoft Visual C++ 10.0 or higher (it comes with Visual Studio 2010), you also now get the AVX-era instructions and Qt will enable them at runtime if it detects that your processor has them.

I’m not done. I have also a couple of other quick wins in terms of performance, all by improving code generation. Those changes are a bit more complex than the previous ones and I haven’t cleaned them up properly after 6 months of rebasing. I hope to add them to Qt 5.1 soon after its branch opens.

Continue reading »]]> Every now and again, someone posts on IRC or to a mailing lists about an issue they’ve had and their description of the problem is that “it doesn’t work”. There’s nothing more annoying to the person giving help than to see that description…

That happened to me twice again today, which is what prompted me to write this blog. At this point, I’d like to shamelessly plug in here my work on Lydia Pintscher’s Open Advice book: I wrote chapter 10 in that book, called “The Art of Problem Solving“. And if you haven’t read the book yet, or even skimmed through it, I recommend you do it. It’s full of great advice from experienced people, in many areas related to open source development, contribution, advocacy, or other forms of participation.

The first section of my chapter is called Phrasing the question correctly, where I wrote:

The most useless problem statement that one can face is “it doesn’t work”, yet we seem to get it far too often. It is a true statement, as evidently something is off. Nevertheless, the phrasing does not provide any clue as to where to start looking for answers.

The question is where we start off. In the context of asking for assistance on IRC, mailing list, or forum, it’s supposed to give the help-giver hints as to what is askew, so that they can begin forming theories as to the root causes of the problem and applying problem-solving techniques to confirm or deny it. But note how all the techniques rest on knowing what exactly is wrong.

I’m not saying I expect a full analysis of the situation by the original poster, just as I don’t expect a person who is not a health professional to do the same when going to the doctor’s. But a minimum of information is necessary. Imagine you were to go to the doctor and tell him or her that “I’m feeling ill”. What do you think the doctor will do with that information? So why do people think that “it doesn’t work” is enough information for an engineering help-giver?

At this point, you might say that “it’s just a conversation starter,” a way to break the ice and begin the discussion. And while I might be inclined to agree with you in a social context, in a live face-to-face discussion, I do not when it comes to interaction via the Internet. It’s definitely the case when the communication is not in real-time: if it takes six hours for an answer to come, then the first usable theory won’t reach the poster until 12 hours after the first post.

But even in real-time communications it’s necessary, as more often than not, it’s a matter of attracting attention of the help-givers. If I’m somewhat busy, you cannot expect me to spend precious minutes asking, “so, what exactly happened? what did you expect to happen?”

How would you know what to say in the original post, then? Here are a couple of suggestions, some of which are, I hope, obvious:

• the description of what actually happened;
• the description of what was supposed to happen;
• the actions that you took that led up to the event;
• a description of the environment, such as versions of the relevant programs and settings you changed;
• the logs of any programs involved that might include relevant information;
• if you’ve tested other conditions and whether they’ve failed or succeeded;
• if it’s a recent issue, when it started happening and when the last time you noticed it not to happen was;
• any theories you might have about the issue;
• what you have already done, so far, to fix the issue;
• what sources of information you’ve used to diagnose the problem.

Try to provide as much information as possible, in a concise manner, appropriate to the medium. For example, on IRC, you cannot write a 20-line description of all the theories you may have, but it’s certainly doable to describe the event that led you to think, “hang on, this isn’t right,” and provide a link to further information such as a pastebin of the logs.

Some other quick advice:

• Use your brain! Exercise it, that’s how it develop. Read the logs that you’ve got, especially compiler error logs, and interpret them. Form your own theories and test them if you can, disproving some and proving others.
• Don’t argue with the evidence. If the compiler tells you there’s an error, then you’ve got an error (the exception is when you suspect a compiler bug);
• Do your homework: use Google and other search tools to find out more information. For example, if you’ve got an error message, search for that specific message. If you don’t do this, you may get the answer in the form of a LMGTFY link.
• Use the appropriate channels: asking the wrong audience will not get you closer to the answer, but might raise your frustration, that of your audience and may delay the process.
• Know your tools, know how to use them. It might be acceptable for a newbie, student or hobbyist not to know them, but it’s not for a professional. Not only should you know how to use them, but also a little of how they work.
• The first error is usually the most important one.
• A warning is (often) not an error, but warnings weren’t meant to be ignored.

I’m sure there are more advice that my readers can give. What else would you suggest as advice or as information a help-giver could use?