forkfd part 3: QProcess’s requirements and current solution

In the previous posts onmy series of blogs about starting and managing sub-processes on Unix, I talked about how it’s implemented and how the current solutions have limitations. On this post, I’ll show how QtCore has solved the problem (to the extent that it can be solved) and what requirements a new solution must fulfill.

Links to the previous posts:

• Part 1: Launching processes on Unix
• Part 2: Finding out that a child process exited on Unix

QProcess’s API

The QProcess class is a very powerful and flexible class. Its API in Qt 4 and Qt 5 is the same, an evolution from the Qt 3 API. Like other Qt I/O classes, QProcess’s API is entirely asynchronous, with functions to make it work under synchronous circumstances.

For those not familiar with Qt’s API design, Qt classes have named callbacks called “signals” (that have nothing to do Unix signals, except the name) that identify a particular change or event that has happened. QProcess has four important signals:

• started
• readyRead (inherited from QIODevice)
• bytesWritten (inherited too)
• finished

Each of these represent one activity or state change in the sub-process. Those signals are emitted by the QProcess object from inside the main application’s event loop, allowing the application code to be entirely asynchronous and simply react to the changes as needed.

In addition, each of those four signals is paired with a synchronous function whose name is waitFor followed by the name of the signal. As the name indicates, the function blocks until either a timeout expires or until that particular signal is emitted. The waitFor functions the application code to be synchronous (blocking) where it needs to.

The first of those four signals, started, is quite simple. It may seem weird at first glance to have a signal indicating that the process has started. After all, shouldn’t the start function return an error condition instead? In fact, it exists because we didn’t want to put any requirements in the OS scheduler: the parent process could continue executing for some time before the scheduler decided to let the child execute and, eventually, execve.

I’m not going to go into the details of how to determine whether the sub-process successfully execve'd. It’s not relevant to our story: in the first blog, I explained how starting a process is a done deal and works just fine.

In turn, the fourth of those signals, the finished signal, is the interesting one for us. The other two are relevant, though, for one particular reason: the parent process does not know what the child process will do next. The child process could write something to its stdout, it could consume its stdin, or it could exit, crash, core dump, or otherwise disappear. That means we need to monitor at all times up to three pipes (stdin, stdout and stderr) as well as whatever mechanism we’re using to be notified that the process has exited. That’s the first of our requirements.

The requirements

The first requirement, as I’ve just explained, is that the main application event loop be able to monitor up to three pipes and the child process’s exit mechanism (whichever we make it). Not only that, the three waitFor functions that pair with runtime signals — that is, waitForReadyRead, waitForBytesWritten, and waitForFinished — need to do the same. In both cases, the requirement boils down to the same: whatever the exit notification mechanism is, it needs to be accessible from a select or poll or it needs to interrupt such a call.

The second requirement is that this mechanism scale for multiple child processes simultaneously. One event loop needs to be able to monitor multiple pipes and the exit notification from multiple processes. Though by the design of the API, this requirement does not apply to the waitFor functions.

The third requirement is that this needs to work in a multithreaded environment. That is, multiple event loops or multiple waitFor functions might be running simultaneously.

And finally, the fourth requirement is that, if we write a UNIX signal handler, we obey the requirements for signal handlers.

The Qt solution

Let’s build it step by step. As I’ve shown in the previous blogs, the current state of the art solution is to install a signal handler. Yes, it has an unfixed and non-workaroundable race condition during the installation, and it has an unfixable problem with uninstallation of the handler. Those issues are out of our control, though of course the point of this series of blogs is to try and solve them, or propose a solution that avoids them completely.

The current solution is also something that exists today and has existed for over 8 years. That means it does not use Linux’s signalfd solution. I’ll explore that possibility later, when we discuss how to improve the current solution.

The signal handler that Qt installs is very similar to the code block I had in part two: it “does something”, then it chains to the previous handler. What is that “something”?

When our SIGCHLD handler is called, we don’t know which child process has exited because we didn’t set the SA_SIGINFO flag (again, more on that later). Since we don’t have the PID of the child process, the only way of getting it is by doing a wait or waitpid call. But as I discussed in the first part, we can’t do a “wait for any child” in Qt, because it could interfere with the operations of other libraries like GLib. Since we can’t be told which child has exited, the only solution we have is to check all of our children to see if any has exited.

Enter the fourth requirement: we can only use functions that are explicitly listed in the list of POSIX signal-safe functions. That excludes locking any mutex: remember that the signal handler could be called in any thread, including a thread that has a particular mutex currently locked.

The Qt solution is to write a single byte on a pipe that ends in a process manager. That way, we exit from the signal handler and can continue operating from the a regular context, where we can lock mutexes and get a list of all children currently managed by QProcess. What happens next is that Qt launches a storm of waitpid calls, one for each child process, with the WNOHANG flag set. We’re hoping that this will yield one child having exited, but it could be more than one and it could also be zero (e.g., a child process that was managed by GLib).

But wait, there’s the third requirement to consider: all the user threads, including the main thread, could be blocked doing something or other. Therefore, this process manager needs to be run in either a thread that is dedicated, is not blocked, or is currently in one of the QProcess::waitFor functions. The Qt solution is the first one: there’s a dedicated thread. This solution makes sense if you consider that, otherwise, all Qt-based event loops would need to add the reading end of the SIGCHLD handler’s pipe to their select or poll list, in which case all of those threads might be woken up by the activity, even if only one can do the work.

If this process manager finds out a child process that has exited, it needs to notify the thread that will emit the finished signal. It does that by writing to a pipe, whose reading end is in the source list for select or poll.

Actually, I embellished the Qt solution a little here. The process manager doesn’t do the waitpid call. It simply writes a byte to all QProcess’s pipes and then lets each QProcess object call waitpid. There’s an obvious improvement opportunity here — though of course, that requires that the process manager communicate what the exit status of the child process was.

Improving by using SA_SIGINFO

Turns out that this is exactly what set me upon this path. A few days ago, I was talking to some colleagues on our internal IRC channel when the subject of signals and SA_SIGINFO came up. It occurred to me that the OS does tell us which child process exited, as part of an extra parameter to the signal handler.

With that information in hand, we don’t have to call waitpid on each and every one of our child processes to figure out which one has exited. In fact, we know from the start which one it is, even if it’s not one of ours. I actually wrote a patch to do this and you can see it in Qt’s code review tool (it’s not approved yet at the time of this writing).

I modified the existing code by writing not a simple byte from the SIGCHLD handler to the process manager, but the 4 bytes of the pid_t type containing the PID of the child that exited. Then the process manager simply needs to search for that PID and notify the exact QProcess object whose child exited.

The improvement is clear: as a response to a SIGCHLD being delivered, the application executes exactly one waitpid call.

What’s the problem with this? Well, two of them. First, what happens if the actual signal handler for SIGCHLD was not ours, but something like I wrote on the second blog:

static struct sigaction old_sigaction;
static void sigchld_handler(int signum)
{
    /* my code goes here */
 
    if (old_sigaction.sa_handler != SIG_IGN
            && old_sigaction.sa_handler != SIG_DFL)
        old_sigaction.sa_handler(signum);
}

Do you see what will happen when our signal handler tries to dereference the second parameter, of type siginfo_t *? Crash.

The other problem is more serious: what happens if a second child process exits while our signal handler is running? POSIX requires that this second signal be queued, so our handler is executed again. And what happens if a third child exits too? Well… in that case we’re toast: the third SIGCHLD simply gets dropped.

That’s pretty much a showstopper. Coupled with the comments I’ve received in the review tool and on IRC, which pointed me to an issue with the pipe buffer being full causing either an unhandled EWOULDBLOCK error or a possible deadlock, it indicates to me that I must abandon this solution.

Improving by using signalfd

Since we’re talking about Linux only since the beginning of the second blog, we could use Linux’s signalfd mechanism and avoid a signal handler altogether. Here are some implementation possibilities and the problems with them:

1. We could have one distinct signalfd on SIGCHLD per thread, but I don’t think it’s determined which of the signalfd get woken up by the delivery of the signal. If the answer isn’t “all of them”, this won’t work for us and I’m sure the answer isn’t that.
2. We could have the same signalfd in all threads, but then we run into the problem of “which select gets woken up” by the activity. What’s more, we don’t know if GLib doesn’t have its own signalfd installed, causing the problems from #1 above.
3. We could have a dedicated “process manager” thread that is the only one listening for the signal. However, this solution is hardly different from the time-tried signal handler. Moreover, it also falls short in the multiple-library-criterion as #2.

Add to that the fact that a signalfd is only delivered if all threads in the process have blocked the signal. If even one of them has it unblocked, the signal will be delivered to a signal handler in that thread, bypassing the signalfd completely. We can’t be sure that the user hasn’t created a thread and cleared the signal blocking mask.

No matter what we do, any solution in userspace will require a signal handler and will run into the unsolved problems from the second blog. In the next issue, I’ll explore what solutions I’m proposing, both in userspace and in kernel space.