QUrl in Qt 5: encoding

One of the classes I used to maintain in Qt was QUrl and I had worked quite extensively with KUrl previously, so I knew what was wrong with the current API. And during the Contributor Summit in Berlin, I volunteered to rewrite QUrl, to add some features KUrl needed.

I described the original goals of the change in the “QUrl in 5″ thread in the qt5-feedback mailing list. After I had done that, I began working on implementing them. This blog is a status update. As I’m not done yet, things might change a bit until the end and the code is accepted into QtCore.

This is the first part and I’ll write more later.

QUrl in Qt 4

To understand the design decisions, I need to explain about what a URL is. It is governed by a pair of relevant specifications: RFC 3986 (Uniform Resource Identifier (URI): Generic Syntax) and RFC 3987 (Internationalized Resource Identifiers (IRIs)). The first of the two is the base definition of URI using 7-bit ASCII only, whereas the latter specifies what you’re supposed to do when you find non-ASCII codepoints. Long story short, RFC 3986 says that a URI is a sequence of letters, digits, hyphens, underscores and the tilde character, separated by some delimiters. Anything else must be represented by its percent-encoding. The latter RFC says non-ASCII characters are allowed and are is equivalent to their UTF-8 encoding in percent-encoded form.

Originally, QUrl in Qt 4.0 tried to implement those specifications: an 8-bit QByteArray was parsed, by searching the boundaries of the components of the URL and then it decoded the percent-encoded forms into proper QStrings, applying the UTF-8-to-UTF-16 transformation in the process. This allowed you to get components like the path or the fragment fully decoded into QStrings, which are nicer to read by us humans. It implements the parsing by following the URI ABNF definition strictly, so we know that the parsing follows the specification to the letter.

However, back in Qt 4.4, that the second operation had a problem: it could not handle an encoded sequence like %80%81, which is not a valid UTF-8 sequence, but is valid in a URL. The solution I found back then was to add an “encoded” form of each of the components of the URL, which returned a QByteArray with proper percent encoding. Those forms are capable of handling anything you throw at it, restoring the full compatibility. So QUrl keeps a pair of forms for each component of the URL: one in UTF-16 decoded form and one in 8-bit fully encoded form.

It also has a problem I had not identified until much later: the QUrl constructor and QUrl::setUrl() functions, which take a fully-decoded QString URL, cannot handle the presence of some special characters in non-delimiter functions (e.g., the ‘#’ character used in a path name or as part of a query). In order for them to be interpreted as non-special, they need to appear percent-encoded — but these functions expect fully decoded forms! Worse, the QUrl::toString() function blindly decodes all the percent-encoded forms it might have and, in the process, may lose the distinction between a delimiter and a regular character.

To avoid those problems, it is recommended in Qt 4 that you use the “encoded” functions, especially QUrl::fromEncoded() and QUrl::toEncoded(). Unfortunately, the output of such functions is not always human-friendly.

Solution

All of those problems had one common cause: using UTF-16 QStrings in the internal representation and in the API, but using an 8-bit strict parser. As I said before, the RFCs say that URIs are a sequence of arbitrary 8-bit characters, some of which may appear in decoded form. The specification does not contemplate or give hints how to write an UTF-16 implementation. You can take the easy way out, like what Java does, and ignore any encoding and decoding. But note how java.net.URL refers to the old URI specification and doesn’t claim to support IRIs. For QUrl, we want to do the right thing.

The centrepieces of QUrl in Qt 5 are a rewritten parser, operating on UTF-16, and what I’ve called the “recoder” (you can also call it a “transcoder”). The principle of both is that QUrl should operate exclusively on QString (UTF-16) data, without having to convert to and from 8-bit, especially for parsing. That is matched in the API by removing all the QByteArray methods: everything is now using QString.

To continue supporting arbitrary sequences that are not valid UTF-8 sequences, the QUrl API now only supports the “encoded” mode. In Qt 4, we had encoded in QByteArray and decoded in QString; in Qt 5, it’s encoded in QString. To avoid the problem of ugly encoded sequences, I’ve introduced a new enum allowing you to specify what you want to see decoded:

   enum ComponentFormattingOption {
        FullyEncoded = 0x0000,
        DecodeSpaces = 0x1000,
        DecodeUnambiguousDelimiters = 0x2000,
        DecodeAllDelimiters = DecodeUnambiguousDelimiters | 0x4000,
        DecodeUnicode = 0x8000,
 
        PrettyDecoded = DecodeSpaces | DecodeUnambiguousDelimiters
                        | DecodeUnicode,
        MostDecoded = PrettyDecoded | DecodeAllDelimiters
    };
    Q_DECLARE_FLAGS(ComponentFormattingOptions, ComponentFormattingOption)

The FullyEncoded type is equivalent to the Qt4 QUrl “encoded” method, strictly following the RFC 3986 specification. If you add the DecodeUnicode option, then it will follow RFC 3987 and decode all the percent-encoded sequences that form valid UTF-8 characters into their QString equivalents. The decoding of delimiters will cause the result to be non-compliant, but won’t lose information, ever. The default value for all functions is PrettyDecoded.

For that reason, since the inputs and outputs of QUrl are always encoded, the function that transforms it according to the flags above doesn’t encode or decode — it simply “recodes”. That’s why I called it like that. And then I spent a day writing unit tests to ensure it works as I want it to.

The new parser will require working directly on QStrings, without converting them to 8-bit like the parser in Qt4 requires. Moreover, given the new possibilities for encoding, the parser needs to be rewritten to be less strict, without sacrificing compatibility. In specific, the new parser will accept certain characters to appear decoded while RFC 3986 would require them to be encoded. With that, I expect to gain in performance quite considerably.

So the new process of parsing a QString will be to find the positions of the components of the URL, then recode straight from the QString into the internal representation (which should be the PrettyDecoded representation to avoid recoding in the common case). The parsing itself should be fast enough so I can drop the lazy-parsing functionality of QUrl (which is currently horribly broken in Qt4).

On the next blog, I’ll go over the front-end API.