«

»

Sep 30

The future of moc

Two days ago, on his blog on some Qt Creator news, Christian wrote:

We’re currently prototyping what would happen if we replaced our own C++ code model with clang’s.

The first comment in the blog was praising that research, but that got me thinking: what do we gain from it? Qt Creator already has a full C++ code model, so what’s the advantage of replacing that with clang’s?

The first thing I thought was reduced maintenance: by using clang’s code, we don’t have to maintain ours. Is that all? What else can we gain from having this code, or the experience in writing it?

Diagnostics

That reminded me of another blog I had read a couple of days before, entitled Making clang GObject aware, where the author was talking about a a plugin he implemented for clang that would produce warnings for setting and getting properties that didn’t exist in a given class. The parallel to moc was quite clear: QObject’s property getter and setter are also string-based, so we could use the same solution. Moreover, traditional signal-slot connections are also string-based and could use the same solution for providing warnings to the developer at code creation time, a feature that is already present inside Qt Creator.

Given that GCC also supports plugins in recent versions, it would be a nice addition to provide those diagnostics during compilation. This would benefit users who choose not to use Qt Creator and it might be even more powerful than what you can do with Qt Creator, seeing as that the IDE must deal with possibly broken code and recover from it — without hogging the CPU and freezing up.

Improving moc itself

Once we have a clang-based code model that can interpret the QObject markings in the source code, another thing we can do is replace moc itself. The moc codebase is extremely old. In fact, you can’t see this, but the old Qt historical repository began with moc being imported into CVS (files moc.l and moc.y). Since then, it’s been rewritten a couple of times, most recently for Qt 4 with a hand-written parser.

Another invention done for Qt 4 was the qt3to4 tool, which required a better understanding of C++ to do more than simple search-and-replace. That tool had a real C++ parser that deal with and recovered from some errors. Later, that C++ parser was reused in the QtScript binding generator and in Qt Creator’s C++ code model (it’s libCPlusPlus.so.1).

Unfortunately, moc was never rewritten to take advantage of that processor. That means today it still has some problems with C++ namespaces in some corner cases. It also doesn’t fail when an include is missing, so it may mis-parse things because a preprocessor expansion was missing.

So instead of rewriting using Roberto’s parser, maybe we should skip it and use clang?

Obviating the need to run moc at all

If we put the two things above together, maybe we wouldn’t have to run moc at all. Think about it: if we have a plugin to clang that can properly understand the Qt Meta Object markings (Q_OBJECT, Q_PROPERTY, etc.) and we have a clang-based codebase that generates the required code for proper operation, can’t we do both operations at the same time? We could teach clang to load a plugin that would interpret the markings, provide the necessary diagnostics as well as emit the necessary symbols at the appropriate place.

At this point, the Qt detractors will be able to correctly claim that Qt requires extensions to the C++ language! Finally they may be right about some thing!

But we still need moc

Of course, we cannot completely get rid of moc, as Qt is supposed to work with other compilers, not just GCC and Clang. So we’d need to provide moc for those people or for versions of the compilers that cannot load plugins.

I know most of the comments are going to be about “why don’t you use a modern C++ signal-slot implementation” or similar, so I’ll address that preemptively. First of all, we are already implementing a modern C++ signal-slot mechanism — Olivier has done most of the work and the Merge Request (number 42) has already been submitted, one that should be on-par with any of the other implementations.

Next, we have some extra requirements that those other implementations don’t fulfill. One of them is the ability to add signals to an existing class without breaking binary compatibility, which immediately excludes most of the other implementations because they use a signal object. Another requirement we have is to keep source compatibility with Qt 4 for the time being (Qt 6 may not require that).

Those two requirements above already require us to have extra code which cannot be done exclusively by templates and inline functions. The only thing we can add to a class that doesn’t break binary compatibility are non-virtual member functions and, unfortunately, pointers-to-member functions (PMFs) are a weird class of citizens in C++. Therefore, not only do we need to implement those functions somewhere, we also need class-specific code that can convert a PMF to something more manageable like an integer ID. Those can only be done with generated code.

What’s more, the most important reason why we need moc is not related to the signals and slots. It’s about everything else that moc allows us to do, which cannot be done without. With the Meta Object system, we can:

  • enumerate the properties, signals, slots and other methods as well as decorations (classinfo) present in a class (this feature is called “type reflection”, which is common in scripted languages, but is also present in Java)
  • get and set properties by name and without knowing th exact type of the property (though technically the latter is a product of the related, but independent Meta Type system)
  • place calls to extracted member functions (the “invokable” functions) by name
  • the last two features together allow us also to integrate C++ classes easily with other languages and IPC mechanisms without the need to write specialised parsers (D-Bus, EcmaScript, QML, but also the many bindings to Qt like Python or C#)
  • instantiate types by simply having access to the meta object

Finally, having code generators isn’t exactly news. I don’t see anyone complaining about uic, rcc, or qdbusxml2cpp, which parse XML and produce C++ code. I also haven’t seen people complain about tools like lex, yacc, bison, gperf that are very commonplace and serve specific purposes. So why do people give moc a hard time? I cannot accept an answer that it’s because you’re forced to use it — it’s possible to live without it, but your life would be harder. When was the last time you wrote a perfect hash by hand?

Besides, adding type reflection/introspection features to static languages isn’t exclusive to Qt. The GObject Introspection that I mentioned before is one such example and it was only created because there are useful features you can get from it. Now, my knowledge of GObject Introspection is incomplete, but from what I gather, the major difference from that to the Qt Meta Object markings is that GObject Introspection is external to the source code (it’s out-of-band). You need to generate the data and place it in separate files or at least a separate code section in the library. The Qt solution places the markings inside the source code.

Moc is here to stay. What will it evolve to, though?

16 comments

  1. avatar
    Giorgos Tsiapaliwkas

    I saw at http://clang.llvm.org/ that c++0x is not supported by clang.

    If i recall correctly Qt Development frameworks want to use c++0x in the future,after the 5.0 release around to 5.2 release.
    With the current status this can’t happen.

    So what’s the priority?clang or c++0x?

    thank you for reading my reply

  2. avatar
    Stefan

    Giorgos: I would definitely vote for clang. It’s one thing to start supporting a new compiler in a 5.x minor release. Supporting a new version of the underlying programming language is a whole different beast, esp. w.r.t. binary compatibility.

  3. avatar
    Thiago Macieira

    @Giorgios: both. Clang’s support for C++11 is improving and will reach full support eventually. As of now, though, Clang is in “unsupported” state for Qt, since it sometimes still fails to properly compile.

    As for what I’ve just described here, this is just a couple of ideas I’m throwing around. There are no plans to implement any of it.

  4. avatar
    Thiago Macieira

    @Stefan: adding support to Clang mid-way through the 5.x series is no problem.

  5. avatar
    Mathias

    The big problem with moc is, that it fails quite often. Well, actually it is qmake that fails, but perception is: “Uh, some vtable missing again!” – “Uh, some include problems!” “Uh, why is it using an old declaration!” – And so on. qmake problems, but moc gets blamed.

  6. avatar
    Thiago Macieira

    @Mathias: yeah, qmake is past its expiration date. It was a perl program called tmake (you can find it in sourceforge) and was replaced by a C++ program that is too close to that perl codebase. It is now unmaintanable.

  7. avatar
    illissius

    The only thing we can add to a class that doesn’t break binary compatibility are non-virtual member functions and, unfortunately, pointers-to-member functions (PMFs) are a weird class of citizens in C++. Therefore, not only do we need to implement those functions somewhere, we also need class-specific code that can convert a PMF to something more manageable like an integer ID. Those can only be done with generated code.

    Could you elaborate on this bit? PMFs are weird, true, but what specific deficiencies do they have which force you to use integer IDs instead? I’m assuming they would be used to tell QMetaObject which signal/slot to call? Deferencing a pointer to member has awful syntax, but that would presumably be QMetaObject’s job, whereas the user would be doing the address-taking part of it which is fairly intuitive (&Class::member). Or I’m guessing the actual problem is that you would need to go bonkers with templates to get the types right for PMFs, whereas ints are conveniently untyped… ?

  8. avatar
    Thiago Macieira

    @Ilissius: here are some of the problems I can think of when using PMFs.

    First, there’s the problem of casting. Unlike pointers, PMFs require static_cast to be cast even to a base class. Just like pointers to classes, you cannot static_cast a PMF through a virtual inheritance. And you cannot reinterpret_cast, since a cast of class type may require adjusting the “this” pointer. And this only applies to casts of actual PMFs: it’s invalid to static_cast a pointer to a PMF to another pointer-to-PMF type.

    Another problem with casting is that you can cast a PMF only through the class it belongs to, but not the function type. So you cannot store a pointer to a QObject member function taking an int parameter in a void (QObject:: *)(). Doing that would require a reinterpret_cast, which I’ve just said is not allowed.

    Even if you solved the problem of casting, there’s the problem of comparisons: a PMF to the same function may take two different representations in memory, as it may be pointing to different sub-objects of the same object. When you do a PMF-to-PMF comparison, the compiler will adjust as needed. But you cannot do a memcmp — again, just like pointers to objects.

    And even if you could memcmp, you cannot provide storage for PMFs appropriately since the size of a PMF may vary according to the class it’s pointing to, depending on the ABI. So you may provide too little storage (bad!) or you may provide excessive and you could end up memcmp’ing uninitialised data.

    With all of that in mind, there’s no way we could have a PMF member in the connection list entry, as there’s no type that will work in all cases, for all classes and all signal types. The only way to do this is to replace the PMF with a wrapper object, like std::function or similar classes. Those would need to be stored indirectly (via a pointer), which one more potential cache miss for each connection where the current object is a sender. To make matters worse, std::function wraps a PMF but we still cannot operate on it directly — we need another level of indirection: a class with virtual functions and a virtual destructor so we dispose properly of memory.

    In the end, instead of a simple 32-bit integer we can compare directly and have no memory allocation issues on, we’d have a pointer to a class of at least 3*sizeof(void*) — the minimum size of a class with virtuals containing one PMF member — where the comparator function is an indirect call (virtual call) and requires its own allocation/deallocation. Purists may say that this is the “proper” way to do it in C++, but I’ll take code generation if it allows me to use 4 bytes at runtime any day.

  9. avatar
    Karl

    Moc is a reasonable solution to work around deficiencies in the base language.

    An issue I often run across is having to deal with circular signal chains. For example, I might have one variable (VAR) having two different widgets (WID1, WID2) which modify it in addition to other programmatic changes (PRO).

    Connections to VAR:
    PRO -> VAR
    WID1 -> VAR
    WID2 -> VAR

    Connections from VAR:
    VAR -> WID1
    VAR -> WID2

    So when PRO changes VAR the WID1/2 are updated and a tweak of WID1 will update WID2 by way of VAR. The problem is that the WID1/2 changes are propagated back to VAR itself and signals from a WID to VAR are sent back to the sender WID.

    What I’d like is for the signaling implementation to handle blocking of an output connected to the same object which is generating the input signal. Currently I have to do various hacks to get this to work. Is there some recommended way of handling this in Qt4?

  10. avatar
    Thiago Macieira

    @Karl: You can call QObject::blockSignals to stop a signal from being emitted in a given object. But that requires knowing which objects are connected to your signal, which is a layering violation.

    Instead, I’d recommend you simply use two signals, one indicating a direct update and another that isn’t direct. A direct update means the receiver emits indirect updates. But indirect updates don’t cause emission of more signals.

    It’s the same solution as cyclic reference counts: you break it with a different type of reference (a weak reference).

  11. avatar
    Karl

    Yes Thiago, those are among the hacks I use. It should be so much easier to deal with this common use case of keeping a group of objects synchronized. I just posted to say that the deeper design issues are more important than a particular implementation (Moc vs no-moc).

  12. avatar
    Thiago Macieira

    @Karl: I disagree. I don’t see a design issue here and I also don’t think this particular type of use is common. So I’d rather keep the meta object system simple and let people build upon it the use-cases that they need.

  13. avatar
    Zeke

    Will MOC ever allow you to derive a class from QObject, that is a template class? This is one of the biggest gripes that I have about MOC.

  14. avatar
    illissius

    Thorough and convincing explanation. Thank you.

  15. avatar
    Thiago Macieira

    @Zeke: a template class with a metaobject is very hard to construct. It’s possible with a lot of effort and under a limited set of conditions.

    Not worth it.

  16. avatar
    John Gutierrez

    “So instead of rewriting using Roberto’s parser, maybe we should skip it and use clang?”

    Sounds like your discussion points to updating moc to include Roberto’s parser as the best interim solution.

Comments have been disabled.

Page optimized by WP Minify WordPress Plugin