Doxygen: Misc ideas/proofs-of-concept

[marzer]

Heyo,

Over the last few months of using m.css I've been working on a 'post-process' script to do some further work on the generated html, and now it's polished enough that I figure you might appreciate it as a set of ideas or proofs-of-concept.

The example implementation (and output html) is here: achilles-doc.zip. The post-process script is rebuild-documentation.py. I don't claim to be any sort of python guru so apologies if there's anything in there that is a bit odd, heh. It uses BeatifulSoup for html parsing, so if you want to actually run it with your own input files, you'd need to install the packages beautifulsoup4 and html5lib. If by some miracle you do want to lift any code out of it, go for it- consider it public domain :)

In no particular order:

Tags for inline namespaces

I know that Doxygen doesn't (currently) emit any sort of metadata indicating that a namespace is inline, so my implementation uses a simple list of namespaces, which could easily be added as one of the additional M_ configuration parameters.

Examples:

Relevant parts of the proof-of-concept:

• rebuild-documentation.py:
  • InlineNamespaceFix1()
  • InlineNamespaceFix2()
  • InlineNamespaceFix3()

TODO for @mosra:

• doxygen feature request: Feature request: C++11 inline namespaces doxygen/doxygen#6741 and a WIP PR: Add namespace inline flag in xml output doxygen/doxygen#7828 (not merged yet)
• show inline for inline namespaces in namespace docs, namespace and class trees -- eacca3e

Collapsing std::enable_if (and aliases of it) in SFINAE contexts

Mostly just because these don't add much value to a function's documentation in the first instance, and tend to be quite verbose- I've allowed them to be expandable on click using some very minimal javascript.

If you pour through my code you'll note that I have to do some pretty gnarly regex to find the relevant part between the relevant '<' and '>' characters, and comes with some pretty annoying limitations:

• It only works if the constraint is the last member of the template arguments
• It fails if the constraint contains any '<' or '>' characters (e.g. sizeof(Foo) >= 4)

These would likely be much be easier to get around when the html was first generated, but also could be more easily automated after-the-fact if the contents of the constraint were wrapped in a <span class="sfinae"> or the like.

Examples:

Relevant parts of the proof-of-concept:

• rebuild-documentation.py:
  • EnableIfFix()
• Achilles.js

TODO for @mosra:

• provide some hooks to postprocess things on m.css side (via plugins!) -- once we have a Python config file

Contextually linking to external documentation

Things like members of the std namespace, types from <windows.h>, etc. I've used a different colour for these to denote 'this is external documentation', though I certainly don't think that's a necessity.

Mine just uses a list of compiled regular expressions and the URI's they relate to, but I guess it might need to be user-customizable, which has the potential to be quite complex. Also this particular step is easily the slowest one since it needs to check that it's not injecting links into existing <a> tags- probably unavoidable?

Examples:

Relevant parts of the proof-of-concept:

• rebuild-documentation.py:
  • ExtDocLinksFix()

TODO for @mosra:

• done (simply use tag files ;))

Lifting calling conventions out into labels

This one is pretty windows-specific, I guess.

Also, in addition to lifting the calling conventions, the code responsible for this also takes care of some edge cases where existing specifiers like constexpr and noexcept don't get properly converted into lables. I haven't dug deeply enough to tell you if that's because doxygen parses them inconsistently, or if it's from somewhere within m.css, but would be happy to help dig into this further.

Example:

Relevant parts of the proof-of-concept:

• rebuild-documentation.py:
  • ModifiersFix1()
  • ModifiersFix2()
• html/namespace_achilles_1_1_math.html

TODO for @mosra:

• Define a setting in m.css (list of names to extract) hook for parsing and extracting info out of this to feed the template with. Similar case is C++14-only constexpr, etc.

Generate sensible defaults for defaulted/deleted rule-of-six

The 'only generating output for explicitly documented code' goal is an amiable one, but does tend to run afoul of some common boilerplate-avoidance practices. For example, if I have the following macro to make immovable type definitions simpler:

#define IMMOVABLE(TYPE)						\
	TYPE(TYPE&&) = delete;					\
	TYPE& operator=(TYPE&&) = delete;		\
	TYPE(const TYPE&) = default;			\
	TYPE& operator=(const TYPE&) = default

I can't then add any documentation to these operations to ensure that someone reading the documentation would see deleted or defaulted.

Some default behaviour for these would be great; maybe even with some default briefs? Something like:

(I haven't actually implemented this one; the example was just some browser DOM editing)

TODO for @mosra:

• none, you have to explicitly list these (sorry)

Automation hints/hooks

Providing more automation hints would be great, since some of the workarounds I've used are quite brittle. One good example is the parsing of Function Documentation; each function gets it's own <section>, but the <section> tags themselves don't have any sort of ID or class that marks them as a function definition. I fetch them by first looking for a <h2>Function Documentation</h2> then collecting all of it's <section> siblings, but that feels brittle as hell.

Relevant parts of the proof-of-concept:

• rebuild-documentation.py:
  • ModifiersFix2()

TODO for @mosra:

• provide some hooks to postprocess things on m.css side (via plugins!) -- once we have a Python config file

---

That's about it. I hope this is useful?

[mosra]

Jesus 😮 😍

This is a very fine collection of very useful additions. Coincidentally I'm just about to publish a larger set of new features and improvements and some of these might be already partially solving a bunch of what you got here (in particular, the noexcept-related stuff). I'll have a blog post with details soon.

Just give me some time to reply properly to all of the above :)

[marzer]

No dramas. I've just realized that the script does some pretty hamfisted stuff with file paths that will only work on Windows (b.c. of case insensitivity), so if you want to test it on Unix you might have to do some hackery :P

[mosra]

Argh! Of course I forgot to reply, sorry 😅

(Blog post here: https://blog.magnum.graphics/meta/improved-doxygen-documentation-and-search/)

Tags for inline namespaces

Yes, definitely, I think this could go as an always-enabled feature, so no need for an option. I didn't have any inline namespace myself yet, that's probably the main reason this wasn't done yet. I hope this is easy to extract from the Doxygen XML files? TODOs for myself:

• show inline for inline namespaces in namespace docs, namespace and class trees -- done in eacca3e
• ... and while at it, show also final for classes in class trees -- done in 9a3c8f6

Collapsing std::enable_if (and aliases of it) in SFINAE contexts

I think the proposal in #56 could help solving this problem as well, in a more general way. It's originally just about hiding these completely, but I like the idea with JavaScript collapsing too, it could be able to do both. What do you say?

Contextually linking to external documentation

Do you know about Doxygen tag files? Those provide a way to link to external documentation (and to make this completely great, cppreference.com provides such a tag file). I'm using this quite heavily in the Magnum documentation, which links to external Corrade docs (a utility library) and also to STL -- see the Animation::TrackView class, for example:

I'm using a non-bold font for to distinguish the external references, experimented also with a different color but that turned out to be way too distracting.

Lifting calling conventions out into labels

I hope I ironed out most of the corner cases with noexcept / constexpr in 2e2b57f, please tell me if you still come across cases that are broken. Doxygen just gives me the full function signature and I have to extract these out of there, so it's very possible I'm making some errors there.

For the additional keywords, I can think of also stuff like new C++ attributes such as [[noreturn]], [[nodiscard]] etc. Since the list is too large and it would both not cover everything for some and be overly verbose for others, I think the best way would be a config option where you could specify what keywords you'd want to have parsed.

• TODO for myself: provide such a setting

Generate sensible defaults for defaulted/deleted rule-of-six

Hmm. Doxygen actually has some pretty powerful preprocessor and I think it could be convinced to expand the IMMOVABLE macro before parsing the comments. So you could have the macro as:

#define IMMOVABLE(TYPE)				 \
        /** @brief Moving is not allowed */      \
	TYPE(TYPE&&) = delete;			 \
        /** @brief Moving is not allowed */      \
	TYPE& operator=(TYPE&&) = delete;	 \
        /** @brief Cop constructor */            \
	TYPE(const TYPE&) = default;	         \
        /** @brief Copy assignment */            \
	TYPE& operator=(const TYPE&) = default;

(Disclaimer: I never tried this myself. Maybe it doesn't work at all.)

Another idea I'm thinking about is that it could be possible to detect what the function is (whether a copy/move constructor or a copy/move assignment) and then providing some implicit comment for it, in case it's defaulted/deleted. But knowing how bad the parser sometimes is (and thinking about the template-heavy cases), I'm not sure if it's at all possible to implement this robustly.

Automation hints/hooks

Hmm, I have to think about this a bit more. I have a similar case, need to integrate for example the following source code annotation, which adds colored rectangle next to custom RGB literals, but didn't find a way to do that cleanly yet without hardcoding stuff directly:

I think this should be done on the Python level, rather than on the resulting HTML (which is too late, in my opinion). There's a well-documented hierarchic structure coming out of the parser which is then fed to Jinja2 templates, so the hooks could be added there I think. Somehow. Don't have an idea how to do that yet.

[marzer]

RE inline namespaces: I had a cursory look through the doxygen XML, did a CTRL+F for 'inline', and didn't find anything relating to the namespaces. That's the extent of my investigation, though; there may have been some other indication that I missed...

RE collapsing templates: Am I reading that correctly, that there be some MCSS-related macros in the C++ source itself? I'm not sure how I feel about that idea, to be honest- inline documentation can be pretty noisy already. I suppose doing it that way means it's opt-in, which is nice. I certainly think there's value in having it hidden by javascript and allowing it to be toggled by the user, since you could then apply some basic parsing to the constraint to make them more readable, like this:

(this was just DOM-editing, I haven't written this parsing logic, since it's pretty hard to do once the document is HTML)

RE external documentation links: I was aware of tag files, but thought the need to generate them was a bit of a non-starter. Using some provided by cppreference.com obviously solves that issue! One thing my approach does that I like is handles plurals etc correctly:

• a list of std::string_views
• becomes this: "a list of std::string_views"
• instead of this: "a list of std::string_views".

Does the tag file approach handle things like that? A minor detail, I suppose.

RE calling conventions and labels: Adding a configurable list of symbols sounds great. At some point over the weekend I'll regenerate our documentation with out the label-related post-processes and see if I can find any missing corner cases for you :)

RE defaults for rule-of-six: I had not even considered adding comments to the macros themselves, to be honest. I'll try that this weekend and see what happens.

[mosra]

inline namespaces

Argh, that's what I feared. Could you try generating the stock HTML output and see if it's there at least? If yes, then it's just not propagated to XML (which would not surpise me at all). If not, then we would need to open a feature request and wait a year or so until it gets implemented, because no way in hell I'm touching their C++ parser. Nope. ☠️

collapsing templates

Yes, that's what I meant, a new macro that gets recognized by m.css. I am doing a similar thing now, see here, but the #ifdef also gets pretty noisy. Doing C++ parsing on the theme side (in python) is also a thing I don't ever want to do ☠️, because unless you have the full context at that point, you'll almost certainly get it wrong. Imagine parsing pathological cases like this:

tempate<std::enable_if<Foo<Bar<Fizz, Buzz>::A>>::value

... is it a type Bar<Fizz, Buzz> with a member ::A or is it template<bool, bool> struct Foo, taking two parameters Bar < Fizz and Buzz > ::A? You'll never know unless Clang has your back. This would probably need to wait until I have Doxygen replaced with Clang-based parser and I can gather all the context easily. Sorry.

external docs

what about a list of @ref std::string_view "std::string_views"? ;) Other that, no, it doesn't do any such thing, you have to be explicit.

labels

defaults for rule-of-6

thanks in advance! 👍

[marzer]

inline namespaces

Guess I should have mentioned that I actually tried the stock HTML output first, didn't see any inline on namespaces, which is what led me to quickly sussing the XML... 😢

labels

Your recent version is almost perfect. The only edge cases my post-process catches now (ignoring the custom calling conventions) are functions that return decltype(auto), e.g.:

(source xml: namespace_achilles.xml.txt)

deleted rule-of-six

I tried adding comments to the macros as in your example, with SKIP_FUNCTION_MACROS = NO and EXPAND_ONLY_PREDEF = NO, and nada :'(

[mosra]

inline namespaces

Feature request submitted: doxygen/doxygen#6741

labels

Oh what the ☠️ 💥 💣. It attempts to "fix" the trailing return by moving the whole function suffix in front, including the constexpr:

<type>decltype(auto) constexpr</type>
<definition>decltype(auto) constexpr Achilles::Coerce</definition>

I bet this will be the case for all other keywords when they go together with trailing return types.

EDIT: oh, actually, looking again, the constexpr leaks all the way to the type. So I think parsing both <definition> and <type> for extra trailing keywords should do the trick, hopefully.

• So I have some fun to do. -- done in 193ab80

rule-of-six

Oh, too bad. Then the only option would be a "is it a constructor" / "is it an operator=" detection... and that will be non-trivial I'm afraid.

[marzer]

Well, good to hear that the labels thing seems a relatively straightforward thing to implemement.

As for the rest of it, I'm totally happy to keep building additional stuff on using a post-process, since m.css has already made that vastly easier. I might have a go at the rule-of-six thing myself by injecting a separate pre-process step on the generated xml before invoking m.css.

edit: come to think of it, that would be a better way of handling the inline namespaces thing too, really. If we assume that at some point between now and the heat death of the universe the doxygen team will inject inline="yes" as part of the namespace's <compounddef>, would you be willing to give m.css the ability to consume that?

[mosra]

injecting a separate pre-process step on the generated xml

When I get back to this (later this week, right now I'm working on stuff that's not related to m.css), I'll have a look. The idea got a bit more stable in my head and I think the detection could be fairly doable. Ugly, but doable.

If we assume that at some point between now and the heat death of the universe the doxygen team will inject inline="yes" as part of the namespace's <compounddef>, would you be willing to give m.css the ability to consume that?

You mean doing that before they have the corresponding code integrated? I would have no way to write a test for this and things that are not tested tend to get broken.

[mosra]

Um, just realized, for the rule-of-six: as far as my experience goes, Doxygen always warns about undocumented functions unless it's a parameterless constructor or a destructor. So even if I implement some copy/move constructor/assignment detection on the m.css side, Doxygen will still warn about an undocumented function. Since (at least for me and a bunch of other users) the goal is to have a completely quiet output (and the script can't prevent Doxygen from complaining), I'm not sure if such a detection is worth implementing.

[mosra]

A bunch of updates, finally:

• the final specifier on classes is shown since 9a3c8f6 (I did not add support for inline="yes" yet because Feature request: C++11 inline namespaces doxygen/doxygen#6741 is so far getting completely ignored and that's making me upset)
• parsing of noexcept for functions returning decltype(auto) is "fixed" in 193ab80 (but seriously, doxygen is just behaving totally random there -- and don't even ask how bad it gets when trailing return types get in the mix. UGH.)
• some further updates and fixes to the keyword parsing done in 38e0d62

[marzer]

Looks great! Just tested it with my script and the custom calling conventions are now the only thing they pick up, so looks like you nailed it (insofar as one can nail anything down given what doxygen produces, anyway :P)

[mosra]

Minor update: as of a246040, the m.code plugin has an ability to postprocess various things. I desperately need this feature for Doxygen as well, so I'll be adding a Python configuration file soon. And that should gradually make most of the above finally possible (while also gradually phasing out the m.css-specific Doxyfile options and moving them to Python).

[mosra]

• As of eacca3e, if you have Doxygen with Add namespace inline flag in xml output doxygen/doxygen#7828 applied, inline namespaces are parsed and shown in the output.
• Since 1c133b1, the Doxygen theme has finally a Python-based config file which allows for additional flexibility needed by the remaining features above. Stay tuned!

[marzer]

Nice! I was just thinking about this a few days ago, funnily enough.

[marzer]

@mosra I just noticed your Doxygen patches note on inline namespaces is now incorrect as the corresponding patch has been merged and released with doxygen 1.8.19

[mosra]

Oh, thanks! I still have to test with 1.8.19 to see what new regressions popped up, and my expectations aren't exactly high. I'm on a fork of 1.8.17 locally.

(And then also go through the 30+ remaining issues and PRs. I fixed the CI at least.)

[marzer]

Well for what's worth I've been using 1.8.20 since it was released in August and haven't noticed anything odd, though that's totally anecdotal.