New extension annotation processor extensibility

[gsmet]

Feedback from @ppalaga.

Let's start a thread per comment below.

---

There are several issues with the current state of the GenerateAsciidocMojo and the annotation processor. Some of them are easy to solve within the current design, some of them are harder:

• Change the qute template so that there is less unused space - see Figure out how to better present config options  quarkiverse/antora-ui-quarkiverse#56 and see how we already render in Quarkus CXF: https://docs.quarkiverse.io/quarkus-cxf/dev/reference/extensions/quarkus-cxf.html#quarkus-cxf_quarkus-cxf-codegen-wsdl2java-includes
• Render @since version if available in JavaDoc. The metadata is correctly passed to yaml. Again, only the Qute template would have to get adjusted. This how we currently render it in Quarkus CXF:
  
• Camel Antora UI required some special AsciiDoc markup, but I do not remember the details. Being able to override the default templates would perhaps solve the issue.
• For Camel Quarkus and Quarkus CXF, we have custom plugins in place, that assemble the whole extension reference page (each extension has one) - see e.g. https://docs.quarkiverse.io/quarkus-cxf/dev/reference/extensions/quarkus-cxf-rt-features-metrics.html or https://camel.apache.org/camel-quarkus/3.8.x/reference/extensions/aws2-s3.html Note that in those pages the config section is optional (because some extensions do not have Quarkus config). The other sections are generated from extension metadata and some adoc chunks stored in src/adoc, such as usage.adoc, limitations.adoc, etc. We could in theory include the config section generated by GenerateAsciidocMojo, which we also did in the past, but that kind of solution was complex and hard to maintain and debug. Therefore we switched to our own custom plugins that assemble the whole reference page. I wonder if Quarkus team would be ready to widen the scope of the plugin to output the whole reference page?
• Our current setup does not require to rebuild the docs module when something in an extension was changed. It is enough to rebuild the extension. The current Quarkus solution requires rebuilding the docs module. I wonder if you have some CI in place that amends the generated changes when a dev forgets to rebuild the docs module?

---

[gsmet]

Change the qute template so that there is less unused space - see Figure out how to better present config options quarkiverse/antora-ui-quarkiverse#56 and see how we already render in Quarkus CXF: https://docs.quarkiverse.io/quarkus-cxf/dev/reference/extensions/quarkus-cxf.html#quarkus-cxf_quarkus-cxf-codegen-wsdl2java-includes

We would either have the notion of themes in the annotation processor and you would then contribute your theme to core Quarkus or you could have the option to provide a directory with some overrides for the templates (or both).

I would lean towards having a proper theme so that we maintain it for everyone.

[ppalaga]

I wonder how would the theming work? Would it be anything else than a custom template?

[gsmet]

I would keep it simple for now:

• I would move the default Qute templates to a default directory
• You could define a theme via a Maven plugin property: we would load the templates from there if they are available (and default to the default ones if not)
• If we do it this way, we could also probably have a local override - but I think it would be nice to maintain things centrally rather than having local templates everywhere

That way you would just override what you need and keep most of the infrastructure.

I already split some things in smaller pieces but we could split things a bit more if needed.

[gsmet]

Render @SInCE version if available in JavaDoc. The metadata is correctly passed to yaml. Again, only the Qute template would have to get adjusted. This how we currently render it in Quarkus CXF

If you have your own theme, you can include it easily.
Not sure we want that in Quarkus given our docs are versioned and a lot of things depend on the version so giving this info for the config properties might not be very helpful.
But given it's hidden with the More info, we could add it to the default theme anyway.

[ppalaga]

Especially in extensions that still undergo some development, since versions on the level of individual config properties are very helpful for the users to figure out whether some property is supported in the version they use. We already had this sort of confusion in Quarkus CXF a couple of times (before we introduced) the rendering of the since tags. Anyway, thanks for being ready to include it in the default theme.

[gsmet]

PR welcome!

[gsmet]

Camel Antora UI required some special AsciiDoc markup, but I do not remember the details. Being able to override the default templates would perhaps solve the issue.

Well, that I can't do much. But point 1 would answer it in any case.

[gsmet]

For Camel Quarkus and Quarkus CXF, we have custom plugins in place, that assemble the whole extension reference page (each extension has one)

I'm not too excited about it. I would like to keep things minimal on our side because I don't want to have the burden to maintain something we don't use at all.

That being said, I moved most of the logic stuff to the annotation processor this week-end so the plugin is now extremely minimal. So if it makes sense for you to have your own, it should be a lot easier than earlier.

Now, just to be clear: this is internal, it's not API, use the config model at your own risk. It might be modified from time to time.

[ppalaga]

I moved most of the logic stuff to the annotation processor this week-end so the plugin is now extremely minimal.

Thanks a lot, I wanted to propose exactly that 🙂

[ppalaga]

if it makes sense for you to have your own, it should be a lot easier than earlier.

I lean towards keeping your our own plugins and use the new code to get the metadata.

[gsmet]

Yeah I can see the value of that.

The nice thing with the new architecture is that you have a model that is completely agnostic of the UI, and you should be able to use this model a lot more easily than the old one as concerns are properly separated.

The idea behind that is that we plan to generate JSON files for IDEs to consume (and these ones will be a lot more simple and also must be a lot more stable).

[gsmet]

Our current setup does not require to rebuild the docs module when something in an extension was changed. It is enough to rebuild the extension. The current Quarkus solution requires rebuilding the docs module. I wonder if you have some CI in place that amends the generated changes when a dev forgets to rebuild the docs module?

We don't commit the generated docs in Quarkus (we might at some point, see #42447) but more to track the changes to the generated output than anything else.

Note that I didn't develop this new infra to have fun (annotation processors are not that fun): we needed to fix the problem of the annotation processor crossing the module boundaries. It was a big problem for us for our Develocity caching (and is not exactly good practice).

Now, there's nothing preventing you from generating the doc in the deployment module and merge the info from the runtime and deployment modules only.

You can already experiment with this sort of things with the Maven plugin we provide (I think).

[ppalaga]

We don't commit the generated docs in Quarkus (we might at some point, see #42447) but more to track the changes to the generated output than anything else.

Maybe you can afford that because you do not use Antora to publish the site?

In Quarkiverse, where the vanilla Antora is used, all .adoc files have to be in Git.

Note that I didn't develop this new infra to have fun (annotation processors are not that fun): we needed to fix the problem of the annotation processor crossing the module boundaries. It was a big problem for us for our Develocity caching (and is not exactly good practice).

I can understand that perfectly. The world of Quarkus extensions would be much nicer place to live if Antora was able to read files outside the docs folder ¯_(ツ)_/¯

[gsmet]

Yes, I'm well aware of that.

I think we should maybe broaden this thing:
https://github.com/quarkiverse/quarkus-github-app/blob/main/.github/workflows/release.yml#L53

So that we commit any do changes to the tree before release.
Probably do it for docs/modules/ROOT/pages/includes/ for now.

/cc @gastaldi we might need your magic to push it everywhere.

[gastaldi]

Can you please open an issue in https://github.com/quarkiverse/quarkiverse-devops/issues so I can add to my never-ending TODO list? ;)

[gsmet]

Ideally, it would be nice to add it at the beginning of the never ending TODO list, instead of at the end :]

[gsmet]

Done here: quarkiverse/quarkiverse-devops#262

I will also adjust the template tomorrow.

[gastaldi]

Don't adjust the template yet, let me create the PR and then we can trigger PRs for all Quarkiverse repositories