Vulkan-Docs/refPageNotes.md

Notes on automatic reference page generation

Main takeaway point: ref page generation from the spec sources is ready to go. It can be tweaked in lots of small ways, but as it stands it's a huge improvement over the current ref pages and should be adopted ASAP.

Approach

This is done with some Python scripts that rely on the rigid structure we've enforced on command and structure definitions in the spec. Everything from the paragraph preceding the protos/structs include:: through the validity:: include is pulled out and then split up into ref page sections. In some cases, it isn't immediately obvious how to reorder to get all the content relevant to a single command or struct blocked together in this fashion, and some simplistic semantic tagging has been added for those cases where the structural organization isn't appropriate.

The changes to the spec are straightforward, though they appear large in some places. What's happening is:

• Validity includes are moved below all the relevant text.
• Structure definitions are moved after the validity includes for commands, so they don't overlap or imbed. This has been done for enumerant definitions in some cases, but those are harder to unwind.
• Blank lines are being removed from parameter / member definition bullet lists, so they don't confuse the extraction script.
• Annotations in the form of asciidoc comments were added where they were needed to assist the extraction scripts. These look like
  // refBegin vkSomething short description of command for ref page title
  // refEnd vkSomething [list of related ref pages]
• A bunch of small consistency edits were done to match the style guide. These (mostly) aren't directly related to the extraction process, but it is an opportune time to do them.
• Pages that don't have extractable descriptive text in the specification, such as VkFlags and some VkFlagBits types, get automatically generated ref pages. These just define the interface and link to other pages using that type.

Things to be done

These are (more or less) in decreasing priority order. Few of them are difficult.

• Something is messed up with the individual page generation targets in the Makefile, so the genRef.py action isn't a dependency of $(MANSOURCES) yet. See https://www.cmcrossroads.com/article/rules-multiple-outputs-gnu-make?page=0%2C2
• Cross-page macro links are broken in the individual ref pages, due to using the wrong macros, but are OK in the single-page apispec.html. Should be easily fixed.
• manpdf (single-page PDF ref document) target is currently broken.
• Turn handle references from sname: to slink: so they link to the right place. Consider using an hlink/hname macro instead. Similarly for basetype: (which doesn't link, but could now with blink:/btext:). In general, most of the ?name: macros can and should turn into ?link:, aside from those directly in the section describing that name.
• Need to tweak asciidoc macros for man pages (manpages.conf).
• "See Also" links are automatically generated based on the type dependencies between pages. This works well, but doesn't include logical groupings (all clear commands, all drawing commands, etc.). These can be added to the explicit crosslinks in // refEnd statements if desired.
• Remove autogenerated files from git and enhance the Makefiles to do all the scripting work required to generate the ref page sources, vkapi.py, etc. at build time.
• Some problems with a2x not finding NOTE icons from the asciidoc distribution (worked around by copying into the man/images directory, for now). It would sure be nice if the build were smart enough to use the icons distributed with asciidoc, instead of needing separate copies.
• I'm not sure how to best deal with KHR extension ref pages. We could generate them separately in their branches and combine with core pages, or generate everything in the wsi_extensions branch - but that will leave us with core ref pages including WSI extension material, without any obvious indication of why or where. Changes for the KHR extension branches will be small once the 1.0 changes are merged in, and I'll undertake to do those once the core changes are accepted.
• Cross-page asciidoc links are broken. This is hard to fix. Probably the best thing to do is strip <<>> links from the source, or perhaps generate an HTML link to the live spec.
• VkResult page needs a better "See Also" section - but not the full list of functions returning VkResult.
• Short descriptions are currently embedded in the spec source, and are missing in some cases. Would it make more sense to add them to vk.xml?