= Vulkan^(R)^ Specification Build Instructions and Notes :toc2: :toclevels: 1 [[intro]] == Introduction This README describes important stuff for getting the Vulkan API specification and reference pages building properly. [[building]] == Building The Spec Once you have all the right tools installed (see <> below), go to `...path-to-git-repo/doc/specs/vulkan` . $ make all or make the individual targets `html`, `pdf`, `styleguide`, `manhtml`, `manpdf`, `manhtmlpages`, `checkinc`, and `checklinks`. will generate a variety of targets in the directory specified by the Makefile variable `$(OUTDIR)` (by default, `../../../out/1.0`). The checked-in file `../../../out/1.0/index.html` links to all these targets, or they can individually be found as follows: * API spec: ** `html` - Single-file HTML5 in `$(OUTDIR)/html/vkspec.html` ** `pdf` - PDF in `$(OUTDIR)/pdf/vkspec.pdf` * "`Vulkan Documentation and Extensions`" guide: ** `styleguide` - Single-file HTML5 in `$(OUTDIR)/styleguide.html` * Reference pages: ** `manhtml` - Single-file HTML in `$(OUTDIR)/apispec.html` ** `manpdf` - Single-file PDF in `$(OUTDIR)/apispec.html` ** `manhtmlpages` - File-per-entry-point HTML in `$(OUTDIR)/man/html/*` * Validator output: ** `checkinc` - List of commands, structs, etc. missing from the API spec in `$(OUTDIR)/checks/notInSpec.txt` ** `checklinks` - Validator script output for API spec in `$(OUTDIR)/checks/specErrs.txt` and for reference pages in `$(OUTDIR)/checks/manErrs.txt` Once you have the basic build working, an appropriate parallelization option to make, such as ---- make -j 8 ---- may significantly speed up the reference page builds. [[building-extensions]] === Building With Extensions Included We now use a "`single-branch model`", where extensions are included in the same `1.0` branch as the core Specification, instead of the older model where each extension lived in a separate Git branch. Whether a given extension is generated in the output depends on asciidoc and generator script options being specified. The extensions included are those specified as a space-separated list of extension names (e.g. `VK_KHR_surface`) in the Makefile variable `$(EXTENSIONS)`, normally set on the make command line. When changing the list of extensions, it is critical to remove all generated files using the `clean_generated` Makefile target, as the contents of generated files depends on `$(EXTENSIONS)`. There are two helper scripts which clean these files and then build one or more specified targets for specified extensions: * `makeExt` - generate outputs with a single extension enabled. Usage is `makeExt extension-name target(s)`, where `extension-name` is the extension name string, such as `VK_EXT_debug_report`. * `makeKHR` - generate outputs with all Khronos (`VK_KHR_*`) extensions enabled. Usage is `makeKHR target(s)`. * `makeKHRAndKHX` - generate outputs with all Khronos (`VK_KHR_*`) and Khronox Experimental (`VK_KHX_*`) extensions enabled. Usage is `makeKHRAndKHX target(s)`. The Makefile variable `$(APITITLE)` defines an additional string which is appended to the specification title. When building with extensions enabled, this should be set to something like `(with extension VK_extension_name)`. The `makeExt`, `makeKHR`, and `makeKHRAndKHX` scripts already do this. [[building-test]] === Alternate and Test Builds If you are just testing asciidoc formatting, macros, stylesheets, etc., you may want to edit `vkspec.txt` to just include your test code. The asciidoctor HTML build is very fast, even for the whole Specification, but PDF builds take several minutes. === Rebuilding The Generated Images There are some images in the `images/` directory which are maintained in one format but need to be converted to another format for corresponding types of output. Most are SVG converted to PDF, some are PPT converted to PDF converted to SVG. SVG are needed by all builds. These files are not automatically converted by the Makefile. Instead, all output forms required are checked into `images/` . On the rare occasions that someone changes a source document and needs to regenerate the other forms: ---- cd images ; make ---- === Validation Scripts There are a several Makefile targets which look for inconsistencies and missing material between the specification and ref pages, and the canonical description of the API in `vk.xml` : * `checkinc` * `checklinks` * `allchecks` - both `checkinc` and `checklinks` They are necessarily heuristic since they're dealing with lots of hand-written material. To use them you'll also need to install: * `python3` The `checkinc` target uses Unix filters to determine which autogenerated API include files are used (and not used) in the spec. It generates several output files, but the only one you're likely to care about is `actual.only`. This is a list of the include files which are *not* referenced anywhere in the spec, and probably correspond to undocumented material in the spec. The `checklinks` target validates the various internal tagged links in the man pages and spec (e.g. the `fname:vkFuncBlah`, `sname:VkStructBlah`, etc.) against the canonical description of the API in `vk.xml`. It generates two output files, `manErrs.txt` and `specErrs.txt`, which report problematic tags and the filenames/lines on which those tags were found. [[macros]] == Our Asciidoc Macros We use a bunch of custom macros in the reference pages and API spec asciidoc sources. The validator scripts rely on these macros as part of their sanity checks, and you should use the macros whenever referring to an API command, struct, token, or enum name, so the documents are semantically tagged and more easily verifiable. The supported macros are defined in the `config/vulkan-macros/extension.rb` asciidoctor extension script. The tags used are described in the style guide (`styleguide.txt`). We (may) eventually tool up the spec and ref pages to the point that anywhere there's a type or token referred to, clicking on (or perhaps hovering over) it in the HTML view and be taken to the definition of that type/token. That will take some more plumbing work to tag the stuff in the autogenerated include files, and do something sensible in the spec (e.g. resolve links to internal references). Most of these macros deeply need more intuitive names. [[refpages]] == Reference Pages The reference pages are extracted from the API Specification source, which has been tagged to help identify boundaries of language talking about different commands, structures, enumerants, and other types. A set of Python scripts extract and lightly massage the relevant tagged language into corresponding ref page. Pages without corresponding content in the API spec are generated automatically, when possible (e.g. for `Vk*FlagBits` pages). If for some reason you want to regenerate the ref pages from scratch yourself, you can do so by ---- rm man/apispec.txt make apispec.txt ---- The `genRef.py` script will generate many warnings, but most are just reminders that some pages are automatically generated. If everything is working correctly, all the `man/*.txt` files will be regenerated, but their contents will not change. If you add new API features to the Specification in a branch, make sure that the commands have the required tagging and that ref pages are generated for them, and build properly. [[styles]] == Our stylesheets NOTE: Section mostly TBD. We use the default Asciidoctor stylesheet. === Marking Normative Language Normative language is marked as *bold*, and also with the [purple]#purple# role for html output. It can be used to mark entire paragraphs or spans of words. In addition, the normative terminology macros, such as must: and may: and cannot:, always use this role. The formatting of normative language depends on the stylesheet. Currently it just comes out in purple. We may add a way to disable this formatting at build time. [[equations]] == Imbedding Equations Where possible, equations should be written using straight asciidoc markup using the _eq_ role. This covers many common equations and is faster than the alternatives. For more complex equations, such as multi-case statements, matrices, and complex fractions, equations should be written using the latexmath: inline and block macros. The contents of the latexmath: blocks should be LaTeX math notation. LaTeX math markup delimiters are now inserted by the asciidoctor toolchain. LaTeX math is passed through unmodified to all HTML output forms, which is subsequently rendered with the KaTeX engine when the html is loaded. A local copy of the KaTeX release is kept in `doc/specs/vulkan/katex` and copied to the HTML output directory during spec generation. Math is processed into SVGs via asciidoctor-mathematical for PDF output. The following caveats apply: * The special characters `<` , `>` , and `&` can currently be used only in +++[latexmath]+++ block macros, not in +++latexmath:[]+++ inline macros. Instead use `\lt`, `\leq`, `\gt`, and `\geq` for `<`, `<=`, `>`, and `>=` respectively. `&` is an alignment construct for multiline equations, and should only appear in block macros anyway. * AMSmath environments (e.g. pass:[\begin{equation*}], pass:[{align*}], etc.) cannot be used in KaTeX at present, and have been replaced with constructs supported by KaTeX such as pass:[{aligned}]. * Arbitrary LaTeX constructs cannot be used. KaTeX and asciidoctor-mathematical are only equation renderers, not full LaTeX engines. Imbedding LaTeX like \Large or pass:[\hbox{\tt\small VK\_FOO}] may not work in any of the backends, and should be avoided. See the "`Vulkan Documentation and Extensions`" document for more details of supported LaTeX math constructs. [[anchors]] == Asciidoc Anchors And Xrefs In the API spec, sections can have anchors (labels) applied with the following syntax. In general the anchor should immediately precede the chapter or section title and should use the form '+++[[chapter-section-label]]+++'. For example, For example, in chapter +synchronization.txt+: ---- [[synchronization-primitives]] Synchronization Primitives ---- Cross-references to those anchors can then be generated with, for example, ---- See the <> section for discussion of fences, semaphores, and events. ---- You can also add anchors on arbitrary paragraphs, using a similar naming scheme. Anything whose definition comes from one of the autogenerated API include files (`.txt` files in the directories `basetypes`, `enums`, `flags`, `funcpointers`, `handles`, `protos`, and `structs`) has a corresponding anchor whose name is the name of the function, struct, etc. being defined. Therefore you can say something like: ---- Fences are used with the +++<>+++ command... ---- [[depends]] == Software Dependencies This section describes the software components used by the Vulkan spec toolchain. Before building the Vulkan spec, you must install the following tools: * GNU make (make version: 4.0.8-1; older versions probably OK) * Python 3 (python, version: 3.4.2) * Ruby (ruby, version: 2.3.3) ** The Ruby development package (ruby-dev) may also be required in some environments. * Git command-line client (git, version: 2.1.4). The build can progress without a git client, but branch/commit information will be omitted from the build. Any version supporting the following operations should work: ** `git symbolic-ref --short HEAD` ** `git log -1 --format="%H"` * Ghostscript (ghostscript, version: 9.10). This is for the PDF build, and it can still progress without it. Ghostscript is used to optimize the size of the PDF, so will be a lot smaller if it is included. The following Ruby Gems and platform package dependencies must also be installed. This process is described in more detail for individual platforms and environment managers below. Please read the remainder of this document (other than platform-specific parts you don't use) completely before trying to install. * Asciidoctor (asciidoctor, version: 1.5.5) * Coderay (coderay, version 1.1.1) * Asciidoctor PDF (asciidoctor-pdf, version: 1.5.0.alpha13) * Asciidoctor Mathematical (asciidoctor-mathematical, version 0.2.2) * https://github.com/asciidoctor/asciidoctor-mathematical#dependencies[Dependencies for asciidoctor-mathematical] (There are a lot of these!) * KaTeX distribution (version 0.7.0 from https://github.com/Khan/KaTeX . This is cached under `doc/specs/vulkan/katex/`, and need not be installed from github. Only the `asciidoctor` and `coderay` gems is needed if you don't intend to build PDF versions of the spec and supporting documents. [NOTE] .Note ==== While it's easier to install just the toolchain components for HTML builds, people submitting MRs with substantial changes to the Specification are responsible for verifying that their branches build *both* `html` and `pdf` targets. ==== Platform-specific toolchain instructions follow: * Microsoft Windows ** <> ** <> (PDF builds not tested) ** <> * <> * <> [[depends-windows]] === Windows (General) Most of the dependencies on Linux packages are light enough that it's possible to build the spec natively in Windows, but it means bypassing the makefile and calling functions directly. This might be solved in future. For now, there are three options for Windows users: Ubuntu / Windows 10, MinGW, or Cygwin. [[depends-ubuntu]] ==== Ubuntu / Windows 10 When using the "`Ubuntu Subsystem`" for Windows 10, most dependencies can be installed via apt-get: ---- sudo apt-get -qq -y install build-essential python3 git cmake bison flex \ libffi-dev libxml2-dev libgdk-pixbuf2.0-dev libcairo2-dev \ libpango1.0-dev ttf-lyx gtk-doc-tools ghostscript ---- The default ruby packages on Ubuntu are fairly out of date. Ubuntu only provides `ruby` and `ruby2.0` - the latter is multiple revisions behind the current stable branch, and would require wrangling to get the makefile working with it. Luckily, there are better options; either https://rvm.io[rvm] or https://github.com/rbenv/rbenv[rbenv] is recommended to install a more recent version. [NOTE] .Note ==== * If you are new to Ruby, you should *completely remove* (through the package manager, e.g. `sudo apt-get remove *packagename*`) all existing Ruby and asciidoctor infrastructure on your machine before trying to use rvm or rbenv for the first time. `dpkg -l | egrep 'asciidoctor|ruby|rbenv|rvm'` will give you a list of candidate package names to remove. ** If you already have a favorite Ruby package manager, ignore this advice, and just install the required OS packages and gems. * In addition, `rvm` and `rbenv` are *mutually incompatible*. They both rely on inserting shims and `$PATH` modifications in your bash shell. If you already have one of these installed and are familiar with it, it's probably best to stay with that one. One of the editors, who is new to Ruby, found `rbenv` far more comprehensible than `rvm`. The other editor likes `rvm` better. ** Neither `rvm` nor `rbenv` work, out of the box, when invoked from non-Bash shells like tcsh. This can be hacked up by setting the right environment variables and PATH additions based on a bash environment. * Most of the tools on Bash for Windows are quite happy with Windows line endings (CR LF), but bash scripts expect Unix line endings (LF). The file `.gitattributes` at the top of the vulkan tree in the 1.0 branch forces such scripts to be checked out with the proper line endings on non-Linux platforms. If you add new scripts whose names don't end in `.sh`, they should be included in .gitattributes as well. ==== [[depends-ubuntu-rbenv]] ===== Ubuntu/Windows 10 Using Rbenv Rbenv is a lighter-weight Ruby environment manager with less functionality than rvm. Its primary task is to manage different Ruby versions, while rvm has additional functionality such as managing "`gemsets`" that is irrelevant to our needs. A complete installation script for the toolchain on Ubuntu for Windows, developed on an essentially out-of-the-box environment, follows. If you try this, don't try to execute the entire thing at once. Do each step separately in case of errors we didn't encounter. ---- # Install packages needed by `ruby_build` and by toolchain components. # See https://github.com/rbenv/ruby-build/wiki and # https://github.com/asciidoctor/asciidoctor-mathematical#dependencies sudo apt-get install autoconf bison build-essential libssl-dev \ libyaml-dev libreadline6-dev zlib1g-dev libncurses5-dev \ libffi-dev libgdbm3 libgdbm-dev install cmake libxml2 \ libxml2-dev flex pkg-config install libglib2.0-dev \ libcairo-dev libpango1.0-dev libgdk-pixbuf2.0-dev \ libpangocairo-1.0 # Install rbenv from https://github.com/rbenv/rbenv git clone https://github.com/rbenv/rbenv.git ~/.rbenv # Set path to shim layers in .bashrc echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> .bashrc ~/.rbenv/bin/rbenv init # Set .rbenv environment variables in .bashrc echo 'eval "$(rbenv init -)"' >> .bashrc # Restart your shell (e.g. open a new terminal window). Note that # you do not need to use the `-l` option, since the modifications # were made to .bashrc rather than .bash_profile. If successful, # `type rbenv` should print 'rbenv is a function' followed by code. # Install `ruby_build` plugin from https://github.com/rbenv/ruby-build git clone https://github.com/rbenv/ruby-build.git ~/.rbenv/plugins/ruby-build # Install Ruby 2.3.3 # This takes in excess of 20 min. to build! # https://github.com/rbenv/ruby-build/issues/1054#issuecomment-276934761 # suggests: # "You can speed up Ruby installs by avoiding generating ri/RDoc # documentation for them: # RUBY_CONFIGURE_OPTS=--disable-install-doc rbenv install 2.3.3 # We have not tried this. rbenv install 2.3.3 # Configure rbenv globally to always use Ruby 2.3.3. echo "2.3.3" > ~/.rbenv/version # Finally, install toolchain components. # asciidoctor-mathematical also takes in excess of 20 min. to build! # The same RUBY_CONFIGURE_OPTS advice above may apply here as well. gem install asciidoctor coderay gem install --pre asciidoctor-pdf MATHEMATICAL_SKIP_STRDUP=1 gem install asciidoctor-mathematical ---- [[depends-ubuntu-rvm]] ===== Ubuntu/Windows 10 Using RVM Here are (sparser) instructions for using rvm to setup version 2.3.x: ---- gpg --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 \curl -sSL https://get.rvm.io | bash -s stable --ruby source ~/.rvm/scripts/rvm rvm install ruby-2.3 rvm use ruby-2.3 ---- NOTE: Windows 10 Bash will need to be launched with the "-l" option appended, so that it runs a login shell; otherwise RVM won't function correctly on future launches. [[depends-ubuntu-sys]] ===== Ubuntu 16.04 using system Ruby The Ubuntu 16.04.1 default Ruby install (version 2.3.1) seems to be up-to-date enough to run all the required gems, but also needs the `ruby-dev` package installed through the package manager. In addition, the library `/var/lib/gems/2.3.0/gems/mathematical-1.6.7/ext/mathematical/lib/liblasem.so` has to be copied or linked into a directory where the loader can find it. This requirement appears to be due to a problem with the asciidoctor-mathematical build process. [[depends-mingw]] ==== MinGW MinGW can be obtained here: http://www.mingw.org/ Once the installer has run its initial setup, following the http://www.mingw.org/wiki/Getting_Started[instructions on the website], you should install the `mingw-developer-tools`, `mingw-base` and `msys-base` packages. The `msys-base` package allows you to use a bash terminal from windows with whatever is normally in your path on Windows, as well as the unix tools installed by MinGW. In the native Windows environment, you should also install the following native packages: * Python 3.x (https://www.python.org/downloads/) * Ruby 2.x (https://rubyinstaller.org/) * Git command-line client (https://git-scm.com/download) Once this is setup, and the necessary <> are installed, launch the `msys` bash shell, and navigate to the spec Makefile. From there, you'll need to set `PYTHON=` to the location of your python executable for version 3.x before your make command - but otherwise everything other than pdf builds should just work. NOTE: Building the PDF spec via this path has not yet been tested but *may* be possible - liblasem is the main issue and it looks like there is now a mingw32 build of it available. [[depends-cygwin]] ==== Cygwin When installing Cygwin, you should install the following packages via `setup`: ---- // "curl" is only used to download fonts, can be done in another way autoconf bison cmake curl flex gcc-core gcc-g++ ghostscript git libbz2-devel libcairo-devel libcairo2 libffi-devel libgdk_pixbuf2.0-devel libiconv libiconv-devel liblasem0.4-devel libpango1.0-devel libpango1.0_0 libxml2 libxml2-devel make python3 ruby ruby-devel ---- NOTE: Native versions of some of these packages are usable, but care should be taken for incompatibilities with various parts of cygwin - e.g. paths. Ruby in particular is unable to resolve Windows paths correctly via the native version. Python and Git for Windows can be used, though for Python you'll need to set the path to it via the PYTHON environment variable, before calling make. When it comes to installing the mathematical ruby gem, there are two things that will require tweaking to get it working. Firstly, instead of: ---- MATHEMATICAL_SKIP_STRDUP=1 gem install asciidoctor-mathematical ---- You should use ---- MATHEMATICAL_USE_SYSTEM_LASEM=1 gem install asciidoctor-mathematical ---- The latter causes it to use the lasem package already installed, rather than trying to build a fresh one. The mathematical gem also looks for "liblasem" rather than "liblasem0.4" as installed by the lasem0.4-devel package, so it is necessary to add a symlink to your /lib directory using: ---- ln -s /lib/liblasem-0.4.dll.a /lib/liblasem.dll.a ---- <> are not installed to a location that is in your path normally. Gems are installed to `~/bin/` - you should add this to your path before calling make: export PATH=~/bin:$PATH Finally, you'll need to manually install fonts for lasem via the following commands: ---- mkdir /usr/share/fonts/truetype cd /usr/share/fonts/truetype curl -LO http://mirrors.ctan.org/fonts/cm/ps-type1/bakoma/ttf/cmex10.ttf \ -LO http://mirrors.ctan.org/fonts/cm/ps-type1/bakoma/ttf/cmmi10.ttf \ -LO http://mirrors.ctan.org/fonts/cm/ps-type1/bakoma/ttf/cmr10.ttf \ -LO http://mirrors.ctan.org/fonts/cm/ps-type1/bakoma/ttf/cmsy10.ttf \ -LO http://mirrors.ctan.org/fonts/cm/ps-type1/bakoma/ttf/esint10.ttf \ -LO http://mirrors.ctan.org/fonts/cm/ps-type1/bakoma/ttf/eufm10.ttf \ -LO http://mirrors.ctan.org/fonts/cm/ps-type1/bakoma/ttf/msam10.ttf \ -LO http://mirrors.ctan.org/fonts/cm/ps-type1/bakoma/ttf/msbm10.ttf ---- [[depends-osx]] === Mac OS X Mac OS X should work in the same way as for ubuntu by using the Homebrew package manager, with the exception that you can simply install the ruby package via `brew` rather than using a ruby-specific version manager. You'll likely also need to install additional fonts for the PDF build via mathematical, which you can do with: ---- cd ~/Library/Fonts curl -LO http://mirrors.ctan.org/fonts/cm/ps-type1/bakoma/ttf/cmex10.ttf \ -LO http://mirrors.ctan.org/fonts/cm/ps-type1/bakoma/ttf/cmmi10.ttf \ -LO http://mirrors.ctan.org/fonts/cm/ps-type1/bakoma/ttf/cmr10.ttf \ -LO http://mirrors.ctan.org/fonts/cm/ps-type1/bakoma/ttf/cmsy10.ttf \ -LO http://mirrors.ctan.org/fonts/cm/ps-type1/bakoma/ttf/esint10.ttf \ -LO http://mirrors.ctan.org/fonts/cm/ps-type1/bakoma/ttf/eufm10.ttf \ -LO http://mirrors.ctan.org/fonts/cm/ps-type1/bakoma/ttf/msam10.ttf \ -LO http://mirrors.ctan.org/fonts/cm/ps-type1/bakoma/ttf/msbm10.ttf ---- Then install the required <>. [[depends-linux]] === Linux (Debian, Ubuntu, etc.) The instructions for the <> installation are generally applicable to native Linux environments using Debian packages, such as Debian and Ubuntu, although the exact list of packages to install may differ. Other distributions using different package managers, such as RPM (Fedora) and Yum (SuSE) will have different requirements. Using `rbenv` or `rvm` is neccessary, since the system Ruby packages are often well out of date. Once the environment manager, Ruby, and `ruby_build` have been installed, install the required <>. [[depends-gems]] === Ruby Gems The following ruby gems can be installed directly via the `gem install` command, once the platform is set up: ---- gem install rake asciidoctor coderay # Required only for pdf builds MATHEMATICAL_SKIP_STRDUP=1 gem install asciidoctor-mathematical gem install --pre asciidoctor-pdf ---- [[history]] == Revision History * 2017-02-13 - Move some comments here from ../../../README.md. Tweak asciidoctor markup to more clearly delineate shell command blocks. * 2017-02-10 - Add more Ruby installation guidelines and reflow the document in accordance with the style guide. * 2017-01-31 - Add rbenv instructions and update the README elsewhere. * 2017-01-16 - Modified dependencies for Asciidoctor * 2017-01-06 - Replace MathJax with KaTeX. * 2016-08-25 - Update for the single-branch model. * 2016-07-10 - Update for current state of spec and ref page generation. * 2015-11-11 - Add new can: etc. macros and DBLATEXPREFIX variable. * 2015-09-21 - Convert document to asciidoc and rename to README.md in the hope the gitlab browser will render it in some fashion. * 2015-09-21 - Add descriptions of LaTeX and MathJax math support for all output formats. * 2015-09-02 - Added Cygwin package info. * 2015-09-02 - Initial version documenting macros, required toolchain components and versions, etc.