nimbus-build-system/README.md

Common parts of the build system used by Nimbus and related projects

We focus on building Nim software on multiple platforms, without having to deal with language-specific package managers.

We care about dependencies specified with commit-level accuracy (including the Nim compiler), reproducible builds, bisectable Git histories and self-contained projects that don't create dirs/files outside their main directory.

We try to minimise complexity, but we will trade implementation complexity increases for a simpler user experience.

Prerequisites

GNU Make, Bash and the usual POSIX utilities.

Usage

Add this repository to your project as a Git submodule. You can use our handy shell script:

curl -OLs https://raw.githubusercontent.com/status-im/nimbus-build-system/master/scripts/add_submodule.sh
less add_submodule.sh # you do read random Internet scripts before running them, right?
chmod 755 add_submodule.sh
./add_submodule.sh status-im/nimbus-build-system

Or you can do it by hand:

git submodule add https://github.com/status-im/nimbus-build-system.git vendor/nimbus-build-system
# specify a branch
git config -f .gitmodules submodule.vendor/nimbus-build-system.branch master
# hide submodule working tree changes in `git diff`
git config -f .gitmodules submodule.vendor/nimbus-build-system.ignore dirty

Write your own top-level Makefile, taking our "Makefile.superproject.example" as an... example.

See also the Makefiles we wrote for Nimbus-eth1, Nimbus-eth2, nim-waku, Stratus, nim-status-client.

Instruct your users to run make update after cloning your project, after a git pull or after switching branches.

Make variables

V

Control the verbosity level. Defaults to 0 for a nice, quiet build.

make V=1 # verbose
make V=2 test # even more verbose

LOG_LEVEL

Set the Chronicles log level to one of: TRACE, DEBUG, INFO, NOTICE, WARN, ERROR, FATAL, NONE.

This Make variable is unset by default, which means that Chronicles' default kicks in (DEBUG in debug builds and INFO in release mode) or some application-specific default takes precedence.

Note that this sets the compile-time log level. If runtime log level selection is implemented (which cannot have larger values than what was set at compile time), additional steps need to be taken to pass the proper command line argument to your binary.

make LOG_LEVEL=DEBUG foo # this is the default
make LOG_LEVEL=TRACE bar # log everything

NIMFLAGS

Pass arbitrary parameters to the Nim compiler. Uses an internal NIM_PARAMS variable that should not be overridden by the user.

make NIMFLAGS="-d:release"

Defaults to Nim parameters mirroring the selected verbosity and log level:

make V=0 # NIMFLAGS="--verbosity:0 --hints:off"
make V=1 LOG_LEVEL=TRACE # NIMFLAGS="--verbosity:1 -d:chronicles_log_level=TRACE"
make V=2 # NIMFLAGS="--verbosity:2"

Projects using this build system may choose to add other default flags like -d:release in their Makefiles (usually those that can't be placed in a top-level "config.nims" or "nim.cfg"). This will be done by appending to the internal variable:

NIM_PARAMS += -d:release

CI_CACHE

Specify a directory where Nim compiler binaries should be cached, in a CI service like AppVeyor:

build_script:
  # the 32-bit build is done on a 64-bit image, so we need to override the architecture
  - mingw32-make -j2 ARCH_OVERRIDE=%PLATFORM% CI_CACHE=NimBinaries update

USE_SYSTEM_NIM

Use the system Nim instead of our shipped version (usually for testing Nim devel versions). Defaults to 0. Setting it to 1 means you're on your own, when it comes to support.

make USE_SYSTEM_NIM=1 test

PARTIAL_STATIC_LINKING

Statically link some libraries (currently libgcc_s and libpcre). Defaults to 0.

This is useful when you can't statically link Glibc because you use NSS functions.

make PARTIAL_STATIC_LINKING=1 beacon_node

LINK_PCRE

Link PCRE, defaults to 1.

make LINK_PCRE=0

QUICK_AND_DIRTY_COMPILER

Skip some Nim compiler bootstrap iterations and tool building. Useful in CI. Defaults to 0.

make QUICK_AND_DIRTY_COMPILER=1 build-nim

QUICK_AND_DIRTY_NIMBLE

Build Nimble when it wouldn't be normally built as part of the tools. Only useful with QUICK_AND_DIRTY_COMPILER=1. Defaults to 0.

make QUICK_AND_DIRTY_COMPILER=1 QUICK_AND_DIRTY_NIMBLE=1 build-nim

NIM_COMMIT

Build and use a different Nim compiler version than the default one.

Possible values: (partial) commit hashes, tags, branches and anything else recognised by git checkout ....

make -j8 NIM_COMMIT="v1.2.6" build-nim

You also need to specify it when using this non-default Nim compiler version:

make -j8 NIM_COMMIT="v1.2.6" nimbus_beacon_node

NIM_COMMIT_REPO

NIM_COMMIT will try to fetch commits from the status-im fork, followed by the official Nim language repo. If you want to use a fork from somewhere else, you can set this to the repo's URL.

make -j8 NIM_COMMIT="4561f01" NIM_COMMIT_REPO="https://github.com/myorg/my-nim-fork"

As before, you will need to specify this again when using this non-default Nim compiler version:

make -j8 NIM_COMMIT="4561f01" NIM_COMMIT_REPO="https://github.com/myorg/my-nim-fork" nimbus_beacon_node

EXCLUDED_NIM_PACKAGES

List of relative paths (incomplete ones also work) to Git submodules that should not end up as Nim packages in "vendor/.nimble" - usually because they duplicate more high-level ones.

For example, say we have:

$ find vendor -name "nim-chronos" | sort
vendor/nim-chronos
vendor/nim-waku/vendor/nim-chronos
vendor/nim-waku/vendor/nim-dnsdisc/vendor/nim-chronos

We only want the top-level "vendor/nim-chronos", so we put the rest in EXCLUDED_NIM_PACKAGES, in the top-level Makefile:

EXCLUDED_NIM_PACKAGES := vendor/nim-waku/vendor/nim-chronos \
			 vendor/nim-waku/vendor/nimbus-build-system \
			 vendor/nim-waku/vendor/nim-dnsdisc/vendor

As you see, we can exclude all those "nim-dnsdisc" submodules with a single line, because the pattern is not anchored during the match.

OVERRIDE

Whether to override any uncommitted changes to Git submodules during make update. Defaults to 1, in order to keep the old behaviour for users building from source.

Set to 0 inside make update-dev, to help developers avoid losing work.

Make targets

build

Internal target that creates the directory with the same name.

sanity-checks

Internal target used to check that a C compiler is installed.

warn-update

Internal target that checks if make update was executed for the current Git commit.

warn-jobs

Internal target that checks if Make's parallelism was enabled by specifying the number of jobs.

deps-common

Internal target that needs to be a dependency for a custom "deps" target which, in turn, will be a dependency for various compilation targets.

The superproject's Makefile would look like this:

deps: | deps-common
	# Have custom deps? Add them above.

# building Nim programs
foo bar: | build deps
	echo -e $(BUILD_MSG) "build/$@" && \
		$(ENV_SCRIPT) nim c -o:build/$@ $(NIM_PARAMS) "$@.nim"

The user should never have to run make deps directly.

build-nim

Internal target that builds the Nim compiler if it's not built yet or if the corresponding submodule points to a newer commit than the existing binary.

It's being executed as part of "update-common" and "$(NIM_BINARY)" targets. It may be executed directly, rarely, when testing new Nim versions.

update-common

Internal target that needs to become the dependency of a custom "update" target.

update: | update-common
	# Do you need to do something extra for this target?

Initialises and updates all Git submodules, with various ugly hacks to account for corner cases like submodules changing URLs or being replaced with regularly committed files.

Tell your users to run make update after cloning the superproject, after a git pull and after changing branches or checking out older commits.

update-dev

Alternative to "update" for developers who want to avoid losing uncommitted work in Git submodules. make update-dev simply runs make OVERRIDE=0 update.

update-remote

Dangerous target that updates all submodules to their latest remote commit.

As you may imagine, it's usually necessary to roll back one or two of these automatic bumps. You do it like this:

git submodule update --recursive vendor/news

clean-cross

Confusingly named target that deletes libnatpmp and miniupnp objects and static libraries. Useful when you're cross-compiling and don't want to make clean and have to rebuild the compiler.

clean-common

Internal target that needs to be a dependency for a custom "clean" target that deletes any additional build artefacts:

clean: | clean-common
	rm -rf build/{foo,bar}

Don't run make clean if you don't really need to, since it also deletes the Nim compiler.

Unlike C/C++ projects, we always recompile our Nim targets (because it's too hard to tell Make what are all the files involved in the build process), so there's no need to delete them to force a recompilation.

mrproper

Dangerous target that, in addition to depending on "clean", deletes the "vendor" directory and any not-yet-pushed modification you may have in there. Don't use it.

github-ssh

Changes submodule URLs, without affecting .gitmodules, so you connect to GitHub using your SSH key - very useful when you have write access to some submodule repos and you want to work on them without cloning them separately.

status

Run git status in all your submodules and in your superproject.

ctags

Run Universal Ctags with a bunch of Nim-specific options.

show-deps

List all Git submodules, including the nested ones.

fetch-dlls

Windows-specific target. Downloads and unpacks in the "build" dir some DLLs we may not want to build ourselves (PCRE, RocksDB, libcurl, pdcurses, SQLite3, OpenSSL, zlib, etc.).

nat-libs

Build "libminiupnpc.a" and "libnatpmp.a" from nim-nat-traversal's submodules - not included in this repo.

Scripts

add_submodule.sh

Add a new Git submodule to your superproject, setting the branch to "master" and hiding submodule working tree changes in git diff.

Usage: ./add_submodule.sh some/repo [destdir] # "destdir" defaults to "vendor/repo"

Examples:

./add_submodule.sh status-im/nimbus-build-system

./vendor/nimbus-build-system/scripts/add_submodule.sh status-im/nim-nat-traversal

build_nim.sh

Build the Nim compiler and some associated tools, parallelising the bootstrap-from-C phase, with binary caching and conditional recompilation support (based on the timestamp of the latest commit in the Nim repo).

Usage: ./build_nim.sh nim_dir csources_dir nimble_dir ci_cache_dir

This script is not usually used directly, but through the update target (which depends on update-common which runs build-nim).

Our build-nim target uses it like this:

build-nim: | sanity-checks
	+ NIM_BUILD_MSG="$(BUILD_MSG) Nim compiler" \
		V=$(V) \
		CC=$(CC) \
		MAKE=$(MAKE) \
		ARCH_OVERRIDE=$(ARCH_OVERRIDE) \
		"$(CURDIR)/$(BUILD_SYSTEM_DIR)/scripts/build_nim.sh" "$(NIM_DIR)" ../Nim-csources ../nimble "$(CI_CACHE)"

Other Nim projects that don't use this build system use the script directly in their CI. From a ".travis.yml":

install:
  - curl -O -L -s -S https://raw.githubusercontent.com/status-im/nimbus-build-system/master/scripts/build_nim.sh
  - env MAKE="make -j${NPROC}" bash build_nim.sh Nim csources dist/nimble NimBinaries
  - export PATH="$PWD/Nim/bin:$PATH"

Or an ".appveyor.yml":

install:
  # [...]
  - curl -O -L -s -S https://raw.githubusercontent.com/status-im/nimbus-build-system/master/scripts/build_nim.sh
  - env MAKE="mingw32-make -j2" ARCH_OVERRIDE=%PLATFORM% bash build_nim.sh Nim csources dist/nimble NimBinaries
  - SET PATH=%CD%\Nim\bin;%PATH%

Notice how the number of Make jobs is set through the "MAKE" env var.

build_rocksdb.sh

Builds RocksDB. No longer used.

Usage: ./build_rocksdb.sh ci_cache_dir

create_nimble_link.sh

Cheeky little script used to fake a Nimble package repository in the $(NIMBLE_DIR) target (a dependency of deps-common which is a dependency of deps):

$(NIMBLE_DIR):
	mkdir -p $(NIMBLE_DIR)/pkgs
	NIMBLE_DIR="$(CURDIR)/$(NIMBLE_DIR)" PWD_CMD="$(PWD)" \
		git submodule foreach --quiet '$(CURDIR)/$(BUILD_SYSTEM_DIR)/scripts/create_nimble_link.sh "$$sm_path"'

That's how the Nim compiler knows how to find all these Nim packages we have in our submodules: we set the "NIMBLE_DIR" env var to the path of this fake Nimble package repo.

env.sh

Script responsible for setting up environment variables and shell aliases necessary for using this build system. It's being sourced by a script with the same name in the superproject's top directory:

#!/usr/bin/env bash

# We use ${BASH_SOURCE[0]} instead of $0 to allow sourcing this file
# and we fall back to a Zsh-specific special var to also support Zsh.
REL_PATH="$(dirname ${BASH_SOURCE[0]:-${(%):-%x}})"
ABS_PATH="$(cd ${REL_PATH}; pwd)"
source ${ABS_PATH}/vendor/nimbus-build-system/scripts/env.sh

Supported usage: ./env.sh nim --version

Unsupported usage: source env.sh; nim --version

An interesting alias is nimble which calls the "nimble.sh" script which pretends to be Nimble:

cd vendor/nim-metrics
../../env.sh nimble test

In order to get autocompletion and code navigation functionality working properly in your text editor, you need the environment variables set by this script:

./env.sh vim

nimble.sh

Simple script that symlinks the first *.nimble file it finds to *.nims and runs it using nim. Easier to access using the nimble alias in "env.sh".

Of very limited use, it can execute *.nimble targets, as long as there are no "before install:" blocks that the real Nimble strips before doing the same thing we do.

If you need the real Nimble, it's in "vendor/nimbus-build-system/vendor/Nim/bin/nimble".

License

Licensed and distributed under either of

• MIT license: LICENSE-MIT or http://opensource.org/licenses/MIT

or

• Apache License, Version 2.0, (LICENSE-APACHEv2 or http://www.apache.org/licenses/LICENSE-2.0)

at your option. These files may not be copied, modified, or distributed except according to those terms.