diff --git a/README.md b/README.md index 47295d2..b88e503 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # Common parts of the build system used by Nimbus and related projects -We focus on building Nim software on multiple platforms without having to deal +We focus on building Nim software on multiple platforms, without having to deal with language-specific package managers. We care about dependencies specified with commit-level accuracy (including the @@ -31,7 +31,7 @@ Or you can do it by hand: git submodule add https://github.com/status-im/nimbus-build-system.git vendor/nimbus-build-system # specify a branch git config -f .gitmodules submodule.vendor/nimbus-build-system.branch master -# hide submodule work tree changes in `git diff` +# hide submodule working tree changes in `git diff` git config -f .gitmodules submodule.vendor/nimbus-build-system.ignore dirty ``` @@ -48,7 +48,7 @@ See also the Makefiles we wrote for Instruct your users to run `make update` after cloning your project, after a `git pull` or after switching branches. -## Make flags +## Make variables ### V @@ -225,7 +225,139 @@ List all Git submodules, including the nested ones. ### fetch-dlls -Windows-specific target. Downloads and unpacks in the "build" dir some DLLs we may not want to build ourselves (PCRE, RocksDB, libcurl, pdcurses, Snappy, SQLite3, SSLeay, etc.). +Windows-specific target. Downloads and unpacks in the "build" dir some DLLs we may not want to build ourselves (PCRE, RocksDB, libcurl, pdcurses, SQLite3, OpenSSL, zlib, etc.). + +## Scripts + +### add_submodule.sh + +Add a new Git submodule to your superproject, setting the branch to "master" +and hiding submodule working tree changes in `git diff`. + +Usage: `./add_submodule.sh some/repo [destdir]` # 'destdir' defaults to 'vendor/repo' + +Examples: + +`./add_submodule.sh status-im/nimbus-build-system` + +`./vendor/nimbus-build-system/scripts/add_submodule.sh status-im/nim-nat-traversal` + +### build_nim.sh + +Build the Nim compiler and some associated tools, parallelising the +bootstrap-from-C phase, with binary caching and conditional recompilation +support (based on the timestamp of the latest commit in the Nim repo). + +Usage: `./build_nim.sh nim_dir csources_dir nimble_dir ci_cache_dir` + +This script is not usually used directly, but through the `update` target +(which depends on `update-common` which runs `build-nim`). + +Our `build-nim` target uses it like this: + +```make +build-nim: | sanity-checks + + NIM_BUILD_MSG="$(BUILD_MSG) Nim compiler" \ + V=$(V) \ + CC=$(CC) \ + MAKE=$(MAKE) \ + ARCH_OVERRIDE=$(ARCH_OVERRIDE) \ + "$(CURDIR)/$(BUILD_SYSTEM_DIR)/scripts/build_nim.sh" "$(NIM_DIR)" ../Nim-csources ../nimble "$(CI_CACHE)" +``` + +Other Nim projects that don't use this build system use the script directly in their CI. From a ".travis.yml": + +```yaml +install: + - curl -O -L -s -S https://raw.githubusercontent.com/status-im/nimbus-build-system/master/scripts/build_nim.sh + - env MAKE="make -j${NPROC}" bash build_nim.sh Nim csources dist/nimble NimBinaries + - export PATH="$PWD/Nim/bin:$PATH" +``` + +Or an ".appveyor.yml": + +```yaml +install: + # [...] + - curl -O -L -s -S https://raw.githubusercontent.com/status-im/nimbus-build-system/master/scripts/build_nim.sh + - env MAKE="mingw32-make -j2" ARCH_OVERRIDE=%PLATFORM% bash build_nim.sh Nim csources dist/nimble NimBinaries + - SET PATH=%CD%\Nim\bin;%PATH% +``` + +Notice how the number of Make jobs is set through the "MAKE" env var. + +### build_p2pd.sh + +Builds the "p2pd" Go daemon. No longer used by a Make target, but needed by +other projects that run it directly in their CI config files, like this: + +```yaml +install: + # [...] + # install and build go-libp2p-daemon + - curl -O -L -s -S https://raw.githubusercontent.com/status-im/nimbus-build-system/master/scripts/build_p2pd.sh + - bash build_p2pd.sh p2pdCache v0.2.1 +``` + +### build_rocksdb.sh + +Builds RocksDB. No longer used. + +Usage: `./build_rocksdb.sh ci_cache_dir` + +### create_nimble_link.sh + +Cheeky little script used to fake a Nimble package repository in the +`$(NIMBLE_DIR)` target (a dependency of `deps-common` which is a dependency of +`deps`): + +```make +$(NIMBLE_DIR): + mkdir -p $(NIMBLE_DIR)/pkgs + NIMBLE_DIR="$(CURDIR)/$(NIMBLE_DIR)" PWD_CMD="$(PWD)" \ + git submodule foreach --quiet '$(CURDIR)/$(BUILD_SYSTEM_DIR)/scripts/create_nimble_link.sh "$$sm_path"' +``` + +That's how the Nim compiler knows how to find all these Nim packages we have in +our submodules: we set the "NIMBLE\_DIR" env var to the path of this fake +Nimble package repo. + +### env.sh + +Script responsible for setting up environment variables and shell aliases +necessary for using this build system. It's being sourced by a script with the +same name in the superproject's top directory: + +```bash +#!/bin/bash + +# We use ${BASH_SOURCE[0]} instead of $0 to allow sourcing this file +# and we fall back to a Zsh-specific special var to also support Zsh. +REL_PATH="$(dirname ${BASH_SOURCE[0]:-${(%):-%x}})" +ABS_PATH="$(cd ${REL_PATH}; pwd)" +source ${ABS_PATH}/vendor/nimbus-build-system/scripts/env.sh +``` + +Supported usage: `./env.sh nim --version` + +Unsupported usage: `source env.sh; nim --version` + +An interesting alias is `nimble` which calls the "nimble.sh" script which pretends to be Nimble: + +```bash +cd vendor/nim-metrics +../../env.sh nimble test +``` + +### nimble.sh + +Simple script that symlinks the first \*.nimble file it finds to \*.nims and +runs it using `nim`. Easier to access using the `nimble` alias in "env.sh". + +Of very limited use, it can execute \*.nimble targets, as long as there are no +"before install:" blocks that the real Nimble strips before doing the same thing we do. + +If you need the real Nimble, it's in "vendor/nimbus-build-system/vendor/Nim/bin/nimble". ## License diff --git a/scripts/add_submodule.sh b/scripts/add_submodule.sh index 189b18f..d4b24e9 100755 --- a/scripts/add_submodule.sh +++ b/scripts/add_submodule.sh @@ -1,6 +1,6 @@ #!/bin/bash -# Copyright (c) 2018-2019 Status Research & Development GmbH. Licensed under +# Copyright (c) 2018-2020 Status Research & Development GmbH. Licensed under # either of: # - Apache License, version 2.0 # - MIT license @@ -8,7 +8,7 @@ # according to those terms. [ -z "$1" -o `echo "$1" | tr '/' '\n' | wc -l` != 2 ] && \ - { echo "usage: `basename $0` some/repo [destdir] # 'destdir' defaults to 'vendor/repo'"; exit 1; } + { echo "Usage: `basename $0` some/repo [destdir] # 'destdir' defaults to 'vendor/repo'"; exit 1; } REPO="$1" DEST="vendor/${REPO#*/}" diff --git a/scripts/build_nim.sh b/scripts/build_nim.sh index e6dcda5..f06944a 100755 --- a/scripts/build_nim.sh +++ b/scripts/build_nim.sh @@ -1,7 +1,7 @@ #!/bin/bash # used in Travis CI and AppVeyor scripts -# Copyright (c) 2018-2019 Status Research & Development GmbH. Licensed under +# Copyright (c) 2018-2020 Status Research & Development GmbH. Licensed under # either of: # - Apache License, version 2.0 # - MIT license @@ -15,7 +15,7 @@ CSOURCES_COMMIT="f72f471adb743bea4f8d8c59d19aa1cb885dcc59" # 0.20.0 NIMBLE_COMMIT="4007b2a778429a978e12307bf13a038029b4c4d9" # 0.11.0 # script arguments -[[ $# -ne 4 ]] && { echo "usage: $0 nim_dir csources_dir nimble_dir ci_cache_dir"; exit 1; } +[[ $# -ne 4 ]] && { echo "Usage: $0 nim_dir csources_dir nimble_dir ci_cache_dir"; exit 1; } NIM_DIR="$1" CSOURCES_DIR="$2" # can be relative to NIM_DIR NIMBLE_DIR="$3" # can be relative to NIM_DIR diff --git a/scripts/build_rocksdb.sh b/scripts/build_rocksdb.sh index 4c19b68..e4ae051 100755 --- a/scripts/build_rocksdb.sh +++ b/scripts/build_rocksdb.sh @@ -1,7 +1,7 @@ #!/bin/bash # used in Travis CI scripts -# Copyright (c) 2018-2019 Status Research & Development GmbH. Licensed under +# Copyright (c) 2018-2020 Status Research & Development GmbH. Licensed under # either of: # - Apache License, version 2.0 # - MIT license @@ -13,7 +13,7 @@ set -e ROCKSDBVER="5.17.2" # script arguments -[[ $# -ne 1 ]] && { echo "usage: $0 ci_cache_dir"; exit 1; } +[[ $# -ne 1 ]] && { echo "Usage: $0 ci_cache_dir"; exit 1; } CI_CACHE="$1" # here we cache the installed files # env vars