document the scripts

This commit is contained in:
Ștefan Talpalaru 2020-06-14 19:12:27 +02:00
parent bb82ba8984
commit a662777d03
No known key found for this signature in database
GPG Key ID: CBF7934204F1B6F9
4 changed files with 142 additions and 10 deletions

140
README.md
View File

@ -1,6 +1,6 @@
# Common parts of the build system used by Nimbus and related projects # Common parts of the build system used by Nimbus and related projects
We focus on building Nim software on multiple platforms without having to deal We focus on building Nim software on multiple platforms, without having to deal
with language-specific package managers. with language-specific package managers.
We care about dependencies specified with commit-level accuracy (including the We care about dependencies specified with commit-level accuracy (including the
@ -31,7 +31,7 @@ Or you can do it by hand:
git submodule add https://github.com/status-im/nimbus-build-system.git vendor/nimbus-build-system git submodule add https://github.com/status-im/nimbus-build-system.git vendor/nimbus-build-system
# specify a branch # specify a branch
git config -f .gitmodules submodule.vendor/nimbus-build-system.branch master git config -f .gitmodules submodule.vendor/nimbus-build-system.branch master
# hide submodule work tree changes in `git diff` # hide submodule working tree changes in `git diff`
git config -f .gitmodules submodule.vendor/nimbus-build-system.ignore dirty git config -f .gitmodules submodule.vendor/nimbus-build-system.ignore dirty
``` ```
@ -48,7 +48,7 @@ See also the Makefiles we wrote for
Instruct your users to run `make update` after cloning your project, after a Instruct your users to run `make update` after cloning your project, after a
`git pull` or after switching branches. `git pull` or after switching branches.
## Make flags ## Make variables
### V ### V
@ -225,7 +225,139 @@ List all Git submodules, including the nested ones.
### fetch-dlls ### fetch-dlls
Windows-specific target. Downloads and unpacks in the "build" dir some DLLs we may not want to build ourselves (PCRE, RocksDB, libcurl, pdcurses, Snappy, SQLite3, SSLeay, etc.). Windows-specific target. Downloads and unpacks in the "build" dir some DLLs we may not want to build ourselves (PCRE, RocksDB, libcurl, pdcurses, SQLite3, OpenSSL, zlib, etc.).
## Scripts
### add_submodule.sh
Add a new Git submodule to your superproject, setting the branch to "master"
and hiding submodule working tree changes in `git diff`.
Usage: `./add_submodule.sh some/repo [destdir]` # 'destdir' defaults to 'vendor/repo'
Examples:
`./add_submodule.sh status-im/nimbus-build-system`
`./vendor/nimbus-build-system/scripts/add_submodule.sh status-im/nim-nat-traversal`
### build_nim.sh
Build the Nim compiler and some associated tools, parallelising the
bootstrap-from-C phase, with binary caching and conditional recompilation
support (based on the timestamp of the latest commit in the Nim repo).
Usage: `./build_nim.sh nim_dir csources_dir nimble_dir ci_cache_dir`
This script is not usually used directly, but through the `update` target
(which depends on `update-common` which runs `build-nim`).
Our `build-nim` target uses it like this:
```make
build-nim: | sanity-checks
+ NIM_BUILD_MSG="$(BUILD_MSG) Nim compiler" \
V=$(V) \
CC=$(CC) \
MAKE=$(MAKE) \
ARCH_OVERRIDE=$(ARCH_OVERRIDE) \
"$(CURDIR)/$(BUILD_SYSTEM_DIR)/scripts/build_nim.sh" "$(NIM_DIR)" ../Nim-csources ../nimble "$(CI_CACHE)"
```
Other Nim projects that don't use this build system use the script directly in their CI. From a ".travis.yml":
```yaml
install:
- curl -O -L -s -S https://raw.githubusercontent.com/status-im/nimbus-build-system/master/scripts/build_nim.sh
- env MAKE="make -j${NPROC}" bash build_nim.sh Nim csources dist/nimble NimBinaries
- export PATH="$PWD/Nim/bin:$PATH"
```
Or an ".appveyor.yml":
```yaml
install:
# [...]
- curl -O -L -s -S https://raw.githubusercontent.com/status-im/nimbus-build-system/master/scripts/build_nim.sh
- env MAKE="mingw32-make -j2" ARCH_OVERRIDE=%PLATFORM% bash build_nim.sh Nim csources dist/nimble NimBinaries
- SET PATH=%CD%\Nim\bin;%PATH%
```
Notice how the number of Make jobs is set through the "MAKE" env var.
### build_p2pd.sh
Builds the "p2pd" Go daemon. No longer used by a Make target, but needed by
other projects that run it directly in their CI config files, like this:
```yaml
install:
# [...]
# install and build go-libp2p-daemon
- curl -O -L -s -S https://raw.githubusercontent.com/status-im/nimbus-build-system/master/scripts/build_p2pd.sh
- bash build_p2pd.sh p2pdCache v0.2.1
```
### build_rocksdb.sh
Builds RocksDB. No longer used.
Usage: `./build_rocksdb.sh ci_cache_dir`
### create_nimble_link.sh
Cheeky little script used to fake a Nimble package repository in the
`$(NIMBLE_DIR)` target (a dependency of `deps-common` which is a dependency of
`deps`):
```make
$(NIMBLE_DIR):
mkdir -p $(NIMBLE_DIR)/pkgs
NIMBLE_DIR="$(CURDIR)/$(NIMBLE_DIR)" PWD_CMD="$(PWD)" \
git submodule foreach --quiet '$(CURDIR)/$(BUILD_SYSTEM_DIR)/scripts/create_nimble_link.sh "$$sm_path"'
```
That's how the Nim compiler knows how to find all these Nim packages we have in
our submodules: we set the "NIMBLE\_DIR" env var to the path of this fake
Nimble package repo.
### env.sh
Script responsible for setting up environment variables and shell aliases
necessary for using this build system. It's being sourced by a script with the
same name in the superproject's top directory:
```bash
#!/bin/bash
# We use ${BASH_SOURCE[0]} instead of $0 to allow sourcing this file
# and we fall back to a Zsh-specific special var to also support Zsh.
REL_PATH="$(dirname ${BASH_SOURCE[0]:-${(%):-%x}})"
ABS_PATH="$(cd ${REL_PATH}; pwd)"
source ${ABS_PATH}/vendor/nimbus-build-system/scripts/env.sh
```
Supported usage: `./env.sh nim --version`
Unsupported usage: `source env.sh; nim --version`
An interesting alias is `nimble` which calls the "nimble.sh" script which pretends to be Nimble:
```bash
cd vendor/nim-metrics
../../env.sh nimble test
```
### nimble.sh
Simple script that symlinks the first \*.nimble file it finds to \*.nims and
runs it using `nim`. Easier to access using the `nimble` alias in "env.sh".
Of very limited use, it can execute \*.nimble targets, as long as there are no
"before install:" blocks that the real Nimble strips before doing the same thing we do.
If you need the real Nimble, it's in "vendor/nimbus-build-system/vendor/Nim/bin/nimble".
## License ## License

View File

@ -1,6 +1,6 @@
#!/bin/bash #!/bin/bash
# Copyright (c) 2018-2019 Status Research & Development GmbH. Licensed under # Copyright (c) 2018-2020 Status Research & Development GmbH. Licensed under
# either of: # either of:
# - Apache License, version 2.0 # - Apache License, version 2.0
# - MIT license # - MIT license
@ -8,7 +8,7 @@
# according to those terms. # according to those terms.
[ -z "$1" -o `echo "$1" | tr '/' '\n' | wc -l` != 2 ] && \ [ -z "$1" -o `echo "$1" | tr '/' '\n' | wc -l` != 2 ] && \
{ echo "usage: `basename $0` some/repo [destdir] # 'destdir' defaults to 'vendor/repo'"; exit 1; } { echo "Usage: `basename $0` some/repo [destdir] # 'destdir' defaults to 'vendor/repo'"; exit 1; }
REPO="$1" REPO="$1"
DEST="vendor/${REPO#*/}" DEST="vendor/${REPO#*/}"

View File

@ -1,7 +1,7 @@
#!/bin/bash #!/bin/bash
# used in Travis CI and AppVeyor scripts # used in Travis CI and AppVeyor scripts
# Copyright (c) 2018-2019 Status Research & Development GmbH. Licensed under # Copyright (c) 2018-2020 Status Research & Development GmbH. Licensed under
# either of: # either of:
# - Apache License, version 2.0 # - Apache License, version 2.0
# - MIT license # - MIT license
@ -15,7 +15,7 @@ CSOURCES_COMMIT="f72f471adb743bea4f8d8c59d19aa1cb885dcc59" # 0.20.0
NIMBLE_COMMIT="4007b2a778429a978e12307bf13a038029b4c4d9" # 0.11.0 NIMBLE_COMMIT="4007b2a778429a978e12307bf13a038029b4c4d9" # 0.11.0
# script arguments # script arguments
[[ $# -ne 4 ]] && { echo "usage: $0 nim_dir csources_dir nimble_dir ci_cache_dir"; exit 1; } [[ $# -ne 4 ]] && { echo "Usage: $0 nim_dir csources_dir nimble_dir ci_cache_dir"; exit 1; }
NIM_DIR="$1" NIM_DIR="$1"
CSOURCES_DIR="$2" # can be relative to NIM_DIR CSOURCES_DIR="$2" # can be relative to NIM_DIR
NIMBLE_DIR="$3" # can be relative to NIM_DIR NIMBLE_DIR="$3" # can be relative to NIM_DIR

View File

@ -1,7 +1,7 @@
#!/bin/bash #!/bin/bash
# used in Travis CI scripts # used in Travis CI scripts
# Copyright (c) 2018-2019 Status Research & Development GmbH. Licensed under # Copyright (c) 2018-2020 Status Research & Development GmbH. Licensed under
# either of: # either of:
# - Apache License, version 2.0 # - Apache License, version 2.0
# - MIT license # - MIT license
@ -13,7 +13,7 @@ set -e
ROCKSDBVER="5.17.2" ROCKSDBVER="5.17.2"
# script arguments # script arguments
[[ $# -ne 1 ]] && { echo "usage: $0 ci_cache_dir"; exit 1; } [[ $# -ne 1 ]] && { echo "Usage: $0 ci_cache_dir"; exit 1; }
CI_CACHE="$1" # here we cache the installed files CI_CACHE="$1" # here we cache the installed files
# env vars # env vars