nix: minor cleanup and documentation updates

Changes:
- Simplified `default.nix` and `shell.nix`
- Moved the default shell to `nix/shell.nix`
- Dropped unnecessary merge from  `nix/shells.nix`
- Fixed `nix/lib/getConfig.nix` to return default on `null`
- Expanded `nix/DETAILS.md` with more info
- Added links to presentations in `nix/README.md`
- Fixed a few typos

Signed-off-by: Jakub Sokołowski <jakub@status.im>
This commit is contained in:
Jakub Sokołowski 2020-06-05 11:23:22 +02:00
parent 496a1fa288
commit dbe3e0a3f9
No known key found for this signature in database
GPG Key ID: 4EF064D0E6D63020
8 changed files with 125 additions and 86 deletions

View File

@ -1,10 +1,10 @@
# for passing build options, see nix/README.md # for passing build options, see nix/README.md
{ config ? { status-im = { }; } }: { config ? { } }:
let let
main = import ./nix/default.nix { inherit config; }; main = import ./nix { inherit config; };
in { in {
# this is where the --attr argument selects the shell or target # this is where the --attr argument selects the shell or target
inherit (main) pkgs targets shells; inherit (main) pkgs targets shells;
inherit config; inherit (main.pkgs) config;
} }

View File

@ -1,30 +1,43 @@
# Description # Description
This document descripts the layout of our Nix setup. This document describes the layout of our Nix setup.
# Folders # Folders
There are four main folders in the `nix` directory: There are four main folders in the `nix` directory:
* `nix/scripts` - Bash scripts for easier usage of Nix * [`nix/scripts`](./scripts) - Bash scripts for easier usage of Nix
* `nix/pkgs` - Packages we add to or modify in `nixpkgs` * [`nix/pkgs`](./pkgs) - Packages we add to or modify in `nixpkgs`
* `nix/deps` - Project dependencies managed by Nix * [`nix/deps`](./deps) - Project dependencies managed by Nix
* `nix/lib` - Our tools we merge into `pkgs.lib` * [`nix/lib`](./lib) - Our tools we merge into `pkgs.lib`
* `nix/tools` - Various tools used by our derivations and shells * [`nix/tools`](./tools) - Various tools used by our derivations and shells
* `nix/status-go` - Derivations for building [`status-go`](https://github.com/status-im/status-go) repo * [`nix/status-go`](./status-go) - Derivations for building [`status-go`](https://github.com/status-im/status-go) repo
# Files # Files
There are a few main files that define the whole build environment: There are a few main files that define the whole build environment:
* `nix/nix.conf` - Binary cache configuration * [`nix/nix.conf`](./nix.conf) - Binary cache configuration
* `nix/default.nix` - Entrypoint for both shells and targets * [`nix/default.nix`](./default.nix) - Entry point for both shells and targets
* `nix/shells.nix` - Definition of Nix shells used in builds * [`nix/shell.nix`](./shell.nix) - Definition of the default Nix shell
* `nix/targets.nix` - Hierarchy of main build targets * [`nix/shells.nix`](./shells.nix) - Definitions of other Nix shells used in builds
* `nix/pkgs.nix` - Definition of a custom `nixpkgs` repo * [`nix/targets.nix`](./targets.nix) - Hierarchy of main build targets
* `nix/overlay.nix` - Overrides for `nixpkgs`, custom packages * [`nix/pkgs.nix`](./pkgs.nix) - Definition of a custom `nixpkgs` repo
* [`nix/config.nix`](./config.nix) - Default config values for instantiating `nixpkgs`
* [`nix/overlay.nix`](./overlay.nix) - Overrides for `nixpkgs`, custom packages
The `default.nix` and `shell.nix` files at th repo root are just a gateway into the `nix` subfolder. The [`default.nix`](../default.nix) and [`shell.nix`](../shell.nix) files at th repo root are just a gateway into the `nix` sub folder.
# Scripts
There's a few scripts in [`nix/scripts`](./scripts) that make use of Nix simpler:
* [`nix/scripts/setup.sh`](./scripts/setup.sh) - Installs Nix Package manager
* [`nix/scripts/source.sh`](./scripts/source.sh) - Sources the Nix profile or installs Nix
* [`nix/scripts/build.sh`](./scripts/build.sh) - A wrapper around `nix-build` with sane defaults
* [`nix/scripts/shell.sh`](./scripts/shell.sh) - A wrapper around `nix-shell` for `Makefile`
* [`nix/scripts/clean.sh`](./scripts/clean.sh) - For cleaning Nix store after builds
* [`nix/scripts/purge.sh`](./scripts/purge.sh) - For purging everything Nix related from system
# Start # Start
@ -32,7 +45,15 @@ The starting point for using our Nix shells and targets is the [`default.nix`](/
It pulls in all the `pkgs`, `targets` and `shells` defined in [`nix/default.nix`](/nix/default.nix). The point is easy access to them via commands like `nix-build` or `nix-shell`, which you'll see next. It pulls in all the `pkgs`, `targets` and `shells` defined in [`nix/default.nix`](/nix/default.nix). The point is easy access to them via commands like `nix-build` or `nix-shell`, which you'll see next.
# Usage # Shells
Normally shells are started using `make shell TARGET=android`, but that is essentially the same as calling:
```bash
nix-shell -A shells.android default.nix
```
The [`nix/scripts/shell.sh`](./scripts/shell.sh) script is essentially a wrapper around that command to make it usable as shell for the `Makefile`.
# Building
We will use the `make jsbundle-android` target as an example of a derivation you can build using Nix: We will use the `make jsbundle-android` target as an example of a derivation you can build using Nix:

View File

@ -10,12 +10,12 @@ __NOTE:__ If you are in Asia you might want to add the `https://nix-cache-cn.sta
## Build arguments ## Build arguments
We leverage the standard nixpkgs `config` argument for our own parameterization of the builds (e.g. to pass a build number or build type). We leverage the `config` argument of standard `nixpkgs` for our own parameterization of the builds (e.g. to pass a build number or build type).
Here is a sample structure of the `config` attribute set: Here is a sample structure of the `config` attribute set:
```nix ```nix
config = { {
status-im = { status-im = {
build-type = "pr"; # Build type (influences which .env file gets used for feature flags) build-type = "pr"; # Build type (influences which .env file gets used for feature flags)
build-number = 9999; # Used for versionCode and CFBundleVersion in Android and iOS respectively build-number = 9999; # Used for versionCode and CFBundleVersion in Android and iOS respectively
@ -29,7 +29,7 @@ config = {
src-override = "$HOME/my/source/status-go"; # local source override src-override = "$HOME/my/source/status-go"; # local source override
}; };
}; };
}; }
``` ```
You can see the defaults in [`nix/config.nix`](./config.nix). You can see the defaults in [`nix/config.nix`](./config.nix).
@ -59,6 +59,15 @@ export STATUS_GO_SRC_OVERRIDE=$GOPATH/src/github.com/status-im/status-go
make release-android make release-android
``` ```
## Resources
You can learn more about Nix by watching these presentations:
* [Nix Fundamentals](https://www.youtube.com/watch?v=m4sv2M9jRLg) ([PDF](https://drive.google.com/file/d/1Tt5R7QOubudGiSuZIGxuFWB1OYgcThcL/view?usp=sharing), [src](https://github.com/status-im/infra-docs/tree/master/presentations/nix_basics))
* [Nix in Status](https://www.youtube.com/watch?v=rEQ1EvRG8Wc) ([PDF](https://drive.google.com/file/d/1Ti0wppMoj40icCPdHy7mJcQj__DeaYBE/view?usp=sharing), [src](https://github.com/status-im/infra-docs/tree/master/presentations/nix_in_status))
And you can read [`nix/DETAILS.md`](./DETAILS.md) for more information.
## Known Issues ## Known Issues
See [`KNOWN_ISSUES.md`](./KNOWN_ISSUES.md). See [`KNOWN_ISSUES.md`](./KNOWN_ISSUES.md).

View File

@ -1,12 +1,11 @@
#
# helper for getting status-im config values in a safe way # helper for getting status-im config values in a safe way
# #
{ lib, config }: { lib, config }:
let inherit (lib) splitString attrByPath;
in name: default:
let let
inherit (lib) splitString attrByPath; path = [ "status-im" ] ++ (splitString "." name);
in value = attrByPath path default config;
name: default: in if value != null then value else default
let path = [ "status-im" ] ++ (splitString "." name);
in attrByPath path default config

View File

@ -16,7 +16,7 @@ assert (lib.stringLength watchmanSockPath) > 0 -> stdenv.isDarwin;
let let
inherit (lib) inherit (lib)
toLower optionalString stringLength toLower optionalString stringLength assertMsg
getConfig makeLibraryPath assertEnvVarSet; getConfig makeLibraryPath assertEnvVarSet;
buildType = getConfig "build-type" "prod"; buildType = getConfig "build-type" "prod";
@ -82,7 +82,7 @@ in stdenv.mkDerivation rec {
phases = [ phases = [
"unpackPhase" "secretsPhase" "secretsCheckPhase" "unpackPhase" "secretsPhase" "secretsCheckPhase"
"buildPhase" "checkPhase" "installPhase" "keystorePhase" "buildPhase" "checkPhase" "installPhase"
]; ];
unpackPhase = '' unpackPhase = ''
@ -90,13 +90,7 @@ in stdenv.mkDerivation rec {
chmod u+w -R ./ chmod u+w -R ./
runHook postUnpack runHook postUnpack
''; '';
postUnpack = postUnpack = ''
assert lib.assertMsg (keystorePath != null) "keystore-file has to be set!";
''
# Keep the same keystore path for determinism
export KEYSTORE_PATH="$PWD/status-im.keystore"
cp -a --no-preserve=ownership "${keystorePath}" "$KEYSTORE_PATH"
# Ensure we have the right .env file # Ensure we have the right .env file
cp -f ./${envFileName} ./.env cp -f ./${envFileName} ./.env
@ -118,10 +112,19 @@ in stdenv.mkDerivation rec {
# Patch build.gradle to use local repo # Patch build.gradle to use local repo
${patchMavenSources} ./android/build.gradle ${patchMavenSources} ./android/build.gradle
''; '';
# if secretsFile is not set we use generate keystore # if secretsFile is not set we use generate keystore
secretsPhase = if (secretsFile != "") then '' secretsPhase = if (secretsFile != "") then ''
source "${secretsFile}" source "${secretsFile}"
'' else keystore.shellHook; '' else keystore.shellHook;
# if keystorePath is set copy it into build directory
keystorePhase =
assert assertMsg (keystorePath != null) "keystorePath has to be set!";
''
export KEYSTORE_PATH="$PWD/status-im.keystore"
cp -a --no-preserve=ownership "${keystorePath}" "$KEYSTORE_PATH"
'';
secretsCheckPhase = '' secretsCheckPhase = ''
${assertEnvVarSet "KEYSTORE_ALIAS"} ${assertEnvVarSet "KEYSTORE_ALIAS"}
${assertEnvVarSet "KEYSTORE_PASSWORD"} ${assertEnvVarSet "KEYSTORE_PASSWORD"}

39
nix/shell.nix Normal file
View File

@ -0,0 +1,39 @@
#
# Defines the default shell that is used when target is not specified.
# It is also merged with all the other shells for a complete set of tools.
#
{ config ? {}
, pkgs ? import ./pkgs.nix { inherit config; } }:
let
inherit (pkgs) mkShell;
in mkShell {
name = "status-react-shell"; # for identifying all shells
buildInputs = with pkgs; lib.unique ([
# core utilities that should always be present in a shell
bash curl wget file unzip flock
git gnumake jq ncurses gnugrep parallel
lsof # used in start-react-native.sh
# build specific utilities
clojure maven watchman
# other nice to have stuff
yarn nodejs python27
] # and some special cases
++ lib.optionals stdenv.isDarwin [ cocoapods clang ]
++ lib.optionals (!stdenv.isDarwin) [ gcc8 ]
);
# avoid terminal issues
TERM="xterm";
# default locale
LANG="en_US.UTF-8";
LANGUAGE="en_US.UTF-8";
# just a nicety for easy access to node scripts
shellHook = ''
export STATUS_REACT_HOME=$(git rev-parse --show-toplevel)
export PATH="$STATUS_REACT_HOME/node_modules/.bin:$PATH"
'';
}

View File

@ -1,48 +1,16 @@
# This file defines custom shells as well as shortcuts # This file defines custom shells as well as shortcuts
# for accessing more nested shells. # for accessing more nested shells.
{ { config ? {}
config ? {}, , pkgs ? import ./pkgs.nix { inherit config; } }:
pkgs ? import ./pkgs.nix { inherit config; }
}:
let let
inherit (pkgs) lib stdenv mkShell; inherit (pkgs) lib mkShell callPackage;
# everything else we define in nix/ dir # everything else we define in nix/ dir
targets = pkgs.callPackage ./targets.nix { inherit config; }; targets = callPackage ./targets.nix { inherit config; };
# the default shell that is used when target is not specified # the default shell with most commonly used tools
# it is also merged with all the other shells default = callPackage ./shell.nix { };
default = mkShell {
name = "status-react-shell"; # for identifying all shells
buildInputs = with pkgs; lib.unique ([
# core utilities that should always be present in a shell
bash curl wget file unzip flock
git gnumake jq ncurses gnugrep parallel
lsof # used in start-react-native.sh
# build specific utilities
clojure maven watchman
# other nice to have stuff
yarn nodejs python27
] # and some special cases
++ lib.optionals stdenv.isDarwin [ cocoapods clang ]
++ lib.optionals (!stdenv.isDarwin) [ gcc8 ]
);
# avoid terinal issues
TERM="xterm";
# default locale
LANG="en_US.UTF-8";
LANGUAGE="en_US.UTF-8";
# just a nicety for easy access to node scripts
shellHook = ''
export STATUS_REACT_HOME=$(git rev-parse --show-toplevel)
export PATH="$STATUS_REACT_HOME/node_modules/.bin:$PATH"
'';
};
# Combines with many other shells # Combines with many other shells
node-sh = mkShell { node-sh = mkShell {
@ -55,6 +23,8 @@ let
# An attrset for easier merging with default shell # An attrset for easier merging with default shell
shells = { shells = {
inherit default;
nodejs = node-sh; nodejs = node-sh;
# for calling clojure targets in CI or Makefile # for calling clojure targets in CI or Makefile
@ -106,6 +76,4 @@ let
# values here can be selected using `nix-shell --attr shells.$TARGET default.nix` # values here can be selected using `nix-shell --attr shells.$TARGET default.nix`
# the nix/scripts/shell.sh wrapper does this for us and expects TARGET to be set # the nix/scripts/shell.sh wrapper does this for us and expects TARGET to be set
in { in lib.mapAttrs mergeDefaultShell shells
inherit default;
} // lib.mapAttrs mergeDefaultShell shells

View File

@ -2,7 +2,7 @@
{ config ? { } }: { config ? { } }:
let let
project = import ./default.nix { inherit config; }; main = import ./nix { inherit config; };
in in
# we use the shell combining most shells as default # use the default shell when calling nix-shell without arguments
project.shells.default main.shells.default