status-react/ci/tools/Jenkinsfile.nix-cache

137 lines
4.2 KiB
Plaintext
Raw Normal View History

2023-06-29 15:40:43 +02:00
#!/usr/bin/env groovy
Update release branch 2.30.x (#20903) Revisions from develop: - 59ceddbaa develop origin/develop fix(wallet): fix bridge transactions (#20902) - 99ccbc338 Cover wallet send events with tests Part 2 #20411 #20533 (#20721) - 8c2d5398b Enabling WalletConnect feature flag (#20906) - 67c83b13e fix(wallet): remove edit routes button in bridging (#20874) - 11a84ba14 feat(wallet): disable complex routing (#20901) - 1f5bb579c chore(wallet): disable bridging on unsupported tokens (#20846) - 4586f8007 Add toggle in advanced settings for mobile data - 55c620e59 fix: create password for small screen (#20645) - 525609f0a Wallet Activity: transactions are not sorted by time #20808 (#20862) - 90653955a chore(settings): Disable telemetry option (#20881) - d27ab756d fix_:display group message using the new ui (#20787) - c6a1db633 ci: enable split apks & build only for arm64-v8a (#20683) - 73777e052 Ensure keycard account can send transaction after upgrading from v1 to v2 #20552 (#20845) - a6d3fc374 [#20524] fix: the missed keypairs are shown in the key pair list screen (#20888) - a671c7083 fix broken screen and navigation when syncing fails (#20887) - a45991b6d 🥅 Filter connected dapps based on testnet mode, reject proposals and requests gracefully (#20799) - 2e9fa22e4 feat: wallet router v2 (#20631) - 737d8c4d5 rename sub to fix error when requesting to join community (#20868) - 3aa7e103f Sync process is blocked on Enabled notifications screen (#20883) - c1d2d44da perf: Fix app freeze after login (#20729) - 0fed8113d e2e: updated testnet switching and added one test into smoke - 53c35cb55 fix(wallet): Linear gradient exception on invalid colors for watched account cards (#20854) - be8236554 chore(settings)_: Remove testnet toggle from legacy advanced settings (#20875) - eae8a6559 feat(wallet)_: Add beta info box in activity tab (#20873) - fe54a25a3 fix: not clearing network & web3-wallet on logout (#20886) - 15a4219ef Reject wallet-connect request by dragging the modal down (#20763) (#20836) - 2ffbdac89 WalletConnect show expired toast (#20857) - 402eb8397 fix Issue with scrolling WalletConnect transaction on Android (#20867) - ff88049a0 Fix WalletConnect header alignment on Android (#20860) - cee21241d WalletConnect no internet edge-cases (#20826) - 60ad7c8a2 chore(tests): New match-strict? cljs.test directive (#20825) - 4989c9278 fix_: Adding own address as saved addresses (#20839)
2024-07-30 11:28:07 -03:00
library 'status-jenkins-lib@v1.9.3'
pipeline {
agent { label params.AGENT_LABEL }
parameters {
string(
name: 'NIX_CACHE_HOST',
description: 'FQDN of Nix binary cache host.',
defaultValue: params.NIX_CACHE_HOST ?: 'cache-01.do-ams3.nix.ci.statusim.net'
)
string(
name: 'NIX_CACHE_USER',
description: 'Username for Nix binary cache host.',
defaultValue: params.NIX_CACHE_USER ?: 'nix-cache'
)
}
environment {
/* we source .bash_profile to be able to use nix-store */
NIX_SSHOPTS = "-oStrictHostKeyChecking=no"
NIX_CONF_DIR = "${env.WORKSPACE}/nix"
NIX_STORE_CMD = '/nix/var/nix/profiles/default/bin/nix-store'
NIX_SSH_REMOTE = "ssh://${params.NIX_CACHE_USER}@${params.NIX_CACHE_HOST}?remote-program=${env.NIX_STORE_CMD}"
}
options {
timestamps()
disableConcurrentBuilds()
/* Prevent Jenkins jobs from running forever */
ci: build generic status-go and all shells When discussing caching of `status-go` with Sid I noticed that the build we cache daily created from our nightly build is different from the build we create locally due to a single input. In a release CI host we can see the IPFS URL is that of Infura: ``` > find /nix/store -maxdepth 1 -name '*-status-go-*android' | tail -n1 /nix/store/2cc8ilhx5g3k2awbn4sla61n4cml2405-status-go-0.130.1-d2cce5e-android > RESULT=$(find /nix/store -maxdepth 1 -name '*-status-go-*android' | tail -n1) > nix show-derivation $RESULT | tr ' ' '\n' | grep IpfsGateway github.com/status-im/status-go/params.IpfsGatewayURL=https://status-im.infura-ipfs.io/ipfs/ ``` But for a local build the URL is the default, which is our own gateway: ``` > nix-build --no-out-link -A targets.status-go.mobile.android /nix/store/1p53m7a6y1kg3vcyd8d06scf3bsyn5rk-status-go-0.157.2-47711c4-android > RESULT=$(nix-build --no-out-link -A targets.status-go.mobile.android) > nix show-derivation $RESULT | tr ' ' '\n' | grep IpfsGateway github.com/status-im/status-go/params.IpfsGatewayURL=https://ipfs.status.im/ ``` This difference causes builds of `status-go` that get uploaded to our Nix cache to not match what developers locally would build, which results in a cache miss. This changes the Nix cache CI jobs to instead of building only dependencies (`buildInuts`) to simply build the generic versions of `status-go` without nightly specific inputs. Signed-off-by: Jakub Sokołowski <jakub@status.im>
2023-06-15 09:39:28 +02:00
timeout(time: 120, unit: 'MINUTES')
/* Limit builds retained */
buildDiscarder(logRotator(
numToKeepStr: '20',
daysToKeepStr: '30',
artifactNumToKeepStr: '1',
))
}
stages {
stage('Setup') {
steps { script {
nix.shell('nix-env -i openssh', sandbox: false, pure: false)
major nix refactor Changes: - Adds a new `nix-gc` Makefile target for removing old packages - Moves all `nix/*.sh` files to `nix/scripts/*.sh` to make things more tidy - Renames `TARGET_OS` into `TARGET` and makes it effective only with `nix/scripts/shell.sh` - Renames `target-os` Nix argument to just `target` and makes it effective only with `shell.nix` - Drops `IN_CI_ENVIRONMENT` env variable which was useless - Drops use of `target-os` argument outside of `shell.nix` (with few exceptions, but just in naming) - `nix/platform.nix` has been made obsolete and removed - Moves the definition of all major targets to `nix/targets.nix` - Moves the definition of all major shells to `nix/shells.nix` - Makes `default.nix` and `shell.nix` just thin wrappers around `nix/default.nix` - `nix/nixpkgs-bootstrap.nix` has been moved to `nix/pkgs.nix` - All package and tool overrides have been moved to `nix/pkgs.nix` - Explicit passing of contents of `pkgs` has been removed in favor of `callPackage` doing it for us - `nix/bootstrapped-shell.nix` has been moved to `nix/tools/mkShell.nix` - A new `mergeSh` tool has been added to `pkgs` from `nix/tools/mergeSh.nix` - This tool is used to merge shells created using `mkShell` - `mobile/targets/jsbundle.nix` has been moved to `mobile/android/jsbundle/default.nix` - Moves `status-go` version sanitization to `nix/status-go/utils.nix` - Renames version to rawVersion and versionName to cleanVersion in status-go derivation - Ports nix/mobile/ios/install-pods-and-status-go.sh to Nix sub-shells - Moves adjustment of `inotify/max_user_watches` out into `scripts/inotify_fix.sh` - Makes iOS builds use the Nix version of Fastlane Signed-off-by: Jakub Sokołowski <jakub@status.im>
2019-11-29 11:20:08 +01:00
/* some build targets don't build on MacOS */
os = sh(script: 'uname', returnStdout: true).trim()
arch = sh(script: 'arch', returnStdout: true).trim()
} }
}
stage('Build status-go') {
steps { script {
ci: build generic status-go and all shells When discussing caching of `status-go` with Sid I noticed that the build we cache daily created from our nightly build is different from the build we create locally due to a single input. In a release CI host we can see the IPFS URL is that of Infura: ``` > find /nix/store -maxdepth 1 -name '*-status-go-*android' | tail -n1 /nix/store/2cc8ilhx5g3k2awbn4sla61n4cml2405-status-go-0.130.1-d2cce5e-android > RESULT=$(find /nix/store -maxdepth 1 -name '*-status-go-*android' | tail -n1) > nix show-derivation $RESULT | tr ' ' '\n' | grep IpfsGateway github.com/status-im/status-go/params.IpfsGatewayURL=https://status-im.infura-ipfs.io/ipfs/ ``` But for a local build the URL is the default, which is our own gateway: ``` > nix-build --no-out-link -A targets.status-go.mobile.android /nix/store/1p53m7a6y1kg3vcyd8d06scf3bsyn5rk-status-go-0.157.2-47711c4-android > RESULT=$(nix-build --no-out-link -A targets.status-go.mobile.android) > nix show-derivation $RESULT | tr ' ' '\n' | grep IpfsGateway github.com/status-im/status-go/params.IpfsGatewayURL=https://ipfs.status.im/ ``` This difference causes builds of `status-go` that get uploaded to our Nix cache to not match what developers locally would build, which results in a cache miss. This changes the Nix cache CI jobs to instead of building only dependencies (`buildInuts`) to simply build the generic versions of `status-go` without nightly specific inputs. Signed-off-by: Jakub Sokołowski <jakub@status.im>
2023-06-15 09:39:28 +02:00
def platforms = ['mobile.android', 'mobile.ios', 'library']
if (os != 'Darwin') { platforms.removeAll { it == 'mobile.ios' } }
/* FIXME: "'x86_64-darwin' with features {} is required to build" */
ci: build generic status-go and all shells When discussing caching of `status-go` with Sid I noticed that the build we cache daily created from our nightly build is different from the build we create locally due to a single input. In a release CI host we can see the IPFS URL is that of Infura: ``` > find /nix/store -maxdepth 1 -name '*-status-go-*android' | tail -n1 /nix/store/2cc8ilhx5g3k2awbn4sla61n4cml2405-status-go-0.130.1-d2cce5e-android > RESULT=$(find /nix/store -maxdepth 1 -name '*-status-go-*android' | tail -n1) > nix show-derivation $RESULT | tr ' ' '\n' | grep IpfsGateway github.com/status-im/status-go/params.IpfsGatewayURL=https://status-im.infura-ipfs.io/ipfs/ ``` But for a local build the URL is the default, which is our own gateway: ``` > nix-build --no-out-link -A targets.status-go.mobile.android /nix/store/1p53m7a6y1kg3vcyd8d06scf3bsyn5rk-status-go-0.157.2-47711c4-android > RESULT=$(nix-build --no-out-link -A targets.status-go.mobile.android) > nix show-derivation $RESULT | tr ' ' '\n' | grep IpfsGateway github.com/status-im/status-go/params.IpfsGatewayURL=https://ipfs.status.im/ ``` This difference causes builds of `status-go` that get uploaded to our Nix cache to not match what developers locally would build, which results in a cache miss. This changes the Nix cache CI jobs to instead of building only dependencies (`buildInuts`) to simply build the generic versions of `status-go` without nightly specific inputs. Signed-off-by: Jakub Sokołowski <jakub@status.im>
2023-06-15 09:39:28 +02:00
if (arch == 'arm64') { platforms.removeAll { it == 'mobile.android' } }
platforms.each { platform ->
/* Allow for Android builds on Apple ARM. */
env.NIXPKGS_SYSTEM_OVERRIDE = nixSysOverride(os, arch, platform)
nix.build(
attr: "targets.status-go.${platform}",
sandbox: false,
link: false
)
}
} }
}
stage('Build android jsbundle') {
steps { script {
/* Build/fetch deps required for jsbundle build. */
nix.build(
attr: 'targets.mobile.jsbundle',
sandbox: false,
pure: false,
link: false
)
} }
}
stage('Build android deps') {
steps { script {
/* Allow for Android builds on Apple ARM. */
env.NIXPKGS_SYSTEM_OVERRIDE = nixSysOverride(os, arch, 'android')
/* Build/fetch deps required to build android release. */
nix.build(
attr: 'targets.mobile.android.build.buildInputs',
sandbox: false,
pure: false,
link: false
)
} }
}
stage('Build nix shell deps') {
steps { script {
ci: build generic status-go and all shells When discussing caching of `status-go` with Sid I noticed that the build we cache daily created from our nightly build is different from the build we create locally due to a single input. In a release CI host we can see the IPFS URL is that of Infura: ``` > find /nix/store -maxdepth 1 -name '*-status-go-*android' | tail -n1 /nix/store/2cc8ilhx5g3k2awbn4sla61n4cml2405-status-go-0.130.1-d2cce5e-android > RESULT=$(find /nix/store -maxdepth 1 -name '*-status-go-*android' | tail -n1) > nix show-derivation $RESULT | tr ' ' '\n' | grep IpfsGateway github.com/status-im/status-go/params.IpfsGatewayURL=https://status-im.infura-ipfs.io/ipfs/ ``` But for a local build the URL is the default, which is our own gateway: ``` > nix-build --no-out-link -A targets.status-go.mobile.android /nix/store/1p53m7a6y1kg3vcyd8d06scf3bsyn5rk-status-go-0.157.2-47711c4-android > RESULT=$(nix-build --no-out-link -A targets.status-go.mobile.android) > nix show-derivation $RESULT | tr ' ' '\n' | grep IpfsGateway github.com/status-im/status-go/params.IpfsGatewayURL=https://ipfs.status.im/ ``` This difference causes builds of `status-go` that get uploaded to our Nix cache to not match what developers locally would build, which results in a cache miss. This changes the Nix cache CI jobs to instead of building only dependencies (`buildInuts`) to simply build the generic versions of `status-go` without nightly specific inputs. Signed-off-by: Jakub Sokołowski <jakub@status.im>
2023-06-15 09:39:28 +02:00
def shells = ['android', 'ios', 'fastlane', 'keytool', 'clojure', 'gradle']
if (os != 'Darwin') { shells.removeAll { it == 'ios' } }
/* FIXME: "'x86_64-darwin' with features {} is required to build" */
if (arch == 'arm64') { shells.removeAll { it == 'android' } }
/* Build/fetch deps required to start default Nix shell. */
ci: build generic status-go and all shells When discussing caching of `status-go` with Sid I noticed that the build we cache daily created from our nightly build is different from the build we create locally due to a single input. In a release CI host we can see the IPFS URL is that of Infura: ``` > find /nix/store -maxdepth 1 -name '*-status-go-*android' | tail -n1 /nix/store/2cc8ilhx5g3k2awbn4sla61n4cml2405-status-go-0.130.1-d2cce5e-android > RESULT=$(find /nix/store -maxdepth 1 -name '*-status-go-*android' | tail -n1) > nix show-derivation $RESULT | tr ' ' '\n' | grep IpfsGateway github.com/status-im/status-go/params.IpfsGatewayURL=https://status-im.infura-ipfs.io/ipfs/ ``` But for a local build the URL is the default, which is our own gateway: ``` > nix-build --no-out-link -A targets.status-go.mobile.android /nix/store/1p53m7a6y1kg3vcyd8d06scf3bsyn5rk-status-go-0.157.2-47711c4-android > RESULT=$(nix-build --no-out-link -A targets.status-go.mobile.android) > nix show-derivation $RESULT | tr ' ' '\n' | grep IpfsGateway github.com/status-im/status-go/params.IpfsGatewayURL=https://ipfs.status.im/ ``` This difference causes builds of `status-go` that get uploaded to our Nix cache to not match what developers locally would build, which results in a cache miss. This changes the Nix cache CI jobs to instead of building only dependencies (`buildInuts`) to simply build the generic versions of `status-go` without nightly specific inputs. Signed-off-by: Jakub Sokołowski <jakub@status.im>
2023-06-15 09:39:28 +02:00
shells.each { shell ->
/* Allow for Android builds on Apple ARM. */
env.NIXPKGS_SYSTEM_OVERRIDE = nixSysOverride(os, arch, shell)
ci: build generic status-go and all shells When discussing caching of `status-go` with Sid I noticed that the build we cache daily created from our nightly build is different from the build we create locally due to a single input. In a release CI host we can see the IPFS URL is that of Infura: ``` > find /nix/store -maxdepth 1 -name '*-status-go-*android' | tail -n1 /nix/store/2cc8ilhx5g3k2awbn4sla61n4cml2405-status-go-0.130.1-d2cce5e-android > RESULT=$(find /nix/store -maxdepth 1 -name '*-status-go-*android' | tail -n1) > nix show-derivation $RESULT | tr ' ' '\n' | grep IpfsGateway github.com/status-im/status-go/params.IpfsGatewayURL=https://status-im.infura-ipfs.io/ipfs/ ``` But for a local build the URL is the default, which is our own gateway: ``` > nix-build --no-out-link -A targets.status-go.mobile.android /nix/store/1p53m7a6y1kg3vcyd8d06scf3bsyn5rk-status-go-0.157.2-47711c4-android > RESULT=$(nix-build --no-out-link -A targets.status-go.mobile.android) > nix show-derivation $RESULT | tr ' ' '\n' | grep IpfsGateway github.com/status-im/status-go/params.IpfsGatewayURL=https://ipfs.status.im/ ``` This difference causes builds of `status-go` that get uploaded to our Nix cache to not match what developers locally would build, which results in a cache miss. This changes the Nix cache CI jobs to instead of building only dependencies (`buildInuts`) to simply build the generic versions of `status-go` without nightly specific inputs. Signed-off-by: Jakub Sokołowski <jakub@status.im>
2023-06-15 09:39:28 +02:00
nix.build(
attr: "shells.${shell}.buildInputs",
sandbox: false,
link: false
)
}
} }
}
stage('Upload') {
steps { script {
sshagent(credentials: ['nix-cache-ssh']) {
nix.shell("""
find /nix/store/ -mindepth 1 -maxdepth 1 -type d \
-not -name '*.links' -and -not -name '*-status-mobile-*' \
| xargs nix copy --to ${NIX_SSH_REMOTE}
""",
pure: false
)
}
} }
}
}
post {
always { script {
nix.shell('nix-store --optimize', pure: false)
} }
}
}
def nixSysOverride(os, arch, target='android') {
return (
os == 'Darwin' &&
arch == 'arm64' &&
target =~ /.*android$/
) ? 'x86_64-darwin' : ''
}