Possible fix for slow upload speeds and failures caused by most probably
hitting per-bucket rate limits of DigitalOcean:
>- 500 total operations per second to any individual bucket.
>- 300 combined PUT, POST, COPY, DELETE, and LIST operations per second
> to any individual Space. We may further limit LIST operations if
> necessary under periods of high load.
https://docs.digitalocean.com/products/spaces/details/limits/#rate-limits
Depends on: https://github.com/status-im/status-jenkins-lib/pull/52
Signed-off-by: Jakub Sokołowski <jakub@status.im>
It needs to be separate script, because once the submodules are broken
and the `variables.mk` file is not available from `nimbus-build-system`
then we cannot even call `make clean-git` successfully because it will
be caught by the `if` clause that checks for `variables.mk`.
Possible fix for issues with submodules not being updated in some
windows release builds.
Signed-off-by: Jakub Sokołowski <jakub@status.im>
This was added to fix some build issues to Linux:
https://github.com/status-im/status-desktop/pull/8233
But other 2 platforms also have had the cleanup stage change which could
have cause these submodules update issues.
Signed-off-by: Jakub Sokołowski <jakub@status.im>
This is a continuation of attempts to fix provlems with Windows CI hosts
getting into a broken state due to `.git` directory becoming a file for
random submodules for unknown reason.
Instead of relying on Jenkins functionality of `cleanWs()` function
provided by the [Workspace Cleanup Plugin](https://plugins.jenkins.io/ws-cleanup)
we use Git which should be more reliable, and possibly also speed up the
initial checkout of the repo and submodules.
A nice side-effect is that the repo checkout on all builds after the first
one takes ~1 minute instead of ~5 minutes.
Previous issues:
- https://github.com/status-im/status-desktop/pull/7968
- https://github.com/status-im/status-desktop/pull/8046
Signed-off-by: Jakub Sokołowski <jakub@status.im>
This is a continuation of a fix done in:
https://github.com/status-im/status-desktop/pull/7968
This adds `deleteDirs` option to delete the whole workspace:
>When deferred wipeout is disabled, the old implementation of
>filesystem content deletion is used. If you want the same
>behavior as with deferred wipeout, you have to set the plugin
>attribute `deleteDirs` to true as well.
https://github.com/jenkinsci/ws-cleanup-plugin#deferred-wipeout
Because otherwise the `vendor` folder remains and causes
issues when `.git` is a file.
Signed-off-by: Jakub Sokołowski <jakub@status.im>
A potential partial fix found for an issue found in:
https://github.com/status-im/status-desktop/pull/7789
Where an incomplete broken checkout of a `vendor` module which crated a
`.git` file instead of a directory in `vendor/nimPNG` caused the error:
```
$ git submodule update --init --recursive vendor/nimPNG
fatal: Needed a single revision
Unable to find current revision in submodule path '../nimPNG'
```
Which then was not correctly cleaned up and in turn caused error:
```
org.jenkinsci.plugins.workflow.steps.MissingContextVariableException: Required context class hudson.FilePath is missing
Perhaps you forgot to surround the code with a step that provides this, such as: node, dockerNode
...
```
The last known successful CI build for that PR showed in logs:
```
00:05:13 [WS-CLEANUP] Deleting project workspace...
00:05:13 [WS-CLEANUP] Deferred wipeout is used...
00:05:13 [WS-CLEANUP] done
```
Which means the `Workspace Cleanup` plugin uses the `Resource Disposer`
plugin to wipe the workspaace in the background, which could potentially fail:
https://plugins.jenkins.io/ws-cleanup/#plugin-content-deferred-wipeout
Signed-off-by: Jakub Sokołowski <jakub@status.im>
By using `s3cmd` through PowerShell instead of Git Bash we improve
upload speeds on Windows slightly, as described here:
https://github.com/status-im/infra-ci/issues/40
Signed-off-by: Jakub Sokołowski <jakub@status.im>
Upgrade of Jenkins to `2.343` has introduced a security fix that breaks
caching plugin when it's configured to store cache on Master host:
https://issues.jenkins.io/browse/JENKINS-67173
Sine the [Caching plugin](https://plugins.jenkins.io/jobcacher/) hasn't
been upgraded in 5 years the only good temporary workaround is just drop
caching of dependencies like Nim compiler entirely.
In the future we can try some other caching methods.
Related: https://github.com/status-im/nimbus-eth2/pull/3594
Signed-off-by: Jakub Sokołowski <jakub@status.im>
It appears this was some leftover from ancient times and wasn't being used.
On the other hand the `RELEASE` environment variable controls if Nim builds
of the client binary include debug symbols:
ba7a6d5d34/Makefile (L177-L184)
The shorthand `?:` symbol means that if this is changed for a given job it stays changed.
Signed-off-by: Jakub Sokołowski <jakub@status.im>
Adds `scripts/sign-linux-file.sh` which expectes the following variables set:
* `LINUX_GPG_PRIVATE_KEY_FILE` - Path to the GPG export of private key.
* `LINUX_GPG_PRIVATE_KEY_PASS` - Password necessary to use the private key.
Given a file it creates a file with a `.asc` suffix containing the signature:
```
> wget -q https://status-im-prs.ams3.digitaloceanspaces.com/StatusIm-210809-104514-156806-pr.tar.gz
> tar xvf StatusIm-210809-104514-156806-pr.tar.gz
StatusIm-210809-104514-156806-pr.AppImage
StatusIm-210809-104514-156806-pr.AppImage.asc
> gpg --verify StatusIm-210809-104514-156806-pr.AppImage.asc
gpg: assuming signed data in 'StatusIm-210809-104514-156806-pr.AppImage'
gpg: Signature made Mon 09 Aug 2021 12:54:49 PM CEST using RSA key ID E20B4DFD
gpg: Good signature from "Status.im Devel Signing (GPG key for signing Status.im development builds.) <devel@status.im>" [ultimate]
Primary key fingerprint: BBF0 5F92 536B ED19 30A9 FD44 009F B3BF E20B 4DFD
```
Issue: https://github.com/status-im/infra-ci/issues/25
Requires: https://github.com/status-im/status-jenkins-lib/pull/32
Signed-off-by: Jakub Sokołowski <jakub@status.im>
This introduces the `scripts/sign-windows-bin.sh` script which is used
by the `Makefile` to sign application libraries and executables. It also
implements the logic necessary to distinguish between different types of
builds: release and non-release builds.
Some other changes:
* Refactore the `Makefile` target that creates the Windows ZIP to make less verbose.
* Added `Microsoft.VisualStudio.Component.Windows10SDK.10240` to VisualStudio component
* Added `BUILD_TYPE` parameter to `Jenkinsfile`s for different platform builds
Signed-off-by: Jakub Sokołowski <jakub@status.im>
This adds a `Publish` stage to the combined `Jenkinsfile` which when the
`PUBLISH` parameters is true pushes a __draft__ release to GitHub.
The job remembers the last value of `PUBLISH` parameter selected.
The release uses the contents of the `VERSION` file at the root of the
repo as the name for the release, and leaves the contents to be filled
in by whoever will approve the release.
The automation overwrites - or to be exact, deletes and recreates - the
release, so releated builds for the same release will simply re-create
it. All the built artifacts are included in the release.
Signed-off-by: Jakub Sokołowski <jakub@status.im>
This is part of a general restructuring in layout of Jenkins job folders
before we can properly introduce Release signing and notarization.
We need this to distinguish between pr/dev builds and release ones to
avoid signing the dev builds with a release certificate.
The meta job managed with `ci/Jenkinsfile.combined` runs a job for all 3
platforms and currently is quite basic, but in the future can be
extended to include - like the mobile one - updating the nightlies page,
or publishing draft GitHub releases.
The addition of `make check-pkg-target-*` steps to other `Jenkinsfile`s
is necessary because the sub-jobs under `platforms` have no option for
checking out Git submodules at the beginning, so I'm making use of how
the `Makefile` works and triggering that with a target that doesn't do much.
Example job:
https://ci.status.im/job/status-desktop/job/release/job/ci-meta-release-job/
Signed-off-by: Jakub Sokołowski <jakub@status.im>
Jakub Sokołowski
Anthony Laibe
Jonathan Rainville
Andrei Smirnov
B.Melnik
B.Melnik