Please do not run 20 different things in a single stage.
It makes debugging much harder than it needs to be, since now you can
see at a glance the startup of which container fails easily.
Changes:
- Starting of Ganache and Nim-Waku containers extracted to separate stages
- Cleanup of containers moved to `cleanup` step after tests are executed
- Many variables moved to `enrivonment` section for job and some stages
- The `throttle` effect narrowed down just to the `Tests` stage and not whole job
- RPC API is used to get the Multiaddress of Nim-Waku node instead of hardcoding key
- Removed no longer necessary `status-go` history node related files
- `Jenkinsfile.uitests` was renamed to `Jenkinsfile.e2e` to match CI job names
Signed-off-by: Jakub Sokołowski <jakub@status.im>
Possible fix for slow upload speeds and failures caused by most probably
hitting per-bucket rate limits of DigitalOcean:
>- 500 total operations per second to any individual bucket.
>- 300 combined PUT, POST, COPY, DELETE, and LIST operations per second
> to any individual Space. We may further limit LIST operations if
> necessary under periods of high load.
https://docs.digitalocean.com/products/spaces/details/limits/#rate-limits
Depends on: https://github.com/status-im/status-jenkins-lib/pull/52
Signed-off-by: Jakub Sokołowski <jakub@status.im>
It needs to be separate script, because once the submodules are broken
and the `variables.mk` file is not available from `nimbus-build-system`
then we cannot even call `make clean-git` successfully because it will
be caught by the `if` clause that checks for `variables.mk`.
Possible fix for issues with submodules not being updated in some
windows release builds.
Signed-off-by: Jakub Sokołowski <jakub@status.im>
This was added to fix some build issues to Linux:
https://github.com/status-im/status-desktop/pull/8233
But other 2 platforms also have had the cleanup stage change which could
have cause these submodules update issues.
Signed-off-by: Jakub Sokołowski <jakub@status.im>
This is a continuation of attempts to fix provlems with Windows CI hosts
getting into a broken state due to `.git` directory becoming a file for
random submodules for unknown reason.
Instead of relying on Jenkins functionality of `cleanWs()` function
provided by the [Workspace Cleanup Plugin](https://plugins.jenkins.io/ws-cleanup)
we use Git which should be more reliable, and possibly also speed up the
initial checkout of the repo and submodules.
A nice side-effect is that the repo checkout on all builds after the first
one takes ~1 minute instead of ~5 minutes.
Previous issues:
- https://github.com/status-im/status-desktop/pull/7968
- https://github.com/status-im/status-desktop/pull/8046
Signed-off-by: Jakub Sokołowski <jakub@status.im>
This is a continuation of a fix done in:
https://github.com/status-im/status-desktop/pull/7968
This adds `deleteDirs` option to delete the whole workspace:
>When deferred wipeout is disabled, the old implementation of
>filesystem content deletion is used. If you want the same
>behavior as with deferred wipeout, you have to set the plugin
>attribute `deleteDirs` to true as well.
https://github.com/jenkinsci/ws-cleanup-plugin#deferred-wipeout
Because otherwise the `vendor` folder remains and causes
issues when `.git` is a file.
Signed-off-by: Jakub Sokołowski <jakub@status.im>
A potential partial fix found for an issue found in:
https://github.com/status-im/status-desktop/pull/7789
Where an incomplete broken checkout of a `vendor` module which crated a
`.git` file instead of a directory in `vendor/nimPNG` caused the error:
```
$ git submodule update --init --recursive vendor/nimPNG
fatal: Needed a single revision
Unable to find current revision in submodule path '../nimPNG'
```
Which then was not correctly cleaned up and in turn caused error:
```
org.jenkinsci.plugins.workflow.steps.MissingContextVariableException: Required context class hudson.FilePath is missing
Perhaps you forgot to surround the code with a step that provides this, such as: node, dockerNode
...
```
The last known successful CI build for that PR showed in logs:
```
00:05:13 [WS-CLEANUP] Deleting project workspace...
00:05:13 [WS-CLEANUP] Deferred wipeout is used...
00:05:13 [WS-CLEANUP] done
```
Which means the `Workspace Cleanup` plugin uses the `Resource Disposer`
plugin to wipe the workspaace in the background, which could potentially fail:
https://plugins.jenkins.io/ws-cleanup/#plugin-content-deferred-wipeout
Signed-off-by: Jakub Sokołowski <jakub@status.im>
- Reviewed `suite_messaging/tst_ChatFlow` feature: Cleanup of duplicated steps and separation of action / validation applied.
- Reviewed `suite_messaging/tst_groupChat` feature.
- Added new tag to `Jenkins` file: `relyon-mailserver`.
- Added new `StartupSteps` class to manage init / startup static methods outside bdd steps. It can be used as a util in `hooks` and `steps`.
Closes#7931
By using `s3cmd` through PowerShell instead of Git Bash we improve
upload speeds on Windows slightly, as described here:
https://github.com/status-im/infra-ci/issues/40
Signed-off-by: Jakub Sokołowski <jakub@status.im>
Fix BigInt compilation on linux
Remove c++20 optimization for NamedType
Add assert for failing getChats that was providing an easy to miss warning
Enable linux CI build
Don't run tests. They fail to run in docker with
"malloc_consolidate(): invalid chunk size", probably due to status-go
Contains minimal account creation and login
Considerations:
- migrated status-go wrapper and login code from the fix/cpp-structure (241eec)
- Minimal refactoring and changes at the moment. Expect further refactoring
follow up to reach the desired state.
- Fix missing keychain initialization
- Fix accounts DB initialization call done by startup -> Controller.openedAccounts -> status-go.OpenAccounts calls
- Small refactoring and todos for other steps
- fix SignalsManager
- fix async access to dereferenced status-go memory from SignalsManager
- fix SignalsManager not starting when registering
- finish dev end to end test for create account and login
- small improvements and added TODOs for future work
- add onboarding test helpers and start messaging test
- Refactoring towards Login UI integration
Closes: #5909Closes: #6028
Linux dockerized build and AppDir generation for compilation errors
Also added AppImage TODOs, basic setup for MacOS and Windows but they
still need work to have at least validation running
Temporary disable CPP windows/mac until limitations are
addressed in a follow up commit
Extra: add build configuration fixes for rest of the platforms
closes: #5998
For some release builds the notarization step fails because Apple
backend takes too long to return results. But the timeout triggered is
the Jenkins job timeout and no the notarization timeout.
Signed-off-by: Jakub Sokołowski <jakub@status.im>
Upgrade of Jenkins to `2.343` has introduced a security fix that breaks
caching plugin when it's configured to store cache on Master host:
https://issues.jenkins.io/browse/JENKINS-67173
Sine the [Caching plugin](https://plugins.jenkins.io/jobcacher/) hasn't
been upgraded in 5 years the only good temporary workaround is just drop
caching of dependencies like Nim compiler entirely.
In the future we can try some other caching methods.
Related: https://github.com/status-im/nimbus-eth2/pull/3594
Signed-off-by: Jakub Sokołowski <jakub@status.im>
It appears this was some leftover from ancient times and wasn't being used.
On the other hand the `RELEASE` environment variable controls if Nim builds
of the client binary include debug symbols:
ba7a6d5d34/Makefile (L177-L184)
The shorthand `?:` symbol means that if this is changed for a given job it stays changed.
Signed-off-by: Jakub Sokołowski <jakub@status.im>
Qt provides support for various input methods through plugins.
Since the fcitx plugin is not delivered with Qt, it needs to be
built and deployed by us.
Fixes: #4436
Adds `scripts/sign-linux-file.sh` which expectes the following variables set:
* `LINUX_GPG_PRIVATE_KEY_FILE` - Path to the GPG export of private key.
* `LINUX_GPG_PRIVATE_KEY_PASS` - Password necessary to use the private key.
Given a file it creates a file with a `.asc` suffix containing the signature:
```
> wget -q https://status-im-prs.ams3.digitaloceanspaces.com/StatusIm-210809-104514-156806-pr.tar.gz
> tar xvf StatusIm-210809-104514-156806-pr.tar.gz
StatusIm-210809-104514-156806-pr.AppImage
StatusIm-210809-104514-156806-pr.AppImage.asc
> gpg --verify StatusIm-210809-104514-156806-pr.AppImage.asc
gpg: assuming signed data in 'StatusIm-210809-104514-156806-pr.AppImage'
gpg: Signature made Mon 09 Aug 2021 12:54:49 PM CEST using RSA key ID E20B4DFD
gpg: Good signature from "Status.im Devel Signing (GPG key for signing Status.im development builds.) <devel@status.im>" [ultimate]
Primary key fingerprint: BBF0 5F92 536B ED19 30A9 FD44 009F B3BF E20B 4DFD
```
Issue: https://github.com/status-im/infra-ci/issues/25
Requires: https://github.com/status-im/status-jenkins-lib/pull/32
Signed-off-by: Jakub Sokołowski <jakub@status.im>
We create a separate make target to avoid tarballing every time.
This is really useful only for releases we upload for users.
Signed-off-by: Jakub Sokołowski <jakub@status.im>
This introduces an automated MacOS notarization process for Jenkins CI.
The process involves:
* Uploading the signed DMG file to the notary service
* Checking periodically if the scanning process has completed
* Stapling the successful scan ticket to the DMG file
This is done by the `scripts/notarize-macos-pkg.sh` via the `make notarize-macos` target.
The whole process is described in more details in `docs/macos_notarization.md`.
Depends on: https://github.com/status-im/status-jenkins-lib/pull/27
Resolves: https://github.com/status-im/status-desktop/issues/2169
Signed-off-by: Jakub Sokołowski <jakub@status.im>
This introduces the `scripts/sign-windows-bin.sh` script which is used
by the `Makefile` to sign application libraries and executables. It also
implements the logic necessary to distinguish between different types of
builds: release and non-release builds.
Some other changes:
* Refactore the `Makefile` target that creates the Windows ZIP to make less verbose.
* Added `Microsoft.VisualStudio.Component.Windows10SDK.10240` to VisualStudio component
* Added `BUILD_TYPE` parameter to `Jenkinsfile`s for different platform builds
Signed-off-by: Jakub Sokołowski <jakub@status.im>
Make the docker-related logic for doing a local packaged build on Linux match
with the Jenkins CI build for Linux.
Also fix the problems with packaged builds for Linux (local or CI) where
gstreamer errors were displayed in the terminal shortly after app start/login
and there was no sound output.
This adds a `Publish` stage to the combined `Jenkinsfile` which when the
`PUBLISH` parameters is true pushes a __draft__ release to GitHub.
The job remembers the last value of `PUBLISH` parameter selected.
The release uses the contents of the `VERSION` file at the root of the
repo as the name for the release, and leaves the contents to be filled
in by whoever will approve the release.
The automation overwrites - or to be exact, deletes and recreates - the
release, so releated builds for the same release will simply re-create
it. All the built artifacts are included in the release.
Signed-off-by: Jakub Sokołowski <jakub@status.im>
This is part of a general restructuring in layout of Jenkins job folders
before we can properly introduce Release signing and notarization.
We need this to distinguish between pr/dev builds and release ones to
avoid signing the dev builds with a release certificate.
The meta job managed with `ci/Jenkinsfile.combined` runs a job for all 3
platforms and currently is quite basic, but in the future can be
extended to include - like the mobile one - updating the nightlies page,
or publishing draft GitHub releases.
The addition of `make check-pkg-target-*` steps to other `Jenkinsfile`s
is necessary because the sub-jobs under `platforms` have no option for
checking out Git submodules at the beginning, so I'm making use of how
the `Makefile` works and triggering that with a target that doesn't do much.
Example job:
https://ci.status.im/job/status-desktop/job/release/job/ci-meta-release-job/
Signed-off-by: Jakub Sokołowski <jakub@status.im>
Allow environmental override at runtime. Also, in the Makefile set a free-tier
default token so that setting up an Infura account isn't strictly necessary for
community contributors to build the app, even though in our docs it should be
recommended they do so.
Core contributors should setup their own free-tier Infura account, create a
key, and set it in the environment variable INFURA_TOKEN in their environment
used to build the desktop app locally.
There is one aspect of this work that is incomplete. Ideally, in the handler
for the `login` event the relevant settings in the database should always be
updated with the resolved Infura key. However, when calling
`getSetting[string](Setting.Networks_Networks)` in the handler it causes a
segfault every time. Neither the reason for the crash nor a workaround have
been worked out at this time.
Changes:
- Adds `ci/Dockerfile` for creating `statusteam/nim-status-client-build:latest` used in builds
- Adds `ci/Jenkinsfile.linux` and `ci/Jenkinsfile.macos` for respective platforms
- Simplifies MacOS signing by adding `scripts/sign-macos-pkg.sh` script
- Makes `Makefile` use `scripts/sign-macos-pkg.sh` to make the DMG
- Makes `APPIMAGE` and `DMG` in `Makefile` modifiable by environment
- Adds `--passL:"-headerpad_max_install_names"` to `NIM_PARAMS` to fix MacOS signing issues
Signed-off-by: Jakub Sokołowski <jakub@status.im>
Jakub Sokołowski
Anthony Laibe
Richard Ramos
Noelia
Jonathan Rainville
Stefan
Patryk Osmaczko
Michele Balistreri
Andrei Smirnov
B.Melnik
B.Melnik
Michael Bradley, Jr