This introduces an automated MacOS notarization process for Jenkins CI.
The process involves:
* Uploading the signed DMG file to the notary service
* Checking periodically if the scanning process has completed
* Stapling the successful scan ticket to the DMG file
This is done by the `scripts/notarize-macos-pkg.sh` via the `make notarize-macos` target.
The whole process is described in more details in `docs/macos_notarization.md`.
Depends on: https://github.com/status-im/status-jenkins-lib/pull/27
Resolves: https://github.com/status-im/status-desktop/issues/2169
Signed-off-by: Jakub Sokołowski <jakub@status.im>
In the repo:
```
$ bin/nim_status_client --help
```
In the packaged app (macOS example):
```
$ cd /Applications/Status.app/Contents/MacOS
$ ./nim_status_client --help
```
Output:
```
Usage:
nim_status_client [OPTIONS]...
The following options are available:
-d, --dataDir Status Desktop data directory.
```
**Using the option**
```
$ cd ~/status-ci-builds/master/Status.app/Contents/MacOS
$ ./nim_status_client --dataDir:"${HOME}/status-dirs/master"
```
In another terminal:
```
$ cd ~/status-ci-builds/PR-4242/Status.app/Contents/MacOS
$ ./nim_status_client --dataDir:"${HOME}/status-dirs/PR-4242"
```
The path supplied can be relative or absolute, and can be specified with
`--dataDir:[path]`, `--dataDir=[path]`, `-d:[path]`, or `-d=[path]`.
Either `:` or `=` must be used, i.e. this *will not* work: `--dataDir [path]`
or `-d [path]`.
The name of the option follows Nim's partial case-insensitivity rules, so
`--dataDir`, `--datadir`, and `--data_dir` are all equivalent. See
[Identifier equality][ieq] in the Nim Manual.
It is possible to run the same build in multiple terminals by supplying
different `--dataDir`, i.e. this works:
```
$ cd /Applications/Status.app/Contents/MacOS
$ ./nim_status_client --dataDir="${HOME}/temp/some1"
```
In another terminal:
```
$ cd /Applications/Status.app/Contents/MacOS
$ ./nim_status_client --dataDir="${HOME}/temp/some2"
```
**Windows**
It is recommended to use a Git Bash or MSYS2 terminal when invoking
`bin/nim_status_client.exe` (development build) or `bin/Status.exe` (production
build) on the command-line. The reason is that if the exe is invoked in a
session of `cmd.exe` it will return to the prompt immediately; the app will run
but there will be no output in the terminal. In any case, the `--dataDir`
option will take effect whether the exe is invoked in `cmd.exe` or a
recommended terminal.
For development builds, when invoking `bin/nim_status_client.exe` directly
instead of via `make run`, because e.g. you wish to use the `--dataDir` option,
it is required to first setup the `PATH` environment variable correctly. See
the `run-windows` target in this repo's Makefile for more information.
**Linux**
The `--dataDir` option may be passed to command-line invocation of a
production (AppImage) build in the same way as passing it to a development
build:
```
$ Status.AppImage --dataDir:/path/to/wherever
```
For development builds, when invoking `bin/nim_status_client` directly instead
of via `make run`, because e.g. you wish to use the `--dataDir` option, it is
required to setup the `LD_LIBRARY_PATH` environment variable correctly. See the
`run-linux` target in this repo's Makefile for more information.
---
BREAKING CHANGE: The `qt` subdir of the app's data directory is now a sibling
of the status-go directory rather than a subdir of the status-go directory:
```
Status (app data directory)
├── data (status-go directory)
├── qt
└── tmp
```
Because app settings are stored in the `qt` directory that means that existing
installations will lose their customized settings.
At app startup, it would be possible to detect `Status/data/qt` and if
`Status/qt` doesn't exist yet then copy `Status/data/qt` to
`Status/qt`. However, there was some concern that behavior could lead to
problems later on if we forget the workaround is in place. So for now that
settings preservation strategy has not been implemented, but it might be before
this commit is merged pending full team awareness/consensus.
---
Command-line option support is provided by
[nim-confutils](https://github.com/status-im/nim-confutils).
The environment variable `NIM_STATUS_CLIENT_DEV` has been removed in favor of
passing a "define" option to the Nim compiler: `-d:development` for development
builds (e.g. `make V=1`) and `-d:production` for packaged builds (e.g. `make
V=1 pkg`). Passing the correct option is handled automatically by the Makefile.
A make variable named `RELEASE` has been introduced, which defaults to
`false`. Presently the `RELEASE` variable should not be set on the command-line
nor in CI as more work needs to be done to toggle the proper compiler flags. In
the case of Status Desktop, "release vs. debug" is a concern orthogonal to
"production vs. development". At present, production builds and development
builds are all debug builds, but that will likely change in the future: we can
have non-release CI production builds and local development builds be debug
builds, while release builds in CI would be production builds with
`RELEASE=true` (the compiled executable will be fully optimized).
Prior to the changes in this PR, symmetry is somewhat lacking between
development and production (packaged) builds with respect to the concept of the
"data directory". In development builds the root of the repo effectively serves
as the `Status` directory used by production builds, e.g. on macOS
`~/Library/Application Support/Status`. Also, there's a bit of confusion as to
whether "data directory" refers to a directory for the desktop app's overall
data (including status-go data) or to the specific directory used by status-go.
This PR attempts to provide symmetry and reduce confusion:
* The term "data directory" means the directory used by the desktop app to
store multiple kinds of data and is not a reference to the subdirectory used by
status-go.
* For development builds the "data directory" defaults to `./Status/` relative
to the root of the repo.
* For production builds the "data directory" default is the same as before,
e.g. on macOS it's ` ~/Library/Application Support/Status/`.
The directory used by status-go is `Status/data/`. To be clear, that should be
referred to as the "status-go directory" and not the app's "data directory". It
would nice if we could rename it from `Status/data/` to `Status/status-go/`. We
can do that, I already checked that it works correctly; however, for existing
installations it would require that at app launch we check for the presence of
`Status/data/` and rename it to `Status/status-go`. While simple enough to do,
I was concerned that there might be edge cases where the directory rename could
cause a problem (e.g. if another copy of the app is running) so chose for now
to stick with the status-go directory being `Status/data/`.
---
**NOTES**
More work needs to be done to ensure that all data written by the app is
contained in the default or cli-specified data directory. Currently, both
development and production (packaged) builds are writing to common directories
outside of the data directory, e.g. located within `~/Library/` on
macOS. Changing that behavior seems like it will mainly involve changing
defaults related to Qt components such as the web engine. See:
https://github.com/status-im/status-desktop/issues/1141.
In general, additional refactoring could be done in the future. For
example, implementing `StatusDesktopConfig` in
`src/status/libstatus/accounts/constants.nim` (as done in this PR) works fine
for now, but better code organization is desirable.
---
Closes#2268
[ieq]: https://nim-lang.org/docs/manual.html#lexical-analysis-identifier-equality
The `TaskManager` threadpool is a memory-safe replacement for the `spawnAndSend` operations that are currently causing memory issues in status-desktop.
From a fundamental memory management point of view, `libstatus/settings`, `libstatus/contracts`, and `libstatus/tokens` (custom tokens) have all been converted to `{.threadvar.}`s and `Atomic[bool]`s to maintain the cache and `dirty` flag across threads, respectively, eliminating the need for thread locks and incorrect `{.gcsafe.}` compiler overrides.
The successful [recyclable threadpool experiment from `nim-task-runner`](https://github.com/status-im/nim-task-runner/blob/test/use-cases/test/use_cases/test_sync.nim) using `AsyncChannel[ThreadSafeString]`s was brought over to `status-desktop` and implemented in somewhat of a hardcoded manner, as we knew this would save some time instead of trying to create a fully fleshed out `nim-task-runner` API and build a miraculous macro that may or may not be able to generate the needed API.
The threadpool is started by the `TaskManager` and both the `TaskManager` and the `TaskManager`'s threadpool are started as early as possible in the application lifecycle (in `nim_status_client.nim`). The `TaskManager` creates a thread to run the threadpool. During its initialization, the threadpool then spools up all the threads it will manage and puts them in an idle thread sequence. This is to prevent expensive thread creation and teardown happening during the app's lifetime as it is quite expensive and blocks the main thread. When tasks comes in to the pool, the task is sent to an idle thread, or put in a queue if all threads are busy. The idle thread is moved to the busy thread sequence. When a task is completed, the thread is taken out of the busy threads sequence and moved back in to the sequence of idle threads, effectively recycling it.
The first `spawnAndSend` we were able to change over to the new threadpool was `estimate`, which estimates the gas of a sticker purchase transaction.
From the consumer point of view, the existing `spawnAndSend` to achieve this looks like:
```nim
proc estimate*(self: StickersView, packId: int, address: string, price: string, uuid: string) {.slot.} =
let status_stickers = self.status.stickers
spawnAndSend(self, "setGasEstimate") do:
var success: bool
var estimate = status_stickers.estimateGas(packId, address, price, success)
if not success:
estimate = 325000
let result: tuple[estimate: int, uuid: string] = (estimate, uuid)
Json.encode(result)
```
And the new syntax looks like this:
```nim
proc estimate*(self: StickersView, packId: int, address: string, price: string, uuid: string) {.slot.} =
self.status.taskManager.threadPool.stickers.stickerPackPurchaseGasEstimate(cast[pointer](self.vptr), "setGasEstimate", packId, address, price, uuid)
```
The logic inside the `spawnAndSend` body was moved to [src/status/tasks/stickers.nim](https://github.com/status-im/status-desktop/compare/experiment/tasks-3?expand=1#diff-09e57eef00b0cee5c4abdb9039f948d8372e7003e09e934a9b4c7e9167d47658).
This is just the first migration of `spawnAndSend`, however moving the majority of the remaining `spawnAndSend`s will likely just be an exercise in copy/pasta. There will be one or two that may require a bit more thinking, depending how they rely on data from the model.
Once the `spawnAndSend`s have been converted to the threadpool, we can start implementing the [long-running process from the task runner use case experiments](https://github.com/status-im/nim-task-runner/blob/test/use-cases/test/use_cases/test_long_running.nim).
And finally, we can then implement the [async tasks](https://github.com/status-im/nim-task-runner/blob/test/use-cases/test/use_cases/test_async.nim) if needed.
@michaelsbradleyjr and I spent many hours digging in to the depths of nim's memory management in an attempt to understand it. We have created [a presentation with our task runner experiment findings](https://docs.google.com/presentation/d/1ItCxAfsVTcIoH_E4bgvmHljhbU-tC3T6K2A6ahwAedk/edit?usp=sharing), and @michaelsbradleyjr has spent time [answering questions off the back of that presentation.](https://gist.github.com/michaelsbradleyjr/1eaa9937b3fbb4ffff3fb814f0dd82a9).
We have created a fork of `edn.nim` at `status-im/edn.nim` and we need the PR to be merged and the commit hash updated before we can merge this PR in to `status-desktop`.
Replaces PR #105.
Implement a `pkg-macos` target that ultimately results in `Status.dmg` being
written to `pkg/`. Due to [limitations][limits] of the OpenSSL `.dylib`s in
`/usr/lib/` on macOS, `libssl.a` and `libcrypto.a` are statically linked into
`bin/nim_status_client` from a [Homebrew][brew] "bottle" that is compatible
with macOS 10.13 (the oldest macOS version compatible with Qt v5.14).
`pkg-macos` creates an `.app` bundle layout in `tmp/macos/dist` based partly on
information in a very helpful [StackOverflow answer][so-answer]. Note the part
of the answer (toward the end) that explains a problem with the working
directory and how to fix it. That's the reason for the `nim_status_client.sh`
script introduced in this commit (it gets copied into the bundle). It's also
the reason for having `Info.plist` copied into the bundle before `macdeployqt`
is run (see below) and then overwriting it with `Info.runner.plist` before
creating the `.dmg` file. The app icons file `status-icon.icns` was taken from
`deployment/macos/` in the [status-react][sr] repo.
The [`macdeployqt`][macdeployqt] tool is used to copy the needed portions of Qt
into the bundle; it automatically updates `rpath`, etc. so the
`nim_status_client` executable in the bundle can locate the libs within the
bundle.
`macdeployqt` is run twice, for the "outer" and "inner" `.app` bundles,
because of an apparent bug in `macdeployqt` that results in QtWebEngine related
resources not being processed correctly on the first pass. This results in some
bloat in the final bundle but it seems unavoidable at present.
The [create-dmg][cdmg] tool is used to package the bundle into a `.dmg`
file. There are two reasons for this:
1. It produces a nice looking icon for the `.dmg` that overlays the Status logo
on an external disk icon.
2. `Info.plist` needs to be overwritten after running `macdeployqt` (see
explanation above) but before creating the `.dmg` file. If we passed the `-dmg`
cli option to `macdeployqt` to have it generate the `.dmg` file then it
wouldn't be possible to overwrite `Info.plist`.
So there is a cosmetic reason and a practical reason for using another
tool. Probably the biggest downside is that `create-dmg` is implemented in
Node.js so it needs to be installed with `npm`; that's the reason this commit
introduces `package.json`, etc. Note that zero code from `node_modules` ends up
in the `.app` bundle or the `.dmg` file.
Code signing of the macOS `.app` bundle and `.dmg` is attempted if the
environment variable `MACOS_CODESIGN_IDENT` is defined. In that case, the
environment variable `MACOS_KEYCHAIN_OPT` may optionally be defined with the
path to a preferred keychain database file.
Refactor a number of sections in the Makefile for consistency's sake, e.g. the
`appimage` target becomes `pkg-linux` and ultimately results in
`NimStatusClient-x86_64.AppImage` being written to `pkg/`.
Make a number of changes to bring the Linux packaging steps up-to-date and use
the `-qmlimport` cli option of `linuxdeployqt` to simplify resolution of Qt
plugins.
Note that `make pkg` will correctly resolve to `make pkg-linux` or `make
pkg-macos` depending on the OS in use.
Consistently use lower-case "c" in the name of *components* directories and
imports.
[limits]: https://developer.apple.com/forums/thread/124782
[brew]: https://brew.sh/
[so-answer]: https://stackoverflow.com/a/3251285
[sr]: https://github.com/status-im/status-react/tree/develop/deployment/macos
[macdeployqt]: https://doc.qt.io/qt-5/macos-deployment.html
[cdmg]: https://github.com/sindresorhus/create-dmg
Generates 5 random accounts with identicons, allows user to enter password, then stores the account and logs in to the statusgo node.
Add EventEmitter that notifies nim_status_client.nim once node has started and is logged in (likely needs some refactoring to include the eventemitter in the base controller class).
Add QML StateMachine for the onboarding view.
Add nimcrytpo, uuids, eventemitter, isaac dependencies via submodules.
Add button to Intro view to randomly gen account.
Work on this PR started before the build system was updated and at one point I
upgraded `nim_status_client.nimble` to use NimScript so the nimble command
would stop warning that the old format was being used. In team chat it was
discussed that since we're no longer using nimble for package management we
could simply delete `nim_status_client.nimble` to avoid confusion, which can be
done in another PR.
Introduce a BrowserLayout stub so the index will be calcualted correctly re:
the active tab.
Reorganize ChatLayout and NodeLayout into subdirs `Chat` and `Node`,
respectively.
Introduce ProfileLayout which uses a "LeftTab" approach similar to that of
WalletLayout. There remains quite a bit of styling work to be done in
ProfileLayout and its LeftTab. Also, it may be better to start over using a
SplitView like the ChatLayout, I'm not really sure.
It wasn't clear what should be the default view for the right-pane when Profile
is selected in the left-most TabBar. In this PR, it defaults to showing the
view corresponding to the ENS usernames button.
In the archived Figma for the desktop design, it seemed a picture could be set,
e.g. there is a headshot of a woman used in the Profile screen. To that end I
explored how to take a square image and clip/mask it so it appears round and I
included a larger placeholder image for that purpose. In the new design, and
with respect to mobile, it may not be possible to set a profile picture so the
code that rounds the image can maybe be dropped.
Onboarding carousel completed
Redo folder restructuring
NOTES:
1. nim_status_client binary is output to `./bin/nim_status_client`, so you'll need to update your workflow to run this file. README has been updated to reflect this. Also, if you're running VSCode, instructions for code reload were added.
2. All `ui` files now live in `ui/`
3. All lib files (ie `libstatus`) now live in `lib/`
4. Removed the Qt Creater user project file as this is unique to each user
5. Moved the project file to `ui/`, so just open up the `ui` folder in Qt creator and it should locate the project file
6. `-d:release` removed from compilation as it wasn't needed
7. Broke up main.qml in to the main app and onboarding components, however this should be broken down further into more reusable components
8. Click "Get started" to show chat
Format some files using nimpretty
Patryk Osmaczko
Richard Ramos
Sale Djenic
Anthony Laibe
Alexandra Betouni
Jakub Sokołowski
Michael Bradley, Jr
Eric Mastro
Jonathan Rainville
Iuri Matias