Fixes#2143.
Methuselah is the a task manager for long-running tasks. It allows fo registration of workers that will each run in their own thead. Each worker can pass messages to and recieve messages from the main thread.
MailserverWoker was also introduced which moves all mailserver model logic to a MethuselahWorker. All communication to/from the model is done via a MethuselahTask. Results of the task are returned to the main thread by way of `signal_handler`, which calls the QtObject slot specified in the task.
Mailsever also provides a way for the model to emit events inside of the worker. These events are forwarded to the main thread via the `receiveEvent` slot of the `MailserverController`.
Co-authored-by: Michael Bradley, Jr <michaelsbradleyjr@gmail.com>
refactor: move threadpool task declarations inline with views
Co-authored-by: Michael Bradley Jr. <michaelsbradleyjr@gmail.com>
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
The `TaskManager` threadpool is a memory-safe replacement for the `spawnAndSend` operations that are currently causing memory issues in status-desktop.
From a fundamental memory management point of view, `libstatus/settings`, `libstatus/contracts`, and `libstatus/tokens` (custom tokens) have all been converted to `{.threadvar.}`s and `Atomic[bool]`s to maintain the cache and `dirty` flag across threads, respectively, eliminating the need for thread locks and incorrect `{.gcsafe.}` compiler overrides.
The successful [recyclable threadpool experiment from `nim-task-runner`](https://github.com/status-im/nim-task-runner/blob/test/use-cases/test/use_cases/test_sync.nim) using `AsyncChannel[ThreadSafeString]`s was brought over to `status-desktop` and implemented in somewhat of a hardcoded manner, as we knew this would save some time instead of trying to create a fully fleshed out `nim-task-runner` API and build a miraculous macro that may or may not be able to generate the needed API.
The threadpool is started by the `TaskManager` and both the `TaskManager` and the `TaskManager`'s threadpool are started as early as possible in the application lifecycle (in `nim_status_client.nim`). The `TaskManager` creates a thread to run the threadpool. During its initialization, the threadpool then spools up all the threads it will manage and puts them in an idle thread sequence. This is to prevent expensive thread creation and teardown happening during the app's lifetime as it is quite expensive and blocks the main thread. When tasks comes in to the pool, the task is sent to an idle thread, or put in a queue if all threads are busy. The idle thread is moved to the busy thread sequence. When a task is completed, the thread is taken out of the busy threads sequence and moved back in to the sequence of idle threads, effectively recycling it.
The first `spawnAndSend` we were able to change over to the new threadpool was `estimate`, which estimates the gas of a sticker purchase transaction.
From the consumer point of view, the existing `spawnAndSend` to achieve this looks like:
```nim
proc estimate*(self: StickersView, packId: int, address: string, price: string, uuid: string) {.slot.} =
let status_stickers = self.status.stickers
spawnAndSend(self, "setGasEstimate") do:
var success: bool
var estimate = status_stickers.estimateGas(packId, address, price, success)
if not success:
estimate = 325000
let result: tuple[estimate: int, uuid: string] = (estimate, uuid)
Json.encode(result)
```
And the new syntax looks like this:
```nim
proc estimate*(self: StickersView, packId: int, address: string, price: string, uuid: string) {.slot.} =
self.status.taskManager.threadPool.stickers.stickerPackPurchaseGasEstimate(cast[pointer](self.vptr), "setGasEstimate", packId, address, price, uuid)
```
The logic inside the `spawnAndSend` body was moved to [src/status/tasks/stickers.nim](https://github.com/status-im/status-desktop/compare/experiment/tasks-3?expand=1#diff-09e57eef00b0cee5c4abdb9039f948d8372e7003e09e934a9b4c7e9167d47658).
This is just the first migration of `spawnAndSend`, however moving the majority of the remaining `spawnAndSend`s will likely just be an exercise in copy/pasta. There will be one or two that may require a bit more thinking, depending how they rely on data from the model.
Once the `spawnAndSend`s have been converted to the threadpool, we can start implementing the [long-running process from the task runner use case experiments](https://github.com/status-im/nim-task-runner/blob/test/use-cases/test/use_cases/test_long_running.nim).
And finally, we can then implement the [async tasks](https://github.com/status-im/nim-task-runner/blob/test/use-cases/test/use_cases/test_async.nim) if needed.
@michaelsbradleyjr and I spent many hours digging in to the depths of nim's memory management in an attempt to understand it. We have created [a presentation with our task runner experiment findings](https://docs.google.com/presentation/d/1ItCxAfsVTcIoH_E4bgvmHljhbU-tC3T6K2A6ahwAedk/edit?usp=sharing), and @michaelsbradleyjr has spent time [answering questions off the back of that presentation.](https://gist.github.com/michaelsbradleyjr/1eaa9937b3fbb4ffff3fb814f0dd82a9).
We have created a fork of `edn.nim` at `status-im/edn.nim` and we need the PR to be merged and the commit hash updated before we can merge this PR in to `status-desktop`.
When the network connection is changed, the sticker packs are cleared and then re-loaded (either loading the offline (installed) sticker packs, or all the sticker packs from the network).
Stickers can be sent while offline, though the sticker images do not load once offline (this is likely a side effect of the bug described below).
There is a known bug in QNetworkAccessManager (https://bugreports.qt.io/browse/QTBUG-55180) that was re-introduced in 5.14.1 that does not allow us to download resources if we go offline then come back online. The workaround employed in this PR manually sets the NetworkAccessible property of QNetworkAccessManager once we have been connected back online. The NetworkAccessible property is marked as obsolete and will be removed in Qt6, so it is something that we need to be aware of when we upgrade. However the hope is that the bug has been fixed.
Close StickersPopup when disconnected from network (so that re-loading of sticker packs doesn't look out of place).
fix: set network status correctly at load
feat: stickers gas estimate async
feat: When network re-connected, any http images that were not properly loaded in an ImageLoader component will automatically be reloaded.
fix: Sticker button loading icon
chore: Bump nimqml and dotherside
NOTE: This PR uses an updated nimqml and dotherside. The respective changes should be merged first, and the commit hash should be bumped in this PR prior to merging. Relevant PRs:
[https://github.com/status-im/dotherside/pull/20](https://github.com/status-im/dotherside/pull/20)
[https://github.com/status-im/nimqml/pull/17](https://github.com/status-im/nimqml/pull/17)
By default the desktop app uses port 30305, unless a value is specifically set
in environment variable `STATUS_PORT`. For convenience of those developing the
app and running development builds, while simultaneously running production
builds (a.k.a. packaged builds, whether built locally or in CI), have `make
run` invoke `bin/nim_status_client` with `STATUS_PORT=30306`. That value can
still be overriden by manually invoking `make run` with a different value,
e.g. `make STATUS_PORT=30307 run` and `STATUS_PORT=30307 make run` are both
valid and achieve the same effect. NOTE: the port "sticks" in the database
after the first clean run, so when changing ports developers will need to
delete the data dir within their local repository, else the existing database
for dev builds will be stuck using whatever port was used previously. In the
future, we can figure out a means to always update the port setting in the
database just after the login event (but see #1505).
Also, for development builds use an icon (for the running app) that is
orangered (`#ff4500`) instead of the official blue color. This makes it much
easier to select between a running production instance and development instance
when Cmd-Tab'ing (on macOS, or equiv on Linux and Windows) through open
applications. Not all icons displayed at runtime have been changed in this
manner for development builds, just the main application icon, and that seems
to be sufficient to achieve the desired effect; though in the future we could
do similar for notification icon, menu bar icon, etc.
In the context of Windows packaged builds, this is consistent with other
resources needed by the application; organizing them in this way keeps the
top-level directory of Windows packaged builds clear of files that most users
should never have to see or think about.
- Changed WebView to WebEngineView
- Created a new controller/view for the web3 provider
- Created a private profile
- Created a channel for comms browser - qml
Necessary cases are:
- Onboarding to show the list of 5 accounts
- In QML when we show it to the user for the backup
- Change it to a Loader, so the component and its memory is cleaned when closed
Implement a `pkg-windows` target that ultimately results in `Status.zip` being
written to `pkg/`.
Note: this commit does not introduce code signing for the Windows build since
that piece is still a work in progress.
`pkg-windows` creates a portable folder in `tmp/windows/dist` with the help of
[`windeployqt`][windeployqt], which copies the needed portions of Qt into the
folder.
Since DLL resolution is relatively inflexible, a launcher `Status.exe` is
created at the top-level of the folder; the launcher opens `bin/Status.exe`
while adding the portable folder's `bin/` to the `PATH`, allowing
`bin/Status.exe` to resolve the DLLs in that folder.
A few additional tools need to be installed (e.g. with [scoop][scoop]) and
availble in `PATH`:
* 7-zip
* dos2unix (provides unix2dos)
* findutils
* go
* rcedit
* wget
The above list builds on the tools list in PR #521, and the other requirements
and instructions in that PR's description still apply.
**Why not build an installer?**
When starting work on packaging for the Windows build, my initial plan was to
build an installer, and for that purpose I researched the [WiX Toolset][wix],
the [Qt Installer Framework][qtif], and some other options.
I found that building an installer is a bit complex. I then recalled, from
personal experience, that [Cmder][cmder]'s [Mini download][mini] is
installer-less. You simply unzip the download and place the `cmder_mini` folder
wherever you prefer. Such an approach was also recommended to me in one of the
Nim language's community chats.
In addition to being simpler, the installer-less approach also gives
installation of Status Desktop a lower profile than an installer-application
would since nothing is written to the Windows registry, added to the *Add or
remove programs* list, etc. I think that's a benefit given the privacy-security
focus of Status, but others may feel differently so please provide feedback on
this point!
[windeployqt]: https://doc.qt.io/qt-5/windows-deployment.html
[scoop]: https://scoop.sh/
[wix]: https://wixtoolset.org/
[qtif]: https://doc.qt.io/qtinstallerframework/index.html
[cmder]: https://cmder.net/
[mini]: https://github.com/cmderdev/cmder/releases/download/v1.3.15/cmder_mini.zip
Move the onboarding/login state machine to the top level in main.qml, so that logout events can trigger new states.
Add Loader to statemachine so that each component is lazy-loaded. Initial tests saved 50MB of memory on startup.
Currently, logging out, then logging back in to the same or different account results in a doubling-up of chats/messages/wallet accounts. These need to be reset, however I need help doing that and it would delayed and blown out this PR further. This reset has been done for Onboarding and Login, but needs to be done for chats, wallet, mailservers, etc.
- unify the "build-..." targets
- enable a debug build by default, to simplify development
- bump vendor/DOtherSide
- avoid DOtherSide checks for docs/tests-specific tools like Doxygen
- switch to an in-place build for DOtherSide
- silence the DOtherSide build when V=0, make it more verbose with V=1
- don't delete checked out submodules in the "clean" target
- update build instructions in the README
- centralise Nim compiler options in a top-level "config.nims" (except
`-d:debug` which needs to be on the command line)
- unify the "build-..." targets
- enable a debug build by default, to simplify development
- bump vendor/DOtherSide
- avoid DOtherSide checks for docs/tests-specific tools like Doxygen
- switch to an in-place build for DOtherSide
- silence the DOtherSide build when V=0, make it more verbose with V=1
- don't delete checked out submodules in the "clean" target
- update build instructions in the README
- centralise Nim compiler options in a top-level "config.nims" (except
`-d:debug` which needs to be on the command line)
Eric Mastro
Michael Bradley, Jr
Jonathan Rainville
Richard Ramos
hydr063n
Iuri Matias
Ștefan Talpalaru