status-desktop/vendor
Eric Mastro 66912fd811 feat: introduce Task Manager threadpool
The `TaskManager` threadpool is a memory-safe replacement for the `spawnAndSend` operations that are currently causing memory issues in status-desktop.

From a fundamental memory management point of view, `libstatus/settings`, `libstatus/contracts`, and `libstatus/tokens` (custom tokens) have all been converted to `{.threadvar.}`s and `Atomic[bool]`s to maintain the cache and `dirty` flag across threads, respectively, eliminating the need for thread locks and incorrect `{.gcsafe.}` compiler overrides.

The successful [recyclable threadpool experiment from `nim-task-runner`](https://github.com/status-im/nim-task-runner/blob/test/use-cases/test/use_cases/test_sync.nim) using `AsyncChannel[ThreadSafeString]`s was brought over to `status-desktop` and implemented in somewhat of a hardcoded manner, as we knew this would save some time instead of trying to create a fully fleshed out `nim-task-runner` API and build a miraculous macro that may or may not be able to generate the needed API.

The threadpool is started by the `TaskManager` and both the `TaskManager` and the `TaskManager`'s threadpool are started as early as possible in the application lifecycle (in `nim_status_client.nim`). The `TaskManager` creates a thread to run the threadpool. During its initialization, the threadpool then spools up all the threads it will manage and puts them in an idle thread sequence. This is to prevent expensive thread creation and teardown happening during the app's lifetime as it is quite expensive and blocks the main thread. When tasks comes in to the pool, the task is sent to an idle thread, or put in a queue if all threads are busy. The idle thread is moved to the busy thread sequence. When a task is completed, the thread is taken out of the busy threads sequence and moved back in to the sequence of idle threads, effectively recycling it.

The first `spawnAndSend` we were able to change over to the new threadpool was `estimate`, which estimates the gas of a sticker purchase transaction.

From the consumer point of view, the existing `spawnAndSend` to achieve this looks like:
```nim
  proc estimate*(self: StickersView, packId: int, address: string, price: string, uuid: string) {.slot.} =
    let status_stickers = self.status.stickers
    spawnAndSend(self, "setGasEstimate") do:
      var success: bool
      var estimate = status_stickers.estimateGas(packId, address, price, success)
      if not success:
        estimate = 325000
      let result: tuple[estimate: int, uuid: string] = (estimate, uuid)
      Json.encode(result)
```
And the new syntax looks like this:
```nim
  proc estimate*(self: StickersView, packId: int, address: string, price: string, uuid: string) {.slot.} =
    self.status.taskManager.threadPool.stickers.stickerPackPurchaseGasEstimate(cast[pointer](self.vptr), "setGasEstimate", packId, address, price, uuid)
```
The logic inside the `spawnAndSend` body was moved to [src/status/tasks/stickers.nim](https://github.com/status-im/status-desktop/compare/experiment/tasks-3?expand=1#diff-09e57eef00b0cee5c4abdb9039f948d8372e7003e09e934a9b4c7e9167d47658).

This is just the first migration of `spawnAndSend`, however moving the majority of the remaining `spawnAndSend`s will likely just be an exercise in copy/pasta. There will be one or two that may require a bit more thinking, depending how they rely on data from the model.

Once the `spawnAndSend`s have been converted to the threadpool, we can start implementing the [long-running process from the task runner use case experiments](https://github.com/status-im/nim-task-runner/blob/test/use-cases/test/use_cases/test_long_running.nim).

And finally, we can then implement the [async tasks](https://github.com/status-im/nim-task-runner/blob/test/use-cases/test/use_cases/test_async.nim) if needed.

@michaelsbradleyjr and I spent many hours digging in to the depths of nim's memory management in an attempt to understand it. We have created [a presentation with our task runner experiment findings](https://docs.google.com/presentation/d/1ItCxAfsVTcIoH_E4bgvmHljhbU-tC3T6K2A6ahwAedk/edit?usp=sharing), and @michaelsbradleyjr has spent time [answering questions off the back of that presentation.](https://gist.github.com/michaelsbradleyjr/1eaa9937b3fbb4ffff3fb814f0dd82a9).

We have created a fork of `edn.nim` at `status-im/edn.nim` and we need the PR to be merged and the commit hash updated before we can merge this PR in to `status-desktop`.
2021-03-18 13:15:05 -04:00
..
DOtherSide@5177a129d4 feat: introduce Task Manager threadpool 2021-03-18 13:15:05 -04:00
QR-Code-generator@d2283a645c build: bump vendor/QR-Code-generator to d2283a6 2020-07-08 11:36:30 -05:00
bearssl@ba5f468798 refactor: use nim-web3 library 2020-09-29 13:28:08 -04:00
chroma@6ec846e917 adding missing deps 2020-09-04 15:57:29 -04:00
edn.nim@3305e41f9d feat: introduce Task Manager threadpool 2021-03-18 13:15:05 -04:00
isaac@45a5cbbd54 feat: onboarding generate new account 2020-05-21 19:33:14 -04:00
nbaser@0c989e0d95 feat: use base32 + status infura ipfs for ens contenthash 2020-10-20 14:42:36 -04:00
news@e1d63564a2 refactor: use nim-web3 library 2020-09-29 13:28:08 -04:00
nim-base32@660680c1cb feat: use base32 + status infura ipfs for ens contenthash 2020-10-20 14:42:36 -04:00
nim-chronicles@fc3f2d3755 feat: add nim-chronicles 2020-05-21 15:16:24 -04:00
nim-chronos@bce0f878d1 feat: add nim-chronicles 2020-05-21 15:16:24 -04:00
nim-eth@3ddb498f2a refactor: use nim-web3 library 2020-09-29 13:28:08 -04:00
nim-faststreams@d3ef34b325 feat: add nim-chronicles 2020-05-21 15:16:24 -04:00
nim-http-utils@33d70b9f37 refactor: use nim-web3 library 2020-09-29 13:28:08 -04:00
nim-json-rpc@dff46c991d refactor: use nim-web3 library 2020-09-29 13:28:08 -04:00
nim-json-serialization@1dccd4b2ef chore: update to latest nim-json-serialization 2020-09-09 15:03:45 -04:00
nim-libp2p@70deac9e0d feat: introduce Task Manager threadpool 2021-03-18 13:15:05 -04:00
nim-metrics@f91deb7422 feat: get collectibles from the contracts and their respective apis 2020-06-18 10:55:48 -04:00
nim-secp256k1@fb9699702b adding missing deps 2020-09-04 15:57:29 -04:00
nim-serialization@474bdbf49c fix: transaction history handling 2020-09-07 12:54:59 -04:00
nim-status-go@f1ec58561d refactor: nim_status -> status_go 2021-03-16 16:37:27 -04:00
nim-stew@1db43c7234 fix: transaction history handling 2020-09-07 12:54:59 -04:00
nim-stint@9e49b00148
nim-task-runner@a87f3f85be feat: introduce Task Manager threadpool 2021-03-18 13:15:05 -04:00
nim-web3@bf6805dff9 refactor: use nim-web3 library 2020-09-29 13:28:08 -04:00
nimPNG@7ff39ec00d adding missing deps 2020-09-04 15:57:29 -04:00
nimage@d683a7319c adding missing deps 2020-09-04 15:57:29 -04:00
nimbus-build-system@e7694f16ce Update nimbus-build-system version 2021-03-11 10:36:18 -05:00
nimcrypto@30d0ceaba0 feat: onboarding generate new account 2020-05-21 19:33:14 -04:00
nimqml@29fca3ce2e feat: load installed stickers while offline 2020-12-28 14:29:38 -05:00
status-go@09942bf200 feat: add community requests, permissions, ENS and more 2021-03-03 16:45:23 -05:00
uuids@c5039c1cc6 feat: onboarding generate new account 2020-05-21 19:33:14 -04:00