refactor: move threadpool task declarations inline with views
Co-authored-by: Michael Bradley Jr. <michaelsbradleyjr@gmail.com>
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Move the remaining stickers spawnAndSend (obtainAvailableStickerPacks) into a threadpool task.
refactor: create a base class for tasks models to inherit from
NOTE: this branch is based off of `experiment/tasks-3` and should be rebased on master once that branch is merged.
The `TaskManager` threadpool is a memory-safe replacement for the `spawnAndSend` operations that are currently causing memory issues in status-desktop.
From a fundamental memory management point of view, `libstatus/settings`, `libstatus/contracts`, and `libstatus/tokens` (custom tokens) have all been converted to `{.threadvar.}`s and `Atomic[bool]`s to maintain the cache and `dirty` flag across threads, respectively, eliminating the need for thread locks and incorrect `{.gcsafe.}` compiler overrides.
The successful [recyclable threadpool experiment from `nim-task-runner`](https://github.com/status-im/nim-task-runner/blob/test/use-cases/test/use_cases/test_sync.nim) using `AsyncChannel[ThreadSafeString]`s was brought over to `status-desktop` and implemented in somewhat of a hardcoded manner, as we knew this would save some time instead of trying to create a fully fleshed out `nim-task-runner` API and build a miraculous macro that may or may not be able to generate the needed API.
The threadpool is started by the `TaskManager` and both the `TaskManager` and the `TaskManager`'s threadpool are started as early as possible in the application lifecycle (in `nim_status_client.nim`). The `TaskManager` creates a thread to run the threadpool. During its initialization, the threadpool then spools up all the threads it will manage and puts them in an idle thread sequence. This is to prevent expensive thread creation and teardown happening during the app's lifetime as it is quite expensive and blocks the main thread. When tasks comes in to the pool, the task is sent to an idle thread, or put in a queue if all threads are busy. The idle thread is moved to the busy thread sequence. When a task is completed, the thread is taken out of the busy threads sequence and moved back in to the sequence of idle threads, effectively recycling it.
The first `spawnAndSend` we were able to change over to the new threadpool was `estimate`, which estimates the gas of a sticker purchase transaction.
From the consumer point of view, the existing `spawnAndSend` to achieve this looks like:
```nim
proc estimate*(self: StickersView, packId: int, address: string, price: string, uuid: string) {.slot.} =
let status_stickers = self.status.stickers
spawnAndSend(self, "setGasEstimate") do:
var success: bool
var estimate = status_stickers.estimateGas(packId, address, price, success)
if not success:
estimate = 325000
let result: tuple[estimate: int, uuid: string] = (estimate, uuid)
Json.encode(result)
```
And the new syntax looks like this:
```nim
proc estimate*(self: StickersView, packId: int, address: string, price: string, uuid: string) {.slot.} =
self.status.taskManager.threadPool.stickers.stickerPackPurchaseGasEstimate(cast[pointer](self.vptr), "setGasEstimate", packId, address, price, uuid)
```
The logic inside the `spawnAndSend` body was moved to [src/status/tasks/stickers.nim](https://github.com/status-im/status-desktop/compare/experiment/tasks-3?expand=1#diff-09e57eef00b0cee5c4abdb9039f948d8372e7003e09e934a9b4c7e9167d47658).
This is just the first migration of `spawnAndSend`, however moving the majority of the remaining `spawnAndSend`s will likely just be an exercise in copy/pasta. There will be one or two that may require a bit more thinking, depending how they rely on data from the model.
Once the `spawnAndSend`s have been converted to the threadpool, we can start implementing the [long-running process from the task runner use case experiments](https://github.com/status-im/nim-task-runner/blob/test/use-cases/test/use_cases/test_long_running.nim).
And finally, we can then implement the [async tasks](https://github.com/status-im/nim-task-runner/blob/test/use-cases/test/use_cases/test_async.nim) if needed.
@michaelsbradleyjr and I spent many hours digging in to the depths of nim's memory management in an attempt to understand it. We have created [a presentation with our task runner experiment findings](https://docs.google.com/presentation/d/1ItCxAfsVTcIoH_E4bgvmHljhbU-tC3T6K2A6ahwAedk/edit?usp=sharing), and @michaelsbradleyjr has spent time [answering questions off the back of that presentation.](https://gist.github.com/michaelsbradleyjr/1eaa9937b3fbb4ffff3fb814f0dd82a9).
We have created a fork of `edn.nim` at `status-im/edn.nim` and we need the PR to be merged and the commit hash updated before we can merge this PR in to `status-desktop`.
We've introduced a regression in https://github.com/status-im/status-desktop/commit/f1e83f74b#diff-f35edd413addd14c1f81816d6b5ee2bcbdf85fa0e3295d324cb78c98e26d4327L364 where we check whether an RPC's `error` is `null`
and its `result` is not `null`.
This breaks the application with an illegal storage access,
as in case of a successful API call, the response will have only a `result` key
(even when it's `null`).
Since we haven't done anything with a possible `error` in the reponse even before
that change was made, this commit removes the `error` check and safe guards around
whether `result` exists.
Fixes#2062
Allow up to 5 images to be dragged and dropped in to one-on-one chats and in the timeline. Can be combined with the existing upload button. The upload file dialog has been changed to allow multiple selections. Drag and dropped images adhere to the following rules, with corresponding validations messages:
- Max 5 image
- Image size must be 0.5 MB or less
- File extension must be one of [".png", ".jpg", ".jpeg", ".heif", "tif", ".tiff"]
Drag and drop and uploaded images are now also deduplicated.
Prior to this commit, communities without an image would render invisible
in the navigation bar of the application. To avoid this, we're now falling
back to our StatusLetterIdenticon component, which renders the first letter
of the community name with the color of the community.
In https://github.com/status-im/status-desktop/commit/31a9d1a6f we've fixed
a bug where a contact's thumbnail hasn't been passed to the `saveContact`
API.
Unfortunately, that fix wasn't memory safe. There are cases when a contact's
`identityImage` is `nil`, resulting in illegal storage access when accessing
a contact's thumbnail.
This commit fixes the issue by safe guarding around `identityImage` possibly
being `nil`.
fixes#935
A bug occurs when someone requests a large amount of funds from you since the gas estimation will fail and there isn't a way of handling errors in the source yet.
This PR handles the error appropriatley for both `estimateGas` and `estimateTokenGas` where the response is only converted from hex to int if the RPC call was successful. Otherwise return the error message as the response and let the UI decide how to display it.
Currently the error for gas estimation in transaction bubbles is displayed in a popup however, ive come to realize that 2 popups open instead of one. This is a new bug of which I can't pinpoint the root cause at the moment and have opted to file a separate issue for it.
When the network connection is changed, the sticker packs are cleared and then re-loaded (either loading the offline (installed) sticker packs, or all the sticker packs from the network).
Stickers can be sent while offline, though the sticker images do not load once offline (this is likely a side effect of the bug described below).
There is a known bug in QNetworkAccessManager (https://bugreports.qt.io/browse/QTBUG-55180) that was re-introduced in 5.14.1 that does not allow us to download resources if we go offline then come back online. The workaround employed in this PR manually sets the NetworkAccessible property of QNetworkAccessManager once we have been connected back online. The NetworkAccessible property is marked as obsolete and will be removed in Qt6, so it is something that we need to be aware of when we upgrade. However the hope is that the bug has been fixed.
Close StickersPopup when disconnected from network (so that re-loading of sticker packs doesn't look out of place).
fix: set network status correctly at load
feat: stickers gas estimate async
feat: When network re-connected, any http images that were not properly loaded in an ImageLoader component will automatically be reloaded.
fix: Sticker button loading icon
chore: Bump nimqml and dotherside
NOTE: This PR uses an updated nimqml and dotherside. The respective changes should be merged first, and the commit hash should be bumped in this PR prior to merging. Relevant PRs:
[https://github.com/status-im/dotherside/pull/20](https://github.com/status-im/dotherside/pull/20)
[https://github.com/status-im/nimqml/pull/17](https://github.com/status-im/nimqml/pull/17)
When sending a profile status update, the message has to be sent to
a specific channel that has the id `@PUBKEY`.
This commit introduces a flag that controls whether the message is
sent to the currently active channel, or tot he profile status channel.
The same is done for the `sendImage` API.
Fixes#1377.
Fixes#1479.
Two sites have been added to the whitelist: giphy.com and tenor.com.
`imageUrls` in its entirety has been removed and instead all links are being handle through the message `linkUrls`. This prevents double-handling of urls that may or may not be images.
The logic to automatically show links previews works like this:
1. If the setting "display chat images" is enabled, all links that *contain* ".png", ".jpg", ".jpeg", ".svg", ".gif" will be automatically shown. If the URL doesn't contain the extension, we are not downloading it. This was meant to be somewhat of a security compromise as we do not want to download each and every link posted in a message just to find out its true content type.
2. If the above setting is *disabled*, then we follow the whitelist settings for tenor and giphy. This allows us to preview gifs that do not have a file extension in their url.
feat: bump status-go to the commit that supports the new whitelist (https://github.com/status-im/status-go/pull/2094), and also lets us get link preview data from urls in the whitelist. NOTE: this commit was branched off status-go `develop`, so once it is merged, and we update this PR to the new commit, we will effectively be getting status-go develop changes. We *could* base that status-go PR off of master if it makes things easier.
fix: height on settings update issue
feat: move date/time of message below links
fix: layout issues when changing setting `neverAskAboutUnfurlingAgain`
feat: Add MessageBorder component to aid in showing rounded corners with different radius
Previously, this API would take a flag `oneToOne` and would use it to determine
whether the chat type is going to be `1` or `2`. However, there are many more chat
types, so it's important this API supports all of them.
Since we already have an enum in place, I'm changing this function to take it instead.
In addition, it also gets a `profile` parameter which is needed to implement
the status timeline functionality.
Primary motivator here was that the current `getContacts` APIs requries
a `ContactModel` which isn't always around. In fact, there's actually
no particular reason this APIs has to live on a model object.
However, to not break existing APIs I'm introducing a `getAllContacts`
API that returns all contacts as profiles, just like `contacts.getContacts`
does, without it being dependent on a `ContactModel`.
The same function is then used to add an API for returning all *added*
contacts as profiles.
When dealing with Timeline and Profile chat data, the `HEAD~1` would break
because we're trying to access `ChatType(4)` inside our `toChat` API.
To fix this issue, we have to make `ChatType` aware of `4` and `5` which are
`Profile` and `Timeline` respectively.
YouTube link unfurling was not working for a couple reasons.
There were two main parts fixed:
1. QML context for messages pertaining to linkUrls and imageUrls was changed from implicit to explicit. By this, I mean that any time we referenced linkUrls/imageUrls, we were relying on the knowledge that those values would be populated by some parent context several levels up. Now, we are referring to properties that have been explicitly defined on the components. This offers the ability to reuse components, and makes reading the code and debugging much easier.
2. Error handling has been added to getting link preview data. An unhandled "error" was thrown each time a link that wasn't whitelisted was passed in, causing the app to crash. For example, when a link to a tenor gif was posted in the chat, that URL was not whitelisted, causing the app to crash.
In timeline status update messages we want to render the ENS name next to a
local nickname in case it exists. This commit extends the `UsernameLabel` to do
just that.
Closes#1488
Allow environmental override at runtime. Also, in the Makefile set a free-tier
default token so that setting up an Infura account isn't strictly necessary for
community contributors to build the app, even though in our docs it should be
recommended they do so.
Core contributors should setup their own free-tier Infura account, create a
key, and set it in the environment variable INFURA_TOKEN in their environment
used to build the desktop app locally.
There is one aspect of this work that is incomplete. Ideally, in the handler
for the `login` event the relevant settings in the database should always be
updated with the resolved Infura key. However, when calling
`getSetting[string](Setting.Networks_Networks)` in the handler it causes a
segfault every time. Neither the reason for the crash nor a workaround have
been worked out at this time.
fix#898
The segmentation fault occured because the RPC response returned json with an error message as oppossed to the usual data required to update the chat. Since the section of the code didn't handle this error message it caused the app to crash. I've handled this error to show an error alert box by emitting a event
fix: SignTransactionModal - set default focused account when none is found
refactor: move token lookup from QML to nim in the toMessage procedure.
fix: 1:1 tx requests - handle case where token contract is not found (ie sending SNT from mainnet and receiving message on testnet)
feat: error checking for building a token transaction
feat: TransactionPreview - add a validation check that disallows continuation if the selected "from" account has insufficient funds
- Create privilege dialogs dynamically for each privilege requested
- Check if a privilege has been granted before to determine if dialog must be shown or not
- If dapp is allowed to use privilege, save it in the settings
- Create privilege dialogs dynamically for each privilege requested
- Check if a privilege has been granted before to determine if dialog must be shown or not
- If dapp is allowed to use privilege, save it in the settings
All tokens are now implemented as a strongly-typed Contract, Erc20Contract, or Erc721Contract. This prevents having two separate lists of overlapping tokens/contracts and normalises how to retreive the current SNT contract (depending on the network).
Fixes: #926.
Gas estimations were not being decoded correctly (indicated with "error getting gas price predictions" in the console) and were preventing transaction dialogs from continuing past the step containing the GasSelector component. This affected mainnet only, because in testnet we have hardcoded gas prices (for when the gas prices on mainnet are insane) which is why it was not apparent in testnet.
fix: Contract address not showing correctly
This was caused by `getStickerMarketContractAddress` being moved to `utilsView` but not updated in QML
Use nim-web3 library and remove internal encoding funcs that were copied from nim-web3.
Remove all instances of EthAddress (and therefore eth/common/eth_types imports)
Currently, exceptions thrown during transactions or gas estimation that were spawned in another thread are not being propagated, due to a limitation in nim (see https://nim-lang.org/docs/manual_experimental.html#parallel-amp-spawn).
This means any exceptions from status-go were not propagated correctly and would cause the app to crash. This includes entering the wrong password when trying to send a transaction.
The issue was addressed by passing a `success` variable by reference, which is set to false if an exception was thrown by status-go.
This commit address two bugs:
1. When importing a mnemonic, the resulting "master" account and its address
was used as account to be stored as opposed to its derived default account
2. The account was stored in the wrong path, resulting in status not being able
to locate accounts when sending transactions.
Fixes#787
Eric Mastro
Pascal Precht
Richard Ramos
Michael Bradley, Jr
Jonathan Rainville
staked-smart-ace
Malik Al-Jabr
Iuri Matias
chainvisions
