Refactor the activity filter session API to account for the new structure
Also:
- Refactor test helpers to mock for different chains with mixed answers
- Test implementation
Updates status-desktop #12120
* chore: move profile showcase structures to the indentity package
* feat: implement proof of membership for unecrypted communities
* feat: implement proof of membership for encrypted communties with grants
Switch from the prototype of duplicating the SQL filter as a runtime
and keeping them in sync on each event that might invalidate the current
filtered entries to a simpler approach of requesting the filter again
and doing the diff to detect the new changes.
Also add a new reset API to model the new entries design requirements.
The new approach shows less corner-case to handle and follows one source
of truth concept making debugging and future maintenance easier.
Other changes
- Fix pending mocking to work with multiple calls
- Refactor tests to account for the new changes
Updates status-desktop #12120
This commit introduces the first steps towards implementing a session-based activity API to support dynamic updates of the current visualized filter in the wallet activity service. This change is necessary to move away from static paginated filtering, which was previously done in SQL, to a more dynamic approach that can handle updates in real-time.
The main changes include:
- Add basic `EventActivitySessionUpdated` support for pending transactions.
- Added a `TODO.md` file outlining the plan and requirements for dynamic activity updates.
- New session-related API to the `activity.Service`
- `session.go` contains the logic for session management and event processing related to activity updates.
- Add test case for incremental filter updates.
The commit also includes:
- various other minor changes and refactoring to support the new session-based approach.
- Deprecation notices added to the `api.go` file for methods that are no longer used by the status-desktop application.
- Clarification comments added to the `scheduler.go` file regarding replacement policies.
Updates: #12120
ghstack-source-id: a61ef74184
Pull Request resolved: https://github.com/status-im/status-go/pull/4480
The reading of the amount for pending transactions was done in the same
way as for transfers table. However, the transfers table has a string
hex representation of the amount, while the pending transactions table
has a binary representation of the amount (*big.Int). This was
triggering the not int warning and value was missing.
Updates status-desktop #12120
In case both to/from addresses are present in the list we were using
the same logic as for transfers. However, this doesn't make sense given
that we can have only one entry in pending activity.
The following cases are still covered
- When the receiver is in addresses we get received
- When both receiver and sender are in the list will get sent
- When the sender is on the list we will get sent
Updates status-desktop #12120
keep restarting on error non-stop, removed some tests accordingly.
Fixed a flaky test for loadBlocksAndTransfers command.
Added tests for async.AtomicGroup.
Made tranfersCommand FiniteCommandWithErrorCounter to prevent infinite
restart.
- favourite column removed from the saved_addresses table
- favourite property removed from the SavedAddress struct
- ens name removed from the primary key, the primary key now is composed of address and is_test columns
- ens parameter removed from wakuext_deleteSavedAddress
- wallet_getSavedAddresses moved to wakuext_getSavedAddresses (to keep them all in a single place)
- saved addresses related endpoints removed from the wallet service, even they logically belong there, a reason for that
is avoiding emitting sync message if one uses calls from the wallet service, while that's not the case in ext service. Once
we refactor this and introduce devices syncing mechanism in the wallet service, we should not only these but other wallet
related endpoints move there (removed: wallet_getSavedAddresses, wallet_addSavedAddress and wallet_deleteSavedAddress).
Affected area:
Saved addresses
Reading the Nonce from the local cache may be incorrect if the tx is made out of the Status app or
if Status app sends a tx prepared by the dapp (via WalletConnect). A submitted tx with a wrong Nonce
results in a failing tx, that's why we need to read the Nonce from the network.
mock it in tests.
Made a configurable timeout interval for Commander interface.
Added tests to verify loadBlocksAndTransfers command is stopped
correctly on max errors limit reached
proper handling of errors and commands restart.
Now:
- Infinite commands started only once and never restarted, stoped on
context.Done.
- Finite commands are joined into AtomicGroup to stop the rest in the
group in case one command fails. Otherwise other commands in the group
will continue running and the failed command is not retried to
restart. Fixed goroutine leakage in case of failure of some commands
fixes#10329
linked with PR #12879
- Adds capacity to mark a message as unread
- Adds capacity to mark a message with mention as unread
- Adds persistence to the marking of the message (change can be seen at
after reboot)
- Adds marking in right click contextual menu
`getLogs` for multiple accounts simultaneously. For now only used for
new transfers detection. Detection of `new` transfers has been changed,
now they are searched from head and forward. Previously they were
searched from last scanned block forward.
- `WalletConnectTransfer` identified as a new transfer type
- Wallet-related endpoints that logically belong to the wallet moved from the wallet connect service
- Wallet connect service now receives `transfer.TransactionManager` instead of `transactions.Transactor`
- Deadlock issue when trying to send the tx with the wrong nonce fixed
* feat: Marking Mentions and Replies AC notifications as read also marks corresponding message as seen
* feat: Marking message as seen marks as read corresponding notification (if there is so)
* chore: make messenger activity center test less flaky
* Update VERSION
Add new APIs to track if valid pairings are available to be used
by application not to run WalletConnect SDK if not needed.
Closes status-desktop: #12794
Refactored transfers loading to reduce blockchain RPC requests (getBaseFee, getTransaction,
getTransactionReceipt) by reusing preloaded transaction and block fee.
Split extraction of subtransaction from logs and from ETH transfer into
different methods.
Refactored log_parser to extract sender and receiver addresses
uniformly for different transfer types.
Replaced info logs with debug where needed.
closes#4221
* feat: Add profile showcase messaging part with ecrypted data
* feat: Separate profile showcase categories to provide ablity to store custom data
* fix: review fixes
* feat: move profile showcase out of contact data
* fix: create index on contact id for profile tables
* chore: remove logger from link preview
* feat: profile showcase preferences basic impl
(squashed)
* feat: save preferences in batch for profile showcase
* chore: add validation for profile showcase settings request
and fix migration order
This functionality is needed in case the user wants to send a transaction and
signs it using the signature provided by the keycard (or any other compatible way).
- use protected topics for communities
- associate chats to pubsub topics and populate these depending if the chat belongs to a community or not
- mailserver functions should be aware of pubsub topics
- generate private key for pubsub topic protection when creating a community
- add shard cluster and index to communities
- setup shards for existing communities
- distribute pubsubtopic password
- fix: do not send the requests to join and cancel in the protected topic
- fix: undefined shard values for backward compatibility
- refactor: use shard message in protobuffers
New contracts and contract go functions.
Adjust owner&master tokens deployment flow.
Create deployment signature.
CommunityTokens API for handling signer pubkey.
Issue #11954
- use `appdatabse.DbInitializer{}` in tests to ensure consistent migrations
- remove protocol's open database functions due to improper
initialization caused by missing node config migration
- introduce `PushNotificationServerConfig` to resolve cyclic dependency
issues
The activity type filtering was not stable in relation to addresses
filter which was generating unexpected Send/Receive type in the
corner-case when both sender and receiver was in the address list.
Updates status-desktop #11960
Brings consistency in case when sender and receiver are both in the
filter address list. This fixes the case of sender and receiver in
addresses and filters out duplicate entries.
Also
- refactor tests to provide support for owners
- adapt TestGetActivityEntriesWithSameTransactionForSenderAndReceiverInDB
to the use of owner instead of from
Trigger async fetching of extra information on each activity filtering
request. Only emit the update event for incomplete entries.
Other changes:
- Make DataEntry light as event payload by making all the fields
optional
- Add new required fields to the activity DataEntry
- Add collectibles.ManagerInterface to aid testing
Note: this PR keeps compatibility with current master by always
providing non-optional multi-transaction ID. The TODO will be executed
before merging the status-desktop PR.
Experienced a hang on FetchAssetsByCollectibleUniqueID call with:
[{{5 0x21263a042aFE4bAE34F08Bb318056C181bD96D3b} 1209},
{{5 0x9A95631794a42d30C47f214fBe02A72585df35e1} 237},
{{5 0x9A95631794a42d30C47f214fBe02A72585df35e1} 236},
{{5 0x9A95631794a42d30C47f214fBe02A72585df35e1} 832},
{{5 0x9A95631794a42d30C47f214fBe02A72585df35e1} 830},
{{5 0x9A95631794a42d30C47f214fBe02A72585df35e1} 853}]
Updates status-desktop #11597
Balance history was not checked for all chains if no history on
some chain.
Removed `SetInitialRange` from wallet API as internal implementation.
This method was called on adding a brand new Status account to initialize
blocks_range table to avoid transfers history checks.
This commit skips deleting a keystore file for account which are not marked as fully operable
and also skips deleting master key keystore file if a keypair is non operable. It also takes into
consideration the operable property of an account when adding/updating/handling a keypair/keycard.
u3
refactor: associate chats to pubsub topics and populate these depending if the chat belongs to a community or not
refactor: add pubsub topic to mailserver batches
chore: ensure default relay messages continue working as they should
refactor: mailserver functions should be aware of pubsub topics
fix: use []byte for communityIDs
Mainly refactor API to have control on pending_transactions operations.
Use the new API to migrate the multi-transaction ID from to transfers
in one SQL transaction.
The refactoring was done to better mirror the purpose of pending_transactions
Also:
- Externalize TransactionManager from WalletService to be used by
other services
- Extract walletEvent as a dependency for all services that need to
propagate events
- Batch chain requests
- Remove unused APIs
- Add auto delete option for clients that fire and forget transactions
Updates status-desktop #11754
There is a desktop app feature where we need to transfer keystore files for selected
keypair/s only via local network using a QR code (of course, which are not migrated
to a keycard, otherwise we wouldn't need to do that).
Added a unit test for changing app and wallet DBs passwords.
Refactored geth_backend to simplify and allow wallet db password changing.
Fixed opening database with wrong password.
interface for initializing db, which is implemented for appdatabase and
walletdatabase. TBD for multiaccounts DB.
Unified DB initializion for all tests using helpers and new interface.
Reduced sqlcipher kdf iterations for all tests to 1.
The only place where appDB is used in wallet is activity,
which refers to `keycards_accounts` table. So a temporary
table `keycards_accounts` is created in wallet db and updated
before each activity query.
Small refactoring in Collectibles service in order to make API code more simple.
Add TokenInstance interface with some common functions.
Implementations of TokenInstance for TokenMaster, Collectibles and Assets contracts.
Issue #11276
Adding new smart contracts and generated go files.
Deploy token owner function and master token address getter.
Adding deployer and privilegesLevel columns to community_tokens table.
Passing addressFrom to API calls.
Issue #11250
1) Only fetch history for networks that match test mode enabled
* Trade Off: it will be only refetch in 12 hours so changing test mode won't trigger a refetch until app is restarted or 12 hours.
I think it is ok as change test mode is not a common use case
2) Do not consider networks that are enabled or not as this can be change more often than every 12 hours
* feat: proposal for collecting community metrics
https://github.com/status-im/status-desktop/issues/11152
* feat: collecting community message metrics with test
* feat: implement both strategies for fetching community metrics
* fix: review fixes
* fix: calc counts for timestamps
**This is a breaking change!**
Prior to this commit we had `AddCommunityToken(token *communities,
croppedImage CroppedImage)` that we used to
1. add a `CommunityToken` to the user's database and
2. to create a `CommunityTokenMetadata` from it which is then added to
the community's `CommunityDescription` and published to its members
However, I've then discovered that we need to separate these two things,
such that we can deploy a community token, then add it to the database
only for tracking purposes, **then** add it to the community description
(and propagate to members) once we know that the deploy tx indeed went
through.
To implement this, this commit introduces a new API
`SaveCommunityToken(token *communities.CommunityToken, croppedImage
CroppedImage)` which adds the token to the database only and doesn't
touch the community description.
The `AddCommunityToken` API is then changed that it's exclusively used
for adding an already saved `CommunityToken` to the community
description so it can be published to members. Hence, the signature is
now `AddCommunityToken(communityID string, chainID int, address
string)`, which makes this a breaking change.
Clients that used `AddCommunityToken()` before now need to ensure that
they first call `SaveCommunityToken()` as `AddCommunityToken()` will
fail otherwise.
If user followed onboarding flow to recover his account using seed phrase or keycard,
then `ProcessBackedupMessages` property of node config json object should be set to
`true`, otherwise it should be set to `false` or be omitted.
* feat: add api to remove private key and separete owner from private key ownership
For https://github.com/status-im/status-desktop/issues/11475
* feat: introduce IsControlNode for Community
* feat: remove community private key from syncing
* feat: add IsControlNode flag to Community json serialisation
* Update protocol/protobuf/pairing.proto
Co-authored-by: Jonathan Rainville <rainville.jonathan@gmail.com>
---------
Co-authored-by: Jonathan Rainville <rainville.jonathan@gmail.com>
* mute and unmute all community chats when community mute status changes
* unmute community when atleast one channel is unmuted
* fix: save community, extend the function to save muted state and mute duration
Modify API to handle also ERC20 tokens.
Modify community_tokens table - keep supply as string since string is easly convertible to bigint.BigInt.
Use bigint.BigInt for supply functions and fields.
Issue #11129
This is the second step of improvements over keypairs/keycards/accounts.
- `SyncKeycardAction` protobuf removed
- `SyncKeypair` protobuf is used for syncing keycards state as well as for all
keycards related changes
- `last_update_clock` column removed from `keypairs` table cause as well as
for accounts, any keycard related change is actually a change made on a related
keypair, thus a keypair's clock keeps the clock of the last change
- `position` column added to `keypairs` table, needed to display keycards in
the same order accross devices
Add wallet events feed to TransactionManager and send pending changed
events on add and delete
Also
- Remove hardcoded values in the filter query
- Small improvement to query
Updates status-desktop #11233
This is the first step of improvements over keypairs/keycards/accounts.
- `SyncKeypairFull` protobuf removed
- `SyncKeypair` protobuf is used for syncing all but the watch only accounts
- `SyncAccount` is used only for syncing watch only accounts
- related keycards are synced together with a keypair
- on any keypair change (either it's just a keypair name or any change made over an
account which belongs to that keypair) entire keypair is synced including related keycards
- on any watch only account related change, that account is synced with all its details
This commit extends the `AddCommunityToken` API to also expect an
optional `CroppedImage`, which will be used instead of the `ImageBase64`
path provided by `CommunityToken`, to calculate the actual base64
encoded image.
This fixes returning address identity after that was changed by
the real to/from fix. Previously the address was wrongly used as from
Updates status-desktop #11233
Added test to validate the fix
Also change JOIN to LEFT JOIN between `multi_transactions` and `transactions` table so that we return all entries from `multi_transactions` table even if there is no matching entry in `transactions` table, which might be the case for multi transactions that that could not be detected. This will postpone the error case until we get into details of the multi transaction.
Updates status-desktop #11233
* feat(share-links): Add protobuf and encode/decode url data methods
* feat(new-links-format): Adds generators for new links format
* feat: add parsing for new links format
* feat: add messenger-level pubkey serialization and tests
* feat: fix and test CreateCommunityURLWithChatKey
* feat: impl and test parseCommunityURLWithChatKey
* feat: fix and test CreateCommunityURLWithData
* feat: impl and test parseCommunityURLWithData (not working)
* feat: UrlDataResponse as response share urls api
* feat: impl& tested ShareCommunityChannelURLWithChatKey
* feat: impl & tested ParseCommunityChannelURLWithChatKey
* fix: bring urls to new format
* feat: add regexp for community channel urls
* feat: impl & test contact urls with chatKey, Ens and data
* fix: encodeDataURL/encodeDataURL patch from Samyoul
* fix: fix unmarshalling protobufs
* fix: fix minor issues, temporary comment TestParseUserURLWithENS
* fix: allow url to contain extra `#` in the signature
* fix: check signatures with SigToPub
* chore: lint fixes
* fix: encode the signature
* feat: Check provided channelID is Uuid
* fix(share-community-url): Remove if community encrypted scope
* fix: review fixes
* fix: use proto.Unmarshal instead of json.Marshal
* feat(share-urls): Adds TagsIndices to community data
* feat: support tag indices to community url data
---------
Co-authored-by: Boris Melnik <borismelnik@status.im>
Implement activity.Scheduler to serialize and limit the number of
calls on the activity service. This way we protect form inefficient
parallel queries and easy support async and rate limiting based on the
API requirements.
Refactor the activity APIs async and use the Scheduler for managing
the activity service calls configured with one of the two rules: cancel
ignore.
Updates status-desktop #11170
- Add ERC20 contract
- Add decimals field to community_tokens db table
- Adjusting API to handle assets deployment
- Add decimals field to CommunityTokenMetadata
Issue #10987
This commit adds new tables to the database and APIs in `Messenger` and
communities `Manager` to store `CheckChannelPermissionsResponse`s.
The responses are stored whenever channel permissions have been checked.
The reason we're doing this is so that clients can retrieve the last
known channel permission state before waiting for onchain checks to
finish.
Main changes:
- Refactor activity API to propagate token identities.
- Extend service to convert token identities to symbols for filtering
multi-transaction
- Filter transfers, pending_transactions and multi-transactions based
on the provided token identities
- Return involved token identities in activity API
- Test token filtering
Also:
- Fixed calling cancel on a filer activity completed task to release
resources
Notes:
- Found limitations with the token identity which complicates things
by not allowing to filter by token groups (like token-code does)
Updates status-desktop #11025
This API is used to get a permission status of all channels of a given
community.
Clients can use this API to get the provided information for all
community channels with a single RPC call instead of doing one call
for each channel separately.
Similar to `CheckPermissionToJoin()` we now get
a `CheckChannelPermissions()` API.
It will rely on the same `PermissionResponse` types, but gives
information about both `ViewOnlyPermissions` and
`ViewAndPostPermissions`.
The new API returns all known recipients of a wallet, by
sourcing transfers, pending_transactions and multi_transactions tables
The API is synchronous. Future work will be to make it async.
In some corner cases, when watching a famous wallet, it can
be that there are too many recipients to be returned in one go. Offset
and limit can be used to paginate through the results.
Updates status-desktop #10025
Stefan
Cuteivist
Sale Djenic
Mikhail Rogachev
Dario Gabriel Lipicar
Patryk Osmaczko
Anthony Laibe
Pablo Lopez
Roman Volosovskyi
Omar Basem
Igor Sirotin
Mohsen
Jonathan Rainville
Icaro Motta
Ivan Belyakov
Ibrahem Khalil
Siddarth Kumar
Richard Ramos
Mykhailo Prakhov
Anthony Laibe
Godfrain Jacques
Stefan Dunca
Andrea Maria Piana
Vitaly Vlasov
osmaczko
Michal Iskierko
Boris Melnik
yqrashawn
Mykhailo Prakhov
frank
Michał
Pascal Precht
Volodymyr Kozieiev
John Ngei