New contracts and contract go functions.
Adjust owner&master tokens deployment flow.
Create deployment signature.
CommunityTokens API for handling signer pubkey.
Issue #11954
- use `appdatabse.DbInitializer{}` in tests to ensure consistent migrations
- remove protocol's open database functions due to improper
initialization caused by missing node config migration
- introduce `PushNotificationServerConfig` to resolve cyclic dependency
issues
The activity type filtering was not stable in relation to addresses
filter which was generating unexpected Send/Receive type in the
corner-case when both sender and receiver was in the address list.
Updates status-desktop #11960
Brings consistency in case when sender and receiver are both in the
filter address list. This fixes the case of sender and receiver in
addresses and filters out duplicate entries.
Also
- refactor tests to provide support for owners
- adapt TestGetActivityEntriesWithSameTransactionForSenderAndReceiverInDB
to the use of owner instead of from
Trigger async fetching of extra information on each activity filtering
request. Only emit the update event for incomplete entries.
Other changes:
- Make DataEntry light as event payload by making all the fields
optional
- Add new required fields to the activity DataEntry
- Add collectibles.ManagerInterface to aid testing
Note: this PR keeps compatibility with current master by always
providing non-optional multi-transaction ID. The TODO will be executed
before merging the status-desktop PR.
Experienced a hang on FetchAssetsByCollectibleUniqueID call with:
[{{5 0x21263a042aFE4bAE34F08Bb318056C181bD96D3b} 1209},
{{5 0x9A95631794a42d30C47f214fBe02A72585df35e1} 237},
{{5 0x9A95631794a42d30C47f214fBe02A72585df35e1} 236},
{{5 0x9A95631794a42d30C47f214fBe02A72585df35e1} 832},
{{5 0x9A95631794a42d30C47f214fBe02A72585df35e1} 830},
{{5 0x9A95631794a42d30C47f214fBe02A72585df35e1} 853}]
Updates status-desktop #11597
Balance history was not checked for all chains if no history on
some chain.
Removed `SetInitialRange` from wallet API as internal implementation.
This method was called on adding a brand new Status account to initialize
blocks_range table to avoid transfers history checks.
This commit skips deleting a keystore file for account which are not marked as fully operable
and also skips deleting master key keystore file if a keypair is non operable. It also takes into
consideration the operable property of an account when adding/updating/handling a keypair/keycard.
u3
refactor: associate chats to pubsub topics and populate these depending if the chat belongs to a community or not
refactor: add pubsub topic to mailserver batches
chore: ensure default relay messages continue working as they should
refactor: mailserver functions should be aware of pubsub topics
fix: use []byte for communityIDs
Mainly refactor API to have control on pending_transactions operations.
Use the new API to migrate the multi-transaction ID from to transfers
in one SQL transaction.
The refactoring was done to better mirror the purpose of pending_transactions
Also:
- Externalize TransactionManager from WalletService to be used by
other services
- Extract walletEvent as a dependency for all services that need to
propagate events
- Batch chain requests
- Remove unused APIs
- Add auto delete option for clients that fire and forget transactions
Updates status-desktop #11754
There is a desktop app feature where we need to transfer keystore files for selected
keypair/s only via local network using a QR code (of course, which are not migrated
to a keycard, otherwise we wouldn't need to do that).
Added a unit test for changing app and wallet DBs passwords.
Refactored geth_backend to simplify and allow wallet db password changing.
Fixed opening database with wrong password.
interface for initializing db, which is implemented for appdatabase and
walletdatabase. TBD for multiaccounts DB.
Unified DB initializion for all tests using helpers and new interface.
Reduced sqlcipher kdf iterations for all tests to 1.
The only place where appDB is used in wallet is activity,
which refers to `keycards_accounts` table. So a temporary
table `keycards_accounts` is created in wallet db and updated
before each activity query.
Small refactoring in Collectibles service in order to make API code more simple.
Add TokenInstance interface with some common functions.
Implementations of TokenInstance for TokenMaster, Collectibles and Assets contracts.
Issue #11276
Adding new smart contracts and generated go files.
Deploy token owner function and master token address getter.
Adding deployer and privilegesLevel columns to community_tokens table.
Passing addressFrom to API calls.
Issue #11250
1) Only fetch history for networks that match test mode enabled
* Trade Off: it will be only refetch in 12 hours so changing test mode won't trigger a refetch until app is restarted or 12 hours.
I think it is ok as change test mode is not a common use case
2) Do not consider networks that are enabled or not as this can be change more often than every 12 hours
* feat: proposal for collecting community metrics
https://github.com/status-im/status-desktop/issues/11152
* feat: collecting community message metrics with test
* feat: implement both strategies for fetching community metrics
* fix: review fixes
* fix: calc counts for timestamps
**This is a breaking change!**
Prior to this commit we had `AddCommunityToken(token *communities,
croppedImage CroppedImage)` that we used to
1. add a `CommunityToken` to the user's database and
2. to create a `CommunityTokenMetadata` from it which is then added to
the community's `CommunityDescription` and published to its members
However, I've then discovered that we need to separate these two things,
such that we can deploy a community token, then add it to the database
only for tracking purposes, **then** add it to the community description
(and propagate to members) once we know that the deploy tx indeed went
through.
To implement this, this commit introduces a new API
`SaveCommunityToken(token *communities.CommunityToken, croppedImage
CroppedImage)` which adds the token to the database only and doesn't
touch the community description.
The `AddCommunityToken` API is then changed that it's exclusively used
for adding an already saved `CommunityToken` to the community
description so it can be published to members. Hence, the signature is
now `AddCommunityToken(communityID string, chainID int, address
string)`, which makes this a breaking change.
Clients that used `AddCommunityToken()` before now need to ensure that
they first call `SaveCommunityToken()` as `AddCommunityToken()` will
fail otherwise.
If user followed onboarding flow to recover his account using seed phrase or keycard,
then `ProcessBackedupMessages` property of node config json object should be set to
`true`, otherwise it should be set to `false` or be omitted.
* feat: add api to remove private key and separete owner from private key ownership
For https://github.com/status-im/status-desktop/issues/11475
* feat: introduce IsControlNode for Community
* feat: remove community private key from syncing
* feat: add IsControlNode flag to Community json serialisation
* Update protocol/protobuf/pairing.proto
Co-authored-by: Jonathan Rainville <rainville.jonathan@gmail.com>
---------
Co-authored-by: Jonathan Rainville <rainville.jonathan@gmail.com>
* mute and unmute all community chats when community mute status changes
* unmute community when atleast one channel is unmuted
* fix: save community, extend the function to save muted state and mute duration
Modify API to handle also ERC20 tokens.
Modify community_tokens table - keep supply as string since string is easly convertible to bigint.BigInt.
Use bigint.BigInt for supply functions and fields.
Issue #11129
This is the second step of improvements over keypairs/keycards/accounts.
- `SyncKeycardAction` protobuf removed
- `SyncKeypair` protobuf is used for syncing keycards state as well as for all
keycards related changes
- `last_update_clock` column removed from `keypairs` table cause as well as
for accounts, any keycard related change is actually a change made on a related
keypair, thus a keypair's clock keeps the clock of the last change
- `position` column added to `keypairs` table, needed to display keycards in
the same order accross devices
Add wallet events feed to TransactionManager and send pending changed
events on add and delete
Also
- Remove hardcoded values in the filter query
- Small improvement to query
Updates status-desktop #11233
This is the first step of improvements over keypairs/keycards/accounts.
- `SyncKeypairFull` protobuf removed
- `SyncKeypair` protobuf is used for syncing all but the watch only accounts
- `SyncAccount` is used only for syncing watch only accounts
- related keycards are synced together with a keypair
- on any keypair change (either it's just a keypair name or any change made over an
account which belongs to that keypair) entire keypair is synced including related keycards
- on any watch only account related change, that account is synced with all its details
This commit extends the `AddCommunityToken` API to also expect an
optional `CroppedImage`, which will be used instead of the `ImageBase64`
path provided by `CommunityToken`, to calculate the actual base64
encoded image.
This fixes returning address identity after that was changed by
the real to/from fix. Previously the address was wrongly used as from
Updates status-desktop #11233
Added test to validate the fix
Also change JOIN to LEFT JOIN between `multi_transactions` and `transactions` table so that we return all entries from `multi_transactions` table even if there is no matching entry in `transactions` table, which might be the case for multi transactions that that could not be detected. This will postpone the error case until we get into details of the multi transaction.
Updates status-desktop #11233
* feat(share-links): Add protobuf and encode/decode url data methods
* feat(new-links-format): Adds generators for new links format
* feat: add parsing for new links format
* feat: add messenger-level pubkey serialization and tests
* feat: fix and test CreateCommunityURLWithChatKey
* feat: impl and test parseCommunityURLWithChatKey
* feat: fix and test CreateCommunityURLWithData
* feat: impl and test parseCommunityURLWithData (not working)
* feat: UrlDataResponse as response share urls api
* feat: impl& tested ShareCommunityChannelURLWithChatKey
* feat: impl & tested ParseCommunityChannelURLWithChatKey
* fix: bring urls to new format
* feat: add regexp for community channel urls
* feat: impl & test contact urls with chatKey, Ens and data
* fix: encodeDataURL/encodeDataURL patch from Samyoul
* fix: fix unmarshalling protobufs
* fix: fix minor issues, temporary comment TestParseUserURLWithENS
* fix: allow url to contain extra `#` in the signature
* fix: check signatures with SigToPub
* chore: lint fixes
* fix: encode the signature
* feat: Check provided channelID is Uuid
* fix(share-community-url): Remove if community encrypted scope
* fix: review fixes
* fix: use proto.Unmarshal instead of json.Marshal
* feat(share-urls): Adds TagsIndices to community data
* feat: support tag indices to community url data
---------
Co-authored-by: Boris Melnik <borismelnik@status.im>
Implement activity.Scheduler to serialize and limit the number of
calls on the activity service. This way we protect form inefficient
parallel queries and easy support async and rate limiting based on the
API requirements.
Refactor the activity APIs async and use the Scheduler for managing
the activity service calls configured with one of the two rules: cancel
ignore.
Updates status-desktop #11170
- Add ERC20 contract
- Add decimals field to community_tokens db table
- Adjusting API to handle assets deployment
- Add decimals field to CommunityTokenMetadata
Issue #10987
This commit adds new tables to the database and APIs in `Messenger` and
communities `Manager` to store `CheckChannelPermissionsResponse`s.
The responses are stored whenever channel permissions have been checked.
The reason we're doing this is so that clients can retrieve the last
known channel permission state before waiting for onchain checks to
finish.
Main changes:
- Refactor activity API to propagate token identities.
- Extend service to convert token identities to symbols for filtering
multi-transaction
- Filter transfers, pending_transactions and multi-transactions based
on the provided token identities
- Return involved token identities in activity API
- Test token filtering
Also:
- Fixed calling cancel on a filer activity completed task to release
resources
Notes:
- Found limitations with the token identity which complicates things
by not allowing to filter by token groups (like token-code does)
Updates status-desktop #11025
This API is used to get a permission status of all channels of a given
community.
Clients can use this API to get the provided information for all
community channels with a single RPC call instead of doing one call
for each channel separately.
Similar to `CheckPermissionToJoin()` we now get
a `CheckChannelPermissions()` API.
It will rely on the same `PermissionResponse` types, but gives
information about both `ViewOnlyPermissions` and
`ViewAndPostPermissions`.
The new API returns all known recipients of a wallet, by
sourcing transfers, pending_transactions and multi_transactions tables
The API is synchronous. Future work will be to make it async.
In some corner cases, when watching a famous wallet, it can
be that there are too many recipients to be returned in one go. Offset
and limit can be used to paginate through the results.
Updates status-desktop #10025
Change smart contract with new API.
Update gas amount for deployment.
Add Burn() and EstimateBurn() functions.
Add RemainingSupply() functions.
Issue #10816
fetch strategy:
Before:
- block fetching commands for different accounts were in the same wait
group, making them dependent on each iteration.
- transfers loading command was checking database for new unloaded
blocks on timeout and was in the same wait group with block fetching, so
it was often blocked until all block fetching commands finish for
iteration.
Now:
- block fetching commands run independently for each account
- transfers fetching command is run once on startup for unloaded blocks
from DB
- fetch history blocks commands are launched once on startup for
accounts with no full history loaded
- transfers are loaded on each iteration of block range check
without waiting for all ranges to be checked
Refactor the filter interface to be an async call which returns
the result using a wallet event
A call to the filter API will cancel the ongoing filter and receive
an error result event
Closes status-desktop #10994
Main changes:
- extend DB query include status for transactions and pending_transactions
- extend DB query to aggregate data from complex activity entries
- extend tests to include the new status values
Other changes:
- Improve tests with mocked addresses in DB to have a true to/from state
Update status-desktop #10746
fix flaky test: TestRetrieveBlockedContact
resolve conflict when rebase origin/develop
Feat/sync activity center notification (#3581)
* feat: sync activity center notification
* add test
* fix lint issue
* fix failed test
* addressed feedback from sale
* fix failed test
* addressed feedback from ilmotta
go generate ./protocol/migrations/sqlite/...
feat: add updated_at for syncing activity center notification
Extended the migration process with a generic way of applying custom
migration code on top of the SQL files. The implementation provides
a safer way to run GO code along with the SQL migrations and possibility
of rolling back the changes in case of failure to keep the database
consistent.
This custom GO migration is needed to extract the status from
the JSON blob receipt and store it in transfers table.
Other changes:
- Add NULL DB value tracking to JSONBlob helper
- Index status column on transfers table
- Remove unnecessary panic calls
- Move log_parser to wallet's common package and use to extract token
identity from the logs
Notes:
- there is already an index on transfers table, sqlite creates one for
each unique constraint therefore add only status to a new index
- the planned refactoring and improvements to the database have been
postponed due to time constraints. Got the time to migrate the data
though, extracting it can be done later for a more efficient
implementation
Update status-desktop #10746
- reverted a change that stopped looking for ERC20 transfers if no nonce
and balance change found within a block range for ETH
- implemented sending EventRecentHistoryReady event at a proper time
- moved EventFetchingRecentHistory event to Strategy type as it does not make
sense to send this event in loop
- moved iterating through blocks logic to inside of `loadTransfers` command, which
now accepts a block range.
- reuse `uniqueHeaders` function in commands.go
- clean up
Updates #10246
* fix(community): stop re-joining comm when receiving a sync community msg
Fixes an issue with chats being reset. Since joining a community resaves the chats with the synced default value, it resets the sate of the chats, losing the unread messages, the muted state and more.
The solution is to block the re-joining of the community. In the case of the sync, we catch that error and just continue on.
* fix(import): fix HandleImport not saving the chat
Doesn't change much, but it could have caused issues in the future, so since we might have modified the chat, we make sure to save them
Also adds a test
* fix tests
- old `accounts` table is moved/mapped to `keypairs` and `keypairs_accounts`
- `keycards` table has foreign key which refers to `keypairs.key_uid`
- `Keypair` introduced as a new type
- api endpoints updated according to this change
It uses the current data only and doesn't extend with new types or
include new features in activity sources DBs.
Major changes:
- Partially filter by chain IDs
- Partially filter by Status if it is the case
- Partially filter by token types
- Filter by counterparty addresses
- Use wallet accounts for TO/FROM instead of filters
Closes: #10634
This commit does a few things:
1) Extend create/import account endpoint to get wallet config, some of
which has been moved to the backend
2) Set up a loop for retrieving balances every 10 minutes, caching the
balances
3) Return information about which checks are not passing when trying to
join a token gated community
4) Add tests to the token gated communities
5) Fixes an issue with addresses not matching when checking for
permissions
The move to the wallet as a background task is not yet complete, I need
to publish a signal, and most likely I will disable it before merging
for now, as it's currently not used by desktop/mobile, but the PR was
getting to big
were stored in cache by pointers, which caused falsy cache hits in loop
because pointers with same address were created for different block
numbers. Now cache uses block numbers of uint64 as key, which can
overflow but it is not a problem since we use this cache for values
comparison, not as user data.
Fix crash on nil pointer in log.
Remove some unused code.
Protect nonceRanges with mutex while reading.
Updates #10246
This is the initial implementation for the new URL unfurling requirements. The
most important one is that only the message sender will pay the privacy cost for
unfurling and extracting metadata from websites. Once the message is sent, the
unfurled data will be stored at the protocol level and receivers will just
profit and happily decode the metadata to render it.
Further development of this URL unfurling capability will be mostly guided by
issues created on clients. For the moment in status-mobile:
https://github.com/status-im/status-mobile/labels/url-preview
- https://github.com/status-im/status-mobile/issues/15918
- https://github.com/status-im/status-mobile/issues/15917
- https://github.com/status-im/status-mobile/issues/15910
- https://github.com/status-im/status-mobile/issues/15909
- https://github.com/status-im/status-mobile/issues/15908
- https://github.com/status-im/status-mobile/issues/15906
- https://github.com/status-im/status-mobile/issues/15905
### Terminology
In the code, I've tried to stick to the word "unfurl URL" to really mean the
process of extracting metadata from a website, sort of lower level. I use "link
preview" to mean a higher level structure which is enriched by unfurled data.
"link preview" is also how designers refer to it.
### User flows
1. Carol needs to see link previews while typing in the chat input field. Notice
from the diagram nothing is persisted and that status-go endpoints are
essentially stateless.
```
#+begin_src plantuml :results verbatim
Client->>Server: Call wakuext_getTextURLs
Server-->>Client: Normalized URLs
Client->>Client: Render cached unfurled URLs
Client->>Server: Unfurl non-cached URLs.\nCall wakuext_unfurlURLs
Server->>Website: Fetch metadata
Website-->>Server: Metadata (thumbnail URL, title, etc)
Server->>Website: Fetch thumbnail
Server->>Website: Fetch favicon
Website-->>Server: Favicon bytes
Website-->>Server: Thumbnail bytes
Server->>Server: Decode & process images
Server-->>Client: Unfurled data (thumbnail data URI, etc)
#+end_src
```
```
,------. ,------. ,-------.
|Client| |Server| |Website|
`--+---' `--+---' `---+---'
| Call wakuext_getTextURLs | |
| ---------------------------------------> |
| | |
| Normalized URLs | |
| <- - - - - - - - - - - - - - - - - - - - |
| | |
|----. | |
| | Render cached unfurled URLs | |
|<---' | |
| | |
| Unfurl non-cached URLs. | |
| Call wakuext_unfurlURLs | |
| ---------------------------------------> |
| | |
| | Fetch metadata |
| | ------------------------------------>
| | |
| | Metadata (thumbnail URL, title, etc)|
| | <- - - - - - - - - - - - - - - - - -
| | |
| | Fetch thumbnail |
| | ------------------------------------>
| | |
| | Fetch favicon |
| | ------------------------------------>
| | |
| | Favicon bytes |
| | <- - - - - - - - - - - - - - - - - -
| | |
| | Thumbnail bytes |
| | <- - - - - - - - - - - - - - - - - -
| | |
| |----. |
| | | Decode & process images |
| |<---' |
| | |
| Unfurled data (thumbnail data URI, etc)| |
| <- - - - - - - - - - - - - - - - - - - - |
,--+---. ,--+---. ,---+---.
|Client| |Server| |Website|
`------' `------' `-------'
```
2. Carol sends the text message with link previews in the RPC request
wakuext_sendChatMessages. status-go assumes the link previews are good
because it can't and shouldn't attempt to re-unfurl them.
```
#+begin_src plantuml :results verbatim
Client->>Server: Call wakuext_sendChatMessages
Server->>Server: Transform link previews to\nbe proto-marshalled
Server->DB: Write link previews serialized as JSON
Server-->>Client: Updated message response
#+end_src
```
```
,------. ,------. ,--.
|Client| |Server| |DB|
`--+---' `--+---' `+-'
| Call wakuext_sendChatMessages| |
| -----------------------------> |
| | |
| |----. |
| | | Transform link previews to |
| |<---' be proto-marshalled |
| | |
| | |
| | Write link previews serialized as JSON|
| | -------------------------------------->
| | |
| Updated message response | |
| <- - - - - - - - - - - - - - - |
,--+---. ,--+---. ,+-.
|Client| |Server| |DB|
`------' `------' `--'
```
3. The message was sent over waku and persisted locally in Carol's device. She
should now see the link previews in the chat history. There can be many link
previews shared by other chat members, therefore it is important to serve the
assets via the media server to avoid overloading the ReactNative bridge with
lots of big JSON payloads containing base64 encoded data URIs (maybe this
concern is meaningless for desktop). When a client is rendering messages with
link previews, they will have the field linkPreviews, and the thumbnail URL
will point to the local media server.
```
#+begin_src plantuml :results verbatim
Client->>Server: GET /link-preview/thumbnail (media server)
Server->>DB: Read from user_messages.unfurled_links
Server->Server: Unmarshal JSON
Server-->>Client: HTTP Content-Type: image/jpeg/etc
#+end_src
```
```
,------. ,------. ,--.
|Client| |Server| |DB|
`--+---' `--+---' `+-'
| GET /link-preview/thumbnail (media server)| |
| ------------------------------------------> |
| | |
| | Read from user_messages.unfurled_links|
| | -------------------------------------->
| | |
| |----. |
| | | Unmarshal JSON |
| |<---' |
| | |
| HTTP Content-Type: image/jpeg/etc | |
| <- - - - - - - - - - - - - - - - - - - - - |
,--+---. ,--+---. ,+-.
|Client| |Server| |DB|
`------' `------' `--'
```
### Some limitations of the current implementation
The following points will become separate issues in status-go that I'll work on
over the next couple weeks. In no order of importance:
- Improve how multiple links are fetched; retries on failure and testing how
unfurling behaves around the timeout limits (deterministically, not by making
real HTTP calls as I did). https://github.com/status-im/status-go/issues/3498
- Unfurl favicons and store them in the protobuf too.
- For this PR, I added unfurling support only for websites with OpenGraph
https://ogp.me/ meta tags. Other unfurlers will be implemented on demand. The
next one will probably be for oEmbed https://oembed.com/, the protocol
supported by YouTube, for example.
- Resize and/or compress thumbnails (and favicons). Often times, thumbnails are
huge for the purposes of link previews. There is already support for
compressing JPEGs in status-go, but I prefer to work with compression in a
separate PR because I'd like to also solve the problem for PNGs (probably
convert them to JPEGs, plus compress them). This would be a safe choice for
thumbnails, favicons not so much because transparency is desirable.
- Editing messages is not yet supported.
- I haven't coded any artificial limit on the number of previews or on the size
of the thumbnail payload. This will be done in a separate issue. I have heard
the ideal solution may be to split messages into smaller chunks of ~125 KiB
because of libp2p, but that might be too complicated at this stage of the
product (?).
- Link preview deletion.
- For the moment, OpenGraph metadata is extracted by requesting data for the
English language (and fallback to whatever is available). In the future, we'll
want to unfurl by respecting the user's local device language. Some websites,
like GoDaddy, are already localized based on the device's IP, but many aren't.
- The website's description text should be limited by a certain number of
characters, especially because it's outside our control. Exactly how much has
not been decided yet, so it'll be done separately.
- URL normalization can be tricky, so I implemented only the basics to help with
caching. For example, the url https://status.im and HTTPS://status.im are
considered identical. Also, a URL is considered valid for unfurling if its TLD
exists according to publicsuffix.EffectiveTLDPlusOne. This was essential,
otherwise the default Go url.Parse approach would consider many invalid URLs
valid, and thus the server would waste resources trying to unfurl the
unfurleable.
### Other requirements
- If the message is edited, the link previews should reflect the edited text,
not the original one. This has been aligned with the design team as well.
- If the website's thumbnail or the favicon can't be fetched, just ignore them.
The only mandatory piece of metadata is the website's title and URL.
- Link previews in clients should be generated in near real-time, that is, as
the user types, previews are updated. In mobile this performs very well, and
it's what other clients like WhatsApp, Telegram, and Facebook do.
### Decisions
- While the user typing in the input field, the client is constantly (debounced)
asking status-go to parse the text and extract normalized URLs and then the
client checks if they're already in its in-memory cache. If they are, no RPC
call is made. I chose this approach to achieve the best possible performance
in mobile and avoid the whole RPC overhead, since the chat experience is
already not smooth enough. The mobile client uses URLs as cache keys in a
hashmap, i.e. if the key is present, it means the preview is readily available
(naive, but good enough for now). This decision also gave me more flexibility
to find the best UX at this stage of the feature.
- Due to the requirement that users should be able to see independent loading
indicators for each link preview, when status-go can't unfurl a URL, it
doesn't return it in the response.
- As an initial implementation, I added the BLOB column unfurled_links to the
user_messages table. The preview data is then serialized as JSON before being
stored in this column. I felt that creating a separate table and the related
code for this initial PR would be inconvenient. Is that reasonable to you?
Once things stabilize I can create a proper table if we want to avoid this
kind of solution with serialized columns.
Add the possibility of retrieving the metadata of wallet activity based
on the given filter criteria.
Current implementation relies that after fetching the metadata, user
will follow up with more requests for details. However, after some
experimenting I'm considering extracting all required information
for the summary viewing while filtering. This way there will be no
need for another batch requests for transfers, multi-transactions and
pending transactions to show the summary. Only when user wants to see
the details for one will specifically request it.
For this first prototype, the filter criteria is limited to:
- time
- type
- addresses
Major changes:
- Add the filter definition to be used in propagating the filter
information
- Add GetActivityEntries API to return the list of activity entries
for the given addresses/chainIDs by a view in the complete list
- GetTransfersForIdentities to batch retrieve further details of the
transfers
- GetPendingTransactionsForIdentities to batch retrieve further details
of the pending transactions
- Added a new package testutils for tests.
- Added tests
Updates status-desktop #10366
Closes status-desktop #10633
This commit renames few api endpoints:
- old `AddMigratedKeyPairOrAddAccountsIfKeyPairIsAdded` renamed to `AddKeycardOrAddAccountsIfKeycardIsAdded`
- old `GetAllMigratedKeyPairs` renamed to `GetAllKnownKeycardsGroupedByKeyUID`
- old `GetMigratedKeyPairByKeyUID` renamed to `GetKeycardByKeyUID`
- old `DeleteKeypair` renamed to `DeleteAllKeycardsWithKeyUID`
This is done in Nim after the watched pending transaction changes state.
Note: tried to delete all transactions, not only multi-transaction,
as soon as the state is downloaded but it didn't work: noticed crashes
with minted transactions after restart.
Updates status-desktop #10474
strategies.
On mobile devices we might not want to load all transfers history
to avoid high internet traffic consumption or to save some space.
On desktop we will load all transfers in background.
- Moved current fetching transfers implementation to
OnDemandFetchStrategy
- Implemented QueuedAtomicGroup to cache outgoing commands above a
threshold
- Refactored some type names, methods for better clarity:
- LastKnownBlock name was misused
- Block renamed to BlockDAO to clarify what it does and avoid
confusion with geth Block type
- Cleanup - removed unused parameters and types' members
- Changed tests to use in memory DB for speedup
- Added more logging and comments
Updates #10246
The "not found" and "no contract" expected error is caught and wrapped
by ClientWithFallback. The fetching of balance history of next blocks
is aborted which is not desired.
Fix by not treating the error as a connection error in
ClientWithFallback.
* fix mobile mention issue #15616
* add state != nil
* clear previous text when clear mentions
* fix: after selected mention user, and type @ not working
* bump version
This commit replaces `os.MkdirTemp` with `t.TempDir` in tests. The
directory created by `t.TempDir` is automatically removed when the test
and all its subtests complete.
Prior to this commit, temporary directory created using `os.MkdirTemp`
needs to be removed manually by calling `os.RemoveAll`, which is omitted
in some tests. The error handling boilerplate e.g.
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Fatal(err)
}
}
is also tedious, but `t.TempDir` handles this for us nicely.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Changes applied here introduce:
- improvements to sync wallet accounts among devices (including all account types)
- backing up wallet accounts to and fetch them from waku (an information about received
wallet accounts is sent via `waku.backedup.wallet-account` signal to a client)
* Community request to join changes
* Fix read state for request to join notification
* Bring back deleted notification when updated with response
* Update Request timeout to 7 days
* Update VERSION
- Display name is now backed up only as a part of `protobuf.BackedUpProfile` message,
it is not backed up via `protobuf.SyncSetting` any more (this refers only to backing up to
and fetching data from waku, regular syncing (among devices) remains unchanged)
- When saving the display name fetched from waku, before a clock was set to the current
time when that operation is made, which was incorrect, now we're using the clock from
backed up message (`SaveSyncDisplayName` function)
This adds an additional check for collectibles when community
permissions are validated.
Specifically this uses opensea to request all NFTs given an
owner wallet and a list of contract addresses (collectibles).
This adds checks to `HandleCommunityRequestToJoin` and
`AcceptRequestToJoinCommunity` that ensure a given user's revealed
wallet addresses own the token funds required by a community.
When community has token permissions of type `BECOME_MEMBER`, the
following happens when the owner receives a request:
1. Upon verifying provided wallet addresses by the requester, the owner
node accumulates all token funds related to the given wallets that
match the token criteria in the configured permissions
2. If the requester does not meet the necessary requirements, the
request to join will be declined. If the requester does have the
funds, he'll either be automatically accepted to the community, or
enters the next stage where an owner needs to manually accept the
request.
3. The the community does not automatically accept users, then the funds
check will happen again, when the owner tries to manually accept the
request. If the necessary funds do not exist at this stage, the
request will be declined
4. Upon accepting, whether automatically or manually, the owner adds the
requester's wallet addresses to the `CommunityDescription`, such that
they can be retrieved later when doing periodic checks or when
permissions have changed.
Also, make AccountManager a dependency of Messenger.
This is needed for community token permissions as we'll need a way to access wallet accounts
and sign messages when sending requests to join a community.
The APIs have been mostly taken from GethStatusBackend and personal service.
`type` column is set for all rows to appropriate value. Before this change
accounts which were generated from the keypair created importing seed phrase
had `generated` value for the `type`.
According to above, a function for generating an account sets the `type`
based on the passed derive from address.
Community tokens has some metadata (image, description) which must be kept in waku(CommunityDescription).
Add CommunityTokenMetadata message to communities.proto.
Add []CommunityTokenMetadata to CommunityDescription.
Issue #9545
prefixes. Changed primary keys and API methods.
Fixed tests and added new ones.
Fixed saved addresses and transaction tests to use ':memory:' sqlite
DB instead of a tmp file to speed up testing by hundred of times.
Fixes#8599
* feat: refactor activity center endpoints
* fix: restore activity center tests using new endpoints
* feat: Remove from activity center endpoints accepted flag
* feat: Activity Center review fixes
Also:
- extend maximum expected DB entries to fetch
- fix test TestBalanceHistoryGetWithoutOverlappingFetch passing because of all-time stride side effect
Closes status-desktop #9624
Changes applied here introduce backing up keycards data to and fetch them from waku.
Information about received keycards data is sent via `sync.from.waku.keycards` signal.
The L2 networks have their own block number and average block duration,
so we need to hardcode to keep the logic simplified.
Updates status-desktop: #9582
`last_update_clock` will be used later for synchronization.
All keypair functions take clock value in consideration when
making a decision whether to perform an action or not.
Adds a new column named `deleted` to the table `activity_center_notifications`.
Related PR in Mobile https://github.com/status-im/status-mobile/pull/15106 for a lot more details of the feature.
Why? Relying on the `dismissed` column for soft deletion is no longer viable because the mobile & desktop clients should display dismissed notifications (sometimes), hence the need for a new column to truly represent soft deletion.
* feat: add fetching notifications by activity center group
* feat: add api for counting notifications by activity center group
* chore: activity center query code refactor
* feat: add endpoint returning ActivityCenterType(s) by ActivityCenterGroup
* chore: Remove activityGroup from status-go level
* feat: add endpoint for counting notifications with different conditions
* chore: upgrade go-waku to v0.5
* chore: add println and logs to check what's being stored in the enr, and preemptively delete the multiaddr field (#3219)
* feat: add wakuv2 test (#3218)
* Outgoing contact requests
* Test fix
* Test fix
* Fixes
* Bugfixes
* Bugfixes
* Almost there
* Removed the activity center notification
* Test update
* Almost ready
* Fixes
* Fixes
* feat: Add seen/unseen activity center state
* feat: ActivityCenterState for grouping ActivityCenter unread messages cnt and seen state
* feat: always use messenger's addActivityCenterNotification & add state to the response
* Remove unused activity center endpoints form api and fix test
Add history.exchange sub-package with following responsibilities
- fetch and caches daily exchange rates for tokens
- Partial update if missing form yesterday back
- Implement all time fetching special case
- Fetches the price of the token after merging entries for the selected
chains
history.service package changes
- Return ValuePoint instead of DataPoint
- Value point contains the value in fiat as float64 instead
- Convert merged values from tokens balance (wei) to fiat
Other changes
- Move start/stop balance history to startWallet/stopWallet
- Move cryptocompare to thirdparty package to avoid recursive import
- Rename DataPoint.Value to DataPoint.Balance
- Don't merge entries for single chain
This commit fixes an issue where when accepting a contact request
the other end would display an extra notification.
It also changes WaitOnResponse to collect results. This should make
tests less flaky, since sometimes messages are processed in different
batches.
Now we need to be though exact on what we expect from the response (i.e
use == instead of >, otherwise the same behavior applies)
This uncovered a couple of issues with messenger.Merge, so I have moved
the struct to use a map based collection instead of an array.
Because the BLOB retrieved from the database can be empty, umarshalling
the data into JSON causes an error.
The API should simply return an empty list of there are no recent gifs.
Both functions `DeleteAccount` and `AddMigratedKeyPair` require password to be provided
in order to delete account from the keystore properly (removing account from the cache and
deleting corresponding local keystore file).
Password parameter can be also an empty string, since there are cases when an account is
not added to the keystore (in case of keycard account), so we have nothing to delete.
This change improves on the previous implementation which used
the block number that doesn't work with incompatible blockchains e.g. L1 vs. L2
Closes: #9205
This tests the entire process of converting a regular account to a keycard
account and then converting that keycard account back to a regular account.
For the need of this test I had to improve `DeleteAccount` function, cause the
previous implementation didn't remove account from the keystore cache, but
only from the keystore.
Extends wallet module with the history package with the following
components:
BalanceDB (balance_db.go)
- Keeps track of balance information (token count, block, block timestamp)
for a token identity (chain, address, currency)
- The cached data is stored in `balance_history` table.
- Uniqueness constrained is enforced by the `balance_history_identify_entry`
UNIQUE index.
- Optimal DB fetching is ensured by the `balance_history_filter_entries`
index
Balance (balance.go)
- Provides two stages:
- Fetch of balance history using RPC calls (Balance.update function)
- Retrieving of cached balance data from the DB it exists (Balance.get
function)
- Fetching and retrieving of data is done for specific time intervals
defined by TimeInterval "enumeration"
- Update process is done for a token identity by the Balance.Update function
- The granularity of data points returned is defined by the constant
increment step define in `timeIntervalToStride` for each time interval.
- The `blocksStride` values have a common divisor to have cache hit
between time intervals.
Service (service.go)
- Main APIs
- StartBalanceHistory: Regularly updates balance history for all
enabled networks, available accounts and provided tokens.
- GetBalanceHistory: retrieves cached token count for a token identity
(chain, address, currency) for multiple chains
- UpdateVisibleTokens: will set the list of tokens to have historical
balance fetched. This is a simplification to limit tokens to a small
list that make sense
Fetch balance history for ECR20 tokens
- Add token.Manager.GetTokenBalanceAt to fetch balance of a specific
block number of ECR20.
- Add tokenChainClientSource concrete implementation of DataSource
to fetch balance of ECR20 tokens.
- Chose the correct DataSource implementation based on the token
"is native" property.
Tests
Tests are implemented using a mock of `DataSource` interface used
to intercept the RPC calls.
Notes:
- the timestamp used for retrieving block balance is constant
Closes status-desktop: #8175, #8226, #8862
Summary
=======
- [x] Changes endpoint ActivityCenterNotificationsBy to support fetching
multiple types of notification in a single query.
- [x] Adds endpoint UnreadAndAcceptedActivityCenterNotificationsCount to
allow the mobile client to fetch the count of unread & accepted
notifications.
- [x] Add `golangci-lint` to Nix shell. This was possible since PR
https://github.com/status-im/status-go/pull/3087 was merged.
Notes
=====
- If you'd like to understand why these changes are needed, please see
the mobile PR https://github.com/status-im/status-mobile/pull/14785,
or issue https://github.com/status-im/status-mobile/issues/14712
- All changes should be completely backwards compatible, and there
should be no impact for the desktop app.
- The mobile client has been already tested using this branch.
* feat(@desktop/wallet): added iso4217 library for fiat currency display decimals
* feat(@desktop/wallet): added token peg info and use numbers for token market values
* feat(@desktop/wallet): added
* feat(@desktop/wallet): added iso4217 library for fiat currency display decimals
* feat(@desktop/wallet): added token peg info and use numbers for token market values
* feat(@desktop/wallet): extend wallet api to fetch prices in multiple currencies
* chore(@desktop/wallet): rename token peg field for clarity
Changes applied here introduce backing up profile data (display name and identity
images) to waku and fetch them from waku. Information about those data is sent
as a separate signal to a client via `sync.from.waku.profile` signal.
New signal `sync.from.waku.progress` is introduced which will be used to notify a client
about the progress of fetching data from waku.
Handling results of `GetAllMigratedKeyPairs` and `GetMigratedKeyPairByKeyUID`
endpoints updated in a way that account address is unique in the address list.
In order to give clients more insights about archive messages being
processed, we're adding this additional signal that informs clients when
the import of downloaded history archive messages has started.
Add functionality to sample and retrieve balance history and cache
it in memory for the current transfer controller.
The end of the balance history is snapped at twice per day to
avoid having to query the blockchain again for each fetching within
12 hours interval
The functionality will be extended with DB caching, API call batching,
"smarter" cache hitting and syncing between devices
Updates: #7662
The following three new functions introduced, for which password should be verified
on the client side (in case of a keycard user we don't have keystores to check pass):
- `AddAccountWithMnemonicPasswordVerified`
- `AddAccountWithMnemonicAndPathPasswordVerified`
- `AddAccountWithPrivateKeyPasswordVerified`
- `GenerateAccountPasswordVerified`
- `GenerateAccountWithDerivedPathPasswordVerified`
update
* feat(ActivityCenter): Add community request AC notification
* feat(ActivityCenter): Add CommunityID to AC notification
* feat(ActivityCenter): Add membership status for community membership AC notifications
* feat(ActivityCenter): Add tests for community notifications and fix naming
* Add notification for kicked from community action
* feat(ActivityCenter): Fix for missing notification objects for tests
Main changes:
- Extend saved addresses DB with sync info: sync timestamp, update timestamp
and soft removed flag
- Create custom protobuf message payload to sync saved addresses
- Cleanup saved addresses on each start of messenger, by deleting
soft removed older entries
- Sync all saved addresses on Messenger.SyncDevices calls
- Sync particular changes to saved addresses
- Add SavedAddressManager instance to messenger
- Note, can't find a clean way to pass the SavedAddressManager to the
messenger, so we create another one
- Add tests for sync and new DB API
Closes: #7229
- added `SpectateCommunity` endpoint, it is supposed to be used in
scenarios where we want to "Go to public Community" and see its
content without joining
- added `spectated` field to `Community`, it means we are observing the
community and its chats but we are not members
Use case:
https://github.com/status-im/status-desktop/issues/7072#issuecomment-1246560885
This adds a new `DownloadingHistoryArchivesFinished` signal to the
family of community archive signals. It's emitted when all to be
downloaded archives have been downloaded and handled.
Remove Favourites APIs and update the saved address APIs
Added up migration scripts that move the favourites from the old table
to the saved_addresses table with true flag and then drop the favourites table.
Required by #6546
`FirstMessageTimestamp` enables members of the community to determine if
there are any messages they can fetch on the community channel(chat).
`FirstMessageTimestamp` is advertised by admin for each community chat
through `CommunityDescription`. It assumes admin is online frequently
enough to capture the first channel message.
For existing communities admin determines first message timestamp by
finding oldest chat message in its local database.
task: status-im/status-desktop#6731
Add image_payload column to chats table.
Add Base64Image to chat struct.
Add ImageChange event to propagate image.
Change EditChat API - use CroppedImage.
Process and crop image in EditGroupChat.
As part of the new Discord <-> Status Community Import functionality,
we're adding an API that extracts all discord categories and channels
from a previously exported discord export file.
These APIs can be used in clients to show the user what categories and
channels will be imported later on.
There are two APIs:
1. `Messenger.ExtractDiscordCategoriesAndChannels(filesToimport
[]string) (*MessengerResponse, map[string]*discord.ImportError)`
This takes a list of exported discord export (JSON) files (typically one per
channel), reads them, and extracts the categories and channels into
dedicated data structures (`[]DiscordChannel` and `[]DiscordCategory`)
It also returns the oldest message timestamp found in all extracted
channels.
The API is synchronous and returns the extracted data as
a `*MessengerResponse`. This allows to make the API available
status-go's RPC interface.
The error case is a `map[string]*discord.ImportError` where each key
is a file path of a JSON file that we tried to extract data from, and
the value a `discord.ImportError` which holds an error message and an
error code, allowing for distinguishing between "critical" errors and
"non-critical" errors.
2. `Messenger.RequestExtractDiscordCategoriesAndChannels(filesToImport
[]string)`
This is the asynchronous counterpart to
`ExtractDiscordCategoriesAndChannels`. The reason this API has been
added is because discord servers can have a lot of message and
channel data, which causes `ExtractDiscordCategoriesAndChannels` to
block the thread for too long, making apps potentially feel like they
are stuck.
This API runs inside a go routine, eventually calls
`ExtractDiscordCategoriesAndChannels`, and then emits a newly
introduced `DiscordCategoriesAndChannelsExtractedSignal` that clients
can react to.
Failure of extraction has to be determined by the
`discord.ImportErrors` emitted by the signal.
**A note about exported discord history files**
We expect users to export their discord histories via the
[DiscordChatExporter](https://github.com/Tyrrrz/DiscordChatExporter/wiki/GUI%2C-CLI-and-Formats-explained#exportguild)
tool. The tool allows to export the data in different formats, such as
JSON, HTML and CSV.
We expect users to have their data exported as JSON.
Closes: https://github.com/status-im/status-desktop/issues/6690
This introduces a flag to configure whether `Messenger.CreateCommunity`
should create a default channel.
As discussed in https://github.com/status-im/status-go/issues/2758, this
is needed for the upcoming functionality to import discord communities.
Closes#2758
fix: add verification request to response
fix: code review
add missing functions and simplify timestamp usage
fix: sync verification requests
feat: add endpoint to fetch all received verification requests
feat: add signal when trusting verification request
Co-authored-by: Jonathan Rainville <rainville.jonathan@gmail.com>
This introduces a simple garbage collection which checks for all soft
deleted bookmarks (`removed = 1`) which have been marked for garbage
collection more than 30 days ago from the time of bootstapping the
messenger.
Closes#2705
These APIs are being introduced to address #2706 and #2704, provided
that clients will move to using these APIs instead of the currently
provided equivalent APIs in the browser service.
The `bookmarks` table is being extended with a `deleted_at` field which
can later be used for garbage collection, as "removing" a bookmark is
merely soft deletion and doesn't actually remove the data from the
database.
In addition to those APIs adding and soft deleting bookmark entries,
they also automatically perform a sync operation to ensure that
bookmarks are synced in real-time (#2704).
Closes#2706, #2704
* Added function to get preffered network IP
Also done some refactor work oon server package to make a lot more reusable
* Added server.Option and simplified handler funcs
* Added serial number deterministically generated from pk
* Debugging TLS server connection
* Implemented configurable server ip
When accessing over the network the server needs to listen on the network port and not localhost or 127.0.0.1 . Also the cert can now have a dedicated IP
* Refactor of URL funcs to use the url package
* Removed redundant Options pattern in favour of config param
* Added full server test using GetOutboundIP
* Remove references and usage of Server.port
The application does not need to set the port, we rely on the net.Listener to pick a port.
* Version bump
* Added ToECDSA func and improved cert testing
* Added error check in test
* Split Server types, embedding raw Server funcs into specialised server types
* localhost
* Implemented DNS and IP based cert gen
ios doesn't allow for restricted ip addresses to be used in a valid tls cert
* Replace listener handling with original port store
Also added handlers as a parameter of the Server
Added functionality to find target address when 6th param in path is added
for ex: "m'/44'/60'/0'/0/500" reperents the Address at the 500th index
Added a api to get the Address derived from a private key
This introduces logic needed to:
- Create WakuMessageArchives and and indices from store waku messages
- History archive torrent data to disk and create .torrent file from
that
- Seed and unseed history archive torrents as necessary
- Starting/stopping the torrent client
- Enabling/disabling community history support for individual components
and starting/stopping the routine intervals accordingly
This does not yet handle magnet links (#2568)
Closes#2567
* Sync Settings
* Added valueHandlers and Database singleton
Some issues remain, need a way to comparing incoming sql.DB to check if the connection is to a different file or not. Maybe make singleton instance per filename
* Added functionality to check the sqlite filename
* Refactor of Database.SaveSyncSettings to be used as a handler
* Implemented inteface for setting sync protobuf factories
* Refactored and completed adhoc send setting sync
* Tidying up
* Immutability refactor
* Refactor settings into dedicated package
* Breakout structs
* Tidy up
* Refactor of bulk settings sync
* Bug fixes
* Addressing feedback
* Fix code dropped during rebase
* Fix for db closed
* Fix for node config related crashes
* Provisional fix for type assertion - issue 2
* Adding robust type assertion checks
* Partial fix for null literal db storage and json encoding
* Fix for passively handling nil sql.DB, and checking if elem has len and if len is 0
* Added test for preferred name behaviour
* Adding saved sync settings to MessengerResponse
* Completed granular initial sync and clock from network on save
* add Settings to isEmpty
* Refactor of protobufs, partially done
* Added syncSetting receiver handling, some bug fixes
* Fix for sticker packs
* Implement inactive flag on sync protobuf factory
* Refactor of types and structs
* Added SettingField.CanSync functionality
* Addressing rebase artifact
* Refactor of Setting SELECT queries
* Refactor of string return queries
* VERSION bump and migration index bump
* Deactiveate Sync Settings
* Deactiveated preferred_name and send_status_updates
Co-authored-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
These are used to store settings for individual communities a
user is part of, either as member or as owner.
This included whether or not the community history archive protocol
is enabled.
This adds a new `CommunitySettings` type and adds
a migration script that introduces a new `communities_settings`
table.
It also extends the `MessengerResponse` type to include
`CommunitySettings` which are honored when communities are being
added, edited, joined or left.
Lastly, this adds a new RPC API to retreive the settings.
Closes#2564
This commit introduces a new `TorrentConfig` as per #2563 with dedicated
default values and new options/flags for the status-go CLI.
Since it's part of `NodeConfig`, which is stored per user in the
database, this commit also adds a migration script that adds a new
`torrent_config` table and database APIs to insert and retreive the
torrent config.
Closes#2563
This commit enables mailserver cycle logic by default and make a few
changes:
1) Nodes are graylisted instead of being blacklisted for a set amount of
time. The reason is that if we blacklist, any cut in connectivity
might result in long delays before reconnecting, especially on spotty
connections.
2) Fixes an issue on the devp2p server, whereby the node would not
connect to one of the static nodes since all the connection slots
where filled. The fix is a bit inelegant, it always connects to
static nodes, ignoring maxpeers, but it's tricky to get it to work
since the code is clearly not written to select a specific node.
3) Adds support to pinned mailservers
4) Add retries to mailservers requests. It uses a closure for now, I
think we should eventually have a channel etc, but I'd leave that for
later.
We periodically check whether we should backup contacts through waku.
Currently it's not tied to any event, it will only schedule a backup on
a tick, or through the provided API endpoint.
Dario Gabriel Lipicar
Cuteivist
Anthony Laibe
Ivan Belyakov
Roman Volosovskyi
Michal Iskierko
Patryk Osmaczko
Stefan
Michał
Sale Djenic
Andrea Maria Piana
frank
Richard Ramos
Mykhailo Prakhov
Igor Sirotin
Jonathan Rainville
Mikhail Rogachev
Pascal Precht
Volodymyr Kozieiev
Anthony Laibe
John Ngei
Boris Melnik
Khushboo-dev-cpp
Icaro Motta
Eng Zer Jun
Mohamed Javid
Ibrahem Khalil
mprakhov
Alexander
Mykhailo Prakhov
Parvesh Monu
Jakub Sokołowski
Omar Basem
Pascal Precht
dlipicar
yqrashawn
Samuel Hawksby-Robinson
Ibrahem Khalil
siddarthkay
Churikova Tetiana
Vitaliy Vlasov
Vitaliy Vlasov
andrey
osmaczko
Michele Balistreri
Pascal Precht
saledjenic
Brian Sztamfater
Shivek Khurana