- old `accounts` table is moved/mapped to `keypairs` and `keypairs_accounts`
- `keycards` table has foreign key which refers to `keypairs.key_uid`
- `Keypair` introduced as a new type
- api endpoints updated according to this change
It uses the current data only and doesn't extend with new types or
include new features in activity sources DBs.
Major changes:
- Partially filter by chain IDs
- Partially filter by Status if it is the case
- Partially filter by token types
- Filter by counterparty addresses
- Use wallet accounts for TO/FROM instead of filters
Closes: #10634
This commit does a few things:
1) Extend create/import account endpoint to get wallet config, some of
which has been moved to the backend
2) Set up a loop for retrieving balances every 10 minutes, caching the
balances
3) Return information about which checks are not passing when trying to
join a token gated community
4) Add tests to the token gated communities
5) Fixes an issue with addresses not matching when checking for
permissions
The move to the wallet as a background task is not yet complete, I need
to publish a signal, and most likely I will disable it before merging
for now, as it's currently not used by desktop/mobile, but the PR was
getting to big
were stored in cache by pointers, which caused falsy cache hits in loop
because pointers with same address were created for different block
numbers. Now cache uses block numbers of uint64 as key, which can
overflow but it is not a problem since we use this cache for values
comparison, not as user data.
Fix crash on nil pointer in log.
Remove some unused code.
Protect nonceRanges with mutex while reading.
Updates #10246
This is the initial implementation for the new URL unfurling requirements. The
most important one is that only the message sender will pay the privacy cost for
unfurling and extracting metadata from websites. Once the message is sent, the
unfurled data will be stored at the protocol level and receivers will just
profit and happily decode the metadata to render it.
Further development of this URL unfurling capability will be mostly guided by
issues created on clients. For the moment in status-mobile:
https://github.com/status-im/status-mobile/labels/url-preview
- https://github.com/status-im/status-mobile/issues/15918
- https://github.com/status-im/status-mobile/issues/15917
- https://github.com/status-im/status-mobile/issues/15910
- https://github.com/status-im/status-mobile/issues/15909
- https://github.com/status-im/status-mobile/issues/15908
- https://github.com/status-im/status-mobile/issues/15906
- https://github.com/status-im/status-mobile/issues/15905
### Terminology
In the code, I've tried to stick to the word "unfurl URL" to really mean the
process of extracting metadata from a website, sort of lower level. I use "link
preview" to mean a higher level structure which is enriched by unfurled data.
"link preview" is also how designers refer to it.
### User flows
1. Carol needs to see link previews while typing in the chat input field. Notice
from the diagram nothing is persisted and that status-go endpoints are
essentially stateless.
```
#+begin_src plantuml :results verbatim
Client->>Server: Call wakuext_getTextURLs
Server-->>Client: Normalized URLs
Client->>Client: Render cached unfurled URLs
Client->>Server: Unfurl non-cached URLs.\nCall wakuext_unfurlURLs
Server->>Website: Fetch metadata
Website-->>Server: Metadata (thumbnail URL, title, etc)
Server->>Website: Fetch thumbnail
Server->>Website: Fetch favicon
Website-->>Server: Favicon bytes
Website-->>Server: Thumbnail bytes
Server->>Server: Decode & process images
Server-->>Client: Unfurled data (thumbnail data URI, etc)
#+end_src
```
```
,------. ,------. ,-------.
|Client| |Server| |Website|
`--+---' `--+---' `---+---'
| Call wakuext_getTextURLs | |
| ---------------------------------------> |
| | |
| Normalized URLs | |
| <- - - - - - - - - - - - - - - - - - - - |
| | |
|----. | |
| | Render cached unfurled URLs | |
|<---' | |
| | |
| Unfurl non-cached URLs. | |
| Call wakuext_unfurlURLs | |
| ---------------------------------------> |
| | |
| | Fetch metadata |
| | ------------------------------------>
| | |
| | Metadata (thumbnail URL, title, etc)|
| | <- - - - - - - - - - - - - - - - - -
| | |
| | Fetch thumbnail |
| | ------------------------------------>
| | |
| | Fetch favicon |
| | ------------------------------------>
| | |
| | Favicon bytes |
| | <- - - - - - - - - - - - - - - - - -
| | |
| | Thumbnail bytes |
| | <- - - - - - - - - - - - - - - - - -
| | |
| |----. |
| | | Decode & process images |
| |<---' |
| | |
| Unfurled data (thumbnail data URI, etc)| |
| <- - - - - - - - - - - - - - - - - - - - |
,--+---. ,--+---. ,---+---.
|Client| |Server| |Website|
`------' `------' `-------'
```
2. Carol sends the text message with link previews in the RPC request
wakuext_sendChatMessages. status-go assumes the link previews are good
because it can't and shouldn't attempt to re-unfurl them.
```
#+begin_src plantuml :results verbatim
Client->>Server: Call wakuext_sendChatMessages
Server->>Server: Transform link previews to\nbe proto-marshalled
Server->DB: Write link previews serialized as JSON
Server-->>Client: Updated message response
#+end_src
```
```
,------. ,------. ,--.
|Client| |Server| |DB|
`--+---' `--+---' `+-'
| Call wakuext_sendChatMessages| |
| -----------------------------> |
| | |
| |----. |
| | | Transform link previews to |
| |<---' be proto-marshalled |
| | |
| | |
| | Write link previews serialized as JSON|
| | -------------------------------------->
| | |
| Updated message response | |
| <- - - - - - - - - - - - - - - |
,--+---. ,--+---. ,+-.
|Client| |Server| |DB|
`------' `------' `--'
```
3. The message was sent over waku and persisted locally in Carol's device. She
should now see the link previews in the chat history. There can be many link
previews shared by other chat members, therefore it is important to serve the
assets via the media server to avoid overloading the ReactNative bridge with
lots of big JSON payloads containing base64 encoded data URIs (maybe this
concern is meaningless for desktop). When a client is rendering messages with
link previews, they will have the field linkPreviews, and the thumbnail URL
will point to the local media server.
```
#+begin_src plantuml :results verbatim
Client->>Server: GET /link-preview/thumbnail (media server)
Server->>DB: Read from user_messages.unfurled_links
Server->Server: Unmarshal JSON
Server-->>Client: HTTP Content-Type: image/jpeg/etc
#+end_src
```
```
,------. ,------. ,--.
|Client| |Server| |DB|
`--+---' `--+---' `+-'
| GET /link-preview/thumbnail (media server)| |
| ------------------------------------------> |
| | |
| | Read from user_messages.unfurled_links|
| | -------------------------------------->
| | |
| |----. |
| | | Unmarshal JSON |
| |<---' |
| | |
| HTTP Content-Type: image/jpeg/etc | |
| <- - - - - - - - - - - - - - - - - - - - - |
,--+---. ,--+---. ,+-.
|Client| |Server| |DB|
`------' `------' `--'
```
### Some limitations of the current implementation
The following points will become separate issues in status-go that I'll work on
over the next couple weeks. In no order of importance:
- Improve how multiple links are fetched; retries on failure and testing how
unfurling behaves around the timeout limits (deterministically, not by making
real HTTP calls as I did). https://github.com/status-im/status-go/issues/3498
- Unfurl favicons and store them in the protobuf too.
- For this PR, I added unfurling support only for websites with OpenGraph
https://ogp.me/ meta tags. Other unfurlers will be implemented on demand. The
next one will probably be for oEmbed https://oembed.com/, the protocol
supported by YouTube, for example.
- Resize and/or compress thumbnails (and favicons). Often times, thumbnails are
huge for the purposes of link previews. There is already support for
compressing JPEGs in status-go, but I prefer to work with compression in a
separate PR because I'd like to also solve the problem for PNGs (probably
convert them to JPEGs, plus compress them). This would be a safe choice for
thumbnails, favicons not so much because transparency is desirable.
- Editing messages is not yet supported.
- I haven't coded any artificial limit on the number of previews or on the size
of the thumbnail payload. This will be done in a separate issue. I have heard
the ideal solution may be to split messages into smaller chunks of ~125 KiB
because of libp2p, but that might be too complicated at this stage of the
product (?).
- Link preview deletion.
- For the moment, OpenGraph metadata is extracted by requesting data for the
English language (and fallback to whatever is available). In the future, we'll
want to unfurl by respecting the user's local device language. Some websites,
like GoDaddy, are already localized based on the device's IP, but many aren't.
- The website's description text should be limited by a certain number of
characters, especially because it's outside our control. Exactly how much has
not been decided yet, so it'll be done separately.
- URL normalization can be tricky, so I implemented only the basics to help with
caching. For example, the url https://status.im and HTTPS://status.im are
considered identical. Also, a URL is considered valid for unfurling if its TLD
exists according to publicsuffix.EffectiveTLDPlusOne. This was essential,
otherwise the default Go url.Parse approach would consider many invalid URLs
valid, and thus the server would waste resources trying to unfurl the
unfurleable.
### Other requirements
- If the message is edited, the link previews should reflect the edited text,
not the original one. This has been aligned with the design team as well.
- If the website's thumbnail or the favicon can't be fetched, just ignore them.
The only mandatory piece of metadata is the website's title and URL.
- Link previews in clients should be generated in near real-time, that is, as
the user types, previews are updated. In mobile this performs very well, and
it's what other clients like WhatsApp, Telegram, and Facebook do.
### Decisions
- While the user typing in the input field, the client is constantly (debounced)
asking status-go to parse the text and extract normalized URLs and then the
client checks if they're already in its in-memory cache. If they are, no RPC
call is made. I chose this approach to achieve the best possible performance
in mobile and avoid the whole RPC overhead, since the chat experience is
already not smooth enough. The mobile client uses URLs as cache keys in a
hashmap, i.e. if the key is present, it means the preview is readily available
(naive, but good enough for now). This decision also gave me more flexibility
to find the best UX at this stage of the feature.
- Due to the requirement that users should be able to see independent loading
indicators for each link preview, when status-go can't unfurl a URL, it
doesn't return it in the response.
- As an initial implementation, I added the BLOB column unfurled_links to the
user_messages table. The preview data is then serialized as JSON before being
stored in this column. I felt that creating a separate table and the related
code for this initial PR would be inconvenient. Is that reasonable to you?
Once things stabilize I can create a proper table if we want to avoid this
kind of solution with serialized columns.
Add the possibility of retrieving the metadata of wallet activity based
on the given filter criteria.
Current implementation relies that after fetching the metadata, user
will follow up with more requests for details. However, after some
experimenting I'm considering extracting all required information
for the summary viewing while filtering. This way there will be no
need for another batch requests for transfers, multi-transactions and
pending transactions to show the summary. Only when user wants to see
the details for one will specifically request it.
For this first prototype, the filter criteria is limited to:
- time
- type
- addresses
Major changes:
- Add the filter definition to be used in propagating the filter
information
- Add GetActivityEntries API to return the list of activity entries
for the given addresses/chainIDs by a view in the complete list
- GetTransfersForIdentities to batch retrieve further details of the
transfers
- GetPendingTransactionsForIdentities to batch retrieve further details
of the pending transactions
- Added a new package testutils for tests.
- Added tests
Updates status-desktop #10366
Closes status-desktop #10633
This commit renames few api endpoints:
- old `AddMigratedKeyPairOrAddAccountsIfKeyPairIsAdded` renamed to `AddKeycardOrAddAccountsIfKeycardIsAdded`
- old `GetAllMigratedKeyPairs` renamed to `GetAllKnownKeycardsGroupedByKeyUID`
- old `GetMigratedKeyPairByKeyUID` renamed to `GetKeycardByKeyUID`
- old `DeleteKeypair` renamed to `DeleteAllKeycardsWithKeyUID`
This is done in Nim after the watched pending transaction changes state.
Note: tried to delete all transactions, not only multi-transaction,
as soon as the state is downloaded but it didn't work: noticed crashes
with minted transactions after restart.
Updates status-desktop #10474
strategies.
On mobile devices we might not want to load all transfers history
to avoid high internet traffic consumption or to save some space.
On desktop we will load all transfers in background.
- Moved current fetching transfers implementation to
OnDemandFetchStrategy
- Implemented QueuedAtomicGroup to cache outgoing commands above a
threshold
- Refactored some type names, methods for better clarity:
- LastKnownBlock name was misused
- Block renamed to BlockDAO to clarify what it does and avoid
confusion with geth Block type
- Cleanup - removed unused parameters and types' members
- Changed tests to use in memory DB for speedup
- Added more logging and comments
Updates #10246
The "not found" and "no contract" expected error is caught and wrapped
by ClientWithFallback. The fetching of balance history of next blocks
is aborted which is not desired.
Fix by not treating the error as a connection error in
ClientWithFallback.
* fix mobile mention issue #15616
* add state != nil
* clear previous text when clear mentions
* fix: after selected mention user, and type @ not working
* bump version
This commit replaces `os.MkdirTemp` with `t.TempDir` in tests. The
directory created by `t.TempDir` is automatically removed when the test
and all its subtests complete.
Prior to this commit, temporary directory created using `os.MkdirTemp`
needs to be removed manually by calling `os.RemoveAll`, which is omitted
in some tests. The error handling boilerplate e.g.
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Fatal(err)
}
}
is also tedious, but `t.TempDir` handles this for us nicely.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Changes applied here introduce:
- improvements to sync wallet accounts among devices (including all account types)
- backing up wallet accounts to and fetch them from waku (an information about received
wallet accounts is sent via `waku.backedup.wallet-account` signal to a client)
* Community request to join changes
* Fix read state for request to join notification
* Bring back deleted notification when updated with response
* Update Request timeout to 7 days
* Update VERSION
- Display name is now backed up only as a part of `protobuf.BackedUpProfile` message,
it is not backed up via `protobuf.SyncSetting` any more (this refers only to backing up to
and fetching data from waku, regular syncing (among devices) remains unchanged)
- When saving the display name fetched from waku, before a clock was set to the current
time when that operation is made, which was incorrect, now we're using the clock from
backed up message (`SaveSyncDisplayName` function)
This adds an additional check for collectibles when community
permissions are validated.
Specifically this uses opensea to request all NFTs given an
owner wallet and a list of contract addresses (collectibles).
This adds checks to `HandleCommunityRequestToJoin` and
`AcceptRequestToJoinCommunity` that ensure a given user's revealed
wallet addresses own the token funds required by a community.
When community has token permissions of type `BECOME_MEMBER`, the
following happens when the owner receives a request:
1. Upon verifying provided wallet addresses by the requester, the owner
node accumulates all token funds related to the given wallets that
match the token criteria in the configured permissions
2. If the requester does not meet the necessary requirements, the
request to join will be declined. If the requester does have the
funds, he'll either be automatically accepted to the community, or
enters the next stage where an owner needs to manually accept the
request.
3. The the community does not automatically accept users, then the funds
check will happen again, when the owner tries to manually accept the
request. If the necessary funds do not exist at this stage, the
request will be declined
4. Upon accepting, whether automatically or manually, the owner adds the
requester's wallet addresses to the `CommunityDescription`, such that
they can be retrieved later when doing periodic checks or when
permissions have changed.
Also, make AccountManager a dependency of Messenger.
This is needed for community token permissions as we'll need a way to access wallet accounts
and sign messages when sending requests to join a community.
The APIs have been mostly taken from GethStatusBackend and personal service.
`type` column is set for all rows to appropriate value. Before this change
accounts which were generated from the keypair created importing seed phrase
had `generated` value for the `type`.
According to above, a function for generating an account sets the `type`
based on the passed derive from address.
Community tokens has some metadata (image, description) which must be kept in waku(CommunityDescription).
Add CommunityTokenMetadata message to communities.proto.
Add []CommunityTokenMetadata to CommunityDescription.
Issue #9545
prefixes. Changed primary keys and API methods.
Fixed tests and added new ones.
Fixed saved addresses and transaction tests to use ':memory:' sqlite
DB instead of a tmp file to speed up testing by hundred of times.
Fixes#8599
* feat: refactor activity center endpoints
* fix: restore activity center tests using new endpoints
* feat: Remove from activity center endpoints accepted flag
* feat: Activity Center review fixes
Also:
- extend maximum expected DB entries to fetch
- fix test TestBalanceHistoryGetWithoutOverlappingFetch passing because of all-time stride side effect
Closes status-desktop #9624
Changes applied here introduce backing up keycards data to and fetch them from waku.
Information about received keycards data is sent via `sync.from.waku.keycards` signal.
The L2 networks have their own block number and average block duration,
so we need to hardcode to keep the logic simplified.
Updates status-desktop: #9582
`last_update_clock` will be used later for synchronization.
All keypair functions take clock value in consideration when
making a decision whether to perform an action or not.
Adds a new column named `deleted` to the table `activity_center_notifications`.
Related PR in Mobile https://github.com/status-im/status-mobile/pull/15106 for a lot more details of the feature.
Why? Relying on the `dismissed` column for soft deletion is no longer viable because the mobile & desktop clients should display dismissed notifications (sometimes), hence the need for a new column to truly represent soft deletion.
* feat: add fetching notifications by activity center group
* feat: add api for counting notifications by activity center group
* chore: activity center query code refactor
* feat: add endpoint returning ActivityCenterType(s) by ActivityCenterGroup
* chore: Remove activityGroup from status-go level
* feat: add endpoint for counting notifications with different conditions
* chore: upgrade go-waku to v0.5
* chore: add println and logs to check what's being stored in the enr, and preemptively delete the multiaddr field (#3219)
* feat: add wakuv2 test (#3218)
* Outgoing contact requests
* Test fix
* Test fix
* Fixes
* Bugfixes
* Bugfixes
* Almost there
* Removed the activity center notification
* Test update
* Almost ready
* Fixes
* Fixes
* feat: Add seen/unseen activity center state
* feat: ActivityCenterState for grouping ActivityCenter unread messages cnt and seen state
* feat: always use messenger's addActivityCenterNotification & add state to the response
* Remove unused activity center endpoints form api and fix test
Add history.exchange sub-package with following responsibilities
- fetch and caches daily exchange rates for tokens
- Partial update if missing form yesterday back
- Implement all time fetching special case
- Fetches the price of the token after merging entries for the selected
chains
history.service package changes
- Return ValuePoint instead of DataPoint
- Value point contains the value in fiat as float64 instead
- Convert merged values from tokens balance (wei) to fiat
Other changes
- Move start/stop balance history to startWallet/stopWallet
- Move cryptocompare to thirdparty package to avoid recursive import
- Rename DataPoint.Value to DataPoint.Balance
- Don't merge entries for single chain
This commit fixes an issue where when accepting a contact request
the other end would display an extra notification.
It also changes WaitOnResponse to collect results. This should make
tests less flaky, since sometimes messages are processed in different
batches.
Now we need to be though exact on what we expect from the response (i.e
use == instead of >, otherwise the same behavior applies)
This uncovered a couple of issues with messenger.Merge, so I have moved
the struct to use a map based collection instead of an array.
Because the BLOB retrieved from the database can be empty, umarshalling
the data into JSON causes an error.
The API should simply return an empty list of there are no recent gifs.
Both functions `DeleteAccount` and `AddMigratedKeyPair` require password to be provided
in order to delete account from the keystore properly (removing account from the cache and
deleting corresponding local keystore file).
Password parameter can be also an empty string, since there are cases when an account is
not added to the keystore (in case of keycard account), so we have nothing to delete.
This change improves on the previous implementation which used
the block number that doesn't work with incompatible blockchains e.g. L1 vs. L2
Closes: #9205
This tests the entire process of converting a regular account to a keycard
account and then converting that keycard account back to a regular account.
For the need of this test I had to improve `DeleteAccount` function, cause the
previous implementation didn't remove account from the keystore cache, but
only from the keystore.
Extends wallet module with the history package with the following
components:
BalanceDB (balance_db.go)
- Keeps track of balance information (token count, block, block timestamp)
for a token identity (chain, address, currency)
- The cached data is stored in `balance_history` table.
- Uniqueness constrained is enforced by the `balance_history_identify_entry`
UNIQUE index.
- Optimal DB fetching is ensured by the `balance_history_filter_entries`
index
Balance (balance.go)
- Provides two stages:
- Fetch of balance history using RPC calls (Balance.update function)
- Retrieving of cached balance data from the DB it exists (Balance.get
function)
- Fetching and retrieving of data is done for specific time intervals
defined by TimeInterval "enumeration"
- Update process is done for a token identity by the Balance.Update function
- The granularity of data points returned is defined by the constant
increment step define in `timeIntervalToStride` for each time interval.
- The `blocksStride` values have a common divisor to have cache hit
between time intervals.
Service (service.go)
- Main APIs
- StartBalanceHistory: Regularly updates balance history for all
enabled networks, available accounts and provided tokens.
- GetBalanceHistory: retrieves cached token count for a token identity
(chain, address, currency) for multiple chains
- UpdateVisibleTokens: will set the list of tokens to have historical
balance fetched. This is a simplification to limit tokens to a small
list that make sense
Fetch balance history for ECR20 tokens
- Add token.Manager.GetTokenBalanceAt to fetch balance of a specific
block number of ECR20.
- Add tokenChainClientSource concrete implementation of DataSource
to fetch balance of ECR20 tokens.
- Chose the correct DataSource implementation based on the token
"is native" property.
Tests
Tests are implemented using a mock of `DataSource` interface used
to intercept the RPC calls.
Notes:
- the timestamp used for retrieving block balance is constant
Closes status-desktop: #8175, #8226, #8862
Summary
=======
- [x] Changes endpoint ActivityCenterNotificationsBy to support fetching
multiple types of notification in a single query.
- [x] Adds endpoint UnreadAndAcceptedActivityCenterNotificationsCount to
allow the mobile client to fetch the count of unread & accepted
notifications.
- [x] Add `golangci-lint` to Nix shell. This was possible since PR
https://github.com/status-im/status-go/pull/3087 was merged.
Notes
=====
- If you'd like to understand why these changes are needed, please see
the mobile PR https://github.com/status-im/status-mobile/pull/14785,
or issue https://github.com/status-im/status-mobile/issues/14712
- All changes should be completely backwards compatible, and there
should be no impact for the desktop app.
- The mobile client has been already tested using this branch.
* feat(@desktop/wallet): added iso4217 library for fiat currency display decimals
* feat(@desktop/wallet): added token peg info and use numbers for token market values
* feat(@desktop/wallet): added
* feat(@desktop/wallet): added iso4217 library for fiat currency display decimals
* feat(@desktop/wallet): added token peg info and use numbers for token market values
* feat(@desktop/wallet): extend wallet api to fetch prices in multiple currencies
* chore(@desktop/wallet): rename token peg field for clarity
Changes applied here introduce backing up profile data (display name and identity
images) to waku and fetch them from waku. Information about those data is sent
as a separate signal to a client via `sync.from.waku.profile` signal.
New signal `sync.from.waku.progress` is introduced which will be used to notify a client
about the progress of fetching data from waku.
Handling results of `GetAllMigratedKeyPairs` and `GetMigratedKeyPairByKeyUID`
endpoints updated in a way that account address is unique in the address list.
In order to give clients more insights about archive messages being
processed, we're adding this additional signal that informs clients when
the import of downloaded history archive messages has started.
Add functionality to sample and retrieve balance history and cache
it in memory for the current transfer controller.
The end of the balance history is snapped at twice per day to
avoid having to query the blockchain again for each fetching within
12 hours interval
The functionality will be extended with DB caching, API call batching,
"smarter" cache hitting and syncing between devices
Updates: #7662
The following three new functions introduced, for which password should be verified
on the client side (in case of a keycard user we don't have keystores to check pass):
- `AddAccountWithMnemonicPasswordVerified`
- `AddAccountWithMnemonicAndPathPasswordVerified`
- `AddAccountWithPrivateKeyPasswordVerified`
- `GenerateAccountPasswordVerified`
- `GenerateAccountWithDerivedPathPasswordVerified`
update
* feat(ActivityCenter): Add community request AC notification
* feat(ActivityCenter): Add CommunityID to AC notification
* feat(ActivityCenter): Add membership status for community membership AC notifications
* feat(ActivityCenter): Add tests for community notifications and fix naming
* Add notification for kicked from community action
* feat(ActivityCenter): Fix for missing notification objects for tests
Main changes:
- Extend saved addresses DB with sync info: sync timestamp, update timestamp
and soft removed flag
- Create custom protobuf message payload to sync saved addresses
- Cleanup saved addresses on each start of messenger, by deleting
soft removed older entries
- Sync all saved addresses on Messenger.SyncDevices calls
- Sync particular changes to saved addresses
- Add SavedAddressManager instance to messenger
- Note, can't find a clean way to pass the SavedAddressManager to the
messenger, so we create another one
- Add tests for sync and new DB API
Closes: #7229
- added `SpectateCommunity` endpoint, it is supposed to be used in
scenarios where we want to "Go to public Community" and see its
content without joining
- added `spectated` field to `Community`, it means we are observing the
community and its chats but we are not members
Use case:
https://github.com/status-im/status-desktop/issues/7072#issuecomment-1246560885
This adds a new `DownloadingHistoryArchivesFinished` signal to the
family of community archive signals. It's emitted when all to be
downloaded archives have been downloaded and handled.
Remove Favourites APIs and update the saved address APIs
Added up migration scripts that move the favourites from the old table
to the saved_addresses table with true flag and then drop the favourites table.
Required by #6546
`FirstMessageTimestamp` enables members of the community to determine if
there are any messages they can fetch on the community channel(chat).
`FirstMessageTimestamp` is advertised by admin for each community chat
through `CommunityDescription`. It assumes admin is online frequently
enough to capture the first channel message.
For existing communities admin determines first message timestamp by
finding oldest chat message in its local database.
task: status-im/status-desktop#6731
Add image_payload column to chats table.
Add Base64Image to chat struct.
Add ImageChange event to propagate image.
Change EditChat API - use CroppedImage.
Process and crop image in EditGroupChat.
As part of the new Discord <-> Status Community Import functionality,
we're adding an API that extracts all discord categories and channels
from a previously exported discord export file.
These APIs can be used in clients to show the user what categories and
channels will be imported later on.
There are two APIs:
1. `Messenger.ExtractDiscordCategoriesAndChannels(filesToimport
[]string) (*MessengerResponse, map[string]*discord.ImportError)`
This takes a list of exported discord export (JSON) files (typically one per
channel), reads them, and extracts the categories and channels into
dedicated data structures (`[]DiscordChannel` and `[]DiscordCategory`)
It also returns the oldest message timestamp found in all extracted
channels.
The API is synchronous and returns the extracted data as
a `*MessengerResponse`. This allows to make the API available
status-go's RPC interface.
The error case is a `map[string]*discord.ImportError` where each key
is a file path of a JSON file that we tried to extract data from, and
the value a `discord.ImportError` which holds an error message and an
error code, allowing for distinguishing between "critical" errors and
"non-critical" errors.
2. `Messenger.RequestExtractDiscordCategoriesAndChannels(filesToImport
[]string)`
This is the asynchronous counterpart to
`ExtractDiscordCategoriesAndChannels`. The reason this API has been
added is because discord servers can have a lot of message and
channel data, which causes `ExtractDiscordCategoriesAndChannels` to
block the thread for too long, making apps potentially feel like they
are stuck.
This API runs inside a go routine, eventually calls
`ExtractDiscordCategoriesAndChannels`, and then emits a newly
introduced `DiscordCategoriesAndChannelsExtractedSignal` that clients
can react to.
Failure of extraction has to be determined by the
`discord.ImportErrors` emitted by the signal.
**A note about exported discord history files**
We expect users to export their discord histories via the
[DiscordChatExporter](https://github.com/Tyrrrz/DiscordChatExporter/wiki/GUI%2C-CLI-and-Formats-explained#exportguild)
tool. The tool allows to export the data in different formats, such as
JSON, HTML and CSV.
We expect users to have their data exported as JSON.
Closes: https://github.com/status-im/status-desktop/issues/6690
This introduces a flag to configure whether `Messenger.CreateCommunity`
should create a default channel.
As discussed in https://github.com/status-im/status-go/issues/2758, this
is needed for the upcoming functionality to import discord communities.
Closes#2758
fix: add verification request to response
fix: code review
add missing functions and simplify timestamp usage
fix: sync verification requests
feat: add endpoint to fetch all received verification requests
feat: add signal when trusting verification request
Co-authored-by: Jonathan Rainville <rainville.jonathan@gmail.com>
This introduces a simple garbage collection which checks for all soft
deleted bookmarks (`removed = 1`) which have been marked for garbage
collection more than 30 days ago from the time of bootstapping the
messenger.
Closes#2705
These APIs are being introduced to address #2706 and #2704, provided
that clients will move to using these APIs instead of the currently
provided equivalent APIs in the browser service.
The `bookmarks` table is being extended with a `deleted_at` field which
can later be used for garbage collection, as "removing" a bookmark is
merely soft deletion and doesn't actually remove the data from the
database.
In addition to those APIs adding and soft deleting bookmark entries,
they also automatically perform a sync operation to ensure that
bookmarks are synced in real-time (#2704).
Closes#2706, #2704
* Added function to get preffered network IP
Also done some refactor work oon server package to make a lot more reusable
* Added server.Option and simplified handler funcs
* Added serial number deterministically generated from pk
* Debugging TLS server connection
* Implemented configurable server ip
When accessing over the network the server needs to listen on the network port and not localhost or 127.0.0.1 . Also the cert can now have a dedicated IP
* Refactor of URL funcs to use the url package
* Removed redundant Options pattern in favour of config param
* Added full server test using GetOutboundIP
* Remove references and usage of Server.port
The application does not need to set the port, we rely on the net.Listener to pick a port.
* Version bump
* Added ToECDSA func and improved cert testing
* Added error check in test
* Split Server types, embedding raw Server funcs into specialised server types
* localhost
* Implemented DNS and IP based cert gen
ios doesn't allow for restricted ip addresses to be used in a valid tls cert
* Replace listener handling with original port store
Also added handlers as a parameter of the Server
Added functionality to find target address when 6th param in path is added
for ex: "m'/44'/60'/0'/0/500" reperents the Address at the 500th index
Added a api to get the Address derived from a private key
This introduces logic needed to:
- Create WakuMessageArchives and and indices from store waku messages
- History archive torrent data to disk and create .torrent file from
that
- Seed and unseed history archive torrents as necessary
- Starting/stopping the torrent client
- Enabling/disabling community history support for individual components
and starting/stopping the routine intervals accordingly
This does not yet handle magnet links (#2568)
Closes#2567
* Sync Settings
* Added valueHandlers and Database singleton
Some issues remain, need a way to comparing incoming sql.DB to check if the connection is to a different file or not. Maybe make singleton instance per filename
* Added functionality to check the sqlite filename
* Refactor of Database.SaveSyncSettings to be used as a handler
* Implemented inteface for setting sync protobuf factories
* Refactored and completed adhoc send setting sync
* Tidying up
* Immutability refactor
* Refactor settings into dedicated package
* Breakout structs
* Tidy up
* Refactor of bulk settings sync
* Bug fixes
* Addressing feedback
* Fix code dropped during rebase
* Fix for db closed
* Fix for node config related crashes
* Provisional fix for type assertion - issue 2
* Adding robust type assertion checks
* Partial fix for null literal db storage and json encoding
* Fix for passively handling nil sql.DB, and checking if elem has len and if len is 0
* Added test for preferred name behaviour
* Adding saved sync settings to MessengerResponse
* Completed granular initial sync and clock from network on save
* add Settings to isEmpty
* Refactor of protobufs, partially done
* Added syncSetting receiver handling, some bug fixes
* Fix for sticker packs
* Implement inactive flag on sync protobuf factory
* Refactor of types and structs
* Added SettingField.CanSync functionality
* Addressing rebase artifact
* Refactor of Setting SELECT queries
* Refactor of string return queries
* VERSION bump and migration index bump
* Deactiveate Sync Settings
* Deactiveated preferred_name and send_status_updates
Co-authored-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
These are used to store settings for individual communities a
user is part of, either as member or as owner.
This included whether or not the community history archive protocol
is enabled.
This adds a new `CommunitySettings` type and adds
a migration script that introduces a new `communities_settings`
table.
It also extends the `MessengerResponse` type to include
`CommunitySettings` which are honored when communities are being
added, edited, joined or left.
Lastly, this adds a new RPC API to retreive the settings.
Closes#2564
This commit introduces a new `TorrentConfig` as per #2563 with dedicated
default values and new options/flags for the status-go CLI.
Since it's part of `NodeConfig`, which is stored per user in the
database, this commit also adds a migration script that adds a new
`torrent_config` table and database APIs to insert and retreive the
torrent config.
Closes#2563
This commit enables mailserver cycle logic by default and make a few
changes:
1) Nodes are graylisted instead of being blacklisted for a set amount of
time. The reason is that if we blacklist, any cut in connectivity
might result in long delays before reconnecting, especially on spotty
connections.
2) Fixes an issue on the devp2p server, whereby the node would not
connect to one of the static nodes since all the connection slots
where filled. The fix is a bit inelegant, it always connects to
static nodes, ignoring maxpeers, but it's tricky to get it to work
since the code is clearly not written to select a specific node.
3) Adds support to pinned mailservers
4) Add retries to mailservers requests. It uses a closure for now, I
think we should eventually have a channel etc, but I'd leave that for
later.
We periodically check whether we should backup contacts through waku.
Currently it's not tied to any event, it will only schedule a backup on
a tick, or through the provided API endpoint.
* feat: Add more attributes from opensea
Add sale / orders information as well as collection traits
* feat: add background color to opensea response
* feat: Add max value and handle float to opensea response
* feat: enable wallet without network binding
* feat: make transfer network aware
* feat: allow to pass initial networks via config
* fix: nil check and feed
* feat: Add documentation with better function name
* fix: do not init the manager more than once
* fix: PR feedbacks
* Bump version
* Update Jenkinsfile.tests
* Convert int to string
Co-authored-by: RichΛrd <info@richardramos.me>
* Protobufs and adapters
* Added basic anon metric service and config init
* Added fibonacci interval incrementer
* Added basic Client.Start func and integrated interval incrementer
* Added new processed field to app metrics table
* Added id column to app metrics table
* Added migration clean up
* Added appmetrics GetUnprocessed and SetToProcessedByIDs and tests
There was a wierd bug where metrics in the db that did not explicitly insert a value would be NULL, so could not be found by . In addition I've added a new primary id field to the app_metrics table so that updates could be done against very specific metric rows.
* Updated adaptors and db to handle proto_id
I need a way to distinguish individual metric items from each other so that I can ignore the ones that have been seen before.
* Moved incrementer into dedicated file
* Resolve incrementer test fail
* Finalised the main loop functionality
* Implemented delete loop framework
* Updated adaptors file name
* Added delete loop delay and quit, and tweak on RawMessage gen
* Completed delete loop logic
* Added DBLock to prevent deletion during mainLoop
* Added postgres DB connection, integrated into anonmetrics.Server
* Removed proto_id from SQL migration and model
* Integrated postgres with Server and updated adaptors
* Function name update
* Added sample config files for client and server
* Fixes and testing for low level e2e
* make generate
* Fix lint
* Fix for receiving an anonMetricBatch not in server mode
* Postgres test fixes
* Tidy up, make vendor and make generate
* delinting
* Fixing database tests
* Attempted fix of does: cannot open `does' (No such file or directory)
not: cannot open `not' (No such file or directory)
exist: cannot open `exist' (No such file or directory) error on sql resource loas
* Moved all anon metric postgres migration logic and sources into a the protocol/anonmetrics package or sub packages. I don't know if this will fix the does: cannot open `does' (No such file or directory)
not: cannot open `not' (No such file or directory)
exist: cannot open `exist' (No such file or directory) error that happens in Jenkins but this could work
* Lint for the lint god
* Why doesn't the linter list all its problems at once?
* test tweaks
* Fix for wakuV2 change
* DB reset change
* Fix for postgres db migrations fails
* More robust implementation of postgres test setup and teardown
* Added block for anon metrics functionality
* Version Bump to 0.84.0
* Added test to check anon metrics broadcast is deactivated
* Protobufs and adapters
* Added basic anon metric service and config init
* Added new processed field to app metrics table
* Added id column to app metrics table
* Added migration clean up
* Added appmetrics GetUnprocessed and SetToProcessedByIDs and tests
There was a wierd bug where metrics in the db that did not explicitly insert a value would be NULL, so could not be found by . In addition I've added a new primary id field to the app_metrics table so that updates could be done against very specific metric rows.
* Updated adaptors and db to handle proto_id
I need a way to distinguish individual metric items from each other so that I can ignore the ones that have been seen before.
* Added postgres DB connection, integrated into anonmetrics.Server
* Removed proto_id from SQL migration and model
* Integrated postgres with Server and updated adaptors
* Added sample config files for client and server
* Fix lint
* Fix for receiving an anonMetricBatch not in server mode
* Postgres test fixes
* Tidy up, make vendor and make generate
* Moved all anon metric postgres migration logic and sources into a the protocol/anonmetrics package or sub packages. I don't know if this will fix the does: cannot open `does' (No such file or directory)
not: cannot open `not' (No such file or directory)
exist: cannot open `exist' (No such file or directory) error that happens in Jenkins but this could work
AllMessagesFromChatsAndCommunitiesWhichMatchTerm method added which returns all messages which
match the search term, if they belong to either any chat from the chatIds array or any channel of
any community from communityIds array.
This api point is necessary for desktop issue 2934.
Fixes: #2934
* feature(@status-go/chat): implement search on sqlcipher (status-go side)
Searching messages by some term for a specific channel is added on the side of status-go.
Fixes: #2912
* Linting
Co-authored-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
* Adding wakunode module
* Adding wakuv2 fleet files
* Add waku fleets to update-fleet-config script
* Adding config items for waku v2
* Conditionally start waku v2 node depending on config
* Adapting common code to use go-waku
* Setting log level to info
* update dependencies
* update fleet config to use WakuNodes instead of BootNodes
* send and receive messages
* use hash returned when publishing a message
* add waku store protocol
* trigger signal after receiving store messages
* exclude linting rule SA1019 to check deprecated packages
* feat: Add edit communities
Allow Communities to be edited, including display name, description, color, membership, and permissions.
* Added EditCommunity request type
* Fix lint errors
* Allow editing community without changing image
Previously, retaining an existing community image was not possible because the existing community image path had to be provided in the `editCommunity` RPC call to retain the image. However, once the image is processed by status-go, it is encoded as a base64 string and therefore it is not possible to get the original file path back from this string.
This commit allows for the original to be retained by passing an empty string for the image field in the RPC call.
* Don't change permissions. Fixed clock updating
Co-authored-by: Volodymyr Kozieiev <vkjr.sp@gmail.com>
* Updated Ramp siteUrl to use URL in docs and use the referral code
see https://docs.ramp.network/configuration
* Moved Ramp to the top
* Removed local cryptos and transak
* add PinMessage and PinnedMessage
* fix gruop pin messages
* add SkipGroupMessageWrap to pin messages
* update pinMessage ID generation to be symmetric
* Add extra event to capture other type of navigations, allow empty screen name, rename cofx to get rid of clj ns, update tests
* Some view ids are greater than 16 characters. Made it 32 to be safe.
* Tab navigation events occur outside nav, add a new validator for them
* Remove navigate to cofx event, capture screens on will focus, get rid of enum and make valid screens a string less than 32 characters
* Run make generate
* Fix test
* Bump version to 0.75.1
This commit introduces the following changes:
- `local-notifications` require as body an interface complying with
`json.Marshaler`
- removed unmarshaling of `Notifications` as not used (we only Marshal
notifications)
- `protocol/messenger.go` creates directly a `Notification` instead of
having an intermediate format
- add community notifications on request to join
- move parsing of text in status-go for notifications
* Removed region field from on ramp struct
* Added basic placeholder for latamex
* Added latamex on ramp option
* fee rate change to Ramp
* Updated VERSION
* Bump to major version
Sometimes eth_getBlockByNumber returns txs with chainId which is not
equal to chanin's id. That caused an error and tx fetching was
interrupted. From now on such txs will be skipped.
* Migrations in place, how to run them?
* Remove down migrations and touch database.go
* Database and Database Test package in place, added functions to get and store app metrics
* make generate output
* Minor bug fix on app metrics insert and select
* Add a validation layer to restrict what can be saved in the database
* Make validation more terse, throw error if schema doesn't exist, expose appmetrics service
* service updates
* Compute all errors before sending them out
* Trying to bring a closjure to appmetrics go
* Expose appmetrics via an api, skip fancy
* Address value as Jason Dawt Rawmasage to ease parsing
* Introduce a buffered chan with magic cap of 8 to minimize writes to DB. Tests for service and API. Also expose GetAppMetrics function.
* Lint issues
* Remove autoincrement, undo waku.json changes, fix error being shadowed, return nil where nil ought to be returned, get rid of buffered channel
* Bump migration number
* Fix API factory usage
* Add comment re:json.RawMessage instead of strings
* Get rid of test vars, throw save error inside the loop
* Update version
Co-authored-by: Samuel Hawksby-Robinson <samuel@samyoul.com>
* Revert "Revert "Expand Local Notifications to support multiple Notification types (#2100)""
This reverts commit 5887337b88.
* Revert "Revert "fix protocol.MessageNotificationBody marshalling""
This reverts commit cf0a16dff1.
* Bump version to 0.70.0
* Added localnotifications for Transaction messages
* Fixed bug where Message.SigPubKey was presumed to be set
* Added lookup for contact existing in Messenger.allContacts
Additionally added functionality to add a contact to the messenger store if it isn't present
* Get chat directly from Messenger.allChats store
Co-authored-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
- avoid making RPC request for `zero - zero` range
- avoid checking of nonce for a lower block in the range if it is zero
in a higher block
- on `wallet_getTransfersByAddress` scanning of history is skipped if
zero block is already reached
- no need to fetch block num before fetching token balances
- old existing ranges are merged when wallet service is started
- a new range is merged with an existing one if possible
This will decrease the number of entries in blocks_range table as
currently it can grow indefinitely (@flexsurfer reported 23307 entries).
This change is also needed for further optimisations of RPC usage.
There was a bug on status-react where it would save filters that were
not listened to.
This commit adds a task to clean up those filters as they might result
in long syncing times.
This commit also returns topics/ranges/mailserves from messenger in
order to make the initialization of the app simpler and start moving
logic to status-go.
It also removes whisper from vendor.
* Initial work on expanding Local Notifications
Adding functionality to support multiple notification types in Notification.Body. Currently have a bug that I think is caused by a the jsonMarshal func not working as intented, need to resolve this next before proceeding
* Fixed json.Marshaller issue and implemented json.Unmarshaller
* Tweak errors, go convention is errors don't begin with capital letters
* Added notificationMessageBody with un/marshalling
Also removed the Body interface
* Added check for bodyType mismatch
* Implement building and sending new message notifications
* Refactor to remove cycle imports
* Resolved linting issue ... Hopefully
* Resolving an implicit memory aliasing in a for loop
* version bump
* Added Notification.Category consts
There was an issue in using the `Wallet` flag when checking accounts to
watch for transactions.
`Wallet` indicates that it's the default wallet, not whether is a wallet
account.
That can only be checked by looking at the type (and the `Wallet` flag).
If the type is `generated`, `key` or `seed` it should be watched for
transactions.
This commit adds an endpoint to batch the sending of messages.
This is useful to simplify client logic when sending a batch of messages
and ensuring the correct order in the message stream.
It currently implements only what's needed, and naively return an error
if any of the messages fail.
We were not actually passing the topics in the request, therefore we
were using bloom filter for query, which resulted in long syncing times
for some users.
- Wallet service is not started on foreground event on status-go side
anymore, it leaves a client side opportunity to decide whether new
blocks should be watched.
- `watchNewBlocks` parameter is added to `StartWallet`.
- Some requests are removed/moved to the place where they are necessary.
If one request failed, the whole batch would fail.
This caused issue as one of the contract is constantly returning an
error now, and essentially there was not way to fetch balance.
Also extend the timeout to 20s as we throw 165 request to Infura in one
go and it takes its time to reply to those, although it seems like we
should batch them on our side instead of sending them all cuncurrently.
Sale Djenic
Michal Iskierko
Stefan
Anthony Laibe
Andrea Maria Piana
IvanBelyakoff
Icaro Motta
frank
Jonathan Rainville
Eng Zer Jun
Mohamed Javid
Dario Gabriel Lipicar
Ibrahem Khalil
Igor Sirotin
Patryk Osmaczko
mprakhov
Pascal Precht
Mikhail Rogachev
Khushboo-dev-cpp
Roman Volosovskyi
RichΛrd
Alexander
Mykhailo Prakhov
Parvesh Monu
Jakub Sokołowski
Omar Basem
Pascal Precht
dlipicar
Boris Melnik
Anthony Laibe
yqrashawn
Samuel Hawksby-Robinson
Ibrahem Khalil
siddarthkay
Churikova Tetiana
Vitaliy Vlasov
Vitaliy Vlasov
andrey
osmaczko
jo-mut
Michele Balistreri
Volodymyr Kozieiev
Pascal Precht
saledjenic
Brian Sztamfater
Shivek Khurana
Eric Mastro
Andrea Franz
Shivek Khurana
Gheorghe Pinzaru