We periodically check whether we should backup contacts through waku.
Currently it's not tied to any event, it will only schedule a backup on
a tick, or through the provided API endpoint.
* feat: Add more attributes from opensea
Add sale / orders information as well as collection traits
* feat: add background color to opensea response
* feat: Add max value and handle float to opensea response
* feat: enable wallet without network binding
* feat: make transfer network aware
* feat: allow to pass initial networks via config
* fix: nil check and feed
* feat: Add documentation with better function name
* fix: do not init the manager more than once
* fix: PR feedbacks
* Bump version
* Update Jenkinsfile.tests
* Convert int to string
Co-authored-by: RichΛrd <info@richardramos.me>
* Protobufs and adapters
* Added basic anon metric service and config init
* Added fibonacci interval incrementer
* Added basic Client.Start func and integrated interval incrementer
* Added new processed field to app metrics table
* Added id column to app metrics table
* Added migration clean up
* Added appmetrics GetUnprocessed and SetToProcessedByIDs and tests
There was a wierd bug where metrics in the db that did not explicitly insert a value would be NULL, so could not be found by . In addition I've added a new primary id field to the app_metrics table so that updates could be done against very specific metric rows.
* Updated adaptors and db to handle proto_id
I need a way to distinguish individual metric items from each other so that I can ignore the ones that have been seen before.
* Moved incrementer into dedicated file
* Resolve incrementer test fail
* Finalised the main loop functionality
* Implemented delete loop framework
* Updated adaptors file name
* Added delete loop delay and quit, and tweak on RawMessage gen
* Completed delete loop logic
* Added DBLock to prevent deletion during mainLoop
* Added postgres DB connection, integrated into anonmetrics.Server
* Removed proto_id from SQL migration and model
* Integrated postgres with Server and updated adaptors
* Function name update
* Added sample config files for client and server
* Fixes and testing for low level e2e
* make generate
* Fix lint
* Fix for receiving an anonMetricBatch not in server mode
* Postgres test fixes
* Tidy up, make vendor and make generate
* delinting
* Fixing database tests
* Attempted fix of does: cannot open `does' (No such file or directory)
not: cannot open `not' (No such file or directory)
exist: cannot open `exist' (No such file or directory) error on sql resource loas
* Moved all anon metric postgres migration logic and sources into a the protocol/anonmetrics package or sub packages. I don't know if this will fix the does: cannot open `does' (No such file or directory)
not: cannot open `not' (No such file or directory)
exist: cannot open `exist' (No such file or directory) error that happens in Jenkins but this could work
* Lint for the lint god
* Why doesn't the linter list all its problems at once?
* test tweaks
* Fix for wakuV2 change
* DB reset change
* Fix for postgres db migrations fails
* More robust implementation of postgres test setup and teardown
* Added block for anon metrics functionality
* Version Bump to 0.84.0
* Added test to check anon metrics broadcast is deactivated
* Protobufs and adapters
* Added basic anon metric service and config init
* Added new processed field to app metrics table
* Added id column to app metrics table
* Added migration clean up
* Added appmetrics GetUnprocessed and SetToProcessedByIDs and tests
There was a wierd bug where metrics in the db that did not explicitly insert a value would be NULL, so could not be found by . In addition I've added a new primary id field to the app_metrics table so that updates could be done against very specific metric rows.
* Updated adaptors and db to handle proto_id
I need a way to distinguish individual metric items from each other so that I can ignore the ones that have been seen before.
* Added postgres DB connection, integrated into anonmetrics.Server
* Removed proto_id from SQL migration and model
* Integrated postgres with Server and updated adaptors
* Added sample config files for client and server
* Fix lint
* Fix for receiving an anonMetricBatch not in server mode
* Postgres test fixes
* Tidy up, make vendor and make generate
* Moved all anon metric postgres migration logic and sources into a the protocol/anonmetrics package or sub packages. I don't know if this will fix the does: cannot open `does' (No such file or directory)
not: cannot open `not' (No such file or directory)
exist: cannot open `exist' (No such file or directory) error that happens in Jenkins but this could work
AllMessagesFromChatsAndCommunitiesWhichMatchTerm method added which returns all messages which
match the search term, if they belong to either any chat from the chatIds array or any channel of
any community from communityIds array.
This api point is necessary for desktop issue 2934.
Fixes: #2934
* feature(@status-go/chat): implement search on sqlcipher (status-go side)
Searching messages by some term for a specific channel is added on the side of status-go.
Fixes: #2912
* Linting
Co-authored-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
* Adding wakunode module
* Adding wakuv2 fleet files
* Add waku fleets to update-fleet-config script
* Adding config items for waku v2
* Conditionally start waku v2 node depending on config
* Adapting common code to use go-waku
* Setting log level to info
* update dependencies
* update fleet config to use WakuNodes instead of BootNodes
* send and receive messages
* use hash returned when publishing a message
* add waku store protocol
* trigger signal after receiving store messages
* exclude linting rule SA1019 to check deprecated packages
* feat: Add edit communities
Allow Communities to be edited, including display name, description, color, membership, and permissions.
* Added EditCommunity request type
* Fix lint errors
* Allow editing community without changing image
Previously, retaining an existing community image was not possible because the existing community image path had to be provided in the `editCommunity` RPC call to retain the image. However, once the image is processed by status-go, it is encoded as a base64 string and therefore it is not possible to get the original file path back from this string.
This commit allows for the original to be retained by passing an empty string for the image field in the RPC call.
* Don't change permissions. Fixed clock updating
Co-authored-by: Volodymyr Kozieiev <vkjr.sp@gmail.com>
* Updated Ramp siteUrl to use URL in docs and use the referral code
see https://docs.ramp.network/configuration
* Moved Ramp to the top
* Removed local cryptos and transak
* add PinMessage and PinnedMessage
* fix gruop pin messages
* add SkipGroupMessageWrap to pin messages
* update pinMessage ID generation to be symmetric
* Add extra event to capture other type of navigations, allow empty screen name, rename cofx to get rid of clj ns, update tests
* Some view ids are greater than 16 characters. Made it 32 to be safe.
* Tab navigation events occur outside nav, add a new validator for them
* Remove navigate to cofx event, capture screens on will focus, get rid of enum and make valid screens a string less than 32 characters
* Run make generate
* Fix test
* Bump version to 0.75.1
This commit introduces the following changes:
- `local-notifications` require as body an interface complying with
`json.Marshaler`
- removed unmarshaling of `Notifications` as not used (we only Marshal
notifications)
- `protocol/messenger.go` creates directly a `Notification` instead of
having an intermediate format
- add community notifications on request to join
- move parsing of text in status-go for notifications
* Removed region field from on ramp struct
* Added basic placeholder for latamex
* Added latamex on ramp option
* fee rate change to Ramp
* Updated VERSION
* Bump to major version
Sometimes eth_getBlockByNumber returns txs with chainId which is not
equal to chanin's id. That caused an error and tx fetching was
interrupted. From now on such txs will be skipped.
* Migrations in place, how to run them?
* Remove down migrations and touch database.go
* Database and Database Test package in place, added functions to get and store app metrics
* make generate output
* Minor bug fix on app metrics insert and select
* Add a validation layer to restrict what can be saved in the database
* Make validation more terse, throw error if schema doesn't exist, expose appmetrics service
* service updates
* Compute all errors before sending them out
* Trying to bring a closjure to appmetrics go
* Expose appmetrics via an api, skip fancy
* Address value as Jason Dawt Rawmasage to ease parsing
* Introduce a buffered chan with magic cap of 8 to minimize writes to DB. Tests for service and API. Also expose GetAppMetrics function.
* Lint issues
* Remove autoincrement, undo waku.json changes, fix error being shadowed, return nil where nil ought to be returned, get rid of buffered channel
* Bump migration number
* Fix API factory usage
* Add comment re:json.RawMessage instead of strings
* Get rid of test vars, throw save error inside the loop
* Update version
Co-authored-by: Samuel Hawksby-Robinson <samuel@samyoul.com>
* Revert "Revert "Expand Local Notifications to support multiple Notification types (#2100)""
This reverts commit 5887337b88.
* Revert "Revert "fix protocol.MessageNotificationBody marshalling""
This reverts commit cf0a16dff1.
* Bump version to 0.70.0
* Added localnotifications for Transaction messages
* Fixed bug where Message.SigPubKey was presumed to be set
* Added lookup for contact existing in Messenger.allContacts
Additionally added functionality to add a contact to the messenger store if it isn't present
* Get chat directly from Messenger.allChats store
Co-authored-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
- avoid making RPC request for `zero - zero` range
- avoid checking of nonce for a lower block in the range if it is zero
in a higher block
- on `wallet_getTransfersByAddress` scanning of history is skipped if
zero block is already reached
- no need to fetch block num before fetching token balances
- old existing ranges are merged when wallet service is started
- a new range is merged with an existing one if possible
This will decrease the number of entries in blocks_range table as
currently it can grow indefinitely (@flexsurfer reported 23307 entries).
This change is also needed for further optimisations of RPC usage.
There was a bug on status-react where it would save filters that were
not listened to.
This commit adds a task to clean up those filters as they might result
in long syncing times.
This commit also returns topics/ranges/mailserves from messenger in
order to make the initialization of the app simpler and start moving
logic to status-go.
It also removes whisper from vendor.
* Initial work on expanding Local Notifications
Adding functionality to support multiple notification types in Notification.Body. Currently have a bug that I think is caused by a the jsonMarshal func not working as intented, need to resolve this next before proceeding
* Fixed json.Marshaller issue and implemented json.Unmarshaller
* Tweak errors, go convention is errors don't begin with capital letters
* Added notificationMessageBody with un/marshalling
Also removed the Body interface
* Added check for bodyType mismatch
* Implement building and sending new message notifications
* Refactor to remove cycle imports
* Resolved linting issue ... Hopefully
* Resolving an implicit memory aliasing in a for loop
* version bump
* Added Notification.Category consts
There was an issue in using the `Wallet` flag when checking accounts to
watch for transactions.
`Wallet` indicates that it's the default wallet, not whether is a wallet
account.
That can only be checked by looking at the type (and the `Wallet` flag).
If the type is `generated`, `key` or `seed` it should be watched for
transactions.
This commit adds an endpoint to batch the sending of messages.
This is useful to simplify client logic when sending a batch of messages
and ensuring the correct order in the message stream.
It currently implements only what's needed, and naively return an error
if any of the messages fail.
We were not actually passing the topics in the request, therefore we
were using bloom filter for query, which resulted in long syncing times
for some users.
- Wallet service is not started on foreground event on status-go side
anymore, it leaves a client side opportunity to decide whether new
blocks should be watched.
- `watchNewBlocks` parameter is added to `StartWallet`.
- Some requests are removed/moved to the place where they are necessary.
If one request failed, the whole batch would fail.
This caused issue as one of the contract is constantly returning an
error now, and essentially there was not way to fetch balance.
Also extend the timeout to 20s as we throw 165 request to Infura in one
go and it takes its time to reply to those, although it seems like we
should batch them on our side instead of sending them all cuncurrently.
For some reason when calling saveChat from desktop with `lastMessage`
set to null, a sigsev is received.
The issue seems to be in logFormat
05280a7ae3/log/format.go (L356)
which for some reason blows up if passed a nil pointer (`lastMessage`).
Can't replicate on any other platform or running it locally, but hey,
this fixes the issue.
StartWallet was called before service initialization.
After the recent changes this call was moved after initialization, but
the geth system automatically start services.
This meant that `IsStarted()` returned true, although the reactor was
not started, and only after calling `StopWallet()` and `StartWallet()`
again the system would reach the right state.
This commit changes the behavior so that we only check whether the
reactor has been started when calling `IsStarted()` and we allow
multiple calls to `Start()` on the signal service, which won't return an
error (it's a noop if callled multiple times).
Incentivisation was an experiment in running an incentivised fleet that
rewarded nodes based on their well behavior. It was heavily influenced
by https://docs.loki.network/ . It is currently not used anymore, so
removing.
* fix: close resultsets so we don't leak them
* Refactor browsers/database
To implement PR suggestions and improve code quality.
* Refactor services/permissions/database
To implement PR suggestions and improve code quality.
Co-authored-by: Samuel Hawksby-Robinson <samuel@samyoul.com>
Why make the change?
As discussed previously, the way we will move across versions is to maintain completely separate
codebases and eventually remove those that are not supported anymore.
This has the drawback of some code duplication, but the advantage is that is more
explicit what each version requires, and changes in one version will not
impact the other, so we won't pile up backward compatible code.
This is the same strategy used by `whisper` in go ethereum and is influenced by
https://www.youtube.com/watch?v=oyLBGkS5ICk .
All the code that is used for the networking protocol is now under `v0/`.
Some of the common parts might still be refactored out.
The main namespace `waku` deals with `host`->`waku` interactions (through RPC),
while `v0` deals with `waku`->`remote-waku` interactions.
In order to support `v1`, the namespace `v0` will be copied over, and changed to
support `v1`. Once `v0` will be not used anymore, the whole namespace will be removed.
This PR does not actually implement `v1`, I'd rather get things looked over to
make sure the structure is what we would like before implementing the changes.
What has changed?
- Moved all code for the common parts under `waku/common/` namespace
- Moved code used for bloomfilters in `waku/common/bloomfilter.go`
- Removed all version specific code from `waku/common/const` (`ProtocolVersion`, status-codes etc)
- Added interfaces for `WakuHost` and `Peer` under `waku/common/protocol.go`
Things still to do
Some tests in `waku/` are still testing by stubbing components of a particular version (`v0`).
I started moving those tests to instead of stubbing using the actual component, which increases
the testing surface. Some other tests that can't be easily ported should be likely moved under
`v0` instead. Ideally no version specif code should be exported from a version namespace (for
example the various codes, as those might change across versions). But this will be a work-in-progress.
Some code that will be common in `v0`/`v1` could still be extract to avoid duplication, and duplicated only
when implementations diverge across versions.
This resolves a dependency conflict we have with MatterBridge
which was using a newer version of the same package.
This resulted in a JSON marshalling bug that would crash the bridge.
Signed-off-by: Jakub Sokołowski <jakub@status.im>
- unused API methods are removed
- some unusued code is removed too
- API docs are updated
That's just a portion of clean up that should be done,
but the rest of it will probably happen in different PR
with changes to the way how we watch to chain updates.
Currently ENS are verified explicitly by status-react, this is not ideal
as if that fails it will have to be explicilty retried in status-react.
This commits changes that behavior so that ENS are verified in a loop
and updated if new messages are received.
- In order to avoid handling of the reorganized blocks we use an offset
from the latest known block when start listening to new blocks. Before
this commit the offset was 15 blocks for all networks. This offset is
too big for mainnet and causes noticeable delay of marking a transfer as
confirmed in Status (comparing to etherscan). So it was changed to be 5
blocks on mainnet and is still 15 blocks on other networks.
- Also before this commit all new blocks were handled one by one with
network specific interval (10s for mainnet), which means that in case of
lost internet connection or application suspension (happens on iOS)
receiving of new blocks would be paused and then resumed with the same
"speed" - 1 blocks per 10s. In case if that pause is big enough the
application would never catch up with the latest block in the network,
and this also causes the state of transfers to be delayed in the
application. In this commit in case if there was more than 40s delay
after receiving of the previous block the whole history in range between
the previous received block and ("latest"-reorgeSafetyDepth) block is
checked at once and app catches up with a recent state of the chain.
*** How it worked before this PR on multiaccount creation:
- On multiacc creation we scanned chain for eth and erc20 transfers. For
each address of a new empty multiaccount this scan required
1. two `eth_getBalance` requests to find out that there is no any
balance change between zero and the last block, for eth transfers
2. and `chain-size/100000` (currently ~100) `eth_getLogs` requests,
for erc20 transfers
- For some reason we scanned an address of the chat account as well, and
also accounts were not deduplicated. So even for an empty multiacc we
scanned chain twice for each chat and main wallet addresses, in result
app had to execute about 400 requests.
- As mentioned above, `eth_getBalance` requests were used to check if
there were any eth transfers, and that caused empty history in case
if user already used all available eth (so that both zero and latest
blocks show 0 eth for an address). There might have been transactions
but we wouldn't fetch/show them.
- There was no upper limit for the number of rpc requests during the
scan, so it could require indefinite number of requests; the scanning
algorithm was written so that we persisted the whole history of
transactions or tried to scan form the beginning again in case of
failure, giving up only after 10 minutes of failures. In result
addresses with sufficient number of transactions would never be fully
scanned and during these 10 minutes app could use gigabytes of
internet data.
- Failures were caused by `eth_getBlockByNumber`/`eth_getBlockByHash`
requests. These requests return significantly bigger responses than
`eth_getBalance`/`eth_transactionsCount` and it is likely that
execution of thousands of them in parallel caused failures for
accounts with hundreds of transactions. Even for an account with 12k
we could successfully determine blocks with transaction in a few
minutes using `eth_getBalance` requests, but `eth_getBlock...`
couldn't be processed for this acc.
- There was no caching for for `eth_getBalance` requests, and this
caused in average 3-4 times more such requests than is needed.
*** How it works now on multiaccount creation:
- On multiacc creation we scan chain for last ~30 eth transactions and
then check erc20 in the range where these eth transactions were found.
For an empty address in multiacc this means:
1. two `eth_getBalance` transactions to determine that there was no
balance change between zero and the last block; two
`eth_transactionsCount` requests to determine there are no outgoing
transactions for this address; total 4 requests for eth transfers
2. 20 `eth_getLogs` for erc20 transfers. This number can be lowered,
but that's not a big deal
- Deduplication of addresses is added and also we don't scan chat
account, so a new multiacc requires ~25 (we also request latest block
number and probably execute a few other calls) request to determine
that multiacc is empty (comparing to ~400 before)
- In case if address contains transactions we:
1. determine the range which contains 20-25 outgoing eth/erc20
transactions. This usually requires up to 10 `eth_transactionCount`
requests
2. then we scan chain for eth transfers using `eth_getBalance` and
`eth_transactionCount` (for double checking zero balances)
3. we make sure that we do not scan db for more than 30 blocks with
transfers. That's important for accounts with mostly incoming
transactions, because the range found on the first step might
contain any number of incoming transfers, but only 20-25 outgoing
transactions
4. when we found ~30 blocks in a given range, we update initial
range `from` block using the oldest found block
5. and now we scan db for erc20transfers using `eth_getLogs`
`oldest-found-eth-block`-`latest-block`, we make not more than 20 calls
6. when all blocks which contain incoming/outgoing transfers for a
given address are found, we save these blocks to db and mark that
transfers from these blocks are still to be fetched
7. Then we select latest ~30 (the number can be adjusted) blocks from
these which were found and fetch transfers, this requires 3-4
requests per transfer.
8. we persist scanned range so that we know were to start next time
9. we dispatch an event which tells client that transactions are found
10. client fetches latest 20 transfers
- when user presses "fetch more" button we check if app's db contains next
20 transfers, if not we scan chain again and return transfers after
small fixes
Move settings table schema from a key-value store to a one row table with many columns.
We now save the first row with initial data in saveAccountAndLogin and follow up saveSetting calls are only saving one setting at a time.
Co-authored-by: Adam Babik <a.babik@designfortress.com>
This commit does a few things:
1) Handle membership updates using protobuf and adds the relevant
endpoints.
2) Store in memory a map of chats + contacts for faster lookups, which
are then flushed to disk on each update
3) Validate incoming messages
Sorry for the large pr, but you know, v1 :)
Account's address was used as a primary key in accounts db and as a
deterministic id of an account in some API calls. Also it was used as a
part of the name of the account specific database. This revealed some
extra information about the account and wasn't necessary.
At first the hash of the address was planned to be used as a
deterministic id, but we already have a keyUid which is calculated as
sha256 hash of account's public key and has similar properties:
- it is deterministic
- doesn't reveal accounts public key or address in plain
* Use a single Message type `v1/message.go` and `message.go` are the same now, and they embed `protobuf.ChatMessage`
* Use `SendChatMessage` for sending chat messages, this is basically the old `Send` but a bit more flexible so we can send different message types (stickers,commands), and not just text.
* Remove dedup from services/shhext. Because now we process in status-protocol, dedup makes less sense, as those messages are going to be processed anyway, so removing for now, we can re-evaluate if bringing it to status-go or not.
* Change the various retrieveX method to a single one:
`RetrieveAll` will be processing those messages that it can process (Currently only `Message`), and return the rest in `RawMessages` (still transit). The format for the response is:
`Chats`: -> The chats updated by receiving the message
`Messages`: -> The messages retrieved (already matched to a chat)
`Contacts`: -> The contacts updated by the messages
`RawMessages` -> Anything else that can't be parsed, eventually as we move everything to status-protocol-go this will go away.
This commits add a field (parsedMessage) to the json payload sent to
status-react.
This field is the parsed version of the transit message.
The code is all in dedup, I will re-organize it once we made all the
necesseary changes.
Wallet database refactored so that every query ensures isolation by the network id.
Network id provided when database object is created, thus it is transparent to other parts
of the wallet module.
Additionally every uniqueness index is changed to ensure that it doesn't prevent adding
object with same id but from a different network.
* WIP accounts implementation
* Accounts datasore and changes to status mobile API
* Add library changes and method to update config
* Handle error after account selection
* Add two methods to start account to backend
* Use encrypted database for settings and add a service for them
* Resolve linter warning
* Bring back StartNode StopNode for tests
* Add sub accounts and get/save api
* Changes to accounts structure
* Login use root address and fetch necessary info from database
* Cover accounts store with tests
* Refactor in progress
* Initialize status keystore instance before starting ethereum node
* Rework library tests
* Resolve failures in private api test and send transaction test
* Pass pointer to initialized config to unmarshal
* Use multiaccounts/accounts naming consistently
Multiaccount is used as a login identifier
Account references an address and a key, if account is not watch-only.
* Add login timestamp stored in the database to accounts.Account object
* Add photo-path field for multiaccount struct
* Add multiaccoutns rpc with updateAccount method
Update to any other account that wasn't used for login will return an error
* Fix linter in services/accounts
* Select account before starting a node
* Save list of accounts on first login
* Pass account manager to accounts service to avoid selecting account before starting a node
* Add logs to login with save and regualr login
* Add Metadata to messages, expose new messenger methods
This commits modifies deduplicator so that it takes a `StatusMessage`
instead of `WhisperMessage` and also returns a `Metadata` field which is
then passed back by the client when confirming messages, which fixes the
issue we had with not confirming pfs messages.
This commit moves envelopes tracking to status-go.
Post endpoint is not going to track envelopes anymore, as that's taken
care on status-protocol-go side, so this is a breaking change, and
version is updated accordingly.
Andrea Maria Piana
Richard Ramos
Volodymyr Kozieiev
Pascal Precht
Anthony Laibe
saledjenic
Samuel Hawksby-Robinson
Brian Sztamfater
Shivek Khurana
andrey
Michele Balistreri
Eric Mastro
Roman Volosovskyi
Jonathan Rainville
Andrea Franz
Shivek Khurana
Gheorghe Pinzaru
André Medeiros
Adam Babik
Jakub Sokołowski
Pedro Pombeiro
Pedro Pombeiro
Adam Babik
yenda
dmitry