Also:
- extend maximum expected DB entries to fetch
- fix test TestBalanceHistoryGetWithoutOverlappingFetch passing because of all-time stride side effect
Closes status-desktop #9624
The L2 networks have their own block number and average block duration,
so we need to hardcode to keep the logic simplified.
Updates status-desktop: #9582
Add history.exchange sub-package with following responsibilities
- fetch and caches daily exchange rates for tokens
- Partial update if missing form yesterday back
- Implement all time fetching special case
- Fetches the price of the token after merging entries for the selected
chains
history.service package changes
- Return ValuePoint instead of DataPoint
- Value point contains the value in fiat as float64 instead
- Convert merged values from tokens balance (wei) to fiat
Other changes
- Move start/stop balance history to startWallet/stopWallet
- Move cryptocompare to thirdparty package to avoid recursive import
- Rename DataPoint.Value to DataPoint.Balance
- Don't merge entries for single chain
This commit fixes an issue where when accepting a contact request
the other end would display an extra notification.
It also changes WaitOnResponse to collect results. This should make
tests less flaky, since sometimes messages are processed in different
batches.
Now we need to be though exact on what we expect from the response (i.e
use == instead of >, otherwise the same behavior applies)
This uncovered a couple of issues with messenger.Merge, so I have moved
the struct to use a map based collection instead of an array.
This change improves on the previous implementation which used
the block number that doesn't work with incompatible blockchains e.g. L1 vs. L2
Closes: #9205
Extends wallet module with the history package with the following
components:
BalanceDB (balance_db.go)
- Keeps track of balance information (token count, block, block timestamp)
for a token identity (chain, address, currency)
- The cached data is stored in `balance_history` table.
- Uniqueness constrained is enforced by the `balance_history_identify_entry`
UNIQUE index.
- Optimal DB fetching is ensured by the `balance_history_filter_entries`
index
Balance (balance.go)
- Provides two stages:
- Fetch of balance history using RPC calls (Balance.update function)
- Retrieving of cached balance data from the DB it exists (Balance.get
function)
- Fetching and retrieving of data is done for specific time intervals
defined by TimeInterval "enumeration"
- Update process is done for a token identity by the Balance.Update function
- The granularity of data points returned is defined by the constant
increment step define in `timeIntervalToStride` for each time interval.
- The `blocksStride` values have a common divisor to have cache hit
between time intervals.
Service (service.go)
- Main APIs
- StartBalanceHistory: Regularly updates balance history for all
enabled networks, available accounts and provided tokens.
- GetBalanceHistory: retrieves cached token count for a token identity
(chain, address, currency) for multiple chains
- UpdateVisibleTokens: will set the list of tokens to have historical
balance fetched. This is a simplification to limit tokens to a small
list that make sense
Fetch balance history for ECR20 tokens
- Add token.Manager.GetTokenBalanceAt to fetch balance of a specific
block number of ECR20.
- Add tokenChainClientSource concrete implementation of DataSource
to fetch balance of ECR20 tokens.
- Chose the correct DataSource implementation based on the token
"is native" property.
Tests
Tests are implemented using a mock of `DataSource` interface used
to intercept the RPC calls.
Notes:
- the timestamp used for retrieving block balance is constant
Closes status-desktop: #8175, #8226, #8862
* feat(@desktop/wallet): added iso4217 library for fiat currency display decimals
* feat(@desktop/wallet): added token peg info and use numbers for token market values
* feat(@desktop/wallet): added
* feat(@desktop/wallet): added iso4217 library for fiat currency display decimals
* feat(@desktop/wallet): added token peg info and use numbers for token market values
* feat(@desktop/wallet): extend wallet api to fetch prices in multiple currencies
* chore(@desktop/wallet): rename token peg field for clarity
Add functionality to sample and retrieve balance history and cache
it in memory for the current transfer controller.
The end of the balance history is snapped at twice per day to
avoid having to query the blockchain again for each fetching within
12 hours interval
The functionality will be extended with DB caching, API call batching,
"smarter" cache hitting and syncing between devices
Updates: #7662
Main changes:
- Extend saved addresses DB with sync info: sync timestamp, update timestamp
and soft removed flag
- Create custom protobuf message payload to sync saved addresses
- Cleanup saved addresses on each start of messenger, by deleting
soft removed older entries
- Sync all saved addresses on Messenger.SyncDevices calls
- Sync particular changes to saved addresses
- Add SavedAddressManager instance to messenger
- Note, can't find a clean way to pass the SavedAddressManager to the
messenger, so we create another one
- Add tests for sync and new DB API
Closes: #7229
Remove Favourites APIs and update the saved address APIs
Added up migration scripts that move the favourites from the old table
to the saved_addresses table with true flag and then drop the favourites table.
Required by #6546
Added functionality to find target address when 6th param in path is added
for ex: "m'/44'/60'/0'/0/500" reperents the Address at the 500th index
Added a api to get the Address derived from a private key
* Sync Settings
* Added valueHandlers and Database singleton
Some issues remain, need a way to comparing incoming sql.DB to check if the connection is to a different file or not. Maybe make singleton instance per filename
* Added functionality to check the sqlite filename
* Refactor of Database.SaveSyncSettings to be used as a handler
* Implemented inteface for setting sync protobuf factories
* Refactored and completed adhoc send setting sync
* Tidying up
* Immutability refactor
* Refactor settings into dedicated package
* Breakout structs
* Tidy up
* Refactor of bulk settings sync
* Bug fixes
* Addressing feedback
* Fix code dropped during rebase
* Fix for db closed
* Fix for node config related crashes
* Provisional fix for type assertion - issue 2
* Adding robust type assertion checks
* Partial fix for null literal db storage and json encoding
* Fix for passively handling nil sql.DB, and checking if elem has len and if len is 0
* Added test for preferred name behaviour
* Adding saved sync settings to MessengerResponse
* Completed granular initial sync and clock from network on save
* add Settings to isEmpty
* Refactor of protobufs, partially done
* Added syncSetting receiver handling, some bug fixes
* Fix for sticker packs
* Implement inactive flag on sync protobuf factory
* Refactor of types and structs
* Added SettingField.CanSync functionality
* Addressing rebase artifact
* Refactor of Setting SELECT queries
* Refactor of string return queries
* VERSION bump and migration index bump
* Deactiveate Sync Settings
* Deactiveated preferred_name and send_status_updates
Co-authored-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
* feat: Add more attributes from opensea
Add sale / orders information as well as collection traits
* feat: add background color to opensea response
* feat: Add max value and handle float to opensea response
* feat: enable wallet without network binding
* feat: make transfer network aware
* feat: allow to pass initial networks via config
* fix: nil check and feed
* feat: Add documentation with better function name
* fix: do not init the manager more than once
* fix: PR feedbacks
* Bump version
* Update Jenkinsfile.tests
* Convert int to string
Co-authored-by: RichΛrd <info@richardramos.me>
* Updated Ramp siteUrl to use URL in docs and use the referral code
see https://docs.ramp.network/configuration
* Moved Ramp to the top
* Removed local cryptos and transak
* Removed region field from on ramp struct
* Added basic placeholder for latamex
* Added latamex on ramp option
* fee rate change to Ramp
* Updated VERSION
* Bump to major version
Sometimes eth_getBlockByNumber returns txs with chainId which is not
equal to chanin's id. That caused an error and tx fetching was
interrupted. From now on such txs will be skipped.
- avoid making RPC request for `zero - zero` range
- avoid checking of nonce for a lower block in the range if it is zero
in a higher block
- on `wallet_getTransfersByAddress` scanning of history is skipped if
zero block is already reached
- no need to fetch block num before fetching token balances
- old existing ranges are merged when wallet service is started
- a new range is merged with an existing one if possible
This will decrease the number of entries in blocks_range table as
currently it can grow indefinitely (@flexsurfer reported 23307 entries).
This change is also needed for further optimisations of RPC usage.
- Wallet service is not started on foreground event on status-go side
anymore, it leaves a client side opportunity to decide whether new
blocks should be watched.
- `watchNewBlocks` parameter is added to `StartWallet`.
- Some requests are removed/moved to the place where they are necessary.
If one request failed, the whole batch would fail.
This caused issue as one of the contract is constantly returning an
error now, and essentially there was not way to fetch balance.
Also extend the timeout to 20s as we throw 165 request to Infura in one
go and it takes its time to reply to those, although it seems like we
should batch them on our side instead of sending them all cuncurrently.
StartWallet was called before service initialization.
After the recent changes this call was moved after initialization, but
the geth system automatically start services.
This meant that `IsStarted()` returned true, although the reactor was
not started, and only after calling `StopWallet()` and `StartWallet()`
again the system would reach the right state.
This commit changes the behavior so that we only check whether the
reactor has been started when calling `IsStarted()` and we allow
multiple calls to `Start()` on the signal service, which won't return an
error (it's a noop if callled multiple times).
- unused API methods are removed
- some unusued code is removed too
- API docs are updated
That's just a portion of clean up that should be done,
but the rest of it will probably happen in different PR
with changes to the way how we watch to chain updates.
- In order to avoid handling of the reorganized blocks we use an offset
from the latest known block when start listening to new blocks. Before
this commit the offset was 15 blocks for all networks. This offset is
too big for mainnet and causes noticeable delay of marking a transfer as
confirmed in Status (comparing to etherscan). So it was changed to be 5
blocks on mainnet and is still 15 blocks on other networks.
- Also before this commit all new blocks were handled one by one with
network specific interval (10s for mainnet), which means that in case of
lost internet connection or application suspension (happens on iOS)
receiving of new blocks would be paused and then resumed with the same
"speed" - 1 blocks per 10s. In case if that pause is big enough the
application would never catch up with the latest block in the network,
and this also causes the state of transfers to be delayed in the
application. In this commit in case if there was more than 40s delay
after receiving of the previous block the whole history in range between
the previous received block and ("latest"-reorgeSafetyDepth) block is
checked at once and app catches up with a recent state of the chain.
*** How it worked before this PR on multiaccount creation:
- On multiacc creation we scanned chain for eth and erc20 transfers. For
each address of a new empty multiaccount this scan required
1. two `eth_getBalance` requests to find out that there is no any
balance change between zero and the last block, for eth transfers
2. and `chain-size/100000` (currently ~100) `eth_getLogs` requests,
for erc20 transfers
- For some reason we scanned an address of the chat account as well, and
also accounts were not deduplicated. So even for an empty multiacc we
scanned chain twice for each chat and main wallet addresses, in result
app had to execute about 400 requests.
- As mentioned above, `eth_getBalance` requests were used to check if
there were any eth transfers, and that caused empty history in case
if user already used all available eth (so that both zero and latest
blocks show 0 eth for an address). There might have been transactions
but we wouldn't fetch/show them.
- There was no upper limit for the number of rpc requests during the
scan, so it could require indefinite number of requests; the scanning
algorithm was written so that we persisted the whole history of
transactions or tried to scan form the beginning again in case of
failure, giving up only after 10 minutes of failures. In result
addresses with sufficient number of transactions would never be fully
scanned and during these 10 minutes app could use gigabytes of
internet data.
- Failures were caused by `eth_getBlockByNumber`/`eth_getBlockByHash`
requests. These requests return significantly bigger responses than
`eth_getBalance`/`eth_transactionsCount` and it is likely that
execution of thousands of them in parallel caused failures for
accounts with hundreds of transactions. Even for an account with 12k
we could successfully determine blocks with transaction in a few
minutes using `eth_getBalance` requests, but `eth_getBlock...`
couldn't be processed for this acc.
- There was no caching for for `eth_getBalance` requests, and this
caused in average 3-4 times more such requests than is needed.
*** How it works now on multiaccount creation:
- On multiacc creation we scan chain for last ~30 eth transactions and
then check erc20 in the range where these eth transactions were found.
For an empty address in multiacc this means:
1. two `eth_getBalance` transactions to determine that there was no
balance change between zero and the last block; two
`eth_transactionsCount` requests to determine there are no outgoing
transactions for this address; total 4 requests for eth transfers
2. 20 `eth_getLogs` for erc20 transfers. This number can be lowered,
but that's not a big deal
- Deduplication of addresses is added and also we don't scan chat
account, so a new multiacc requires ~25 (we also request latest block
number and probably execute a few other calls) request to determine
that multiacc is empty (comparing to ~400 before)
- In case if address contains transactions we:
1. determine the range which contains 20-25 outgoing eth/erc20
transactions. This usually requires up to 10 `eth_transactionCount`
requests
2. then we scan chain for eth transfers using `eth_getBalance` and
`eth_transactionCount` (for double checking zero balances)
3. we make sure that we do not scan db for more than 30 blocks with
transfers. That's important for accounts with mostly incoming
transactions, because the range found on the first step might
contain any number of incoming transfers, but only 20-25 outgoing
transactions
4. when we found ~30 blocks in a given range, we update initial
range `from` block using the oldest found block
5. and now we scan db for erc20transfers using `eth_getLogs`
`oldest-found-eth-block`-`latest-block`, we make not more than 20 calls
6. when all blocks which contain incoming/outgoing transfers for a
given address are found, we save these blocks to db and mark that
transfers from these blocks are still to be fetched
7. Then we select latest ~30 (the number can be adjusted) blocks from
these which were found and fetch transfers, this requires 3-4
requests per transfer.
8. we persist scanned range so that we know were to start next time
9. we dispatch an event which tells client that transactions are found
10. client fetches latest 20 transfers
- when user presses "fetch more" button we check if app's db contains next
20 transfers, if not we scan chain again and return transfers after
small fixes
Wallet database refactored so that every query ensures isolation by the network id.
Network id provided when database object is created, thus it is transparent to other parts
of the wallet module.
Additionally every uniqueness index is changed to ensure that it doesn't prevent adding
object with same id but from a different network.
* Store tx and receipt in db and cast it to TransferView on read
* Store Log instead of log index
* Use contract from log and bring back address field
* Add tx status and id fields
[services/wallet] Several changes in API after feedback
- Timestamp from block header stored in blocks table and added to each transfers
- From field is computed from signature to offload this computation from client side
- `history` event is added back, so that client can ignore historical blocks when watching
only for new blocks
- block number and timestamp are marshalled in hex. consistent with ethereum data structures
Stefan
Anthony Laibe
Khushboo-dev-cpp
dlipicar
Andrea Maria Piana
Michal Iskierko
Sale Djenic
frank
dlipicar
Anthony Laibe
Roman Volosovskyi
Richard Ramos
Samuel Hawksby-Robinson
Churikova Tetiana
Jakub Sokołowski
Vitaliy Vlasov
andrey
Eric Mastro
Gheorghe Pinzaru
Adam Babik
Pedro Pombeiro
André Medeiros
dmitry