Commit Graph

6 Commits

Author SHA1 Message Date
Roman Volosovskyi c2f22f1fbc
[status-im/status-react#9927] Fast blocks sync after delay
- In order to avoid handling of the reorganized blocks we use an offset
from the latest known block when start listening to new blocks. Before
this commit the offset was 15 blocks for all networks. This offset is
too big for mainnet and causes noticeable delay of marking a transfer as
confirmed in Status (comparing to etherscan). So it was changed to be 5
blocks on mainnet and is still 15 blocks on other networks.
- Also before this commit all new blocks were handled one by one with
network specific interval (10s for mainnet), which means that in case of
lost internet connection or application suspension (happens on iOS)
receiving of new blocks would be paused and then resumed with the same
"speed" - 1 blocks per 10s. In case if that pause is big enough the
application would never catch up with the latest block in the network,
and this also causes the state of transfers to be delayed in the
application. In this commit in case if there was more than 40s delay
after receiving of the previous block the whole history in range between
the previous received block and ("latest"-reorgeSafetyDepth) block is
checked at once and app catches up with a recent state of the chain.
2020-01-30 17:25:56 +02:00
Roman Volosovskyi a92a95cf83
status-im/status-react#9203 Faster tx fetching with less request
*** How it worked before this PR on multiaccount creation:
- On multiacc creation we scanned chain for eth and erc20 transfers. For
  each address of a new empty multiaccount this scan required
  1. two `eth_getBalance` requests to find out that there is no any
     balance change between zero and the last block, for eth transfers
  2. and `chain-size/100000` (currently ~100) `eth_getLogs` requests,
     for erc20 transfers
- For some reason we scanned an address of the chat account as well, and
  also accounts were not deduplicated. So even for an empty multiacc we
  scanned chain twice for each chat and main wallet addresses, in result
  app had to execute about 400 requests.
- As mentioned above, `eth_getBalance` requests were used to check if
  there were any eth transfers, and that caused empty history in case
  if user already used all available eth (so that both zero and latest
  blocks show 0 eth for an address). There might have been transactions
  but we wouldn't fetch/show them.
- There was no upper limit for the number of rpc requests during the
  scan, so it could require indefinite number of requests; the scanning
  algorithm was written so that we persisted the whole history of
  transactions or tried to scan form the beginning again in case of
  failure, giving up only after 10 minutes of failures. In result
  addresses with sufficient number of transactions would never be fully
  scanned and during these 10 minutes app could use gigabytes of
  internet data.
- Failures were caused by `eth_getBlockByNumber`/`eth_getBlockByHash`
  requests. These requests return significantly bigger responses than
  `eth_getBalance`/`eth_transactionsCount` and it is likely that
  execution of thousands of them in parallel caused failures for
  accounts with hundreds of transactions. Even for an account with 12k
  we could successfully determine blocks with transaction in a few
  minutes using `eth_getBalance` requests, but `eth_getBlock...`
  couldn't be processed for this acc.
- There was no caching for for `eth_getBalance` requests, and this
  caused in average 3-4 times more such requests than is needed.

*** How it works now on multiaccount creation:
- On multiacc creation we scan chain for last ~30 eth transactions and
  then check erc20 in the range where these eth transactions were found.
  For an empty address in multiacc this means:
  1. two `eth_getBalance` transactions to determine that there was no
     balance change between zero and the last block; two
     `eth_transactionsCount` requests to determine there are no outgoing
     transactions for this address; total 4 requests for eth transfers
  2. 20 `eth_getLogs` for erc20 transfers. This number can be lowered,
     but that's not a big deal
- Deduplication of addresses is added and also we don't scan chat
  account, so a new multiacc requires ~25 (we also request latest block
  number and probably execute a few other calls) request to determine
  that multiacc is empty (comparing to ~400 before)
- In case if address contains transactions we:
  1. determine the range which contains 20-25 outgoing eth/erc20
     transactions. This usually requires up to 10 `eth_transactionCount`
     requests
  2. then we scan chain for eth transfers using `eth_getBalance` and
     `eth_transactionCount` (for double checking zero balances)
  3. we make sure that we do not scan db for more than 30 blocks with
     transfers. That's important for accounts with mostly incoming
     transactions, because the range found on the first step might
     contain any number of incoming transfers, but only 20-25 outgoing
     transactions
  4. when we found ~30 blocks in a given range, we update initial
     range `from` block using the oldest found block
  5. and now we scan db for erc20transfers using `eth_getLogs`
     `oldest-found-eth-block`-`latest-block`, we make not more than 20 calls
  6. when all blocks which contain incoming/outgoing transfers for a
     given address are found, we save these blocks to db and mark that
     transfers from these blocks are still to be fetched
  7. Then we select latest ~30 (the number can be adjusted) blocks from
     these which were found and fetch transfers, this requires 3-4
     requests per transfer.
  8. we persist scanned range so that we know were to start next time
  9. we dispatch an event which tells client that transactions are found
  10. client fetches latest 20 transfers
- when user presses "fetch more" button we check if app's db contains next
  20 transfers, if not we scan chain again and return transfers after

small fixes
2020-01-23 10:36:11 +02:00
Pedro Pombeiro c8a911ebd1 Use goimports instead of gofmt 2020-01-06 10:17:23 +01:00
Dmitry Shulyak 0165b028c9
Watch new accounts aftter they were saved to accounts table (#1569)
* Watch new accounts once they are saved in accounts table

* Add test that reactor can be restarted and watch new accounts
2019-08-28 10:49:03 +03:00
Dmitry Shulyak e20648ecc7
[services/wallet] Several changes in API after feedback (#1517)
[services/wallet] Several changes in API after feedback

- Timestamp from block header stored in blocks table and added to each transfers
- From field is computed from signature to offload this computation from client side
- `history` event is added back, so that client can ignore historical blocks when watching
only for new blocks
-  block number and timestamp are marshalled in hex. consistent with ethereum data structures
2019-07-10 12:08:43 +03:00
Dmitry Shulyak 047c9b5263
Download transfers starting from latest block header (#1467) 2019-06-14 13:16:30 +03:00