Roman Volosovskyi a92a95cf83
status-im/status-react#9203 Faster tx fetching with less request
*** How it worked before this PR on multiaccount creation:
- On multiacc creation we scanned chain for eth and erc20 transfers. For
  each address of a new empty multiaccount this scan required
  1. two `eth_getBalance` requests to find out that there is no any
     balance change between zero and the last block, for eth transfers
  2. and `chain-size/100000` (currently ~100) `eth_getLogs` requests,
     for erc20 transfers
- For some reason we scanned an address of the chat account as well, and
  also accounts were not deduplicated. So even for an empty multiacc we
  scanned chain twice for each chat and main wallet addresses, in result
  app had to execute about 400 requests.
- As mentioned above, `eth_getBalance` requests were used to check if
  there were any eth transfers, and that caused empty history in case
  if user already used all available eth (so that both zero and latest
  blocks show 0 eth for an address). There might have been transactions
  but we wouldn't fetch/show them.
- There was no upper limit for the number of rpc requests during the
  scan, so it could require indefinite number of requests; the scanning
  algorithm was written so that we persisted the whole history of
  transactions or tried to scan form the beginning again in case of
  failure, giving up only after 10 minutes of failures. In result
  addresses with sufficient number of transactions would never be fully
  scanned and during these 10 minutes app could use gigabytes of
  internet data.
- Failures were caused by `eth_getBlockByNumber`/`eth_getBlockByHash`
  requests. These requests return significantly bigger responses than
  `eth_getBalance`/`eth_transactionsCount` and it is likely that
  execution of thousands of them in parallel caused failures for
  accounts with hundreds of transactions. Even for an account with 12k
  we could successfully determine blocks with transaction in a few
  minutes using `eth_getBalance` requests, but `eth_getBlock...`
  couldn't be processed for this acc.
- There was no caching for for `eth_getBalance` requests, and this
  caused in average 3-4 times more such requests than is needed.

*** How it works now on multiaccount creation:
- On multiacc creation we scan chain for last ~30 eth transactions and
  then check erc20 in the range where these eth transactions were found.
  For an empty address in multiacc this means:
  1. two `eth_getBalance` transactions to determine that there was no
     balance change between zero and the last block; two
     `eth_transactionsCount` requests to determine there are no outgoing
     transactions for this address; total 4 requests for eth transfers
  2. 20 `eth_getLogs` for erc20 transfers. This number can be lowered,
     but that's not a big deal
- Deduplication of addresses is added and also we don't scan chat
  account, so a new multiacc requires ~25 (we also request latest block
  number and probably execute a few other calls) request to determine
  that multiacc is empty (comparing to ~400 before)
- In case if address contains transactions we:
  1. determine the range which contains 20-25 outgoing eth/erc20
     transactions. This usually requires up to 10 `eth_transactionCount`
     requests
  2. then we scan chain for eth transfers using `eth_getBalance` and
     `eth_transactionCount` (for double checking zero balances)
  3. we make sure that we do not scan db for more than 30 blocks with
     transfers. That's important for accounts with mostly incoming
     transactions, because the range found on the first step might
     contain any number of incoming transfers, but only 20-25 outgoing
     transactions
  4. when we found ~30 blocks in a given range, we update initial
     range `from` block using the oldest found block
  5. and now we scan db for erc20transfers using `eth_getLogs`
     `oldest-found-eth-block`-`latest-block`, we make not more than 20 calls
  6. when all blocks which contain incoming/outgoing transfers for a
     given address are found, we save these blocks to db and mark that
     transfers from these blocks are still to be fetched
  7. Then we select latest ~30 (the number can be adjusted) blocks from
     these which were found and fetch transfers, this requires 3-4
     requests per transfer.
  8. we persist scanned range so that we know were to start next time
  9. we dispatch an event which tells client that transactions are found
  10. client fetches latest 20 transfers
- when user presses "fetch more" button we check if app's db contains next
  20 transfers, if not we scan chain again and return transfers after

small fixes
2020-01-23 10:36:11 +02:00
..
2020-01-06 10:17:23 +01:00
2020-01-06 10:17:23 +01:00
2019-07-15 14:16:07 +03:00
2020-01-06 10:17:23 +01:00

Wallet

Wallet service starts a loop that watches for new transfers (eth and erc20). To correctly start the service two values need to be changed in the config:

  1. Set Enable to true in WalletConfig
{
  "WalletConfig": {
    "Enabled": true,
  }
}
  1. And expose wallet API with APIModules
{
  APIModules: "eth,net,web3,peer,wallet",
}

API

wallet_getTransfers

Returns avaiable transfers in a given range.

Parameters
  • start: BIGINT - start of the range
  • end: BIGINT - end of the range. if nil query will return all transfers from start.
Examples
{"jsonrpc":"2.0","id":14,"method":"wallet_getTransfers","params":[0,20]}
{"jsonrpc":"2.0","id":14,"method":"wallet_getTransfers","params":[0,null]}
{"jsonrpc":"2.0","id":13,"method":"wallet_getTransfers","params":[0]}
Returns

List of objects like:

[

  {
    "id": "0xac14e5fb9a81fd7d0517e51e23c4f3a8040459bfe0c4bee97b813db2d0438e2e",
    "type": "eth",
    "blockNumber": "0x1",
    "blockhash": "0x1471b02682f2308ce74314d89009251afb1f2d5dedc6835d069b1ad6edf98257",
    "timestamp": "0x5d25a873",
    "gasPrice": "0xa",
    "gasLimit": "0xf4240",
    "gasUsed": "0x5208",
    "nonce": "0x0",
    "input": "0x",
    "txStatus": "0x1",
    "txHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
    "value": "0xde0b6b3a7640000",
    "from": "0xd1c9bfa31ae8c085ba4672b165151245b9bfc25e",
    "to": "0x9dfc85106d84405a83271c2fe0cdfc1ca311a1f5",
    "contract": "0x0000000000000000000000000000000000000000"
  },
  {
    "id": "0x2629ee5f443d558ee4ae9e1cf202d76c04e262051b8d8acde7b766bb9d95068e",
    "type": "erc20",
    "blockNumber": "0x2",
    "blockhash": "0x046ad915b86a5eaa6026c8cdd09ea2f09fd3e603dd6e1ea86e8318f4a4b7d4e0",
    "timestamp": "0x5d25a88a",
    "gasPrice": "0x1",
    "gasLimit": "0xb0b8",
    "gasUsed": "0xb0b8",
    "nonce": "0x1",
    "txStatus": "0x1",
    "input": "0xa9059cbb000000000000000000000000f759c6683dfc5dad899eb86529dfaf4d0b25af1b0000000000000000000000000000000000000000000000000000000000000064",
    "txHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
    "value": "0x64",
    "from": "0xbd691e87d65b2857de55ac44598161ea135f73f6",
    "to": "0xf759c6683dfc5dad899eb86529dfaf4d0b25af1b",
    "contract": "0xd2439b0e20823e1e4c08df2d19c3b6a4c5f8f2d1"
  }
]
Examples
{"jsonrpc":"2.0","id":14,"method":"wallet_getTransfers","params":[0,20]}
{"jsonrpc":"2.0","id":14,"method":"wallet_getTransfers","params":[0,null]}
{"jsonrpc":"2.0","id":13,"method":"wallet_getTransfers","params":[0]}

wallet_getTransfersByAddress

Returns avaiable transfers in a given range.

Parameters
  • address: HEX - ethereum address encoded in hex
  • start: BIGINT - start of the range
  • end: BIGINT - end of the range. if nil query will return all transfers from start.
Examples
{"jsonrpc":"2.0","id":7,"method":"wallet_getTransfersByAddress","params":["0xb81a6845649fa8c042dfaceb3f7a684873406993","0x0"]}
Returns

Objects in the same format.

wallet_getTokensBalances

Returns tokens balances mapping for every account. See section below for the response example.

Parameters
  • accounts HEX - list of ethereum addresses encoded in hex
  • tokens HEX - list of ethereum addresses encoded in hex
{"jsonrpc":"2.0","id":11,"method":"wallet_getTokensBalances","params":[["0x066ed5c2ed45d70ad72f40de0b4dd97bd67d84de", "0x0ed535be4c0aa276942a1a782669790547ad8768"], ["0x5e4bbdc178684478a615354d83c748a4393b20f0", "0x5e4bbdc178684478a615354d83c748a4393b20f0"]]}

Returns

First level keys accounts, second level keys are tokens.

{
  "0x066ed5c2ed45d70ad72f40de0b4dd97bd67d84de": {
    "0x1dfb2099f936b3e98bfc9b7059a8fb04edcce5b3": 12,
    "0x5e4bbdc178684478a615354d83c748a4393b20f0": 12
  },
  "0x0ed535be4c0aa276942a1a782669790547ad8768": {
    "0x1dfb2099f936b3e98bfc9b7059a8fb04edcce5b3": 14,
    "0x5e4bbdc178684478a615354d83c748a4393b20f0": 14
  }
}

Signals

Two signals can be emitted:

  1. newblock signal

Emitted when transfers from new block were added to the database. In this case block number if the number of this new block. Client expected to request transfers starting from received block.

{
  "type": "wallet",
  "event": {
    "type": "newblock",
    "blockNumber": 0,
    "accounts": [
      "0x42c8f505b4006d417dd4e0ba0e880692986adbd8",
      "0x3129mdasmeo132128391fml1130410k312312mll"
    ]
  }
}
  1. reorg signal.

Emitted when part of blocks were removed. Starting from a given block number all transfers were removed. Client expected to request new transfers from received block and replace transfers that were received previously.

{
  "type": "wallet",
  "event": {
    "type": "reorg",
    "blockNumber": 0,
    "accounts": [
      "0x42c8f505b4006d417dd4e0ba0e880692986adbd8"
    ]
  }
}
  1. history signal

Emmited when historical transfers were downloaded. Block number will refer the first block where historical transfers were found.

{
  "type": "wallet",
  "event": {
    "type": "history",
    "blockNumber": 0,
    "accounts": [
      "0x42c8f505b4006d417dd4e0ba0e880692986adbd8"
    ]
  }
}