Commit Graph

2838 Commits

Author SHA1 Message Date
Kim De Mey 33015ea4a3
Adjust to the newly specified SSZ types for Block Body and Receipts (#1147) 2022-07-01 21:51:51 +02:00
Jordan Hrycaj 134fe26997
Store proved snap accounts (#1145)
* Relocated `IntervalSets` to nim-stew repo

* Accumulate accounts on temporary kv-DB

why:
  Explore the data as returned from snap/1. Will be converted to a
  `eth/db` next.

details:
  Verify and accumulate per/state-root accounts downloaded via snap.

also:
  Some unit tests

* Replace `Table` by `TrieDatabaseRef` for accounts accumulator

* update ticker statistics

details:
  mean/variance based counter update

* allow persistent db for proved accounts

* rebase, and globally activate unit test

* fix statistics
2022-07-01 12:42:17 +01:00
jangko e274b347ae
add haveBlockAndState to BaseChainDB and use it in engine api 2022-06-30 16:11:43 +07:00
KonradStaniec fdb048ed21
Add getLogs implementation (#1143)
* Add eth_getLogs json-rpc endpoint in Fluffy and Nimbus
2022-06-29 17:44:08 +02:00
jangko 7f0bc71b65
add invalidMissingAncestorReOrg test case 2022-06-27 20:18:54 +07:00
jangko 4a50b00c37
rpc: change getStorageAt param type to HexDataStr
if using HexQuantityStr, it will fail when given hex string with leading zeros
2022-06-27 17:47:59 +07:00
jangko 3f99ece6c5
rpc: change receipt status field type to HexQuantityStr 2022-06-27 13:01:32 +07:00
jangko 71ac6b7de5
engine-api-test: fix transactionReorg test case 2022-06-27 13:01:32 +07:00
jangko b80eca0718
json-rpc: able to query finalized block and safe block header
engine-api:
- store safe block hash and finalized block hash

engine-api test:
- fix test case related to safe block hash and finalized block hash
2022-06-27 11:15:54 +07:00
Kim De Mey cad74db423
Support for multiple content items over Portal stream (#1090)
Each content item send gets prefix with varint (max u32) allowing
to send multiple content items as requested by offer/accept flow.
2022-06-24 15:35:31 +02:00
Kim De Mey d4d4e8c28f
Add accumulator buildProof, verifyProof and verifyHeader (#1132) 2022-06-23 21:00:59 +02:00
Kim De Mey bf511b091e
Docs: Add separate section for seeding history data (#1140)
- Move section of eth_data_exporter out of default way to run
Fluffy
- Update section to explain full flow of seeding history data
into the network
2022-06-23 14:14:31 +02:00
Kim De Mey 589ba8367e
Adjust Grafana dashboard to represent the one used for the fleet (#1139)
This means adding the instance and container variables which can
be used to select the metrics of the specific fleet node.
2022-06-22 17:22:07 +02:00
KonradStaniec 196428dcf5
Use one ws connection to transfer data from source (#1136) 2022-06-22 08:50:58 +02:00
jangko 8dce76c49f
add engine api test fixtures: PoW chain rlp 2022-06-22 08:10:10 +07:00
jangko d07ef2ee56
fix engine api and angine api test 2022-06-22 08:10:01 +07:00
Jacek Sieka 9e8d2d0550
sqlite: bump 2022-06-21 16:18:41 +02:00
KonradStaniec 489b975189
Add seed db file format (#1126)
* Add seed db file format
2022-06-20 16:52:48 +02:00
andri lim 5374874552
bump serialization family(serialization, json, toml) (#1135) 2022-06-20 11:32:41 +02:00
jangko 00a43234d7
fix enable rpc logic in makeConfig 2022-06-17 07:54:13 +07:00
Jordan Hrycaj c123e1eb93
Updated account scheduler (#1124)
* Using `IntervalSet` type data for `LeafRange`

* Updated log ticker

* Update to `eth67`

details:
  Disabled by default, use `ENABLE_LEGACY_ETH66=0` to enable
  No support for `Get/NodeData` dialogue via eth, anymore

* Dissolved fetch/common.nim

details;
  the log/ticker part becomes ticker.nim
  the interval range management is merged into fetch.nim

* Updated account scheduler

why:
  The previous scheduler fetched each account once (for different state
  roots.) The updated scheduler re-calibrates after a change of the state
  root and potentially (until told otherwise) fetches all possible
  accounts.

* Fix `high(P)` fringe cases in `IntervalSet` handling

why:
  The `high(P)` value for a point type `P` cannot be represented with
  half open intervals `[a,b)` for a,b points of `P`. So this single value
  needs extra treatment which was slightly wrong.

* Updated docu/comments

also:
  rebased

* Update scheduler

details:
  Change the `pivot` management when creating new accounts lists. It is
  strictly increasing (and wrapping around) depending on last updated
  accounts list.
2022-06-16 09:58:50 +01:00
Kim De Mey de2051dcf8
Add Accumulator build helper calls and refactors (#1127)
- Move the accumulator definitions to a history accumulator file
- Add accumulator build helper calls + temporary database
- Add a header gossip content key encoding test
- Refactor & some cleanup
2022-06-16 08:50:29 +02:00
jangko 69a1000d77
more engine api test 2022-06-15 15:56:45 +07:00
jangko ee13e5fdae
completing 'PrevRandao Opcode Transactions' test case in engine api test 2022-06-15 07:55:14 +07:00
jangko 77be2f66d2
handle PoA block difficulty during block creation
- in vmState
- in txpool
- in sealing engine

fix #1105
2022-06-15 07:55:00 +07:00
Kim De Mey 6325712aa6
Add accumulator content keys for history network (#1123) 2022-06-14 23:38:34 +02:00
Kim De Mey 49bdaa112e
Increase wait time for content propagation to avoid CI failures (#1122) 2022-06-14 14:23:07 +02:00
jangko a37f8b17e2
fix rpc and websocket server config if they share the same port with engine api
also fixes json-rpc-engine-api server and websocket-engine-api server shutdown code,
checking if they actually created or not at startup.

fix #1119
2022-06-13 19:32:19 +07:00
KonradStaniec b23789d107
Add getting receipts from history network (#1118)
* Add getting receipts from history network
2022-06-10 12:24:53 +02:00
Nikolay Mitev c1f7503402 Use asyncSpawn instead of deprecated asyncCheck 2022-06-09 12:51:06 +03:00
Nikolay Mitev 84a89e918a Add nimbus_addPeer rpc call 2022-06-09 12:51:06 +03:00
KonradStaniec 730013bde3
Change test data and propagation format (#1115)
* Change test data and propagation format

* Add tool to download data from local node
2022-06-08 15:14:01 +02:00
Jordan Hrycaj ee77d704bc
Non-adjacent intervals set management (#1117)
why:
  extracted from snap as separate helper utility

also:
  should go to `stew`, later on
2022-06-07 15:55:42 +01:00
Jordan Hrycaj 76f6de8059
Normalise snap objects (#1114)
* Fix/recover download flag

why:
  The fetch indicator used to control the data download somehow got
  lost during re-org.

* Updated chronicles/logger topics

* Reorganised run state flags

why:
  The original code used a pair of boolean flags `(stopped,stopThisState)`
  which was translated to three states running, stoppedPending, and
  stopped. It is currently not clear whether collapsing some states was
  correct. So the original logic has been re-stored, albeit wrapped into
  directives like `isStopped()` etc.

also:
  Moving some function bodies in `worker.nim`

* Moved `reply_data.nim` and `validate_trienode.nim` to sub-directory `fetch_trie`

why:
  Only used in `fetch_trie.nim`.

* Move `fetch_*` file and directory objects to `fetch` subdirectory

why:
  Only used in `fetch.nim`

* Added start/stop and/or setup/release methods for all sub-modules

why:
  good housekeeping

also:
  updated getters/setters for ctrl states
  updated trace messages
2022-06-06 14:42:08 +01:00
KonradStaniec 0776f35e0c
Use queries with custom xor function (#1113)
* Use queries with custom xor function
2022-06-03 13:44:42 +02:00
jangko 5bd134e2f0
more engine api tests 2022-06-01 20:32:07 +07:00
Kim De Mey 6d8b25a5f5
Update header accumulator test to start from Mainnet genesis (#1111) 2022-06-01 15:21:22 +02:00
Kim De Mey 7216160ad0
Update Ultralight bootstrap nodes for testnet (#1110) 2022-06-01 08:52:54 +02:00
KonradStaniec 5c78fe64e4
Fix utp connection leak on cancel (#1107)
* Fix utp connection leak on cancel
2022-05-31 14:28:02 +02:00
KonradStaniec b975d35c84
Bump sqlite-abi (#1108) 2022-05-31 12:37:10 +02:00
jangko 0ad7ffcdbe
hive: proper rpc test 2022-05-31 15:42:01 +07:00
jangko b03c2dca4f
bump nim-web3: engine api version alpha.9 2022-05-29 11:23:57 +07:00
jangko 9843c9428f
fix related to engine api alpha.9 2022-05-29 11:23:03 +07:00
jangko 8827e04bc0
remove unused coinbase from sealing engine
header.coinbase is handled by txpool, clique, and engine api.
so the sealing engine no need to touch it anymore.
2022-05-29 10:29:56 +07:00
jangko 3f0994f80b
fix clique_sealer: reset block header coinbase
if we are not voting, coinbase should be filled with zero
because other subsystem e.g txpool can produce block header
with non zero coinbase. if that coinbase is one of the signer
and the nonce is zero, that signer will be vote out from
signer list
2022-05-29 10:26:11 +07:00
KonradStaniec af10e8f179
Do not use vacuum when pruning (#1103)
* Do not use vacuum when pruning
2022-05-26 08:26:08 +02:00
jangko 8b0d700b45
fix EVM stack.[] bug 2022-05-25 12:04:47 +07:00
Kim De Mey f219c69840
Rework of Portal json-rpc debug API and related functionality (#1102)
- More consistent naming in calls used by the JSON-RPC debug API
- Use storeContent everywhere to make sure content is only stored
if in range
- Remove populateHistoryDb subcommand and replace by JSON-RPC call
storeContent
- Remove some whitespace and fix some indentations
2022-05-24 13:27:22 +02:00
Jordan Hrycaj 96bb09457e
Snap sync rename objects (#1099)
* Disentangle `collect` module from `reply_data`

why:
  Now the module visible from `collect` for fetching data is `peer/fetch`
  only.

* Merge `SnapPeerHunt` into `collect`

why:
  This part needs to be known by `collect`, only

* rename collect => worker

* Dissolve `sync_fetch_xdesc` module into `common`

why:
  Descriptor is only used in `common` and `fetch_trie`

* rename `snap/peer` directory => `snap/worker`

* rename `SnapSync` -> `Worker`, `SnapPeer` -> `WorkerBuddy`

* moved `snap/base_desc.nim` -> `snap/worker/worker_desc.nim`

* Unified opaque object ref naming in `worker_desc.nim`

details:
  indicated my inheriting module (exactly one, always)
2022-05-24 09:07:39 +01:00
Kim De Mey 553be51217
Change radius-config option to radius and allow also logRadius (#1100) 2022-05-23 23:23:24 +02:00