Commit Graph

4060 Commits

Author SHA1 Message Date
Kim De Mey 438e183586
Rename to HistoricalHashesAccumulator as per Portal spec (#2663) 2024-09-27 22:25:26 +02:00
Jordan Hrycaj debb68b3a7
Flare sync: Remove `debug` modules (#2665)
why:
  Not needed anymore but deletion kept in a separate PR to make it easy
  to refer back and find these modules.
2024-09-27 16:59:16 +00:00
Jordan Hrycaj 0d2a72d2a9
Flare sync (#2627)
* Cosmetics, small fixes, add stashed headers verifier

* Remove direct `Era1` support

why:
  Era1 is indirectly supported by using the import tool before syncing.

* Clarify database persistent save function.

why:
  Function relied on the last saved state block number which was wrong.
  It now relies on the tx-level. If it is 0, then data are saved directly.
  Otherwise the task that owns the tx will do it.

* Extracted configuration constants into separate file

* Enable single peer mode for debugging

* Fix peer losing issue in multi-mode

details:
  Running concurrent download peers was previously programmed as running
  a batch downloading and storing ~8k headers and then leaving the `async`
  function to be restarted by a scheduler.

  This was unfortunate because of occasionally occurring long waiting
  times for restart.

  While the time gap until restarting were typically observed a few
  millisecs, there were always a few outliers which well exceed several
  seconds. This seemed to let remote peers run into timeouts.

* Prefix function names `unprocXxx()` and `stagedYyy()` by `headers`

why:
  There will be other `unproc` and `staged` modules.

* Remove cruft, update logging

* Fix accounting issue

details:
  When staging after fetching headers from the network, there was an off
  by 1 error occurring when the result was by one smaller than requested.
  Also, a whole range was mis-accounted when a peer was terminating
  connection immediately after responding.

* Fix slow/error header accounting when fetching

why:
  Originally set for detecting slow headers in a row, the counter
  was wrongly extended to general errors.

* Ban peers for a while that respond with too few headers continuously

why:
  Some peers only returned one header at a time. If these peers sit on a
  farm, they might collectively slow down the download process.

* Update RPC beacon header updater

why:
  Old function hook has slightly changed its meaning since it was used
  for snap sync. Also, the old hook is used by other functions already.

* Limit number of peers or set to single peer mode

details:
  Merge several concepts, single peer mode being one of it.

* Some code clean up, fixings for removing of compiler warnings

* De-noise header fetch related sources

why:
  Header download looks relatively stable, so general debugging is not
  needed, anymore. This is the equivalent of removing the scaffold from
  the part of the building where work has completed.

* More clean up and code prettification for headers stuff

* Implement body fetch and block import

details:
  Available headers are used stage blocks by combining existing headers
  with newly fetched blocks. Then these blocks are imported/executed via
  `persistBlocks()`.

* Logger cosmetics and cleanup

* Remove staged block queue debugging

details:
  Feature still available, just not executed anymore

* Docu, logging update

* Update/simplify `runDaemon()`

* Re-calibrate block body requests and soft config for import blocks batch

why:
* For fetching, larger fetch requests are mostly truncated anyway on
  MainNet.
* For executing, smaller batch sizes reduce the memory needed for the
  price of longer execution times.

* Update metrics counters

* Docu update

* Some fixes, formatting updates, etc.

* Update `borrowed` type: uint -. uint64

also:
  Always convert to `uint64` rather than `uint` where appropriate
2024-09-27 15:07:42 +00:00
andri lim db8b68a28c
ForkedChainRef.forkchoice: Skip newBase calculation and skip chain finalization if finalizedHash is zero (#2654)
* ForkedChainRef.forkchoice: Skip newBase calculation and skip chain finalization if finalizedHash is zero

* Fix ForkedChainRef.forkChoice: do nothing if headHash is the same with cursorHash

* Fix stupid bug in engine API FCU when calling ForkedChainRef.forkChoice

* Wire RPC server API to nimbus RPC manager

* Add test case

* Use default(Hash256) in ForkedChainRef
2024-09-27 07:53:27 +07:00
Jacek Sieka f3e3c6bbe0
init style for Hash256 (#2661)
* init style for Hash256

https://github.com/status-im/nim-eth/pull/733 updates `Hash256` to
become an array instead of an object - unfortunately, nim does not allow
constructing arrays with `name()`, so this PR changes it to `default`
which works with both.

* lint
2024-09-26 13:24:36 +02:00
Bhaskar Metiya b0a4a9ef84
Fix multiple instances running from same dataDir (#2647)
* Fix multiple instances running from same dataDir

* Add exclusive lock on lock file

* Unlock lock file on process exit

* Fix minor issues in lock file implementation
2024-09-26 10:09:34 +02:00
Kim De Mey 5e3f3db165
Remove client-side json-rpc calls that were removed server-side (#2659) 2024-09-25 22:35:19 +02:00
Kim De Mey dd5bb695ed
Tune revalidateMax + defaultMaxGossipNodes values in wire protocol (#2658)
The revalidateMax value is lowered to have a quicker ramp up of
the radiusCache + to keep it healthier.

The defaultMaxGossipNodes value is lowered because with the
current value a Nodes lookup is triggered almost always.
This value is dependant on the content replication value. This
is dependant on the network (and subnetwork) because of the amount
of nodes and their radius/storage capacity.
2024-09-25 17:38:33 +02:00
web3-developer 69d58e8215
Fluffy: Configure RPC APIs via CLI parameter (#2657)
* Support RPC API namespaces as cli parameter.

* Fluffy now uses rpcFlags on startup.

* Update testnet script to enable all RPC APIs.

* Update Fluffy book and move web3 call into eth calls.
2024-09-25 22:44:46 +08:00
Kim De Mey 3820b15f28
Fix bug that would cause no nodes to be selected for revalidation (#2656)
This bug would have as effect that our radius cache would not get
filled by any outgoing pings, causing:
- Node lookups to always be occurring on NH gossip
- POKEs to much more rarely

Also add metrics for the amount of offers done via POKE mechanism.
2024-09-25 11:30:42 +02:00
web3-developer cb69723ff3
Fluffy: Make number of nodes to gossip content to configurable (#2653) 2024-09-25 15:15:20 +08:00
web3-developer b1dc1578f0
Move web3 rpc handler into eth handler. Create debug rpc handler and move existing debug endpoints. (#2655) 2024-09-25 11:28:45 +08:00
Jacek Sieka 513f11f911
bumps (#2652)
eth/stew/unittest2 in preparation for eth refactoring
2024-09-24 13:19:09 +02:00
Kim De Mey e603952df3
Refactor portal test directory structure (#2650) 2024-09-24 13:07:20 +02:00
andri lim 38d651c9c8
FCU should consider ForkedChainRef when calculating valid ancestor (#2651) 2024-09-24 10:53:18 +00:00
Kim De Mey 0fb9581b96
Refactor and apply renaming to post merge block proofs (#2648) 2024-09-23 18:56:28 +02:00
tersec 35d4529ef2
rm outdated development information (#2646) 2024-09-23 09:22:58 +00:00
Advaita Saha 379592e711
Fix import stuck with era history behind (#2629)
* fix: nimbus state ahead of era history

* comments

* fix: suggestions

* fix: messages

* fix edge case resume

* check from last file

* formatting

* fix: typo

* fix: unwanted quit before rlp import
2024-09-21 08:38:38 +02:00
web3-developer 07193968ca
Implement graceful shutdown in Fluffy (#2645)
* Make stop functions wait for completion before return.

* Implement graceful shutdown.

* Shutdown rpc and metric servers if enabled.

* Move metrics and rpc servers out of PortalNode.
2024-09-20 20:54:36 +08:00
Jacek Sieka 7a15aa2a3a
clean up vertex delete (#2644)
avoid allocating and updating the trie twice when the branch is fully
removed
2024-09-20 10:31:29 +02:00
Jacek Sieka b4b4d16729
speed up key computation (#2642)
* batch database key writes during `computeKey` calls
* log progress when there are many keys to update
* avoid evicting the vertex cache when traversing the trie for key
computation purposes
* avoid storing trivial leaf hashes that directly can be loaded from the
vertex
2024-09-20 07:43:53 +02:00
Kim De Mey c77c6239b0
Use discv5's maxDiscv5TalkRespPayload const for payload size calc (#2643) 2024-09-19 20:30:10 +02:00
web3-developer a9ad10cadc
Fluffy state network now enabled by default and improve status logs (#2640)
* Enable state network by default. Create status log loop for state and beacon networks. Create status log loop for portal node. Implement stop functions.
2024-09-19 21:38:49 +08:00
Jacek Sieka 2fe8cc4551
leaf cache fixes (#2637)
* Add missing leaf cache update when a leaf turns to a branch with two
leaves (on merge) and vice versa (on delete) - this could lead to stale
leaves being returned from the cache causing validation failures - it
didn't happen because the leaf caches were not being used efficiently :)
* Replace `seq` with `ArrayBuf` in `Hike` allowing it to become
allocation-free - this PR also works around an inefficiency in nim in
returning large types via a `var` parameter
* Use the leaf cache instead of `getVtxRc` to fetch recent leaves - this
makes the vertex cache more efficient at caching branches because fewer
leaf requests pass through it.
2024-09-19 10:39:06 +02:00
web3-developer ea74e035ab
Add WebSocket support to Fluffy. (#2636)
* Add WebSocket support to Fluffy.

* Support websocket compression.

* Create setupRpcServer closure to remove code duplication.
2024-09-18 21:27:24 +08:00
tersec 3fb2e080ea
rm exp_ RPC API infrastructure; had no actual RPC endpoints (#2635)
* rm exp_ RPC API infrastructure; had no actual RPC endpoints

* update command-line flag descriptions
2024-09-18 08:53:26 +00:00
web3-developer 0ee8e61a3a
Remove RpcProxy from Fluffy and use RpcHttpServer instead. (#2634)
* Remove RpcProxy from Fluffy and use RpcHttpServer instead.

* Cleanup test.
2024-09-18 15:46:50 +08:00
web3-developer c652f5efc2
Fluffy state bridge docs (#2632)
* Don't proxy implemented state JSON-RPC endpoints. Fix minor issue in testnet script.

* Add docs covering usage of the Fluffy state bridge.
2024-09-18 11:56:20 +08:00
tersec 74db8af702
keep macOS CPU targeting consistent with EOL/supported macOS versions (#2633) 2024-09-17 10:22:06 +00:00
tersec 6bf4cd55b9
rm some commented-out/stub/obsolete RPC endpoints (#2630) 2024-09-16 19:33:20 +00:00
Jacek Sieka 5cd0297462
fix missed cache opportunity (#2628)
The storage leaf cache was being circumvented when actually fetching
leaves and was instead only being filled with items :/

Also avoids an expensive copy when fetching account data (broadly,
variant objects are comparatively expensive to copy and fetching
accounts is a hotspot)
2024-09-14 09:47:32 +02:00
Jacek Sieka adb8d64377
simplify VertexRef (#2626)
* move pfx out of variant which avoids pointless field type panic checks
and copies on access
* make `VertexRef` a non-inheritable object which reduces its memory
footprint and simplifies its use - it's also unclear from a semantic
point of view why inheritance makes sense for storing keys
2024-09-13 18:55:17 +02:00
andri lim 0be6291fba
Bump nim-eth and nim-web3 (#2625) 2024-09-13 15:48:27 +02:00
Jacek Sieka 5c1e2e7d3b
Migrate `keyed_queue` to `minilru` (#2608)
Compared to `keyed_queue`, `minilru` uses significantly less memory, in
particular for the 32-byte hash keys where `kq` stores several copies of
the key redundantly.
2024-09-13 15:47:50 +02:00
web3-developer d17ddacf39
More improvements to the Fluffy local testnet script (#2624)
* More improvements to the Fluffy local testnet script.
2024-09-13 20:40:48 +08:00
Kim De Mey d96196d01d
Add support for BlockHeader by number in portal_bridge (#2621) 2024-09-13 13:41:34 +02:00
web3-developer b11701c75a
Update state JSON-RPC endpoints to directly lookup by blockNumber and skip lookup by blockHash. (#2623) 2024-09-13 14:51:16 +08:00
web3-developer 9c1594b417
Implement eth_getProof JSON-RPC API in Fluffy (#2622)
* Improve state endpoint genesis test and cover cases when accounts, code and slots doesn't exist.

* Refactor state endpoints to support returning partial proofs.

* Implement getProofs in state endpoints.

* Add tests for getProofs and improve code.

* Implement eth_getProof JSON-RPC api in Fluffy.
2024-09-13 11:46:35 +08:00
Kim De Mey 45867bc214
Add BlockHeader by number to Portal history network (#2620)
- Add new content + content key functionality for header by number
- Remove EpochRecords from the network
- Add pruning call for the EpochRecords + required deprecated
functionality
- Adjust getBlock and getBlockHashByNumber to make use of the
new functionality instead
- Delete content_verifier as it was only verifying the now
deprecated EpochRecord
2024-09-12 19:03:58 +02:00
tersec aaefac0795
key eth_syncing off correct indication, not peer count (#2619) 2024-09-12 16:42:38 +00:00
andri lim 178d77ab31
Implement EIP-7002 and EIP-7251 (#2616) 2024-09-12 16:09:46 +07:00
andri lim 6503d51b44
Implement EIP-6110: Execution layer triggered deposits (#2612)
* Implement EIP-6110: Execution layer triggered deposits

* Implement EIP-6110 of t8n tool

* Avoid unnecessary DepositRequestType check

* Avoid using 'result' in t8n helpers

* Fix logs collection and deposits validation
2024-09-12 16:09:46 +07:00
jangko 8e8258e460
Prague types conversion 2024-09-12 16:09:42 +07:00
Kim De Mey fd3475ea3e
Add a brief doc page on how to add documentation (#2617) 2024-09-12 10:50:45 +02:00
Kim De Mey 4f0bc49a84
Remove the early-fin and add a 4s timeout on socket destroy instead (#2614) 2024-09-12 10:47:02 +02:00
web3-developer 3a73b948c6
Fluffy testnet script updates (#2615)
* Update local testnet script to support additional options.

* Increase state bridge retry sleep times.

* Fix run fluffy testnet tests in CI.
2024-09-12 16:18:24 +08:00
web3-developer e8a9cfe555
Re-enable eth_getProof implementation (#2599)
* Re-enable eth_getProof implementation.

* Update to use latest Aristo proof changes.

* Refactor and cleanup.
2024-09-12 09:06:31 +08:00
Jordan Hrycaj c6674311eb
Fringe case portal proof for existing account without storage tree (#2613)
detail:
  For practical reasons, ifsuch an account is asked for a slot, an empty
  proof list is returned. It is up to the user to provide an account
  proof that shows that there is no storage tree.
2024-09-11 20:27:42 +00:00
Jordan Hrycaj 75808bc03b
Add portal proof functionality for non-existing keys/paths (#2610) 2024-09-11 09:39:45 +00:00
Jordan Hrycaj 1ced684d8f
Update flare header download mechanism (#2607)
* Reverse order in staged blob lists

why:
  having the largest block number with the least header list index `0`
  makes it easier to grow the list with parent headers, i.e. decreasing
  block numbers.

* Set a header response threshold when to ditch peer

* Refactor extension of staged header chains record

why:
  Was cobbled together as a proof of concept after several approaches of
  how to run the download.

* TODO update

* Make debugging code independent of `release` flag

* Update import from jacek
2024-09-10 11:37:49 +00:00