Commit Graph

4055 Commits

Author SHA1 Message Date
Bhaskar Metiya b0a4a9ef84
Fix multiple instances running from same dataDir (#2647)
* Fix multiple instances running from same dataDir

* Add exclusive lock on lock file

* Unlock lock file on process exit

* Fix minor issues in lock file implementation
2024-09-26 10:09:34 +02:00
Kim De Mey 5e3f3db165
Remove client-side json-rpc calls that were removed server-side (#2659) 2024-09-25 22:35:19 +02:00
Kim De Mey dd5bb695ed
Tune revalidateMax + defaultMaxGossipNodes values in wire protocol (#2658)
The revalidateMax value is lowered to have a quicker ramp up of
the radiusCache + to keep it healthier.

The defaultMaxGossipNodes value is lowered because with the
current value a Nodes lookup is triggered almost always.
This value is dependant on the content replication value. This
is dependant on the network (and subnetwork) because of the amount
of nodes and their radius/storage capacity.
2024-09-25 17:38:33 +02:00
web3-developer 69d58e8215
Fluffy: Configure RPC APIs via CLI parameter (#2657)
* Support RPC API namespaces as cli parameter.

* Fluffy now uses rpcFlags on startup.

* Update testnet script to enable all RPC APIs.

* Update Fluffy book and move web3 call into eth calls.
2024-09-25 22:44:46 +08:00
Kim De Mey 3820b15f28
Fix bug that would cause no nodes to be selected for revalidation (#2656)
This bug would have as effect that our radius cache would not get
filled by any outgoing pings, causing:
- Node lookups to always be occurring on NH gossip
- POKEs to much more rarely

Also add metrics for the amount of offers done via POKE mechanism.
2024-09-25 11:30:42 +02:00
web3-developer cb69723ff3
Fluffy: Make number of nodes to gossip content to configurable (#2653) 2024-09-25 15:15:20 +08:00
web3-developer b1dc1578f0
Move web3 rpc handler into eth handler. Create debug rpc handler and move existing debug endpoints. (#2655) 2024-09-25 11:28:45 +08:00
Jacek Sieka 513f11f911
bumps (#2652)
eth/stew/unittest2 in preparation for eth refactoring
2024-09-24 13:19:09 +02:00
Kim De Mey e603952df3
Refactor portal test directory structure (#2650) 2024-09-24 13:07:20 +02:00
andri lim 38d651c9c8
FCU should consider ForkedChainRef when calculating valid ancestor (#2651) 2024-09-24 10:53:18 +00:00
Kim De Mey 0fb9581b96
Refactor and apply renaming to post merge block proofs (#2648) 2024-09-23 18:56:28 +02:00
tersec 35d4529ef2
rm outdated development information (#2646) 2024-09-23 09:22:58 +00:00
Advaita Saha 379592e711
Fix import stuck with era history behind (#2629)
* fix: nimbus state ahead of era history

* comments

* fix: suggestions

* fix: messages

* fix edge case resume

* check from last file

* formatting

* fix: typo

* fix: unwanted quit before rlp import
2024-09-21 08:38:38 +02:00
web3-developer 07193968ca
Implement graceful shutdown in Fluffy (#2645)
* Make stop functions wait for completion before return.

* Implement graceful shutdown.

* Shutdown rpc and metric servers if enabled.

* Move metrics and rpc servers out of PortalNode.
2024-09-20 20:54:36 +08:00
Jacek Sieka 7a15aa2a3a
clean up vertex delete (#2644)
avoid allocating and updating the trie twice when the branch is fully
removed
2024-09-20 10:31:29 +02:00
Jacek Sieka b4b4d16729
speed up key computation (#2642)
* batch database key writes during `computeKey` calls
* log progress when there are many keys to update
* avoid evicting the vertex cache when traversing the trie for key
computation purposes
* avoid storing trivial leaf hashes that directly can be loaded from the
vertex
2024-09-20 07:43:53 +02:00
Kim De Mey c77c6239b0
Use discv5's maxDiscv5TalkRespPayload const for payload size calc (#2643) 2024-09-19 20:30:10 +02:00
web3-developer a9ad10cadc
Fluffy state network now enabled by default and improve status logs (#2640)
* Enable state network by default. Create status log loop for state and beacon networks. Create status log loop for portal node. Implement stop functions.
2024-09-19 21:38:49 +08:00
Jacek Sieka 2fe8cc4551
leaf cache fixes (#2637)
* Add missing leaf cache update when a leaf turns to a branch with two
leaves (on merge) and vice versa (on delete) - this could lead to stale
leaves being returned from the cache causing validation failures - it
didn't happen because the leaf caches were not being used efficiently :)
* Replace `seq` with `ArrayBuf` in `Hike` allowing it to become
allocation-free - this PR also works around an inefficiency in nim in
returning large types via a `var` parameter
* Use the leaf cache instead of `getVtxRc` to fetch recent leaves - this
makes the vertex cache more efficient at caching branches because fewer
leaf requests pass through it.
2024-09-19 10:39:06 +02:00
web3-developer ea74e035ab
Add WebSocket support to Fluffy. (#2636)
* Add WebSocket support to Fluffy.

* Support websocket compression.

* Create setupRpcServer closure to remove code duplication.
2024-09-18 21:27:24 +08:00
tersec 3fb2e080ea
rm exp_ RPC API infrastructure; had no actual RPC endpoints (#2635)
* rm exp_ RPC API infrastructure; had no actual RPC endpoints

* update command-line flag descriptions
2024-09-18 08:53:26 +00:00
web3-developer 0ee8e61a3a
Remove RpcProxy from Fluffy and use RpcHttpServer instead. (#2634)
* Remove RpcProxy from Fluffy and use RpcHttpServer instead.

* Cleanup test.
2024-09-18 15:46:50 +08:00
web3-developer c652f5efc2
Fluffy state bridge docs (#2632)
* Don't proxy implemented state JSON-RPC endpoints. Fix minor issue in testnet script.

* Add docs covering usage of the Fluffy state bridge.
2024-09-18 11:56:20 +08:00
tersec 74db8af702
keep macOS CPU targeting consistent with EOL/supported macOS versions (#2633) 2024-09-17 10:22:06 +00:00
tersec 6bf4cd55b9
rm some commented-out/stub/obsolete RPC endpoints (#2630) 2024-09-16 19:33:20 +00:00
Jacek Sieka 5cd0297462
fix missed cache opportunity (#2628)
The storage leaf cache was being circumvented when actually fetching
leaves and was instead only being filled with items :/

Also avoids an expensive copy when fetching account data (broadly,
variant objects are comparatively expensive to copy and fetching
accounts is a hotspot)
2024-09-14 09:47:32 +02:00
Jacek Sieka adb8d64377
simplify VertexRef (#2626)
* move pfx out of variant which avoids pointless field type panic checks
and copies on access
* make `VertexRef` a non-inheritable object which reduces its memory
footprint and simplifies its use - it's also unclear from a semantic
point of view why inheritance makes sense for storing keys
2024-09-13 18:55:17 +02:00
andri lim 0be6291fba
Bump nim-eth and nim-web3 (#2625) 2024-09-13 15:48:27 +02:00
Jacek Sieka 5c1e2e7d3b
Migrate `keyed_queue` to `minilru` (#2608)
Compared to `keyed_queue`, `minilru` uses significantly less memory, in
particular for the 32-byte hash keys where `kq` stores several copies of
the key redundantly.
2024-09-13 15:47:50 +02:00
web3-developer d17ddacf39
More improvements to the Fluffy local testnet script (#2624)
* More improvements to the Fluffy local testnet script.
2024-09-13 20:40:48 +08:00
Kim De Mey d96196d01d
Add support for BlockHeader by number in portal_bridge (#2621) 2024-09-13 13:41:34 +02:00
web3-developer b11701c75a
Update state JSON-RPC endpoints to directly lookup by blockNumber and skip lookup by blockHash. (#2623) 2024-09-13 14:51:16 +08:00
web3-developer 9c1594b417
Implement eth_getProof JSON-RPC API in Fluffy (#2622)
* Improve state endpoint genesis test and cover cases when accounts, code and slots doesn't exist.

* Refactor state endpoints to support returning partial proofs.

* Implement getProofs in state endpoints.

* Add tests for getProofs and improve code.

* Implement eth_getProof JSON-RPC api in Fluffy.
2024-09-13 11:46:35 +08:00
Kim De Mey 45867bc214
Add BlockHeader by number to Portal history network (#2620)
- Add new content + content key functionality for header by number
- Remove EpochRecords from the network
- Add pruning call for the EpochRecords + required deprecated
functionality
- Adjust getBlock and getBlockHashByNumber to make use of the
new functionality instead
- Delete content_verifier as it was only verifying the now
deprecated EpochRecord
2024-09-12 19:03:58 +02:00
tersec aaefac0795
key eth_syncing off correct indication, not peer count (#2619) 2024-09-12 16:42:38 +00:00
andri lim 178d77ab31
Implement EIP-7002 and EIP-7251 (#2616) 2024-09-12 16:09:46 +07:00
andri lim 6503d51b44
Implement EIP-6110: Execution layer triggered deposits (#2612)
* Implement EIP-6110: Execution layer triggered deposits

* Implement EIP-6110 of t8n tool

* Avoid unnecessary DepositRequestType check

* Avoid using 'result' in t8n helpers

* Fix logs collection and deposits validation
2024-09-12 16:09:46 +07:00
jangko 8e8258e460
Prague types conversion 2024-09-12 16:09:42 +07:00
Kim De Mey fd3475ea3e
Add a brief doc page on how to add documentation (#2617) 2024-09-12 10:50:45 +02:00
Kim De Mey 4f0bc49a84
Remove the early-fin and add a 4s timeout on socket destroy instead (#2614) 2024-09-12 10:47:02 +02:00
web3-developer 3a73b948c6
Fluffy testnet script updates (#2615)
* Update local testnet script to support additional options.

* Increase state bridge retry sleep times.

* Fix run fluffy testnet tests in CI.
2024-09-12 16:18:24 +08:00
web3-developer e8a9cfe555
Re-enable eth_getProof implementation (#2599)
* Re-enable eth_getProof implementation.

* Update to use latest Aristo proof changes.

* Refactor and cleanup.
2024-09-12 09:06:31 +08:00
Jordan Hrycaj c6674311eb
Fringe case portal proof for existing account without storage tree (#2613)
detail:
  For practical reasons, ifsuch an account is asked for a slot, an empty
  proof list is returned. It is up to the user to provide an account
  proof that shows that there is no storage tree.
2024-09-11 20:27:42 +00:00
Jordan Hrycaj 75808bc03b
Add portal proof functionality for non-existing keys/paths (#2610) 2024-09-11 09:39:45 +00:00
Jordan Hrycaj 1ced684d8f
Update flare header download mechanism (#2607)
* Reverse order in staged blob lists

why:
  having the largest block number with the least header list index `0`
  makes it easier to grow the list with parent headers, i.e. decreasing
  block numbers.

* Set a header response threshold when to ditch peer

* Refactor extension of staged header chains record

why:
  Was cobbled together as a proof of concept after several approaches of
  how to run the download.

* TODO update

* Make debugging code independent of `release` flag

* Update import from jacek
2024-09-10 11:37:49 +00:00
andri lim 38c58c4feb
Implement EIP-2935: Serve historical block hashes from state (#2606)
* Implement EIP-2935: Serve historical block hashes from state

* Fix EIP-2935 in t8n
2024-09-10 09:52:03 +00:00
andri lim 5464f8e5f1
Fix EIP-2537: Precompile for BLS12-381 curve operations (#2603)
* Fix EIP-2537: Precompile for BLS12-381 curve operations

* Update test vectors
2024-09-10 06:56:08 +00:00
tersec 838c9854e7
increase Python dependencies to address urllib3 vuln and certifi root cert (#2605) 2024-09-10 06:36:28 +00:00
tersec 2bbf98e3c8
bump nim-eth and nim-nat-traversal (#2604) 2024-09-10 06:33:34 +00:00
tersec 1b173d420d
small cleanups (#2598)
* small cleanups

* stop hiding ConvFromXtoItselfNotNeeded hints

* lowmem optimization flag is no-op
2024-09-10 05:24:45 +00:00