* Remove `--sync-mode` option from nimbus config
why:
Currently there is only one sync mode available.
* Rename `flare` -> `beacon`, but not base module folder and nim source
why:
The name `flare` was used do designate an alternative `beacon` mode that.
Leaving the base folder and source as-is for a moment, makes it easier
to read change diffs.
* Rename `flare` base module folder and nim source: `flare` -> `beacon`
* Dissolve legacy `sync/types.nim` into `*/eth/eth_types.nim`
* Flare sync: Simplify scheduler and remove `runSingle()` method
why:
`runSingle()` is not used anymore (main purpose was for negotiating
best headers in legacy full sync.)
Also, `runMulti()` was renamed `runPeer()`
* Flare sync: Move `chain` field from `sync_desc` -> `worker_desc`
* Flare sync: Remove handler descriptor lateral reference
why:
Not used anymore. It enabled to turn on/off eth handler activity with
regards to the sync state, i.e.from with in the sync worker.
* Flare sync: Update `Hash256` and other deprecated `std/eth` symbols
* Protocols: Update `Hash256` and other deprecated `std/eth` symbols
* Eth handler: Update `Hash256` and other deprecated `std/eth` symbols
* Update flare TODO
* Remove redundant `sync/type` import
why:
The import module `type` has been removed
* Remove duplicate implementation
This is a minimal set of changes to make things work with the new types
in nim-eth - this is the minimal PR that merely resolves
incompatibilities while the full change set would include more cleanup
and migration.
* Cosmetics, small fixes, add stashed headers verifier
* Remove direct `Era1` support
why:
Era1 is indirectly supported by using the import tool before syncing.
* Clarify database persistent save function.
why:
Function relied on the last saved state block number which was wrong.
It now relies on the tx-level. If it is 0, then data are saved directly.
Otherwise the task that owns the tx will do it.
* Extracted configuration constants into separate file
* Enable single peer mode for debugging
* Fix peer losing issue in multi-mode
details:
Running concurrent download peers was previously programmed as running
a batch downloading and storing ~8k headers and then leaving the `async`
function to be restarted by a scheduler.
This was unfortunate because of occasionally occurring long waiting
times for restart.
While the time gap until restarting were typically observed a few
millisecs, there were always a few outliers which well exceed several
seconds. This seemed to let remote peers run into timeouts.
* Prefix function names `unprocXxx()` and `stagedYyy()` by `headers`
why:
There will be other `unproc` and `staged` modules.
* Remove cruft, update logging
* Fix accounting issue
details:
When staging after fetching headers from the network, there was an off
by 1 error occurring when the result was by one smaller than requested.
Also, a whole range was mis-accounted when a peer was terminating
connection immediately after responding.
* Fix slow/error header accounting when fetching
why:
Originally set for detecting slow headers in a row, the counter
was wrongly extended to general errors.
* Ban peers for a while that respond with too few headers continuously
why:
Some peers only returned one header at a time. If these peers sit on a
farm, they might collectively slow down the download process.
* Update RPC beacon header updater
why:
Old function hook has slightly changed its meaning since it was used
for snap sync. Also, the old hook is used by other functions already.
* Limit number of peers or set to single peer mode
details:
Merge several concepts, single peer mode being one of it.
* Some code clean up, fixings for removing of compiler warnings
* De-noise header fetch related sources
why:
Header download looks relatively stable, so general debugging is not
needed, anymore. This is the equivalent of removing the scaffold from
the part of the building where work has completed.
* More clean up and code prettification for headers stuff
* Implement body fetch and block import
details:
Available headers are used stage blocks by combining existing headers
with newly fetched blocks. Then these blocks are imported/executed via
`persistBlocks()`.
* Logger cosmetics and cleanup
* Remove staged block queue debugging
details:
Feature still available, just not executed anymore
* Docu, logging update
* Update/simplify `runDaemon()`
* Re-calibrate block body requests and soft config for import blocks batch
why:
* For fetching, larger fetch requests are mostly truncated anyway on
MainNet.
* For executing, smaller batch sizes reduce the memory needed for the
price of longer execution times.
* Update metrics counters
* Docu update
* Some fixes, formatting updates, etc.
* Update `borrowed` type: uint -. uint64
also:
Always convert to `uint64` rather than `uint` where appropriate
* ForkedChainRef.forkchoice: Skip newBase calculation and skip chain finalization if finalizedHash is zero
* Fix ForkedChainRef.forkChoice: do nothing if headHash is the same with cursorHash
* Fix stupid bug in engine API FCU when calling ForkedChainRef.forkChoice
* Wire RPC server API to nimbus RPC manager
* Add test case
* Use default(Hash256) in ForkedChainRef
* init style for Hash256
https://github.com/status-im/nim-eth/pull/733 updates `Hash256` to
become an array instead of an object - unfortunately, nim does not allow
constructing arrays with `name()`, so this PR changes it to `default`
which works with both.
* lint
* Fix multiple instances running from same dataDir
* Add exclusive lock on lock file
* Unlock lock file on process exit
* Fix minor issues in lock file implementation
The revalidateMax value is lowered to have a quicker ramp up of
the radiusCache + to keep it healthier.
The defaultMaxGossipNodes value is lowered because with the
current value a Nodes lookup is triggered almost always.
This value is dependant on the content replication value. This
is dependant on the network (and subnetwork) because of the amount
of nodes and their radius/storage capacity.
* Support RPC API namespaces as cli parameter.
* Fluffy now uses rpcFlags on startup.
* Update testnet script to enable all RPC APIs.
* Update Fluffy book and move web3 call into eth calls.
This bug would have as effect that our radius cache would not get
filled by any outgoing pings, causing:
- Node lookups to always be occurring on NH gossip
- POKEs to much more rarely
Also add metrics for the amount of offers done via POKE mechanism.
* fix: nimbus state ahead of era history
* comments
* fix: suggestions
* fix: messages
* fix edge case resume
* check from last file
* formatting
* fix: typo
* fix: unwanted quit before rlp import
* Make stop functions wait for completion before return.
* Implement graceful shutdown.
* Shutdown rpc and metric servers if enabled.
* Move metrics and rpc servers out of PortalNode.
* batch database key writes during `computeKey` calls
* log progress when there are many keys to update
* avoid evicting the vertex cache when traversing the trie for key
computation purposes
* avoid storing trivial leaf hashes that directly can be loaded from the
vertex
* Enable state network by default. Create status log loop for state and beacon networks. Create status log loop for portal node. Implement stop functions.
* Add missing leaf cache update when a leaf turns to a branch with two
leaves (on merge) and vice versa (on delete) - this could lead to stale
leaves being returned from the cache causing validation failures - it
didn't happen because the leaf caches were not being used efficiently :)
* Replace `seq` with `ArrayBuf` in `Hike` allowing it to become
allocation-free - this PR also works around an inefficiency in nim in
returning large types via a `var` parameter
* Use the leaf cache instead of `getVtxRc` to fetch recent leaves - this
makes the vertex cache more efficient at caching branches because fewer
leaf requests pass through it.
The storage leaf cache was being circumvented when actually fetching
leaves and was instead only being filled with items :/
Also avoids an expensive copy when fetching account data (broadly,
variant objects are comparatively expensive to copy and fetching
accounts is a hotspot)
* move pfx out of variant which avoids pointless field type panic checks
and copies on access
* make `VertexRef` a non-inheritable object which reduces its memory
footprint and simplifies its use - it's also unclear from a semantic
point of view why inheritance makes sense for storing keys
Compared to `keyed_queue`, `minilru` uses significantly less memory, in
particular for the 32-byte hash keys where `kq` stores several copies of
the key redundantly.
Jacek Sieka
Jordan Hrycaj
bhartnett
tersec
Advaita Saha
andri lim
Kim De Mey
Bhaskar Metiya
web3-developer