When running the import, currently blocks are loaded in batches into a
`seq` then passed to the importer as such.
In reality, blocks are still processed one by one, so the batching does
not offer any performance advantage. It does however require that the
client wastes memory, up to several GB, on the block sequence while
they're waiting to be processed.
This PR introduces a persister that accepts these potentially large
blocks one by one and at the same time removes a number of redundant /
unnecessary copies, assignments and resets that were slowing down the
import process in general.
The radius sort performance improvement in content lookups was
not implemented in the trace version.
Also cleanup some part of the logging related to uTP connection
setup.
The idea of the beacon-lc-bridge was to allow to bridge data into
the Portal network while only using p2p protocols to get access
to the data.
It is however incomplete as for history content the receipts are
missing. These could be added by also adding devp2p access.
But for the beacon content, there would be no way for getting the
historical summaries over p2p.
And then we did not even look yet on how to do this for state.
Considering it is incomplete it was also not being used by anyone
and thus we remove it.
There is an assertion hitting due to the additon of an iterator
that deletes items from the sequence while iteratting over it.
Before the keepIf helper was used that has different code for
doing this similar work.
startedAtMs is the time passed since UNIX epoch in milliseconds.
Cannot use a monotime clock for that. As we want to keep mono
for other Moment.now() usage, imported epochTime from std/times
instead.
* Remove two simple sleeps from history network test
Replace simple sleeps in test_history_network with retries +
sleeps. Current simple sleep setting would occasionally still
fail on CI.
* Increase retries on test_portal_testnet from 2 to 3.
* Add 1 second sleep after headers with proof get gossiped
- Move any validation related code to new file
- Move any type conversion related code to new file
- Prepare validation code for adding different type of canonical
proofs
- Prepare for validation code for json-rpc api
- several other clean-ups and renames
Sort of a first pass as the validation code specifically can
use some further changes.
* Add two, not so great, quick fix attempts for flaky CI
* Another attempt
* Remove the increased sleep in test_history_network
Need something better here.
* Remove getSszDecoded from ContentDb.
* Update ContentDb get to use onData callback to reduce copies.
* Use templates for helper procs in ContentDb.
* Add contains handler to portal protocol.
* Improve performance of DbGetHandler.
* switch to Nim v2.0.12
* fix LruCache capitalization for styleCheck
* KzgProof/KzgCommitment for styleCheck
* TxEip4844 for styleCheck
* styleCheck issues in nimbus/beacon/payload_conv.nim
* ENode for styleCheck
* isOk for styleCheck
* some more styleCheck fixes
* more styleCheck fixes
---------
Co-authored-by: jangko <jangko128@gmail.com>
* Set offer processing logs to debug level and add offer counts to state network.
* Use metrics instead of int counters and remove from logs.
* More logging improvements. Make decoding and validation failures use error log level.
* Add protocol_id to metrics.
* remove redundant abstraction
* fix misleading raises - the implementation actually swallows errors or
panics (depending on how many other layers of abstraction we penetrate
before detecting it)
* Make content request retries configurable and add to state network and history network.
* Set retries to 1. Use uint to prevent negative values. Make contentRequestRetries a debug parameter.
* blocks can be bigger than the default 1mb when json-rpc-encoded - this
happens on sepolia for example
* json-rpc bump improves debug logging and fixes a number of bugs
* json-serialization bump fixes a crash on invalid arrays in json data
At some point, it would probably be better to compute the maximum block
size from actual block constraints, though this is somewhat tricky and
depends on gas limits etc. Until then, 16mb should be plenty.
With this, sepolia can be synced :)
* Cache content after lookups in history network.
* Cleanup config in PortalProtocol.
* Use local content lookup in history network to enable using cache.
* Enable content cache for state network.
* Update state json-rpc endpoints to return local content which uses cache if enabled.
* Add content cache metrics.
* Make content cache configurable.
* Add content cache tests.
Jacek Sieka
andri lim
bhartnett
Kim De Mey
Kim De Mey
tersec