* partial commit
* fixes
* remove converters too
* revert changes on nimbus_verified_proxy
* revert changes in converter
* revert changes(re-xport) in rpc_types
* update copyright year
* replace types in other binaries
* chain config bug
* fix rebase conflict imcomplete buffer
* fix more rebase buffers
* remove ditto types and converters
* fix the tests
* update copyright year
This was the first API created for this, it has been superseded
by the API that makes use of era1 files and/or the portal_bridge.
Only usage was still in test_portal_testnet which has now been
altered to make use if API calls from Portal specification.
* Add validation functions to be used in state portal rpc.
* Add validation to remaining state portal rpc methods.
* Lookup local content in recursiveFindContent rpc methods.
* portal_stateFindContent and portal_stateOffer no longer store in db.
Portal beacon LC sync can be started now from a provided trusted
block root or, in case it has been running before, from a
previously verified and stored LC bootstrap.
This required altering the the beacon db on how the bootstraps
are stored.
These create only confusion as if they are actual different types
and it is within their usage already clear what they are about
because of the name of the variable or the function.
They are also nowhere aliased like this in any of the Portal
specification.
* Create separate portal rpc handlers for each sub-network.
* Remove random gossip from history and state networks.
* Move shared portal rpc endpoints into common handler.
* Use enum instead of network string.
* bump nimbus-build-system to use Nim v2.0.10
* 2.0.10 fixes
* fluffy linting
* make trivial change which should trigger whole-nimbus+fluffy rebuild/ci
* Nim v2.0.10 chronicles.error/macros.error ambiguity workaround
* another contentType enum specifier
* fluffy linting
* Fix eth/common & web3 related deprecation warnings for fluffy
This commit uses the new types in the new eth/common/ structure
to remove deprecation warnings.
It is however more than just a mass replace as also all places
where eth/common or eth/common/eth_types or eth/common/eth_types_rlp
got imported have been revised and adjusted to a better per submodule
based import.
There are still a bunch of toMDigest deprecation warnings but that
convertor is not needed for fluffy code anymore so in theory it
should not be used (bug?). It seems to still get imported via export
leaks ffrom imported nimbus code I think.
* Address review comments
* Remove two more unused eth/common imports
This is a minimal set of changes to make things work with the new types
in nim-eth - this is the minimal PR that merely resolves
incompatibilities while the full change set would include more cleanup
and migration.
* Fix multiple instances running from same dataDir
* Add exclusive lock on lock file
* Unlock lock file on process exit
* Fix minor issues in lock file implementation
The revalidateMax value is lowered to have a quicker ramp up of
the radiusCache + to keep it healthier.
The defaultMaxGossipNodes value is lowered because with the
current value a Nodes lookup is triggered almost always.
This value is dependant on the content replication value. This
is dependant on the network (and subnetwork) because of the amount
of nodes and their radius/storage capacity.
* Support RPC API namespaces as cli parameter.
* Fluffy now uses rpcFlags on startup.
* Update testnet script to enable all RPC APIs.
* Update Fluffy book and move web3 call into eth calls.
This bug would have as effect that our radius cache would not get
filled by any outgoing pings, causing:
- Node lookups to always be occurring on NH gossip
- POKEs to much more rarely
Also add metrics for the amount of offers done via POKE mechanism.
* Make stop functions wait for completion before return.
* Implement graceful shutdown.
* Shutdown rpc and metric servers if enabled.
* Move metrics and rpc servers out of PortalNode.
* Enable state network by default. Create status log loop for state and beacon networks. Create status log loop for portal node. Implement stop functions.
* Improve state endpoint genesis test and cover cases when accounts, code and slots doesn't exist.
* Refactor state endpoints to support returning partial proofs.
* Implement getProofs in state endpoints.
* Add tests for getProofs and improve code.
* Implement eth_getProof JSON-RPC api in Fluffy.
- Add new content + content key functionality for header by number
- Remove EpochRecords from the network
- Add pruning call for the EpochRecords + required deprecated
functionality
- Adjust getBlock and getBlockHashByNumber to make use of the
new functionality instead
- Delete content_verifier as it was only verifying the now
deprecated EpochRecord
This avoid restarting the node always with a full radius, which
causes the node the be bombarded with offers which it later has
to delete anyhow.
In order to implement this functionality, several changes were
made as the radius needed to move from the Portal wire protocol
current location to the contentDB and beaconDB, which is
conceptually more correct anyhow.
So radius is now part of the database objects and a handler is
used in the portal wire protocol to access its value.
* Return default values when account, slot or code doesn't exist.
* Handle case when storage doesn't exist due to account not existing or being a non contract account.
* Cleaning up, removing cruft and debugging statements
* Make `aristo_delta` fluffy compatible
why:
A sub-module that uses `chronicles` must import all possible
modules used by a parent module that imports the sub-module.
* update TODO
* Extract sub-tree deletion functions into separate sub-modules
* Move/rename `aristo_desc.accLruSize` => `aristo_constants.ACC_LRU_SIZE`
* Lazily delete sub-trees
why:
This gives some control of the memory used to keep the deleted vertices
in the cached layers. For larger sub-trees, keys and vertices might be
on the persistent backend to a large extend. This would pull an amount
of extra information from the backend into the cached layer.
For lazy deleting it is enough to remember sub-trees by a small set of
(at most 16) sub-roots to be processed when storing persistent data.
Marking the tree root deleted immediately allows to let most of the code
base work as before.
* Comments and cosmetics
* No need to import all for `Aristo` here
* Kludge to make `chronicle` usage in sub-modules work with `fluffy`
why:
That `fluffy` would not run with any logging in `core_deb` is a problem
I have known for a while. Up to now, logging was only used for debugging.
With the current `Aristo` PR, there are cases where logging might be
wanted but this works only if `chronicles` runs without the
`json[dynamic]` sinks.
So this should be re-visited.
* More of a kludge
* Update state network to use addressHash instead of address in contract trie and contract code content keys.
* Fix path calculation bug in getParent when working with extension nodes.
* Bump portal spec tests repo.
* Finish updating tests due to portal test vector changes.
* Update Fluffy state bridge to use addressHash.
* Update Fluffy book with correct commands for running portal hive tests.
* Use RPC batching to send offer requests and filter out duplicate offers.
* Lookup offers after gossip to check if gossip successful.
* Use multiple workers for gossiping offers.
* Update Fluffy state network logging.
* Use single RPC calls instead of batching.
* Update cli parameters.
* Fix bug in contract trie offer building.
* Create block offers queue and collect account preimages.
* Implement iterators to return account and storage proofs and bytecode from updatedCaches.
* Implement building offers from proofs.
* Refactor BlockDataRef type to only include required fields.
* Store block data in database.
* Improve state diff types.
* Implement start state backfill from specific block.
* Record last persisted block number in database.
* Persist account preimages in db.
* Apply state updates for DAO hard fork.
* Implement state gossip of block offers via portal JSON RPC.
- Add basic validation for LC bootstrap gossip, validating either
by trusted block root (only 1) when not synced, or by comparing
with the header of the latest finality update when synced.
- Update portal_bridge beacon to also gossip bootstraps into the
network on each end of epoch.
* Started state bridge.
* Implement call to fetch stateDiffs using trace_replayBlockTransactions.
* Convert JSON responses to stateDiff types.
* State updates working for first few blocks.
* Correctly building state for first 200K blocks.
* Add storage of code and cleanup.
* Start state bridge refactor.
* More cleanup and fixes.
* Use RocksDb as backend for state.
* Implement transactions.
* Build RocksDb dependency when building fluffy tools.
* Move code to world state helper.
* Implement producer and consumer queue.
* Cleanup exceptions.
* Improve logging.
* Add update caches to DatabaseRef backends.
The 3 proofs can be reworked to two proofs as we can use the
BeaconBlock directly instead of BeaconBlockHeader and
BeaconBlockBody. This is possible because the HTR of the
BeaconBlock is the same as the one of the BeaconBlockHeader.
This results in 32 bytes less as an intermediate hash can be
removed. But more importantly looks more clean and compact in
structure and code.
- EpochAccumulator got renamed to EpochRecord
- MasterAccumulator is not HistoricalHashesAccumulator
- The List size for the accumulator got a different maximum which
also result in a different encoding and HTR
- Use --portal-rpc-url as url option to connect to Portal JSON-RPC
interface, just as --web3-url for EL JSON RPC interface.
- Improve logging to know beter which call on which JSON-RPC
interface fails
Chirag Parmar
Kim De Mey
bhartnett
Jacek Sieka
andri lim
tersec
Bhaskar Metiya
web3-developer
Jordan Hrycaj