* Updated to the latest nim-eth, nim-rocksdb, nim-web3
* Bump nimbus-eth2 module and fix related issues
Temporarily disabling Portal beacon light client network as it is
a lot of copy pasted code that did not yet take into account
forks. This will require a bigger rework and was not yet tested
in an actual network anyhow.
* More nimbus fixes after module bumps
---------
Co-authored-by: Adam Spitz <adamspitz@status.im>
Co-authored-by: jangko <jangko128@gmail.com>
Two unresolved items currently:
- Three tests that are temporarily disabled as they fail in the
macro_assembler code, which seems to be due to an ambigious
identifier Stop (Ops and chronos ServerCommand enum).
- i386 CI disabled as it fails at Nim compilation already. Failed
tests where already ignored for this target.
* Move BlockHeaderWithProof content to content key selector 0
- Remove as content type with content key selector 4
- Replace regular block header with BlockHeaderWithProof at
content key selector 0
* Apply blockHeader content key also to bridge
* Add tests for header with proof generation and verification
* Fix Portal Hive fails by correcting Portal history JSON RPC API
- Field naming in discv5_nodeInfo
- Call naming of portal_historyStore
- Other: some proc to func adjustements
- Switch from using Option to Opt which allows for smoother
usage with already existing Result types
- With all moved to Opt, make more use of valueOr to avoid
too many if else clause indentation and unstead have a more
clear error path at each step
- Remove dead code, char limits, style guide, etc.
- Replace getEncodedKeyForContent with ContentKey.init
and use ContentKey.init for each type
* Add headers with proof content type and use it for verification
- Add BlockHeaderWithProof content type & content
- Use BlockHeaderWithProof content to verify if chain data is
part of the canonical chain
- Adjust parser & seeder code to be able to seed these headers
with proof
- Adjust eth_data_exporter to be able to export custom header
ranges for which to build proofs (mostly for testing)
There is currently quite some ugliness & clean-up needed for which
a big part is due tos upporting both BlockHeader and
BlockHeaderWithProof on the network.
* Change accumulator proof to array / SSZ vector type
- Change accumulator proof to SSZ vector instead of SSZ list.
- Add and use general buildProof and buildHeaderWithProof func.
* Make the BlockHeaderWithProof an SSZ Union with None option
* Update portal-spec-tests to master commit
- Can write epoch accumulators to files now with eth_data_exporter
- RPC requests to gossip epoch accumulators now uses these files
instead of building on the fly
- Other build accumulator calls are adjusted and only used for
tests and thus moved to testing folder
For accumulator building we now use intermediary header epoch
files which renders the use of a temporary db for this no longer
needed. Code was already no longer in use.
Portal master accumulator was removed from the network specs as a
content type shared on the network, as since the merge this is
a finite accumulator (pre-merge only).
So in this PR the accumulator gets removed as network type and
gets instead baked into the library. Building it is done by
seperate tooling (eth_data_exporter).
Because of this a lot of extra code can be removed that was
located in history_network, content_db, portal_protocol, etc.
Also removed to option to build the accumulator at start-up
of fluffy as this takes several minutes making it not viable.
It can still be loaded from a provided file however.
The ssz accumulator file is for now stored in the recently
created portal-spec-tests repository.
- Add exportHeaders to write headers in e2s file per epoch
- Add verifyHeaders to verify decoding the above files
- Adjust exportAccumulatorData to use generated header epoch files
and write accumulator in SSZ (no hex)
- Add printAccumulatorData to print all root hashes
* Improvements to the propagation and seeding of data
- Use a lookup for nodes selection in neighborhoodGossip
- Rework populate db code and add `propagateBlockHistoryDb` call
and portal_history__propagateBlock json-rpc call
- Small adjustment to blockwalk
* Avoid storing out-of-range data in the propagate db calls
Currently re-using http connections for the json-rpc requests
seems to slow down the requests considerably. Avoid this by
forcing a close after each requests in the blockwalk tool.
* Sharing block header data around in a Portal history network (PoC)
- Rework PortalStream to have an instance per PortalProtocol (this
needs to be improved eventually). Each instance uses the same
UtpDiscv5Protocol instance.
- Add processContent on receival of accepted data
- Add dumb neighborhoodGossip: dumb in the sense that it only
offers one piece of content at a time.
- Add to / adjust populate_db to also allow for propagation of
the data and add debug rpc call: portal_history_propagate
- Add eth_rpc_client
- Add eth_getBlockbyHash (no txs or uncles) to eth API
- Add additional test to test_portal_testnet which loads 5 block
headers to 1 node, and offers this data to few nodes, which should
propagate it over the network further. Next query every node for
this data.
* Adjust paths on which Fluffy CI is triggered
* Add documentation on the local testnet
- Allow access to contentDB from portal wire protocol
- Use this to do the db.get in `handleFindContent` directly
- Use this to check the `contentKeys` list in `handleOffer`
- Add portal network ping and findNodes JSON-RPC endpoints
- clean-up of some cli arguments
- Add protocol_interop.md document and adjust/fix other
documentation
Currently bootstrap nodes for discv5 and for the Portal nodes
were provided through separate cli arguments. This is however
confusing and cumbersome as typically when (currently) testing
a node will have both discv5 and the Portal networks enabled.
We merge them into one argument.
If a node happens not to support a Portal network, it will be
removed after message request failure.
Commit also contains additional clean-up and nim-eth bump.
* Rename FindNode to FindNodes as per spec
* Use consistently lower case starting camelCase for consts
Style guide nep1 allows for both CamelCase and camelCase for
consts, but we seem to use more often camelCase. Using this now
consistently.
* Change some procs to func
* Add SSZ Unions through case objects
* Add connection id content response test and improve other test vectors
* Implement content keys and ids for state network as per spec
Content keys case object is used so that it can be serialized and
deserialized as an SSZ Union.
* Let message Union in Portal wire protocol start at 0 as per new spec
Kim De Mey
KonradStaniec