* Updates to Fluffy book for hive tests.
* Add support to disable state root checks for state content from the command line.
* Update portal_stateStore endpoint to support decoding offer and storing retrieval value.
* Started implementation of state endpoints.
* Add rpc calls and server stubs.
* Initial implementation of getAccountProof and getStorageProof.
* Refactor validation to use toAccount utils functions.
* Add state endpoints tests.
`initTable` is obsolete since nim 0.19 and can introduce significant
memory overhead while providing no benefit (since the table will be
grown to the default initial size on first use anyway).
In particular, aristo layers will not necessarily use all tables they
initialize, for exampe when many empty accounts are being created.
* Refactor get parent gossip code and add Nibbles helper function.
* Add logging to state gossip.
* Unit test recursive gossip using state gossip getParent functions.
* Add recursive gossip genesis json test and fix bug in state gossip getParent.
* Add Nibbles len function.
* Add toContentKey helper functions.
* Implement typed lookup functions for getting account and storage trie nodes and bytecode.
* Re-implementation of recursive gossip.
* Offer validation improvements. Don't allow key end in Nibbles for leafs as defined in the spec.
* Minor refactor to validation and add addition bytecode check.
* Implemented offer validation interface.
* Implement high level offer validation steps.
* Completed state validation tests.
* Update validation to use result type.
* Update state proof verification tests to test offer verification.
* Query history network to get state root by block hash.
* Fix state network test and remove usage of CoreDb.
* Fix state network gossip test and PR comment updates.
* Add trieproof state validation tests and fix for short nodes.
* Add retrieval content validation to state network.
* Complete state validation tests for retrieval content.
* Improve naming functions and parameters.
- Add post Bellatrix Beacon block proof exporting to
eth_data_exporter
- Rework the Bellatrix Beacon block proof exports to us yaml
format
- Move some common code of the two different beacon block proofs
to beacon_chain_block_proof_common module
- Move + add to yaml utils to be usable not just for the tests but
also for eth data exporter
* Remove references to CoreDb in Fluffy code.
* Add missing import.
* Remove coredb imports and use dynamic server port in tests.
* Fix formatting using nph.
- The fluffy test vector repo got forked (well, copied rather) to
become the official one under ethereum github org, so we change
to that repo now and archive ours.
- Our repo also stored accumulator / historical_roots, replace
that with a new repo which is only for network configs.
- Several changes needed to be made due to test vectors that got
updated + some of them got changed to / are yaml format instead of
json.
* Add nph check to fluffy CI lint
* Add a section on nph usage in the fluffy.guide
* Update copyright years for altered files
* Avoid chained methods formatting style in db code
* Update nph in CI to v0.5
* Remove leftover commented import
* Move comment to avoid nph turning complex list into simple list (nph bug)
* Update nph in CI to v0.5.1
* Formatting fluffy with nph v0.5.1
* Completed draft implementation of witness JSON-RPC endpoints for portal network bridge.
* Updated Nimbus RPC configuration to support enabling experimental endpoints.
* Moved witness verification tests.
* Added json test for getProof.
* Added main procs to new tests to fix test suite.
* Added getBlockWitness test to blockchain json test suite.
* Added tests for experimental RPC endpoints and improved the API to support returning state proofs from before or after block execution.
* Correctly rollback transaction in getBlockWitness proc.
* Remove support for validation of missing leafs.
* Move proof types into separate file and borrow required procs.
* Added state proof types.
* Completed initial implementation of witness verification.
* Reject empty proofs.
* Added tests for verification of block witnesses.
* Initial implementation of account and storage proof verification for the portal state network.
* Completed initial state proof verification loop test.
* Completed proof verification tests.
* Minor updates based on PR feedback and comments.
* Add state proof verification test to test runner.
- The current setting of 0.25 was very big. Set to 0.5, which is
potentially still large. This change did expose some issues with
the current implementation and especially testing.
- General clean-up, renaming for consistency, and
re-ordering/restructuring/deletion of some code.
- Fixed several typos
- ...
- Link network gossip validation to LC processor validation
- QuickFix put/get for optimistic and finality updates
- Minor fixes and clean-up
- Improve bridge gossip for new LC Updates
- Adjust local testnet script to be able to locally test this
- Adjust test and skip broken test
* Add discv5 provided ENR directly to Portal protocol routing table
Previous version of getting the ENR from the discv5 routing table
would not work due to the order of first calling the talk protocol
handler and only after that the addEnr to the disc5 routing table.
Instead of changing this order, pass along the ENR directly to
avoid this additional getNode call.
* Still request ENR from discv5 if it wasn't passed via handshake
* Nimbus folder environment update
details:
* Integrated `CoreDbRef` for the sources in the `nimbus` sub-folder.
* The `nimbus` program does not compile yet as it needs the updates
in the parallel `stateless` sub-folder.
* Stateless environment update
details:
* Integrated `CoreDbRef` for the sources in the `stateless` sub-folder.
* The `nimbus` program compiles now.
* Premix environment update
details:
* Integrated `CoreDbRef` for the sources in the `premix` sub-folder.
* Fluffy environment update
details:
* Integrated `CoreDbRef` for the sources in the `fluffy` sub-folder.
* Tools environment update
details:
* Integrated `CoreDbRef` for the sources in the `tools` sub-folder.
* Nodocker environment update
details:
* Integrated `CoreDbRef` for the sources in the
`hive_integration/nodocker` sub-folder.
* Tests environment update
details:
* Integrated `CoreDbRef` for the sources in the `tests` sub-folder.
* The unit tests compile and run cleanly now.
* Generalise `CoreDbRef` to any `select_backend` supported database
why:
Generalisation was just missed due to overcoming some compiler oddity
which was tied to rocksdb for testing.
* Suppress compiler warning for `newChainDB()`
why:
Warning was added to this function which must be wrapped so that
any `CatchableError` is re-raised as `Defect`.
* Split off persistent `CoreDbRef` constructor into separate file
why:
This allows to compile a memory only database version without linking
the backend library.
* Use memory `CoreDbRef` database by default
detail:
Persistent DB constructor needs to import `db/core_db/persistent
why:
Most tests use memory DB anyway. This avoids linking `-lrocksdb` or
any other backend by default.
* fix `toLegacyBackend()` availability check
why:
got garbled after memory/persistent split.
* Clarify raw access to MPT for snap sync handler
why:
Logically, `kvt` is not the raw access for the hexary trie (although
this holds for the legacy database)
It is only (mostly) a skeleton, not further developed and not
used. It is unlikely the way forward either when further
developing the Portal state network.
* Refactoring in preparation for time-based forking.
* Timestamp-based hard-fork-transition.
* Workaround SideEffect issue / compiler bug for both failing locations in Portal history code
---------
Co-authored-by: kdeme <kim.demey@gmail.com>
* Reduce Nim 1.6 compiler warnings/hints for Fluffy and Nimbus proxy
Mostly raises Defect removals, TaintedString removal and some
unnecessary imports.
Also updating the copyright years alongside.
* Further reduce Nim 1.6 compiler warnings/hints for Nimbus
* Move BlockHeaderWithProof content to content key selector 0
- Remove as content type with content key selector 4
- Replace regular block header with BlockHeaderWithProof at
content key selector 0
* Apply blockHeader content key also to bridge
* Add tests for header with proof generation and verification
This is an example of how the beacon state historical_roots could
be added with a proof to be able to provide on the network and
verify by means of proof against the state root.
* Fix Portal Hive fails by correcting Portal history JSON RPC API
- Field naming in discv5_nodeInfo
- Call naming of portal_historyStore
- Other: some proc to func adjustements
- Switch from using Option to Opt which allows for smoother
usage with already existing Result types
- With all moved to Opt, make more use of valueOr to avoid
too many if else clause indentation and unstead have a more
clear error path at each step
- Remove dead code, char limits, style guide, etc.
- Replace getEncodedKeyForContent with ContentKey.init
and use ContentKey.init for each type
* Add headers with proof content type and use it for verification
- Add BlockHeaderWithProof content type & content
- Use BlockHeaderWithProof content to verify if chain data is
part of the canonical chain
- Adjust parser & seeder code to be able to seed these headers
with proof
- Adjust eth_data_exporter to be able to export custom header
ranges for which to build proofs (mostly for testing)
There is currently quite some ugliness & clean-up needed for which
a big part is due tos upporting both BlockHeader and
BlockHeaderWithProof on the network.
* Change accumulator proof to array / SSZ vector type
- Change accumulator proof to SSZ vector instead of SSZ list.
- Add and use general buildProof and buildHeaderWithProof func.
* Make the BlockHeaderWithProof an SSZ Union with None option
* Update portal-spec-tests to master commit
- Can write epoch accumulators to files now with eth_data_exporter
- RPC requests to gossip epoch accumulators now uses these files
instead of building on the fly
- Other build accumulator calls are adjusted and only used for
tests and thus moved to testing folder
Portal master accumulator was removed from the network specs as a
content type shared on the network, as since the merge this is
a finite accumulator (pre-merge only).
So in this PR the accumulator gets removed as network type and
gets instead baked into the library. Building it is done by
seperate tooling (eth_data_exporter).
Because of this a lot of extra code can be removed that was
located in history_network, content_db, portal_protocol, etc.
Also removed to option to build the accumulator at start-up
of fluffy as this takes several minutes making it not viable.
It can still be loaded from a provided file however.
The ssz accumulator file is for now stored in the recently
created portal-spec-tests repository.
- Let accumulator finish its last pre merge epoch (hash_tree_root
on incomplete epoch).
- Adjust code to use isPreMerge and remove isCurrentEpoch
- Split up tests to a set that runs with a mainnet merge block
number and a set that runs with a testing value.
* Fix bug in inCurrentEpoch and improve accumulator related tests
- Fix negative wraparound / underflow in inCurrentEpoch
- Add tests in accumulator tests to verify the above
- Add header offer tests with accumulator that does and doesn't
contain historical epochs
- Additional clean-up of history tests
- enable canonicalVerify in the tests
web3-developer
Jacek Sieka
Kim De Mey
Daniel Sobol
jangko
Jordan Hrycaj
Adam Spitz
KonradStaniec