Commit Graph

99 Commits

Author SHA1 Message Date
Kim De Mey 2fb90cfb43
Initial implementation of era1 + export + verify (#1998)
* Initial implementation of era1 + export + verify

Initial implementation of era1 + export and verify command in the
existing eth_data_exporter.

* Quick verify version to test with geth created era1 files

* Add Era1File object and getter for specific block tuple

* Add getAccumulatorRoot and update AccumulatorRoot type name

+ clean-up of some comments.

* Implement BlockIndex based verify + additional helper calls

+ fix bug for reading last (incomplete) Era1 file

* Fix init BlockIndex offsets for last era + extra check on reading

* Remove duplicated e2store code
2024-02-09 11:13:12 +01:00
Kim De Mey 3199857ed5
Further fluffy code clean-up from warnings/hints (#1987) 2024-01-25 11:04:09 +01:00
Kim De Mey dbc1ae86e2
Vendor bumps + related fixes + warning fixes (#1985)
- Vendor bump of stew, nim-eth, chronos, nimbus-eth2 and libp2p
- Bump related fixes + fixes of deprecation warnings
- Several other warnings fixed.
2024-01-24 16:28:03 +01:00
Daniel Sobol 7ecd38347f
New state network types update (#1976) 2024-01-19 20:18:57 +03:00
Kim De Mey 2df9e2b7b1
Cleanup utp_test and make the code more flexible to change amounts (#1972) 2024-01-15 17:29:29 +01:00
jangko 3e21281d12
Bump nim-web3 and others
Bump nim-json-rpc and nimbus-eth2 too.
Reason: both nim-json-rpc and nim-web3 migrate from
stdlib/json to nim-json-serialization
2024-01-14 10:41:23 +07:00
Kim De Mey cbf5f59681
Bump chronos and other sub modules + related fixes (#1966)
* Bump chronos and additional modules + fixes

* Additional chronos bump + nim-eth bump for utp and discv5 fixes

* Fix the raw Exception errors in async evm

---------

Co-authored-by: jangko <jangko128@gmail.com>
2024-01-12 22:06:19 +01:00
Daniel Sobol 70013422cd
Initial impl of state network bridge 1902 (#1948) 2024-01-09 12:32:29 +03:00
jangko 5fd54961e3
Bump nim-web3: remove rpc types duplicates 2023-12-13 07:58:12 +07:00
Kim De Mey c99c0687f8
Rename beacon bridge to portal_bridge and prep for 1 bridge binary (#1936)
Renaming the beacon_chain_bridge to portal_bridge and preparing
for stuffing all bridge functionality for all networks under this
2023-12-12 21:08:58 +01:00
Daniel Sobol dc55ee0c87
fix local dev container for portal-hive (#1931)
* fix local dev container for portal-hive
2023-12-11 14:56:00 +01:00
Kim De Mey b27c2ff449
Add fcli_db tool for offline interactions with a fluffy ContentDb (#1929) 2023-12-07 13:08:49 +01:00
Kim De Mey aad4adde07
Move all Fluffy database code to database folder (#1918) 2023-12-01 17:20:52 +01:00
Kim De Mey 86f93fdba5
ValidIpAddress -> IpAddress related vendor bumps and changes (#1892) 2023-11-10 19:38:11 +01:00
Kim De Mey 436ef0e40e
Portal BeaconLightClientNetwork -> BeaconNetwork renaming (#1832) 2023-10-20 12:06:25 +02:00
Kim De Mey 0472b75e23
Rework fluffy beacon lc db to allow storing processed content (#1821) 2023-10-18 16:59:44 +02:00
andri lim 04c7ed8ec9
Change EthTime from std.Time to distinct uint64 (#1820)
* Change EthTime from std.Time to distinct uint64

* Bump nimbus-eth2
2023-10-18 09:16:11 +07:00
Kim De Mey ca61a7009d
Bump submodules and related fixes (#1729)
* Bump submodules and related fixes

* Fix some warnings due to chronos and Nim bumps

* Seems macOS and Windows have different behavior
2023-10-17 14:19:50 +02:00
Kim De Mey 7b313ec1ac
Use signature slot instead of attested header slot (#1811)
Use signature slot instead of attested header slot in the
OptimisticUpdate content key.
2023-10-06 17:42:34 +02:00
Kim De Mey f72f02c88b
Remove optimistic slot from FinalityUpdate content key (#1810) 2023-10-06 15:46:53 +02:00
Kim De Mey 43b6014ffe
Further Portal beacon network improvements (#1808) 2023-10-05 19:29:39 +02:00
Kim De Mey 2f2c5127ea
Further implementation to working prototype for Portal beacon LC (#1781)
- Link network gossip validation to LC processor validation
- QuickFix put/get for optimistic and finality updates
- Minor fixes and clean-up
- Improve bridge gossip for new LC Updates
- Adjust local testnet script to be able to locally test this
- Adjust test and skip broken test
2023-09-28 18:16:41 +02:00
Daniel Sobol bd1a647987
Dockerfile for portal-hive development (#1759)
improves build time for a local dev container for a portal-hive test suite
2023-09-20 19:07:09 +03:00
andri lim 7af9e3dc53
Refactor beacon skeleton (#1761)
* Fix unlisted exception due to recent modification in CoreDB interface

* Refactor beacon skeleton

* More unlisted exception fix
2023-09-19 11:52:28 +07:00
Kim De Mey cb4626b488
Initial implementation of Portal fluffy beacon chain bridge (#1751)
* Initial implementation of Portal fluffy beacon chain bridge

* Run windows ci for fluffy with -j1 to avoid OOM
2023-09-15 16:21:00 +02:00
andri lim 56215ed83f
Bump stint to v2.0: new array backend (#1747)
* Bump stint to v2.0: new array backend
2023-09-13 09:32:38 +07:00
Daniel Sobol 69a1a988b0
don't gossip data back to the node we rcvd it from [#1227] (#1719) 2023-09-04 12:21:01 +02:00
andri lim bdaeedb09f
rename data gas to blob gas (#1659)
* rename data gas to blob gas

* bump more submodules
* extend evmc tx_context with EIP-4844 blob_hashes
2023-08-04 19:43:30 +07:00
Jordan Hrycaj 221e6c9e2f
Unified database frontend integration (#1670)
* Nimbus folder environment update

details:
* Integrated `CoreDbRef` for the sources in the `nimbus` sub-folder.
* The `nimbus` program does not compile yet as it needs the updates
  in the parallel `stateless` sub-folder.

* Stateless environment update

details:
* Integrated `CoreDbRef` for the sources in the `stateless` sub-folder.
* The `nimbus` program compiles now.

* Premix environment update

details:
* Integrated `CoreDbRef` for the sources in the `premix` sub-folder.

* Fluffy environment update

details:
* Integrated `CoreDbRef` for the sources in the `fluffy` sub-folder.

* Tools environment update

details:
* Integrated `CoreDbRef` for the sources in the `tools` sub-folder.

* Nodocker environment update

details:
* Integrated `CoreDbRef` for the sources in the
  `hive_integration/nodocker` sub-folder.

* Tests environment update

details:
* Integrated `CoreDbRef` for the sources in the `tests` sub-folder.
* The unit tests compile and run cleanly now.

* Generalise `CoreDbRef` to any `select_backend` supported database

why:
  Generalisation was just missed due to overcoming some compiler oddity
  which was tied to rocksdb for testing.

* Suppress compiler warning for `newChainDB()`

why:
  Warning was added to this function which must be wrapped so that
  any `CatchableError` is re-raised as `Defect`.

* Split off persistent `CoreDbRef` constructor into separate file

why:
  This allows to compile a memory only database version without linking
  the backend library.

* Use memory `CoreDbRef` database by default

detail:
 Persistent DB constructor needs to import `db/core_db/persistent

why:
 Most tests use memory DB anyway. This avoids linking `-lrocksdb` or
 any other backend by default.

* fix `toLegacyBackend()` availability check

why:
  got garbled after memory/persistent split.

* Clarify raw access to MPT for snap sync handler

why:
  Logically, `kvt` is not the raw access for the hexary trie (although
  this holds for the legacy database)
2023-08-04 12:10:09 +01:00
Kim De Mey 31cff910b1
Rename beacon_chain_bridge to beacon_lc_bridge (#1655) 2023-07-25 20:53:31 +02:00
Kim De Mey c20892f805
Allow starting beacon network in local testnet and add lc bridge (#1654)
Add the option to start the fluffy nodes with the beacon network
activated and allow for running a beacon lc bridge in the local
testnet script.
2023-07-25 14:52:44 +02:00
Kim De Mey 89b3e679e2
Add new portal beacon chain lc content encoding tests (#1636) 2023-07-08 17:01:33 +02:00
Kim De Mey ab13e43db8
Add export of beacon LC data to eth_data_exporter (#1629)
* Add export of beacon LC data to eth_data_exporter

* Update content seeding docs with new eth_data_exporter cli
2023-07-05 20:17:03 +02:00
Kim De Mey b3d06c14b6
Build Fluffy tools individually through Makefile (#1626)
* Build Fluffy tools individually through Makefile

Before the builds were launched sequentually from a nimble task,
this would not allow for concurrent builds of the individual tools

* Some Makefile / nimble clean-up
2023-07-05 16:44:18 +02:00
jangko 67aaf92c1d
bump submodules 2023-06-07 18:12:02 +07:00
Kim De Mey c9f3f82877
Add new Portal BlockBody type for Shanghai fork (#1589) 2023-05-30 06:56:54 +02:00
Kim De Mey 408394a2bd
Bump nim-eth and remove unneeded Defect raises (#1575) 2023-05-10 18:04:35 +02:00
Kim De Mey ff90f4fd22
Improve logging and logging options in Fluffy (#1548)
* Improve logging and logging options in Fluffy

- Allow selection of log format, including:
  - JSON
  - automatic selection based on tty
- Allow log levels per topic configured on cli
2023-04-19 17:01:01 +02:00
Kim De Mey 3eafb15d7c
Make beacon LC adjustements to support capella in light clients (#1545)
- Bump nimbus-eth2 repo
- Beacon LC adjustements in fluffy
- Beacon LC adjustements in nimbus_verified_proxy
2023-04-13 18:40:01 +02:00
Kim De Mey 9fde5c96e8
Prepare Portal bridge for Capella by adding withdrawalsRoot (#1543) 2023-04-13 00:02:38 +02:00
Kim De Mey 034d0e6305
Add exporting of headersWithProof and configurable web3 url (#1539) 2023-04-11 14:06:45 +02:00
Kim De Mey 4c19a0e48e
Add support for injecting receipts into the Portal Network (#1523)
Allow for the Fluffy Portal bridge to inject receipts. This
requires a web3 endpoint to be provided, and currently only
Alchemy is supported due to the used JSON-RPC endpoint.
2023-03-28 15:50:23 +02:00
Kim De Mey 84d9b9889f
Add beacon chain lc content bridging to the Fluffy bridge (#1517)
Also delete the old, obsolete bridge that only stores content into
the db.
2023-03-23 08:49:16 +01:00
Adam Spitz d8a1adacaa
More work on withdrawals (#1482)
* Part of EIP-4895: add withdrawals processing to block processing.

* Refactoring: extracted the engine API handler bodies into procs.

Intending to implement the V2 versions next. (I need the bodies to be
in separate procs so that multiple versions can use them.)

* Working on Engine API changes for Shanghai.

* Updated nim-web3, resolved ambiguity in Hash256 type.

* Updated nim-eth3 to point to master, now that I've merged that.

* I'm confused about what's going on with engine_client.

But let's try resolving this Hash256 ambiguity.

* Still trying to fix this conflict with the Hash256 types.

* Does this work now that nimbus-eth2 has been updated?

* Corrected blockValue in getPayload responses back to UInt256.

c834f67a37

* Working on getting the withdrawals-related tests to pass.

* Fixing more of those Hash256 ambiguities.

(I'm not sure why the nim-web3 library introduced a conflicting type
named Hash256, but right now I just want to get this code to compile again.)

* Bumped a couple of libraries to fix some error messages.

* Needed to get "make fluffy-tools" to pass, too.

* Getting "make nimbus_verified_proxy" to build.
2023-03-09 18:40:55 -05:00
Kim De Mey b64135b830
Bump nimbus-eth2 and fix beacon LC related changes (#1473)
* Bump nimbus-eth2 and fix beacon LC related changes

* Disable local_testnet test in CI
Latest version is no longer compatible with the old nightlies that are being downloaded for the test
2023-02-26 16:44:05 +01:00
Kim De Mey 77135e7001
Avoid raw Exception raises by moving confutils config load (#1461) 2023-02-07 17:21:12 +01:00
Kim De Mey 73e93f1f11
Reduce Nim 1.6 compiler warnings & hints
* Reduce Nim 1.6 compiler warnings/hints for Fluffy and Nimbus proxy

Mostly raises Defect removals, TaintedString removal and some
unnecessary imports.

Also updating the copyright years alongside.

* Further reduce Nim 1.6 compiler warnings/hints for Nimbus
2023-01-31 13:38:08 +01:00
Kim De Mey 8523d8b40e
Bump submodules and fix related issues (#1448)
* Updated to the latest nim-eth, nim-rocksdb, nim-web3

* Bump nimbus-eth2 module and fix related issues

Temporarily disabling Portal beacon light client network as it is
a lot of copy pasted code that did not yet take into account
forks. This will require a bigger rework and was not yet tested
in an actual network anyhow.

* More nimbus fixes after module bumps

---------

Co-authored-by: Adam Spitz <adamspitz@status.im>
Co-authored-by: jangko <jangko128@gmail.com>
2023-01-27 15:57:48 +01:00
Kim De Mey a669b51ec5
Bump Nim to 1.6 and resolve the related issues (#1445)
Two unresolved items currently:
- Three tests that are temporarily disabled as they fail in the
macro_assembler code, which seems to be due to an ambigious
identifier Stop (Ops and chronos ServerCommand enum).
- i386 CI disabled as it fails at Nim compilation already. Failed
tests where already ignored for this target.
2023-01-26 13:37:19 +01:00
Kim De Mey 12f66ae598
Move BlockHeaderWithProof content to content key selector 0 (#1379)
* Move BlockHeaderWithProof content to content key selector 0

- Remove as content type with content key selector 4
- Replace regular block header with BlockHeaderWithProof at
content key selector 0

* Apply blockHeader content key also to bridge

* Add tests for header with proof generation and verification
2023-01-20 23:04:58 +01:00