Commit Graph

30 Commits

Author SHA1 Message Date
Kim De Mey d996e60347
Rename to EpochRecord and other accumulator spec changes (#2473)
- EpochAccumulator got renamed to EpochRecord
- MasterAccumulator is not HistoricalHashesAccumulator
- The List size for the accumulator got a different maximum which
also result in a different encoding and HTR
2024-07-11 17:42:45 +02:00
Kim De Mey 4a20756e6b
Remove unused seed_db and related code (#2471) 2024-07-10 23:02:15 +02:00
andri lim 5a18537450
Bump nim-eth, nim-web3, nimbus-eth2 (#2344)
* Bump nim-eth, nim-web3, nimbus-eth2

- Replace std.Option with results.Opt
- Fields name changes

* More fixes

* Fix Portal stream async raises and portal testnet Opt usage

* Bump eth + nimbus-eth2 + more fixes related to eth_types changes

* Fix in utp test app and nimbus-eth2 bump

* Fix test_blockchain_json rebase conflict

* Fix EVMC block_timestamp conversion plus commentary

---------

Co-authored-by: kdeme <kim.demey@gmail.com>
2024-06-14 14:31:08 +07:00
Kim De Mey 9c64b44c89
Add post Bellatrix beacon block proof exports (#2158)
- Add post Bellatrix Beacon block proof exporting to
eth_data_exporter
- Rework the Bellatrix Beacon block proof exports to us yaml
format
- Move some common code of the two different beacon block proofs
to beacon_chain_block_proof_common module
- Move + add to yaml utils to be usable not just for the tests but
also for eth data exporter
2024-05-03 11:50:18 +02:00
Kim De Mey 5c1cf0410c
Add exportBlockProofBellatrix command to eth_data_exporter (#2093)
Add functionality to  export bellatrix block proofs from era files
to the eth_data_exporter tool.
2024-03-21 13:25:32 +01:00
Kim De Mey 49a199dbc3
Add beacon state historical_roots to fluffy (#2085)
- Add historical_roots in the binary for proof verification
pre-capella
- Add command to export historical_roots to file in
eth_data_exporter tool
2024-03-19 16:45:32 +01:00
Kim De Mey 2d76b8c010
Formatting fluffy with nph v0.5.1 + CI check (#2020)
* Add nph check to fluffy CI lint

* Add a section on nph usage in the fluffy.guide

* Update copyright years for altered files

* Avoid chained methods formatting style in db code

* Update nph in CI to v0.5

* Remove leftover commented import

* Move comment to avoid nph turning complex list into simple list (nph bug)

* Update nph in CI to v0.5.1

* Formatting fluffy with nph v0.5.1
2024-02-28 18:31:45 +01:00
Kim De Mey 2fb90cfb43
Initial implementation of era1 + export + verify (#1998)
* Initial implementation of era1 + export + verify

Initial implementation of era1 + export and verify command in the
existing eth_data_exporter.

* Quick verify version to test with geth created era1 files

* Add Era1File object and getter for specific block tuple

* Add getAccumulatorRoot and update AccumulatorRoot type name

+ clean-up of some comments.

* Implement BlockIndex based verify + additional helper calls

+ fix bug for reading last (incomplete) Era1 file

* Fix init BlockIndex offsets for last era + extra check on reading

* Remove duplicated e2store code
2024-02-09 11:13:12 +01:00
Kim De Mey aad4adde07
Move all Fluffy database code to database folder (#1918) 2023-12-01 17:20:52 +01:00
Kim De Mey cb4626b488
Initial implementation of Portal fluffy beacon chain bridge (#1751)
* Initial implementation of Portal fluffy beacon chain bridge

* Run windows ci for fluffy with -j1 to avoid OOM
2023-09-15 16:21:00 +02:00
Kim De Mey 89b3e679e2
Add new portal beacon chain lc content encoding tests (#1636) 2023-07-08 17:01:33 +02:00
Kim De Mey ab13e43db8
Add export of beacon LC data to eth_data_exporter (#1629)
* Add export of beacon LC data to eth_data_exporter

* Update content seeding docs with new eth_data_exporter cli
2023-07-05 20:17:03 +02:00
Kim De Mey ff90f4fd22
Improve logging and logging options in Fluffy (#1548)
* Improve logging and logging options in Fluffy

- Allow selection of log format, including:
  - JSON
  - automatic selection based on tty
- Allow log levels per topic configured on cli
2023-04-19 17:01:01 +02:00
Kim De Mey 034d0e6305
Add exporting of headersWithProof and configurable web3 url (#1539) 2023-04-11 14:06:45 +02:00
Kim De Mey 73e93f1f11
Reduce Nim 1.6 compiler warnings & hints
* Reduce Nim 1.6 compiler warnings/hints for Fluffy and Nimbus proxy

Mostly raises Defect removals, TaintedString removal and some
unnecessary imports.

Also updating the copyright years alongside.

* Further reduce Nim 1.6 compiler warnings/hints for Nimbus
2023-01-31 13:38:08 +01:00
Kim De Mey a669b51ec5
Bump Nim to 1.6 and resolve the related issues (#1445)
Two unresolved items currently:
- Three tests that are temporarily disabled as they fail in the
macro_assembler code, which seems to be due to an ambigious
identifier Stop (Ops and chronos ServerCommand enum).
- i386 CI disabled as it fails at Nim compilation already. Failed
tests where already ignored for this target.
2023-01-26 13:37:19 +01:00
Kim De Mey 90f91e5ebd
Reorganize eth data files parsing code (#1407) 2023-01-05 15:26:58 +01:00
Kim De Mey 36d430aaa2
Add headers with proof content type and use it for verification (#1281)
* Add headers with proof content type and use it for verification

- Add BlockHeaderWithProof content type & content
- Use BlockHeaderWithProof content to verify if chain data is
part of the canonical chain
- Adjust parser & seeder code to be able to seed these headers
with proof
- Adjust eth_data_exporter to be able to export custom header
ranges for which to build proofs (mostly for testing)

There is currently quite some ugliness & clean-up needed for which
a big part is due tos upporting both BlockHeader and
BlockHeaderWithProof on the network.

* Change accumulator proof to array / SSZ vector type

- Change accumulator proof to SSZ vector instead of SSZ list.
- Add and use general buildProof and buildHeaderWithProof func.

* Make the BlockHeaderWithProof an SSZ Union with None option

* Update portal-spec-tests to master commit
2022-11-04 09:27:01 +01:00
kdeme e6d8bb4f2f Optionally write all epoch accumulators when building accumulator
- Can write epoch accumulators to files now with eth_data_exporter
- RPC requests to gossip epoch accumulators now uses these files
instead of building on the fly
- Other build accumulator calls are adjusted and only used for
tests and thus moved to testing folder
2022-10-18 16:23:10 +03:00
Kim De Mey 09766ef283
Remove accumulator_db and minor clean-up in eth_data_exporter (#1268)
For accumulator building we now use intermediary header epoch
files which renders the use of a temporary db for this no longer
needed. Code was already no longer in use.
2022-10-18 10:53:26 +02:00
Kim De Mey 9430619272
Remove Accumulator as network content type and add it to binary (#1267)
Portal master accumulator was removed from the network specs as a
content type shared on the network, as since the merge this is
a finite accumulator (pre-merge only).
So in this PR the accumulator gets removed as network type and
gets instead baked into the library. Building it is done by
seperate tooling (eth_data_exporter).
Because of this a lot of extra code can be removed that was
located in history_network, content_db, portal_protocol, etc.

Also removed to option to build the accumulator at start-up
of fluffy as this takes several minutes making it not viable.
It can still be loaded from a provided file however.

The ssz accumulator file is for now stored in the recently
created portal-spec-tests repository.
2022-10-17 20:38:51 +02:00
Kim De Mey d667346834
Adjustments in eth_data_exporter related to accumulator generation (#1261)
- Add exportHeaders to write headers in e2s file per epoch
- Add verifyHeaders to verify decoding the above files
- Adjust exportAccumulatorData to use generated header epoch files
and write accumulator in SSZ (no hex)
- Add printAccumulatorData to print all root hashes
2022-10-14 08:57:17 +02:00
Kim De Mey 1bd4911e78
Remove ChaindId from BlockKey and update test vectors (#1242) 2022-09-28 09:09:38 +02:00
Kim De Mey 87f28458a5
StyleCheck fixes for Fluffy and lcproxy (#1220)
Also add the compiler options in the nim.cfg of Fluffy and
lcproxy (only visible with direct nimble usage).
2022-09-10 21:00:27 +02:00
KonradStaniec c5101c16e9
Bump nim eth (#1188)
* Bump nim eth

* Bump actions ubuntu
2022-08-16 09:17:57 +02:00
Kim De Mey 50af402f59
Add verification of block data by use of accumulator (#1178) 2022-08-01 21:00:21 +02:00
Kim De Mey 33015ea4a3
Adjust to the newly specified SSZ types for Block Body and Receipts (#1147) 2022-07-01 21:51:51 +02:00
KonradStaniec 196428dcf5
Use one ws connection to transfer data from source (#1136) 2022-06-22 08:50:58 +02:00
KonradStaniec 489b975189
Add seed db file format (#1126)
* Add seed db file format
2022-06-20 16:52:48 +02:00
KonradStaniec 730013bde3
Change test data and propagation format (#1115)
* Change test data and propagation format

* Add tool to download data from local node
2022-06-08 15:14:01 +02:00