Commit Graph

66 Commits

Author SHA1 Message Date
Kim De Mey 9097548a10
Adjust discv5_updateNodeInfo JSON-RPC to specify key and value (#1569) 2023-05-09 14:34:01 +02:00
Kim De Mey 2ba974a7cd
Adjust portal FindContent JSON-RPC according to latest specs (#1566)
Also remove the FindContentFull unspecced JSON-RPC as the new
FindContent specification replaces this functionality basically.
2023-04-28 09:04:21 +02:00
Kim De Mey acc2d53b52
Further clean-up of Portal and discv5 JSON-RPC (#1565)
- Removal of TODOs fixed in specification instead
- Fix some other TODOs
- General clean-up such as to0xHex usage
2023-04-27 14:07:57 +02:00
Kim De Mey 21ef655b9c
Fix dataRadius in result of Portal ping JSON-RPC (#1553) 2023-04-21 17:00:55 +02:00
Kim De Mey b195f0893b
Minor naming adjustement to a discv5 JSON-RPC call (#1540) 2023-04-13 18:39:46 +02:00
Kim De Mey fb81937708
Add a version to fluffy and add web3_clientVersion JSON-RPC (#1531) 2023-04-05 16:14:42 +02:00
Kim De Mey 4c19a0e48e
Add support for injecting receipts into the Portal Network (#1523)
Allow for the Fluffy Portal bridge to inject receipts. This
requires a web3 endpoint to be provided, and currently only
Alchemy is supported due to the used JSON-RPC endpoint.
2023-03-28 15:50:23 +02:00
Kim De Mey e6964a029b
Remove unused, underdeveloped BridgeClient from Fluffy (#1520)
It is only (mostly) a skeleton, not further developed and not
used. It is unlikely the way forward either when further
developing the Portal state network.
2023-03-23 13:41:40 +01:00
Kim De Mey 249cabd64e
Portal JSON-RPC API fixes (#1511) 2023-03-20 17:25:34 +01:00
Kim De Mey 11fc2de060
Prepare Fluffy for beacon light client bridge (#1506) 2023-03-17 10:19:17 +01:00
Kim De Mey 73e93f1f11
Reduce Nim 1.6 compiler warnings & hints
* Reduce Nim 1.6 compiler warnings/hints for Fluffy and Nimbus proxy

Mostly raises Defect removals, TaintedString removal and some
unnecessary imports.

Also updating the copyright years alongside.

* Further reduce Nim 1.6 compiler warnings/hints for Nimbus
2023-01-31 13:38:08 +01:00
Kim De Mey a669b51ec5
Bump Nim to 1.6 and resolve the related issues (#1445)
Two unresolved items currently:
- Three tests that are temporarily disabled as they fail in the
macro_assembler code, which seems to be due to an ambigious
identifier Stop (Ops and chronos ServerCommand enum).
- i386 CI disabled as it fails at Nim compilation already. Failed
tests where already ignored for this target.
2023-01-26 13:37:19 +01:00
Kim De Mey 52a2257922
Adjust Portal JSON-RPC Offer call and add Gossip call (#1418) 2023-01-17 14:47:10 +01:00
Kim De Mey 90f91e5ebd
Reorganize eth data files parsing code (#1407) 2023-01-05 15:26:58 +01:00
KonradStaniec 2bce04afc7
Add light client mode to fluffy (#1396) 2022-12-27 15:57:10 +01:00
Kim De Mey 591a1e7062
Add content verifier tool check data availability on the network (#1388) 2022-12-16 21:11:09 +01:00
Kim De Mey 733d7e5ceb
Add new portal JSON-RPC OfferReal as Offer is doing gossip now (#1387)
The portal_*Offer call was changed in the specs to actually do
gossip. Make it no longer possible to test purely an offer with
one node. Add OfferReal call for now until spec potentially gets
adjusted.

Also Add some Node information logging for FindContent and Offer
to be able to better debug failures and interoperability.
2022-12-16 17:47:52 +01:00
Kim De Mey 3f74f084c3
Add LocalContent call to Portal JSON-RPC API (#1383) 2022-12-16 11:00:10 +01:00
Kim De Mey 727b2445b7
Fix Portal history/state JSON-RPC API and add missing calls (#1380) 2022-12-16 08:49:18 +01:00
Kim De Mey 095854107c
Fix Portal discv5 JSON-RPC API where it deviates from spec (#1378) 2022-12-15 16:24:23 +01:00
Kim De Mey c962bafd5a
Fix Portal Hive fails by correcting Portal history JSON RPC API (#1365)
* Fix Portal Hive fails by correcting Portal history JSON RPC API

- Field naming in discv5_nodeInfo
- Call naming of portal_historyStore
- Other: some proc to func adjustements
2022-12-13 19:22:36 +01:00
Kim De Mey c5ecba83cd
Refactor and clean-up of history network and eth rpc (#1357)
- Switch from using Option to Opt which allows for smoother
usage with already existing Result types
- With all moved to Opt, make more use of valueOr to avoid
too many if else clause indentation and unstead have a more
clear error path at each step
- Remove dead code, char limits, style guide, etc.
- Replace getEncodedKeyForContent with ContentKey.init
and use ContentKey.init for each type
2022-12-09 17:59:36 +01:00
jangko 94a94c5b65 implement better hardfork management 2022-12-02 13:51:42 +07:00
KonradStaniec d75afd9f8a
Make portal network generic in db store and db get (#1290)
* Make portal network generic in db store and db get
2022-11-08 18:31:45 +01:00
Kim De Mey 36d430aaa2
Add headers with proof content type and use it for verification (#1281)
* Add headers with proof content type and use it for verification

- Add BlockHeaderWithProof content type & content
- Use BlockHeaderWithProof content to verify if chain data is
part of the canonical chain
- Adjust parser & seeder code to be able to seed these headers
with proof
- Adjust eth_data_exporter to be able to export custom header
ranges for which to build proofs (mostly for testing)

There is currently quite some ugliness & clean-up needed for which
a big part is due tos upporting both BlockHeader and
BlockHeaderWithProof on the network.

* Change accumulator proof to array / SSZ vector type

- Change accumulator proof to SSZ vector instead of SSZ list.
- Add and use general buildProof and buildHeaderWithProof func.

* Make the BlockHeaderWithProof an SSZ Union with None option

* Update portal-spec-tests to master commit
2022-11-04 09:27:01 +01:00
kdeme e6d8bb4f2f Optionally write all epoch accumulators when building accumulator
- Can write epoch accumulators to files now with eth_data_exporter
- RPC requests to gossip epoch accumulators now uses these files
instead of building on the fly
- Other build accumulator calls are adjusted and only used for
tests and thus moved to testing folder
2022-10-18 16:23:10 +03:00
KonradStaniec 181243b6c4
Make portal_historyOffer spec compliant (#1259)
* Make portal_historyOffer spec compliant
2022-10-11 12:10:54 +02:00
KonradStaniec 0a2f1378e4
Make fluffy rpc portal call compatible with spec (#1258) 2022-10-10 15:15:57 +02:00
Kim De Mey 4e08f774d5
Fix issue where invalid / not validated would get gossiped (#1247)
Also requires us to split header data propagation from block body
and receipts propagation as the now fixed bug would allow for more
data to be gossiped even when data does not get validated (which
requires the headers).
2022-09-29 08:42:54 +02:00
Kim De Mey 1bd4911e78
Remove ChaindId from BlockKey and update test vectors (#1242) 2022-09-28 09:09:38 +02:00
Kim De Mey 87f28458a5
StyleCheck fixes for Fluffy and lcproxy (#1220)
Also add the compiler options in the nim.cfg of Fluffy and
lcproxy (only visible with direct nimble usage).
2022-09-10 21:00:27 +02:00
Kim De Mey 4a396bdd25
Add eth_chainId rpc to Nimbus and Fluffy (#1219) 2022-09-10 15:05:32 +02:00
Jacek Sieka c2ed731fa5
eth: adapt to smaller eth_types (#1210) 2022-09-03 20:15:35 +02:00
KonradStaniec f0cd340163
Adhere to transport limits in seed methods (#1186)
* Add means to limit offered content to fit talkreq

* Add test for history network limits

* Change seed method api to return num of items offered
2022-08-09 14:32:41 +02:00
KonradStaniec 71f9e37482
Add getting block by number through accumulator (#1182) 2022-08-04 08:34:53 +02:00
Kim De Mey 50af402f59
Add verification of block data by use of accumulator (#1178) 2022-08-01 21:00:21 +02:00
KonradStaniec e3cabaff7f
Add bulk seeding to multiple peers (#1170)
* Add bulk seeding to multiple peers
2022-07-26 13:14:56 +02:00
KonradStaniec aa945f1ed9
Additional seed endpoints (#1164)
* Additional seed endpoints
2022-07-20 12:46:42 +02:00
Kim De Mey facd7342fb
Add network retries for fluffy block requests (#1153)
- Add retries when the network request failed on validation of
block header, body or receipts.
- Shuffle the first set of neighbours that the request is send to
in order to not always hit the same peer first for the same content
- Some general clean-up
2022-07-05 14:42:55 +02:00
KonradStaniec fdb048ed21
Add getLogs implementation (#1143)
* Add eth_getLogs json-rpc endpoint in Fluffy and Nimbus
2022-06-29 17:44:08 +02:00
Kim De Mey f219c69840
Rework of Portal json-rpc debug API and related functionality (#1102)
- More consistent naming in calls used by the JSON-RPC debug API
- Use storeContent everywhere to make sure content is only stored
if in range
- Remove populateHistoryDb subcommand and replace by JSON-RPC call
storeContent
- Remove some whitespace and fix some indentations
2022-05-24 13:27:22 +02:00
Kim De Mey a69b16abff
Cleanup some imports and fix some warnings in the process (#1096) 2022-05-22 22:44:15 +02:00
KonradStaniec d3f231e3da
Add put method which preserves max size (#1065)
* Add put method which preserves max size
2022-05-09 17:18:57 +02:00
Kim De Mey 0fba19b81a
Improvements to the propagation and seeding of data (#1058)
* Improvements to the propagation and seeding of data

- Use a lookup for nodes selection in neighborhoodGossip
- Rework populate db code and add `propagateBlockHistoryDb` call
and portal_history__propagateBlock json-rpc call
- Small adjustment to blockwalk

* Avoid storing out-of-range data in the propagate db calls
2022-04-12 15:49:19 +02:00
Kim De Mey 8c3f210526
Add blockwalk tool to test range of blocks on Portal network (#1020) 2022-03-30 08:55:38 +02:00
Kim De Mey 392a540eb5
Remove unused import of config to avoid select_backend db import (#1018)
* Remove unused import of config to avoid select_backend db import

- Importing nimbus-eth1 config.nim causes import of select_backend
which will default cause an import of kvstore_rocksdb and thus a
require rocksdb. Remove unused one to avoid rocksdb dependency
for Fluffy.
- Remove some whitespace in bridge_client (to make fluffy CI
trigger for sure).

* Use specific cache keys for fluffy CI workflow

* Disable Fluffy CI reproducibility test
2022-03-29 15:51:48 +02:00
Kim De Mey 74127644db
Adjust findNodes proc naming for consistency (#1002) 2022-03-19 08:54:42 +01:00
KonradStaniec b951139af9
Add basic validation of history network (#983)
* Add basic validation of history network

* Save only valid objects to db
2022-03-02 15:29:12 +01:00
Kim De Mey 6f21c232ae
Add block bodies to the propagation and lookups (#975)
* Add block bodies to the propagation and lookups

- Read and propagate block bodies next to the headers
- Add block bodies content (via lookups) to the eth_getBlockByHash
call
- Test the above in test_portal_testnet

* Fix storage/propagation of block bodies

- Data format is an actual block: [header, txs, uncles], which
requires some adjustment to store the block body
- Added also `eth_getBlockTransactionCountByHash` json rpc call
2022-02-22 11:52:44 +01:00
Kim De Mey 0060462dc0
Sharing block header data around in a Portal history network (PoC) (#960)
* Sharing block header data around in a Portal history network (PoC)

- Rework PortalStream to have an instance per PortalProtocol (this
needs to be improved eventually). Each instance uses the same
UtpDiscv5Protocol instance.
- Add processContent on receival of accepted data
- Add dumb neighborhoodGossip: dumb in the sense that it only
offers one piece of content at a time.
- Add to / adjust populate_db to also allow for propagation of
the data and add debug rpc call: portal_history_propagate
- Add eth_rpc_client
- Add eth_getBlockbyHash (no txs or uncles) to eth API
- Add additional test to test_portal_testnet which loads 5 block
headers to 1 node, and offers this data to few nodes, which should
propagate it over the network further. Next query every node for
this data.

* Adjust paths on which Fluffy CI is triggered

* Add documentation on the local testnet
2022-02-11 14:43:10 +01:00