Commit Graph

50 Commits

Author SHA1 Message Date
Daniel Sobol 85134eb24b
add hidden --disable-poke option into fluffy [#1640] (#1711) 2023-08-30 10:01:00 +02:00
Kim De Mey ff90f4fd22
Improve logging and logging options in Fluffy (#1548)
* Improve logging and logging options in Fluffy

- Allow selection of log format, including:
  - JSON
  - automatic selection based on tty
- Allow log levels per topic configured on cli
2023-04-19 17:01:01 +02:00
Kim De Mey fb81937708
Add a version to fluffy and add web3_clientVersion JSON-RPC (#1531) 2023-04-05 16:14:42 +02:00
Kim De Mey e6964a029b
Remove unused, underdeveloped BridgeClient from Fluffy (#1520)
It is only (mostly) a skeleton, not further developed and not
used. It is unlikely the way forward either when further
developing the Portal state network.
2023-03-23 13:41:40 +01:00
Kim De Mey 11fc2de060
Prepare Fluffy for beacon light client bridge (#1506) 2023-03-17 10:19:17 +01:00
Kim De Mey 8989141446
Renames on Portal beacon LC code (#1496) 2023-03-13 21:30:57 +01:00
Kim De Mey 06512409ea
Fix, rework and re-enable Portal beacon light client (#1481) 2023-02-26 19:18:03 +01:00
Kim De Mey 73e93f1f11
Reduce Nim 1.6 compiler warnings & hints
* Reduce Nim 1.6 compiler warnings/hints for Fluffy and Nimbus proxy

Mostly raises Defect removals, TaintedString removal and some
unnecessary imports.

Also updating the copyright years alongside.

* Further reduce Nim 1.6 compiler warnings/hints for Nimbus
2023-01-31 13:38:08 +01:00
Kim De Mey 8523d8b40e
Bump submodules and fix related issues (#1448)
* Updated to the latest nim-eth, nim-rocksdb, nim-web3

* Bump nimbus-eth2 module and fix related issues

Temporarily disabling Portal beacon light client network as it is
a lot of copy pasted code that did not yet take into account
forks. This will require a bigger rework and was not yet tested
in an actual network anyhow.

* More nimbus fixes after module bumps

---------

Co-authored-by: Adam Spitz <adamspitz@status.im>
Co-authored-by: jangko <jangko128@gmail.com>
2023-01-27 15:57:48 +01:00
Kim De Mey 90f91e5ebd
Reorganize eth data files parsing code (#1407) 2023-01-05 15:26:58 +01:00
KonradStaniec 2bce04afc7
Add light client mode to fluffy (#1396) 2022-12-27 15:57:10 +01:00
Kim De Mey 9430619272
Remove Accumulator as network content type and add it to binary (#1267)
Portal master accumulator was removed from the network specs as a
content type shared on the network, as since the merge this is
a finite accumulator (pre-merge only).
So in this PR the accumulator gets removed as network type and
gets instead baked into the library. Building it is done by
seperate tooling (eth_data_exporter).
Because of this a lot of extra code can be removed that was
located in history_network, content_db, portal_protocol, etc.

Also removed to option to build the accumulator at start-up
of fluffy as this takes several minutes making it not viable.
It can still be loaded from a provided file however.

The ssz accumulator file is for now stored in the recently
created portal-spec-tests repository.
2022-10-17 20:38:51 +02:00
Kim De Mey 011f44abea
Split parsing and seeding functionality for history data (#1218) 2022-09-09 21:21:48 +02:00
KonradStaniec 5f2117d081
Refactor portal stream (#1191) 2022-08-17 09:32:06 +02:00
KonradStaniec c5101c16e9
Bump nim eth (#1188)
* Bump nim eth

* Bump actions ubuntu
2022-08-16 09:17:57 +02:00
Kim De Mey 50af402f59
Add verification of block data by use of accumulator (#1178) 2022-08-01 21:00:21 +02:00
Kim De Mey f219c69840
Rework of Portal json-rpc debug API and related functionality (#1102)
- More consistent naming in calls used by the JSON-RPC debug API
- Use storeContent everywhere to make sure content is only stored
if in range
- Remove populateHistoryDb subcommand and replace by JSON-RPC call
storeContent
- Remove some whitespace and fix some indentations
2022-05-24 13:27:22 +02:00
KonradStaniec 4e8c5f292e
Dynamic radius adjustments (#1079)
* Add config for node radius

* Adjust radius when deleting content
2022-05-12 18:04:37 +02:00
KonradStaniec d3f231e3da
Add put method which preserves max size (#1065)
* Add put method which preserves max size
2022-05-09 17:18:57 +02:00
Kim De Mey 7bd9afa465
Bump nim-eth for max discv5 packet size (#1070) 2022-05-03 09:18:33 +02:00
KonradStaniec 0365d030fb
Increase packet size of utp over discv5 (#1042)
* Increase packet size of utp over discv5
2022-04-07 13:54:49 +02:00
Kim De Mey 08aa20a791
Fix error when dataDir does not exist (#1000)
- When dataDir does not exist, attempt to create the dir
- Some additional logging improvements
- Whitespace clean-up
2022-03-18 13:06:57 +01:00
Kim De Mey ae4aef6065
Add bootstrap nodes at compile time for testnet0 (#998)
- Add bootstrap nodes from a bootstrap_nodes.txt file at compile
time
- Add --network flag to select network specific bootstrap nodes
2022-03-17 18:39:24 +01:00
Kim De Mey d1127d77b1
Add cli option to start fluffy with a netkey file (#974)
Default the network key will be taken from a network key file
instead of randomly generated on each run. This is done because
the data that gets stored in the content database is dependant on
the network key used, as the node id is derived from this.
2022-02-17 14:13:39 +01:00
Kim De Mey f29e307fd0
Add radius cli option to Fluffy and use it in local testnet script (#968) 2022-02-15 13:11:27 +01:00
Kim De Mey 0060462dc0
Sharing block header data around in a Portal history network (PoC) (#960)
* Sharing block header data around in a Portal history network (PoC)

- Rework PortalStream to have an instance per PortalProtocol (this
needs to be improved eventually). Each instance uses the same
UtpDiscv5Protocol instance.
- Add processContent on receival of accepted data
- Add dumb neighborhoodGossip: dumb in the sense that it only
offers one piece of content at a time.
- Add to / adjust populate_db to also allow for propagation of
the data and add debug rpc call: portal_history_propagate
- Add eth_rpc_client
- Add eth_getBlockbyHash (no txs or uncles) to eth API
- Add additional test to test_portal_testnet which loads 5 block
headers to 1 node, and offers this data to few nodes, which should
propagate it over the network further. Next query every node for
this data.

* Adjust paths on which Fluffy CI is triggered

* Add documentation on the local testnet
2022-02-11 14:43:10 +01:00
Kim De Mey 38036966a6
Improve the tests of the local testnet (#953)
* Improve the tests of the local testnet

The local testnet test was rather flaky and would occasionally
fail. It has been made more robust by adding the ENRs directly
to the routing table instead of doing some random lookups.

Additionally, the amount of nodes were increased (=64), ip limits
configuration was added, and the bits-per-hop value was set to 1
in order to make the lookups more likely to hit the network
instead of only the local routing table.

Failure is obviously still possible to happen when sufficient
packets get lost. If this turns out to be the case with the current
amount of nodes, we might have to revise the testing strategy here.

* Disable lookup test for State network

Disable lookup test for State network due to issue with custom
distance function causing the lookup to not always converging
towards the target.
2022-02-02 22:48:33 +01:00
Kim De Mey 14dd763900
Add populate history db option to fluffy (#929) 2022-01-18 15:08:02 +01:00
Kim De Mey 81ebfd2b2a
Add Portal protocol config and add config options to fluffy cli (#928) 2022-01-18 09:01:22 +01:00
Kim De Mey f19a64a2fe
Add uTP over discv5 in portal wire protocol (#922)
* Add uTP over discv5 in portal wire protocol
2022-01-14 16:07:14 +01:00
Kim De Mey f4ca598c03
Some version of the findContent JSON-RPC proc (#918)
Also includes fixing instantiation of the history network json-rpc
API handles.
2021-12-20 11:57:55 +01:00
Kim De Mey f3d0e97997
Add Portal protocol testing docs and more (#912)
- Add portal network ping and findNodes JSON-RPC endpoints
- clean-up of some cli arguments
- Add protocol_interop.md document and adjust/fix other
documentation
2021-12-13 14:12:51 +01:00
Kim De Mey 70625bc02c
Use one bootstrap nodes argument for discv5 and Portal (#911)
Currently bootstrap nodes for discv5 and for the Portal nodes
were provided through separate cli arguments. This is however
confusing and cumbersome as typically when (currently) testing
a node will have both discv5 and the Portal networks enabled.
We merge them into one argument.

If a node happens not to support a Portal network, it will be
removed after message request failure.

Commit also contains additional clean-up and nim-eth bump.
2021-12-13 09:06:29 +01:00
Kim De Mey 5f4ea7822c
Have one generic installPortalApiHandlers to be used for different Portal networks (#893) 2021-11-24 14:53:01 +01:00
Kim De Mey 77cc9b6215
Add support for passing bootstrap file and write local enr file (#892)
Also use this now in the local testnet script instead of hardcoded
values.
2021-11-24 12:12:25 +01:00
Kim De Mey 903350bdde
Add local testnet script and required json-rpc calls (#891)
- Add basic discv5 and portal json-rpc calls and activate them in
fluffy
- Renames in the rpc folder
- Add local testnet script and run this script in CI
- bump nim-eth
2021-11-24 08:45:55 +01:00
Kim De Mey 447deb29ff
Add HistoryNetwork that uses Portal wire protocol (#856)
* Add HistoryNetwork that uses Portal wire protocol

* Further implement getContent and adjust naming of distance calc calls
2021-10-09 13:22:03 +02:00
Kim De Mey 51626c5831
Add a basic ContentDB for portal networks (#848)
* Add a basic ContentDB for Portal networks

* Use ContentDB in StateNetwork

* Avoid probably some form of sandwich problem by re-exporting kvstore_sqlite3 from content_db
2021-09-28 19:58:41 +02:00
Kim De Mey 785a3b47b0
Allow for passing Portal specific bootstrap nodes (#844)
* Allow for passing Portal specific bootstrap nodes

* Fix to also replaceNode when decodeMessage fails

* Add portal bootstrap node tests and reorder test cases
2021-09-23 14:26:41 +02:00
Kim De Mey 6f6345a022
Move Portal wire protocol out of state specific code (#843)
* Allow for custom protocol id in portal wire protocol

* Move Portal wire protocol out of state specific code
2021-09-22 17:07:14 +02:00
KonradStaniec a083fb30fc
Generalize network layer for portal (#814)
* Generalize netork layer for portal

* Make messages free from any content references

* Use portal network in main fluffy module

* Fix cli

* Use lookup in portal network

* Avoid using result
2021-09-03 10:57:19 +02:00
KonradStaniec f0637cfbfc
Add rpc for local node info (#810)
* Add rpc for local node info

* Group let statement
2021-08-27 18:04:55 +02:00
KonradStaniec edd642c048
Move state network to separate folder (#800) 2021-08-17 10:58:40 +02:00
KonradStaniec 4b1fa050e9
Use websockets in proxy (#779) 2021-08-05 08:14:25 +02:00
Kim De Mey e2b50d2339
Add routing table and required lookup code to Portal protocol (#773)
* Add routing table and required lookup code to Portal protocol

* Enable Portal lookup and validate loops + minor adjustments
2021-07-30 21:19:03 +02:00
KonradStaniec 0fb0fc4680
[Minor] use proper constructor for proxy in fluffy (#759) 2021-07-13 15:49:53 +02:00
Kim De Mey 424fd405e9
Move Portal wire protocol from nim-eth to fluffy (#749)
* Move Portal wire protocol from nim-eth to fluffy

* Rename fluffy make targets

* Move all_fluffy_tests a folder down
2021-07-09 13:34:16 +02:00
KonradStaniec 1c5c2cba8b
[FEATURE] Use rpc proxy (#747) 2021-07-07 14:13:27 +02:00
KonradStaniec 65ea8a12a9
[FEATURE] Stub for bridge client (#739)
* [FEATURE] Stub for bridge client

* [FEATURE] Add client to fluffy runner

* [FEATURE] Fix params in logs

* [FEATURE] Rename config option
2021-07-02 07:30:48 +02:00
Kim De Mey f8b3922eb5
Change name from nlpn to fluffy (#734) 2021-06-28 17:53:13 +02:00