Commit Graph

181 Commits

Author SHA1 Message Date
Jacek Sieka c2ed731fa5
eth: adapt to smaller eth_types (#1210) 2022-09-03 20:15:35 +02:00
Kim De Mey 5355f4e73a
A bit of consistency clean-up in accumulator code (#1201) 2022-08-24 22:12:56 +02:00
Kim De Mey c0a6c9a75c
Remove the usage of AccumulatorDB in the accumulator tests (#1200) 2022-08-24 16:41:04 +02:00
KonradStaniec 5f2117d081
Refactor portal stream (#1191) 2022-08-17 09:32:06 +02:00
KonradStaniec c5101c16e9
Bump nim eth (#1188)
* Bump nim eth

* Bump actions ubuntu
2022-08-16 09:17:57 +02:00
KonradStaniec f0cd340163
Adhere to transport limits in seed methods (#1186)
* Add means to limit offered content to fit talkreq

* Add test for history network limits

* Change seed method api to return num of items offered
2022-08-09 14:32:41 +02:00
KonradStaniec 4f8d10f338
Bump nim-eth (#1185) 2022-08-06 10:11:36 +02:00
KonradStaniec 71f9e37482
Add getting block by number through accumulator (#1182) 2022-08-04 08:34:53 +02:00
Kim De Mey 50af402f59
Add verification of block data by use of accumulator (#1178) 2022-08-01 21:00:21 +02:00
KonradStaniec e3cabaff7f
Add bulk seeding to multiple peers (#1170)
* Add bulk seeding to multiple peers
2022-07-26 13:14:56 +02:00
KonradStaniec aa945f1ed9
Additional seed endpoints (#1164)
* Additional seed endpoints
2022-07-20 12:46:42 +02:00
Kim De Mey 5abf05e455
Add validation for offered block bodies and receipts (#1158)
Also use a AsyncQueue now to handle offered data instead of the
network specific validation callbacks.
2022-07-11 16:29:16 +02:00
Kim De Mey f29cbebb9d
Move block data test files to one location (#1154) 2022-07-05 16:14:52 +02:00
Kim De Mey fb12793482
Bump nim-eth and nim-bearssl and accompanying fixes (#1150) 2022-07-04 09:38:02 +02:00
Kim De Mey 33015ea4a3
Adjust to the newly specified SSZ types for Block Body and Receipts (#1147) 2022-07-01 21:51:51 +02:00
Kim De Mey cad74db423
Support for multiple content items over Portal stream (#1090)
Each content item send gets prefix with varint (max u32) allowing
to send multiple content items as requested by offer/accept flow.
2022-06-24 15:35:31 +02:00
Kim De Mey d4d4e8c28f
Add accumulator buildProof, verifyProof and verifyHeader (#1132) 2022-06-23 21:00:59 +02:00
Kim De Mey de2051dcf8
Add Accumulator build helper calls and refactors (#1127)
- Move the accumulator definitions to a history accumulator file
- Add accumulator build helper calls + temporary database
- Add a header gossip content key encoding test
- Refactor & some cleanup
2022-06-16 08:50:29 +02:00
Kim De Mey 6325712aa6
Add accumulator content keys for history network (#1123) 2022-06-14 23:38:34 +02:00
KonradStaniec b23789d107
Add getting receipts from history network (#1118)
* Add getting receipts from history network
2022-06-10 12:24:53 +02:00
KonradStaniec 730013bde3
Change test data and propagation format (#1115)
* Change test data and propagation format

* Add tool to download data from local node
2022-06-08 15:14:01 +02:00
KonradStaniec 0776f35e0c
Use queries with custom xor function (#1113)
* Use queries with custom xor function
2022-06-03 13:44:42 +02:00
Kim De Mey 6d8b25a5f5
Update header accumulator test to start from Mainnet genesis (#1111) 2022-06-01 15:21:22 +02:00
KonradStaniec af10e8f179
Do not use vacuum when pruning (#1103)
* Do not use vacuum when pruning
2022-05-26 08:26:08 +02:00
Kim De Mey 6e05c7588e
Add Header Accumulator for the Header Gossip Network (#1094) 2022-05-18 22:49:35 +02:00
KonradStaniec 6a9d875fe7
Add pruning metrics (#1093) 2022-05-17 12:47:24 +02:00
Kim De Mey 56c3e31efc
Refactor and clean-up of Portal wire protocol test (#1089) 2022-05-16 13:17:42 +02:00
KonradStaniec 4e8c5f292e
Dynamic radius adjustments (#1079)
* Add config for node radius

* Adjust radius when deleting content
2022-05-12 18:04:37 +02:00
KonradStaniec d3f231e3da
Add put method which preserves max size (#1065)
* Add put method which preserves max size
2022-05-09 17:18:57 +02:00
Kim De Mey ed38ed494f
Add validation handler to Portal wire protocol (#1055)
- Validation handler allows for network specific validation of
the content
- Added also pruning of the allowed connections on PortalStream
2022-04-11 19:42:38 +02:00
KonradStaniec 203d8e8b70
Inital poke implementation (#1037)
* Initial poke implementation
2022-04-06 13:47:23 +02:00
KonradStaniec b3570fae6a
Add api to get n furthest elements from db (#1026)
* Add api to get n furthest elements from db
2022-04-03 15:14:44 +02:00
Kim De Mey 74127644db
Adjust findNodes proc naming for consistency (#1002) 2022-03-19 08:54:42 +01:00
KonradStaniec a330f0a4ef
Add api to get db size to content db (#987)
* Add size proc to content_db

* Add way to reclaim used space
2022-03-08 14:49:41 +01:00
jangko 9a1c8fc779
refactor genesis.nim 2022-03-03 08:28:00 +07:00
KonradStaniec b951139af9
Add basic validation of history network (#983)
* Add basic validation of history network

* Save only valid objects to db
2022-03-02 15:29:12 +01:00
jangko 6374c9f66d
fixes regression caused by #962 2022-02-14 21:22:39 +07:00
Kim De Mey 0060462dc0
Sharing block header data around in a Portal history network (PoC) (#960)
* Sharing block header data around in a Portal history network (PoC)

- Rework PortalStream to have an instance per PortalProtocol (this
needs to be improved eventually). Each instance uses the same
UtpDiscv5Protocol instance.
- Add processContent on receival of accepted data
- Add dumb neighborhoodGossip: dumb in the sense that it only
offers one piece of content at a time.
- Add to / adjust populate_db to also allow for propagation of
the data and add debug rpc call: portal_history_propagate
- Add eth_rpc_client
- Add eth_getBlockbyHash (no txs or uncles) to eth API
- Add additional test to test_portal_testnet which loads 5 block
headers to 1 node, and offers this data to few nodes, which should
propagate it over the network further. Next query every node for
this data.

* Adjust paths on which Fluffy CI is triggered

* Add documentation on the local testnet
2022-02-11 14:43:10 +01:00
Kim De Mey 38036966a6
Improve the tests of the local testnet (#953)
* Improve the tests of the local testnet

The local testnet test was rather flaky and would occasionally
fail. It has been made more robust by adding the ENRs directly
to the routing table instead of doing some random lookups.

Additionally, the amount of nodes were increased (=64), ip limits
configuration was added, and the bits-per-hop value was set to 1
in order to make the lookups more likely to hit the network
instead of only the local routing table.

Failure is obviously still possible to happen when sufficient
packets get lost. If this turns out to be the case with the current
amount of nodes, we might have to revise the testing strategy here.

* Disable lookup test for State network

Disable lookup test for State network due to issue with custom
distance function causing the lookup to not always converging
towards the target.
2022-02-02 22:48:33 +01:00
Kim De Mey f19a64a2fe
Add uTP over discv5 in portal wire protocol (#922)
* Add uTP over discv5 in portal wire protocol
2022-01-14 16:07:14 +01:00
Kim De Mey 79647e5580
Allow access to contentDB from portal wire protocol (#920)
- Allow access to contentDB from portal wire protocol
- Use this to do the db.get in `handleFindContent` directly
- Use this to check the `contentKeys` list in `handleOffer`
2022-01-06 09:06:05 +01:00
Kim De Mey a8640fe57c
History content keys (#919)
* Change History content key to us SSZ Union and adjust tests

* Change slot to byteBE instead of LE

This is currently not specified in the Portal network
specifications, but we are using already BE for the actual content
key, so change it also here to remain consistent.
2022-01-05 09:49:49 +01:00
Kim De Mey 796787e0e5
Add state and history content key tests (#916) 2021-12-20 09:53:00 +01:00
Kim De Mey 092d946c72
Add state ContentKey test and adjust history ContentKey test (#914) 2021-12-15 08:48:56 +01:00
Kim De Mey 70625bc02c
Use one bootstrap nodes argument for discv5 and Portal (#911)
Currently bootstrap nodes for discv5 and for the Portal nodes
were provided through separate cli arguments. This is however
confusing and cumbersome as typically when (currently) testing
a node will have both discv5 and the Portal networks enabled.
We merge them into one argument.

If a node happens not to support a Portal network, it will be
removed after message request failure.

Commit also contains additional clean-up and nim-eth bump.
2021-12-13 09:06:29 +01:00
Kim De Mey 10a3946194
Bit of cleanup (#903)
* Rename FindNode to FindNodes as per spec

* Use consistently lower case starting camelCase for consts

Style guide nep1 allows for both CamelCase and camelCase for
consts, but we seem to use more often camelCase. Using this now
consistently.

* Change some procs to func
2021-12-08 11:54:22 +01:00
Kim De Mey ed04aad203
Fix fluffy discovery rpc tests after nim-json-rpc bump (#897) 2021-11-30 16:04:24 +01:00
Kim De Mey 903350bdde
Add local testnet script and required json-rpc calls (#891)
- Add basic discv5 and portal json-rpc calls and activate them in
fluffy
- Renames in the rpc folder
- Add local testnet script and run this script in CI
- bump nim-eth
2021-11-24 08:45:55 +01:00
Kim De Mey 78f585399f
Bump submodules (#887)
* Bump submodules

* Fix missing import for state and history networks
2021-11-18 12:06:53 +01:00
Kim De Mey bdbd8e4c87
Add SSZ Unions through case objects (#882)
* Add SSZ Unions through case objects

* Add connection id content response test and improve other test vectors

* Implement content keys and ids for state network as per spec

Content keys case object is used so that it can be serialized and
deserialized as an SSZ Union.

* Let message Union in Portal wire protocol start at 0 as per new spec
2021-11-17 17:11:17 +01:00
Kim De Mey 0ba3c682e4
Use SSZ code from nim-ssz-serialization module (#875) 2021-10-23 14:28:12 +02:00
Kim De Mey 02e8b2eb12
Clean up old unused code and unused imports (#877) 2021-10-22 21:36:53 +02:00
Kim De Mey 92dfb02ad7
Use ByteList.init instead of List 2048 (#876) 2021-10-22 17:49:23 +02:00
Kim De Mey 4b85c73f43
Update portal wire protocol ids to latest spec (#873) 2021-10-20 22:31:05 +02:00
Kim De Mey b77034c870
Update portal wire protocol according to latest spec changes (#860)
This is not complete, in order to be according to specification
two SSZ Union types need to be used.
2021-10-13 21:35:54 +02:00
Kim De Mey 447deb29ff
Add HistoryNetwork that uses Portal wire protocol (#856)
* Add HistoryNetwork that uses Portal wire protocol

* Further implement getContent and adjust naming of distance calc calls
2021-10-09 13:22:03 +02:00
KonradStaniec c62905db25
Re enable using custom distance function (#854) 2021-10-05 21:16:33 +02:00
Kim De Mey 51626c5831
Add a basic ContentDB for portal networks (#848)
* Add a basic ContentDB for Portal networks

* Use ContentDB in StateNetwork

* Avoid probably some form of sandwich problem by re-exporting kvstore_sqlite3 from content_db
2021-09-28 19:58:41 +02:00
Kim De Mey 16a36453b5
Rework ContentId, ContentKey, GetContent to be more clear (#845)
* Small clarification in naming and note where GetContent is actually needed

* Rework contentId and contentKey
2021-09-24 11:22:07 +02:00
Kim De Mey 785a3b47b0
Allow for passing Portal specific bootstrap nodes (#844)
* Allow for passing Portal specific bootstrap nodes

* Fix to also replaceNode when decodeMessage fails

* Add portal bootstrap node tests and reorder test cases
2021-09-23 14:26:41 +02:00
Kim De Mey 6f6345a022
Move Portal wire protocol out of state specific code (#843)
* Allow for custom protocol id in portal wire protocol

* Move Portal wire protocol out of state specific code
2021-09-22 17:07:14 +02:00
Kim De Mey 11c120e400
Update wire protocol of state network spec (#829)
* Update wire protocol of state network spec

* BytesN should be SSZ Vector not List

* Add specific lists and bitlists for content keys
2021-09-22 11:28:04 +02:00
Kim De Mey 0135189e4c
Oops, isMainModule will not run of course for all_tests (#841) 2021-09-21 18:38:08 +02:00
Kim De Mey 1afdfdf2ff
Disable mixed usage of cliBuilder to fix incorrect test failure reporting (#836) 2021-09-21 14:35:41 +02:00
jangko a3badea928
config: fix new config based on input from jamie and zahary 2021-09-18 17:34:51 +07:00
KonradStaniec 6192cd7dc1
Use custom distance function in state network (#831)
* Use custom distance calculator in state network
2021-09-16 16:13:36 +02:00
KonradStaniec 58b11e683d
Implement custom distance function (#827)
* Implement custom distance function

* More docs about function equivalence
2021-09-14 13:36:55 +02:00
jangko bf87c6e83c
tests: fixes fluffy test due to recent changes
- CustomGenesis -> CustomNetwork
- loadCustomGenesis -> loadCustomNetwork
- BaseChainDB constructor changes
2021-09-09 08:09:35 +07:00
KonradStaniec 2c620c007d
Fix bugs in lookup and add test case (#820) 2021-09-07 14:22:25 +02:00
KonradStaniec a083fb30fc
Generalize network layer for portal (#814)
* Generalize netork layer for portal

* Make messages free from any content references

* Use portal network in main fluffy module

* Fix cli

* Use lookup in portal network

* Avoid using result
2021-09-03 10:57:19 +02:00
KonradStaniec df3e7bb368
Add content lookup function (#807)
* Add content lookup function

* Pr comments

Refactor contentLookup function
Add additional logginf for protocol breaches
in find content responses
2021-09-02 14:35:25 +02:00
KonradStaniec f0637cfbfc
Add rpc for local node info (#810)
* Add rpc for local node info

* Group let statement
2021-08-27 18:04:55 +02:00
KonradStaniec cff5c5afc4
Update just seen on message replies (#805) 2021-08-20 16:02:29 +02:00
KonradStaniec f4d9421836
Fix format of content message (#803) 2021-08-18 09:23:57 +02:00
KonradStaniec edd642c048
Move state network to separate folder (#800) 2021-08-17 10:58:40 +02:00
Kim De Mey e2b50d2339
Add routing table and required lookup code to Portal protocol (#773)
* Add routing table and required lookup code to Portal protocol

* Enable Portal lookup and validate loops + minor adjustments
2021-07-30 21:19:03 +02:00
Kim De Mey 66532e8ad5
Use ssz vectors for the hashes and add enrs in the tests (#763) 2021-07-20 14:04:57 +02:00
Kim De Mey e37bafd47e
Add a first, simple, content network test (#760)
* Add a first, simple, content network test

* Use sqlite as nimbus_db_backend for fluffy tests

* Fix backend selection for sqlite
2021-07-15 15:12:33 +02:00
Kim De Mey 25522b2c24
Further implement Portal Nodes and FoundContent messages (#757) 2021-07-13 15:15:33 +02:00
Kim De Mey 424fd405e9
Move Portal wire protocol from nim-eth to fluffy (#749)
* Move Portal wire protocol from nim-eth to fluffy

* Rename fluffy make targets

* Move all_fluffy_tests a folder down
2021-07-09 13:34:16 +02:00
KonradStaniec a59a2c61b6
[FEATURE] Add support for handling experimental api call (#746)
* [FEATURE] Add support for handling experimental api call
2021-07-07 11:04:18 +02:00