124 Commits

Author SHA1 Message Date
KonradStaniec
71f9e37482
Add getting block by number through accumulator (#1182) 2022-08-04 08:34:53 +02:00
Kim De Mey
50af402f59
Add verification of block data by use of accumulator (#1178) 2022-08-01 21:00:21 +02:00
KonradStaniec
e3cabaff7f
Add bulk seeding to multiple peers (#1170)
* Add bulk seeding to multiple peers
2022-07-26 13:14:56 +02:00
KonradStaniec
aa945f1ed9
Additional seed endpoints (#1164)
* Additional seed endpoints
2022-07-20 12:46:42 +02:00
Kim De Mey
5abf05e455
Add validation for offered block bodies and receipts (#1158)
Also use a AsyncQueue now to handle offered data instead of the
network specific validation callbacks.
2022-07-11 16:29:16 +02:00
Kim De Mey
f29cbebb9d
Move block data test files to one location (#1154) 2022-07-05 16:14:52 +02:00
Kim De Mey
fb12793482
Bump nim-eth and nim-bearssl and accompanying fixes (#1150) 2022-07-04 09:38:02 +02:00
Kim De Mey
33015ea4a3
Adjust to the newly specified SSZ types for Block Body and Receipts (#1147) 2022-07-01 21:51:51 +02:00
Kim De Mey
cad74db423
Support for multiple content items over Portal stream (#1090)
Each content item send gets prefix with varint (max u32) allowing
to send multiple content items as requested by offer/accept flow.
2022-06-24 15:35:31 +02:00
Kim De Mey
d4d4e8c28f
Add accumulator buildProof, verifyProof and verifyHeader (#1132) 2022-06-23 21:00:59 +02:00
Kim De Mey
de2051dcf8
Add Accumulator build helper calls and refactors (#1127)
- Move the accumulator definitions to a history accumulator file
- Add accumulator build helper calls + temporary database
- Add a header gossip content key encoding test
- Refactor & some cleanup
2022-06-16 08:50:29 +02:00
Kim De Mey
6325712aa6
Add accumulator content keys for history network (#1123) 2022-06-14 23:38:34 +02:00
KonradStaniec
b23789d107
Add getting receipts from history network (#1118)
* Add getting receipts from history network
2022-06-10 12:24:53 +02:00
KonradStaniec
730013bde3
Change test data and propagation format (#1115)
* Change test data and propagation format

* Add tool to download data from local node
2022-06-08 15:14:01 +02:00
KonradStaniec
0776f35e0c
Use queries with custom xor function (#1113)
* Use queries with custom xor function
2022-06-03 13:44:42 +02:00
Kim De Mey
6d8b25a5f5
Update header accumulator test to start from Mainnet genesis (#1111) 2022-06-01 15:21:22 +02:00
KonradStaniec
af10e8f179
Do not use vacuum when pruning (#1103)
* Do not use vacuum when pruning
2022-05-26 08:26:08 +02:00
Kim De Mey
6e05c7588e
Add Header Accumulator for the Header Gossip Network (#1094) 2022-05-18 22:49:35 +02:00
KonradStaniec
6a9d875fe7
Add pruning metrics (#1093) 2022-05-17 12:47:24 +02:00
Kim De Mey
56c3e31efc
Refactor and clean-up of Portal wire protocol test (#1089) 2022-05-16 13:17:42 +02:00
KonradStaniec
4e8c5f292e
Dynamic radius adjustments (#1079)
* Add config for node radius

* Adjust radius when deleting content
2022-05-12 18:04:37 +02:00
KonradStaniec
d3f231e3da
Add put method which preserves max size (#1065)
* Add put method which preserves max size
2022-05-09 17:18:57 +02:00
Kim De Mey
ed38ed494f
Add validation handler to Portal wire protocol (#1055)
- Validation handler allows for network specific validation of
the content
- Added also pruning of the allowed connections on PortalStream
2022-04-11 19:42:38 +02:00
KonradStaniec
203d8e8b70
Inital poke implementation (#1037)
* Initial poke implementation
2022-04-06 13:47:23 +02:00
KonradStaniec
b3570fae6a
Add api to get n furthest elements from db (#1026)
* Add api to get n furthest elements from db
2022-04-03 15:14:44 +02:00
Kim De Mey
74127644db
Adjust findNodes proc naming for consistency (#1002) 2022-03-19 08:54:42 +01:00
KonradStaniec
a330f0a4ef
Add api to get db size to content db (#987)
* Add size proc to content_db

* Add way to reclaim used space
2022-03-08 14:49:41 +01:00
jangko
9a1c8fc779
refactor genesis.nim 2022-03-03 08:28:00 +07:00
KonradStaniec
b951139af9
Add basic validation of history network (#983)
* Add basic validation of history network

* Save only valid objects to db
2022-03-02 15:29:12 +01:00
jangko
6374c9f66d
fixes regression caused by #962 2022-02-14 21:22:39 +07:00
Kim De Mey
0060462dc0
Sharing block header data around in a Portal history network (PoC) (#960)
* Sharing block header data around in a Portal history network (PoC)

- Rework PortalStream to have an instance per PortalProtocol (this
needs to be improved eventually). Each instance uses the same
UtpDiscv5Protocol instance.
- Add processContent on receival of accepted data
- Add dumb neighborhoodGossip: dumb in the sense that it only
offers one piece of content at a time.
- Add to / adjust populate_db to also allow for propagation of
the data and add debug rpc call: portal_history_propagate
- Add eth_rpc_client
- Add eth_getBlockbyHash (no txs or uncles) to eth API
- Add additional test to test_portal_testnet which loads 5 block
headers to 1 node, and offers this data to few nodes, which should
propagate it over the network further. Next query every node for
this data.

* Adjust paths on which Fluffy CI is triggered

* Add documentation on the local testnet
2022-02-11 14:43:10 +01:00
Kim De Mey
38036966a6
Improve the tests of the local testnet (#953)
* Improve the tests of the local testnet

The local testnet test was rather flaky and would occasionally
fail. It has been made more robust by adding the ENRs directly
to the routing table instead of doing some random lookups.

Additionally, the amount of nodes were increased (=64), ip limits
configuration was added, and the bits-per-hop value was set to 1
in order to make the lookups more likely to hit the network
instead of only the local routing table.

Failure is obviously still possible to happen when sufficient
packets get lost. If this turns out to be the case with the current
amount of nodes, we might have to revise the testing strategy here.

* Disable lookup test for State network

Disable lookup test for State network due to issue with custom
distance function causing the lookup to not always converging
towards the target.
2022-02-02 22:48:33 +01:00
Kim De Mey
f19a64a2fe
Add uTP over discv5 in portal wire protocol (#922)
* Add uTP over discv5 in portal wire protocol
2022-01-14 16:07:14 +01:00
Kim De Mey
79647e5580
Allow access to contentDB from portal wire protocol (#920)
- Allow access to contentDB from portal wire protocol
- Use this to do the db.get in `handleFindContent` directly
- Use this to check the `contentKeys` list in `handleOffer`
2022-01-06 09:06:05 +01:00
Kim De Mey
a8640fe57c
History content keys (#919)
* Change History content key to us SSZ Union and adjust tests

* Change slot to byteBE instead of LE

This is currently not specified in the Portal network
specifications, but we are using already BE for the actual content
key, so change it also here to remain consistent.
2022-01-05 09:49:49 +01:00
Kim De Mey
796787e0e5
Add state and history content key tests (#916) 2021-12-20 09:53:00 +01:00
Kim De Mey
092d946c72
Add state ContentKey test and adjust history ContentKey test (#914) 2021-12-15 08:48:56 +01:00
Kim De Mey
70625bc02c
Use one bootstrap nodes argument for discv5 and Portal (#911)
Currently bootstrap nodes for discv5 and for the Portal nodes
were provided through separate cli arguments. This is however
confusing and cumbersome as typically when (currently) testing
a node will have both discv5 and the Portal networks enabled.
We merge them into one argument.

If a node happens not to support a Portal network, it will be
removed after message request failure.

Commit also contains additional clean-up and nim-eth bump.
2021-12-13 09:06:29 +01:00
Kim De Mey
10a3946194
Bit of cleanup (#903)
* Rename FindNode to FindNodes as per spec

* Use consistently lower case starting camelCase for consts

Style guide nep1 allows for both CamelCase and camelCase for
consts, but we seem to use more often camelCase. Using this now
consistently.

* Change some procs to func
2021-12-08 11:54:22 +01:00
Kim De Mey
ed04aad203
Fix fluffy discovery rpc tests after nim-json-rpc bump (#897) 2021-11-30 16:04:24 +01:00
Kim De Mey
903350bdde
Add local testnet script and required json-rpc calls (#891)
- Add basic discv5 and portal json-rpc calls and activate them in
fluffy
- Renames in the rpc folder
- Add local testnet script and run this script in CI
- bump nim-eth
2021-11-24 08:45:55 +01:00
Kim De Mey
78f585399f
Bump submodules (#887)
* Bump submodules

* Fix missing import for state and history networks
2021-11-18 12:06:53 +01:00
Kim De Mey
bdbd8e4c87
Add SSZ Unions through case objects (#882)
* Add SSZ Unions through case objects

* Add connection id content response test and improve other test vectors

* Implement content keys and ids for state network as per spec

Content keys case object is used so that it can be serialized and
deserialized as an SSZ Union.

* Let message Union in Portal wire protocol start at 0 as per new spec
2021-11-17 17:11:17 +01:00
Kim De Mey
0ba3c682e4
Use SSZ code from nim-ssz-serialization module (#875) 2021-10-23 14:28:12 +02:00
Kim De Mey
02e8b2eb12
Clean up old unused code and unused imports (#877) 2021-10-22 21:36:53 +02:00
Kim De Mey
92dfb02ad7
Use ByteList.init instead of List 2048 (#876) 2021-10-22 17:49:23 +02:00
Kim De Mey
4b85c73f43
Update portal wire protocol ids to latest spec (#873) 2021-10-20 22:31:05 +02:00
Kim De Mey
b77034c870
Update portal wire protocol according to latest spec changes (#860)
This is not complete, in order to be according to specification
two SSZ Union types need to be used.
2021-10-13 21:35:54 +02:00
Kim De Mey
447deb29ff
Add HistoryNetwork that uses Portal wire protocol (#856)
* Add HistoryNetwork that uses Portal wire protocol

* Further implement getContent and adjust naming of distance calc calls
2021-10-09 13:22:03 +02:00
KonradStaniec
c62905db25
Re enable using custom distance function (#854) 2021-10-05 21:16:33 +02:00