86 Commits

Author SHA1 Message Date
Kim De Mey
78f7de1344
Make accumulator finite at merge block (#1256)
- Let accumulator finish its last pre merge epoch (hash_tree_root
on incomplete epoch).
- Adjust code to use isPreMerge and remove isCurrentEpoch
- Split up tests to a set that runs with a mainnet merge block
number and a set that runs with a testing value.
2022-10-10 12:59:55 +02:00
Kim De Mey
1bd4911e78
Remove ChaindId from BlockKey and update test vectors (#1242) 2022-09-28 09:09:38 +02:00
Kim De Mey
87f28458a5
StyleCheck fixes for Fluffy and lcproxy (#1220)
Also add the compiler options in the nim.cfg of Fluffy and
lcproxy (only visible with direct nimble usage).
2022-09-10 21:00:27 +02:00
Kim De Mey
011f44abea
Split parsing and seeding functionality for history data (#1218) 2022-09-09 21:21:48 +02:00
Kim De Mey
621c6a31a7
Fix bug in inCurrentEpoch and improve accumulator related tests (#1217)
* Fix bug in inCurrentEpoch and improve accumulator related tests

- Fix negative wraparound / underflow in inCurrentEpoch
- Add tests in accumulator tests to verify the above
- Add header offer tests with accumulator that does and doesn't
contain historical epochs
- Additional clean-up of history tests
- enable canonicalVerify in the tests
2022-09-09 13:12:09 +02:00
Jacek Sieka
c2ed731fa5
eth: adapt to smaller eth_types (#1210) 2022-09-03 20:15:35 +02:00
Kim De Mey
5355f4e73a
A bit of consistency clean-up in accumulator code (#1201) 2022-08-24 22:12:56 +02:00
Kim De Mey
c0a6c9a75c
Remove the usage of AccumulatorDB in the accumulator tests (#1200) 2022-08-24 16:41:04 +02:00
KonradStaniec
5f2117d081
Refactor portal stream (#1191) 2022-08-17 09:32:06 +02:00
KonradStaniec
c5101c16e9
Bump nim eth (#1188)
* Bump nim eth

* Bump actions ubuntu
2022-08-16 09:17:57 +02:00
KonradStaniec
f0cd340163
Adhere to transport limits in seed methods (#1186)
* Add means to limit offered content to fit talkreq

* Add test for history network limits

* Change seed method api to return num of items offered
2022-08-09 14:32:41 +02:00
KonradStaniec
4f8d10f338
Bump nim-eth (#1185) 2022-08-06 10:11:36 +02:00
KonradStaniec
71f9e37482
Add getting block by number through accumulator (#1182) 2022-08-04 08:34:53 +02:00
Kim De Mey
50af402f59
Add verification of block data by use of accumulator (#1178) 2022-08-01 21:00:21 +02:00
KonradStaniec
e3cabaff7f
Add bulk seeding to multiple peers (#1170)
* Add bulk seeding to multiple peers
2022-07-26 13:14:56 +02:00
KonradStaniec
aa945f1ed9
Additional seed endpoints (#1164)
* Additional seed endpoints
2022-07-20 12:46:42 +02:00
Kim De Mey
5abf05e455
Add validation for offered block bodies and receipts (#1158)
Also use a AsyncQueue now to handle offered data instead of the
network specific validation callbacks.
2022-07-11 16:29:16 +02:00
Kim De Mey
f29cbebb9d
Move block data test files to one location (#1154) 2022-07-05 16:14:52 +02:00
Kim De Mey
fb12793482
Bump nim-eth and nim-bearssl and accompanying fixes (#1150) 2022-07-04 09:38:02 +02:00
Kim De Mey
33015ea4a3
Adjust to the newly specified SSZ types for Block Body and Receipts (#1147) 2022-07-01 21:51:51 +02:00
Kim De Mey
cad74db423
Support for multiple content items over Portal stream (#1090)
Each content item send gets prefix with varint (max u32) allowing
to send multiple content items as requested by offer/accept flow.
2022-06-24 15:35:31 +02:00
Kim De Mey
d4d4e8c28f
Add accumulator buildProof, verifyProof and verifyHeader (#1132) 2022-06-23 21:00:59 +02:00
Kim De Mey
de2051dcf8
Add Accumulator build helper calls and refactors (#1127)
- Move the accumulator definitions to a history accumulator file
- Add accumulator build helper calls + temporary database
- Add a header gossip content key encoding test
- Refactor & some cleanup
2022-06-16 08:50:29 +02:00
Kim De Mey
6325712aa6
Add accumulator content keys for history network (#1123) 2022-06-14 23:38:34 +02:00
KonradStaniec
b23789d107
Add getting receipts from history network (#1118)
* Add getting receipts from history network
2022-06-10 12:24:53 +02:00
KonradStaniec
730013bde3
Change test data and propagation format (#1115)
* Change test data and propagation format

* Add tool to download data from local node
2022-06-08 15:14:01 +02:00
KonradStaniec
0776f35e0c
Use queries with custom xor function (#1113)
* Use queries with custom xor function
2022-06-03 13:44:42 +02:00
Kim De Mey
6d8b25a5f5
Update header accumulator test to start from Mainnet genesis (#1111) 2022-06-01 15:21:22 +02:00
KonradStaniec
af10e8f179
Do not use vacuum when pruning (#1103)
* Do not use vacuum when pruning
2022-05-26 08:26:08 +02:00
Kim De Mey
6e05c7588e
Add Header Accumulator for the Header Gossip Network (#1094) 2022-05-18 22:49:35 +02:00
KonradStaniec
6a9d875fe7
Add pruning metrics (#1093) 2022-05-17 12:47:24 +02:00
Kim De Mey
56c3e31efc
Refactor and clean-up of Portal wire protocol test (#1089) 2022-05-16 13:17:42 +02:00
KonradStaniec
4e8c5f292e
Dynamic radius adjustments (#1079)
* Add config for node radius

* Adjust radius when deleting content
2022-05-12 18:04:37 +02:00
KonradStaniec
d3f231e3da
Add put method which preserves max size (#1065)
* Add put method which preserves max size
2022-05-09 17:18:57 +02:00
Kim De Mey
ed38ed494f
Add validation handler to Portal wire protocol (#1055)
- Validation handler allows for network specific validation of
the content
- Added also pruning of the allowed connections on PortalStream
2022-04-11 19:42:38 +02:00
KonradStaniec
203d8e8b70
Inital poke implementation (#1037)
* Initial poke implementation
2022-04-06 13:47:23 +02:00
KonradStaniec
b3570fae6a
Add api to get n furthest elements from db (#1026)
* Add api to get n furthest elements from db
2022-04-03 15:14:44 +02:00
Kim De Mey
74127644db
Adjust findNodes proc naming for consistency (#1002) 2022-03-19 08:54:42 +01:00
KonradStaniec
a330f0a4ef
Add api to get db size to content db (#987)
* Add size proc to content_db

* Add way to reclaim used space
2022-03-08 14:49:41 +01:00
jangko
9a1c8fc779
refactor genesis.nim 2022-03-03 08:28:00 +07:00
KonradStaniec
b951139af9
Add basic validation of history network (#983)
* Add basic validation of history network

* Save only valid objects to db
2022-03-02 15:29:12 +01:00
jangko
6374c9f66d
fixes regression caused by #962 2022-02-14 21:22:39 +07:00
Kim De Mey
0060462dc0
Sharing block header data around in a Portal history network (PoC) (#960)
* Sharing block header data around in a Portal history network (PoC)

- Rework PortalStream to have an instance per PortalProtocol (this
needs to be improved eventually). Each instance uses the same
UtpDiscv5Protocol instance.
- Add processContent on receival of accepted data
- Add dumb neighborhoodGossip: dumb in the sense that it only
offers one piece of content at a time.
- Add to / adjust populate_db to also allow for propagation of
the data and add debug rpc call: portal_history_propagate
- Add eth_rpc_client
- Add eth_getBlockbyHash (no txs or uncles) to eth API
- Add additional test to test_portal_testnet which loads 5 block
headers to 1 node, and offers this data to few nodes, which should
propagate it over the network further. Next query every node for
this data.

* Adjust paths on which Fluffy CI is triggered

* Add documentation on the local testnet
2022-02-11 14:43:10 +01:00
Kim De Mey
38036966a6
Improve the tests of the local testnet (#953)
* Improve the tests of the local testnet

The local testnet test was rather flaky and would occasionally
fail. It has been made more robust by adding the ENRs directly
to the routing table instead of doing some random lookups.

Additionally, the amount of nodes were increased (=64), ip limits
configuration was added, and the bits-per-hop value was set to 1
in order to make the lookups more likely to hit the network
instead of only the local routing table.

Failure is obviously still possible to happen when sufficient
packets get lost. If this turns out to be the case with the current
amount of nodes, we might have to revise the testing strategy here.

* Disable lookup test for State network

Disable lookup test for State network due to issue with custom
distance function causing the lookup to not always converging
towards the target.
2022-02-02 22:48:33 +01:00
Kim De Mey
f19a64a2fe
Add uTP over discv5 in portal wire protocol (#922)
* Add uTP over discv5 in portal wire protocol
2022-01-14 16:07:14 +01:00
Kim De Mey
79647e5580
Allow access to contentDB from portal wire protocol (#920)
- Allow access to contentDB from portal wire protocol
- Use this to do the db.get in `handleFindContent` directly
- Use this to check the `contentKeys` list in `handleOffer`
2022-01-06 09:06:05 +01:00
Kim De Mey
a8640fe57c
History content keys (#919)
* Change History content key to us SSZ Union and adjust tests

* Change slot to byteBE instead of LE

This is currently not specified in the Portal network
specifications, but we are using already BE for the actual content
key, so change it also here to remain consistent.
2022-01-05 09:49:49 +01:00
Kim De Mey
796787e0e5
Add state and history content key tests (#916) 2021-12-20 09:53:00 +01:00
Kim De Mey
092d946c72
Add state ContentKey test and adjust history ContentKey test (#914) 2021-12-15 08:48:56 +01:00
Kim De Mey
70625bc02c
Use one bootstrap nodes argument for discv5 and Portal (#911)
Currently bootstrap nodes for discv5 and for the Portal nodes
were provided through separate cli arguments. This is however
confusing and cumbersome as typically when (currently) testing
a node will have both discv5 and the Portal networks enabled.
We merge them into one argument.

If a node happens not to support a Portal network, it will be
removed after message request failure.

Commit also contains additional clean-up and nim-eth bump.
2021-12-13 09:06:29 +01:00