Commit Graph

3046 Commits

Author SHA1 Message Date
Ivan Yonchovski 7031ef88fc Fix windows build instructions 2022-09-03 20:44:07 +07:00
Jordan Hrycaj 72a31593a9
Snap fetch account storage data (#1211)
* Removed database write comparison statistics

* Provide life storage tests data

details:
  database dumps on external repo `nimbus-eth1`-blobs`

* Update hexary tree interpolation for storage bulk tests

* fetch storage update
2022-09-02 19:16:09 +01:00
KonradStaniec 5aaf16049f
Use separeate client and server in lcproxy (#1208)
* Use separate client and server in lc proxy
2022-08-31 07:48:48 +02:00
KonradStaniec 0353803012
Code Cleanup. Specific config for app (#1206)
* Code Cleanup. Specific config for app
2022-08-30 07:40:03 +02:00
jangko 1ab5c29530
remove duplication of EIP1559 base fee calculation
fix #1193
2022-08-29 17:07:53 +07:00
jangko 112a6f7d76
move sealing engine gaslimit calculator to txpool
fix #1032
2022-08-29 16:54:59 +07:00
jangko c6f35142a8
simple peer manager to handle static peers reconnection
fix #618
2022-08-26 22:07:56 +07:00
KonradStaniec caa5e009ff
Initial Binary for light client proxy (#1202)
* Initial Binary for light client proxy
2022-08-26 13:54:10 +02:00
Kim De Mey 5355f4e73a
A bit of consistency clean-up in accumulator code (#1201) 2022-08-24 22:12:56 +02:00
Kim De Mey c0a6c9a75c
Remove the usage of AccumulatorDB in the accumulator tests (#1200) 2022-08-24 16:41:04 +02:00
Jordan Hrycaj de2c13e136
Update snap offline tests (#1199)
* Re-implemented `hexaryFollow()` in a more general fashion

details:
+ New name for re-implemented `hexaryFollow()` is `hexaryPath()`
+ Renamed `rTreeFollow()` as `hexaryPath()`

why:
  Returning similarly organised structures, the results of the
  `hexaryPath()` functions become comparable when running over
  the persistent and the in-memory databases.

* Added traversal functionality for persistent ChainDB

* Using `Account` values as re-packed Blob

* Repack samples as compressed data files

* Produce test data

details:
+ Can force pivot state root switch after minimal coverage.
+ For emulating certain network behaviour, downloading accounts stops for
  a particular pivot state root if 30% (some static number) coverage is
  reached. Following accounts are downloaded for a later pivot state root.
2022-08-24 14:44:18 +01:00
Kim De Mey 91a3425731
Bump nim-secp256k1 module for signature recovery id check (#1196) 2022-08-23 16:28:24 +02:00
Kim De Mey 55dcbe9e58
Make the default network for portalcli the history network (#1195) 2022-08-23 16:28:10 +02:00
jangko fe54f93ab6
bump nim eth
better method not implemented message of AbstractChainDB
2022-08-22 20:11:36 +07:00
KonradStaniec 88263e664b
Clean up (#1197) 2022-08-22 12:23:26 +02:00
Zahary Karadjov daac75796f
Extract the EIP1559 gas fee calculation in nim-eth, so it can be reused in nimbus-eth2 2022-08-22 10:52:20 +07:00
Nikolay Mitev 1e4f138574 Add nimbus-eth2 as submodule. Add launch_local_testnet script to CI. 2022-08-18 18:06:38 +03:00
KonradStaniec 5f2117d081
Refactor portal stream (#1191) 2022-08-17 09:32:06 +02:00
Jordan Hrycaj f07945d37b
Misc snap sync updates (#1192)
* Bump nim-stew

why:
  Need fixed interval set

* Keep track of accumulated account ranges over all state roots

* Added comments and explanations to unit tests

* typo
2022-08-17 08:30:11 +01:00
KonradStaniec c5101c16e9
Bump nim eth (#1188)
* Bump nim eth

* Bump actions ubuntu
2022-08-16 09:17:57 +02:00
Jordan Hrycaj 7489784ba8
Snap sync accounts db code reorg (#1189)
* Extracted functionality into sub-modules for maintainability

* Setting SST bulk load as default in `accounts_db`

details:
+ currently, the same data are stored via rocksdb if available, and
  the same via embedded `storage_type` with (non-standard) prefix 200
  for time comparisons
+ fallback to normal `put()` unless rocksdb is accessible
2022-08-15 16:51:50 +01:00
Jordan Hrycaj 7d7e26d45f
Experimental bulk loader tests (#1187)
why:
  Rocksdb bulk loading might provide a slight advantage when loading
  larger data sets into the system
2022-08-12 16:42:07 +01:00
KonradStaniec f0cd340163
Adhere to transport limits in seed methods (#1186)
* Add means to limit offered content to fit talkreq

* Add test for history network limits

* Change seed method api to return num of items offered
2022-08-09 14:32:41 +02:00
KonradStaniec 4f8d10f338
Bump nim-eth (#1185) 2022-08-06 10:11:36 +02:00
jangko 11908c78ec
engine api test: handle JsonRpcError for each client rpc call 2022-08-05 09:51:16 +07:00
Jordan Hrycaj 5f0e89a41e
Snap accounts bulk import preparer (#1183)
* Provided common scheduler API, applied to `full` sync

* Use hexary trie as storage for proofs_db records

also:
 + Store metadata with account for keeping track of account state
 + add iterator over accounts

* Common scheduler API applied to `snap` sync

* Prepare for accounts bulk import

details:
+ Added some ad-hoc checks for proving accounts data received from the
  snap/1 (will be replaced by proper database version when ready)
+ Added code that dumps some of the received snap/1 data into a file
  (turned of by default, see `worker_desc.nim`)
2022-08-04 09:04:30 +01:00
KonradStaniec 71f9e37482
Add getting block by number through accumulator (#1182) 2022-08-04 08:34:53 +02:00
jangko 8117032c67
fix chain config parser regression
fixes #1180
2022-08-02 16:41:53 +07:00
jangko 21140191da
attempt to fix flaky simulator test 2022-08-02 11:48:59 +07:00
Kim De Mey 50af402f59
Add verification of block data by use of accumulator (#1178) 2022-08-01 21:00:21 +02:00
jangko a5d4759bfd
enhance net-key command line option to accept random, hex, and path
- previously it only accept hex
- fix #587
2022-07-30 08:46:11 +07:00
jangko a087d65542
reduce test suite time consumption 2022-07-30 08:43:15 +07:00
jangko 2c0a8fa26d
update to supported macOS in Github CI 2022-07-30 08:42:44 +07:00
KonradStaniec a1253c67bd
Fix testnet tests (#1176)
* Testnet improvements

Increase timeout for reading
Add more logs
Offer endpoint can fail due to talkReq timeout, to avoid
test failure, retry it few times until success.
2022-07-29 14:24:07 +02:00
KonradStaniec e3cabaff7f
Add bulk seeding to multiple peers (#1170)
* Add bulk seeding to multiple peers
2022-07-26 13:14:56 +02:00
jangko eb972fd598
attemp to fix simulator ci 2022-07-24 15:47:06 +07:00
jangko e3cf999033
fix .nimble dependency 2022-07-22 09:05:42 +07:00
Jordan Hrycaj 73b628491d
Clique snapshots reorg (#1169)
* Add persistent snapshot size logging

why:
  Suspecting too much space used

snapshot statistic:
  [..]
  blockNumber=2214912 nSnaps=2236 snapsTotal=1.14m
  blockNumber=2215936 nSnaps=2237 snapsTotal=1.14m
  [..]
  Persisting blocks fromBlock=2216449 toBlock=2216640
  36458496	datadir-nimbus-goerlish/data/nimbus/

* Replace legacy `lru_cache` by `keyed_queue`

why:
  `keyed_queue` generalises `lru_cache`

snapshot statistic:
  [..]
  blockNumber=2234368 nSnaps=2259 snapsTotal=1.15m
  blockNumber=2235392 nSnaps=2260 snapsTotal=1.15m
  [..]
  Persisting blocks fromBlock=2235649 toBlock=2235840
  37627288	datadir-nimbus-goerlish/data/nimbus/

* Increase persistent snapshot storage interval by 300%

snapshot statistic:
      [..]
      blockNumber=2232320 nSnaps=620 snapsTotal=0.30m
      blockNumber=2236416 nSnaps=621 snapsTotal=0.30m
      [..]
      Persisting blocks fromBlock=2237185 toBlock=2237376
      37627288	datadir-nimbus-goerlish/data/nimbus/

* Cull legacy debugging environment for clique

why:
  Chronicles provides a better choice (when properly set up)
2022-07-21 19:16:28 +01:00
Jordan Hrycaj 5d98f68c09
Sync update to work with sepolia reorgs (#1168)
* Error return in `persistBlocks()` on initial `VmState` roblem

why:
  previously threw an exception

* Updated sync mode option

why:
 using enum rather than bool => space for more

* Added sync mode `full`, re-factued legacy sync

also:
  rebased

* Fix typo (crashes `pesistBlocks()` otherwise)

also:
  rebase to master

* Reduce log ticker noise by suppressing duplicate messages

* Clarify staged queue overflow handling

why:
  backtrack/re-org mode in `stageItem()` should be detected by both,
  the global indicator or the work item where it might have moved into.

also:
  rebased
2022-07-21 13:14:41 +01:00
KonradStaniec aa945f1ed9
Additional seed endpoints (#1164)
* Additional seed endpoints
2022-07-20 12:46:42 +02:00
jangko 4721fc7a54
add CORS support for HTTP local services(json-rpc, engine-api, graphql)
There is also a stub for websocket CORS handler. We still need to add
non immediate response management to websock.
2022-07-19 16:12:36 +07:00
jangko a48f69a89a
hive: engine simulator add jwt auth test cases 2022-07-18 16:56:44 +07:00
jangko 76e438da2a
bump submodule: json-rpc and graphql
this is an update to support jwt auth and cors hook
2022-07-18 16:56:44 +07:00
jangko e6938af437
apply jwt auth to rpcHttpServer and update jwt auth of rpcWebsocketServer
fixes #967
2022-07-18 16:56:44 +07:00
jangko 6cfaaf5b45
bump json-rpc, secp256k1, blscurve, and accompanying fixes 2022-07-12 20:01:02 +07:00
Kim De Mey 5abf05e455
Add validation for offered block bodies and receipts (#1158)
Also use a AsyncQueue now to handle offered data instead of the
network specific validation callbacks.
2022-07-11 16:29:16 +02:00
Kim De Mey 87ea6ffb92
Bump most of the submodules (#1156) 2022-07-08 20:28:31 +02:00
Jordan Hrycaj 1a5e2e53dd
treat `persistBlocks()` exception as failure return code, unless `Defect` (#1155) 2022-07-06 16:00:51 +01:00
Kim De Mey f29cbebb9d
Move block data test files to one location (#1154) 2022-07-05 16:14:52 +02:00
Kim De Mey facd7342fb
Add network retries for fluffy block requests (#1153)
- Add retries when the network request failed on validation of
block header, body or receipts.
- Shuffle the first set of neighbours that the request is send to
in order to not always hit the same peer first for the same content
- Some general clean-up
2022-07-05 14:42:55 +02:00