Commit Graph

147 Commits

Author SHA1 Message Date
andri lim 198e9c6c98
Remove more goerli data (#2249) 2024-05-30 11:49:47 +00:00
tersec 75ff54fef3
rm --terminal-total-difficulty command-line option (#2221) 2024-05-26 08:15:18 +02:00
tersec e895c0baeb
rm Clique consensus method support and Goerli network (#2219)
* rm Clique consensus method support and Goerli network

* rm a few more SealingEngineRef and GoerliNets
2024-05-25 16:12:14 +02:00
Jacek Sieka fe296213cf
les: remove (#2211)
LES was [removed from
geth](https://github.com/ethereum/go-ethereum/pull/28586) and is
generally no longer supported on the network.
2024-05-23 16:53:51 +02:00
Jordan Hrycaj 2c322054e0
Attempt to roll back stateless mode implementation in a single PR (#2209)
* Attempt to roll back stateless mode implementation in a single PR

why:
+ Stateless mode is not fully working and in the way
+ Single PR should make it feasible to investigate for a possible
  re-implementation

* Fix copyright year

* Fix annotation for exception (evmc mode)
2024-05-22 21:01:19 +00:00
Jordan Hrycaj ee9aea171d
Culling legacy DB and accounts cache (#2197)
details:
+ Compiles nimbus all_tests
+ Failing tests have been commented out
2024-05-20 10:17:51 +00:00
Jordan Hrycaj e8eb3268f5
Generalise prune mode option 4 different db models (#2139)
* Update README

* Nimbus-main: replaced `PruneMode` options by `ChainDbMode` options

details:
  For the legacy database, this changes the phrase
  - `conf.pruneMode == PruneMode.Full` to the expression
  + `conf.chainDbMode == ChainDbMode.Prune`.

* Fix issues moaned about by NIM compiler

* Fix copyright year
2024-04-17 18:09:55 +00:00
web3-developer 332d294e77
Storage of block witnesses (#1986)
* Added procs to get and store block witness in db and add generate-witness cli flag.

* Completed initial implementation of block witness storage.

* Added test to verify witness is persisted to db after call to persistBlock.

* Update getBlockWitness to return witness using Result type.

* Make generate witness parameter hidden.
2024-02-13 17:49:41 +08:00
jangko d8ec962b6a
Bump confutils: no more exception leak 2024-02-12 13:01:05 +07:00
andri lim a441ec3cb1
Remove obsolete select_backend and fix simulators CI (#2007)
* Remove obsolete select_backend

* Fix copyright year
2024-02-04 21:28:20 +07:00
andri lim c635e160d9
Implement combo http server for rpc, engine_api, and graphql services (#1992)
* Combo HTTP server implementation

* Use json flavor for jwt_auth decoder
2024-01-29 20:20:04 +07:00
Vit∀ly Vlasov 048fc380a9
Nimbus light client integration with status-go rebased version (#1971)
* Nimbus light client integration with status-go

* Add cleanup code, address review comments

* Disable metrics for libverifproxy only

* Update confutils

* missing import

* build proxy in tests

* more build stuff

* namespace make vars

* export NimMain for windows

* reduce dependency on Nim compiler in header file

* copyright

---------

Co-authored-by: Vitaliy Vlasov <siphiuel@protonmail.com>
Co-authored-by: Jacek Sieka <jacek@status.im>
2024-01-26 23:04:08 +01:00
Kim De Mey dbc1ae86e2
Vendor bumps + related fixes + warning fixes (#1985)
- Vendor bump of stew, nim-eth, chronos, nimbus-eth2 and libp2p
- Bump related fixes + fixes of deprecation warnings
- Several other warnings fixed.
2024-01-24 16:28:03 +01:00
web3-developer 48630ccb03
Experimental RPC endpoints for generating block witnesses (#1977)
* Completed draft implementation of witness JSON-RPC endpoints for portal network bridge.

* Updated Nimbus RPC configuration to support enabling experimental endpoints.

* Moved witness verification tests.

* Added json test for getProof.

* Added main procs to new tests to fix test suite.

* Added getBlockWitness test to blockchain json test suite.

* Added tests for experimental RPC endpoints and improved the API to support returning state proofs from before or after block execution.

* Correctly rollback transaction in getBlockWitness proc.
2024-01-22 17:11:37 +08:00
Kim De Mey 86f93fdba5
ValidIpAddress -> IpAddress related vendor bumps and changes (#1892) 2023-11-10 19:38:11 +01:00
jangko 1db8253a2c
Fix --static-peers-file description 2023-10-31 17:53:06 +07:00
jangko 26ede94a73
Add Holesky testnet configuration 2023-10-25 14:06:36 +07:00
jangko 63ff17efbf
Remove unused testnet configuration 2023-10-24 16:00:44 +07:00
jangko e3041347fd
Add --trusted-setup-file command line option 2023-10-01 21:00:32 +07:00
Kim De Mey d0e46d9075
Add cli option for providing bootstrap and static peers by ENR (#1634)
* Add cli option for providing bootstrap and static peers by ENR

* Minor bootstrap and static peers loading clean-up
2023-07-07 09:47:26 +02:00
jangko 67aaf92c1d
bump submodules 2023-06-07 18:12:02 +07:00
Kim De Mey ff90f4fd22
Improve logging and logging options in Fluffy (#1548)
* Improve logging and logging options in Fluffy

- Allow selection of log format, including:
  - JSON
  - automatic selection based on tty
- Allow log levels per topic configured on cli
2023-04-19 17:01:01 +02:00
Adam Spitz c58d83efd5
More work on stateless mode (#1535)
* Reviving more of the code for stateless mode.

* Made asyncProcessTransaction.

* More work on stateless mode.
2023-04-12 08:39:11 -04:00
Jordan Hrycaj 9facab91cb
Prepare snap client for continuing with full sync (#1534)
* Somewhat tighten error handling

why:
  Zombie state is invoked when the current peer turns out to be useless
  for further communication. While there is a chance to further talk
  to a peer about another topic (aka healing) after some protocol failure,
  it makes no sense to do so after a network problem.

  The latter state is explained bu the `peerDegraded` flag that goes
  together with the `zombie` state flag. A degraded peer is dropped
  immediately.

* Remove `--sync-mode=snapCtx` option, always start snap in recovery mode

why:
  No need for a snap sync option without recovery mode, can be achieved
  by deleting the database.

* Code cosmetics, typos, prettify logging, debugging helper, etc.

* Split off snap sync sub-mode handler into separate modules

details:
  The original `worker.nim` source has become a multiplexer for several
  snap sync sub-modes `full` and `snap`. The source modules of the
  incarnations of a particular sync sub-mode are places into the
  `worker/play` directory.

* Update ticker for snap and full sync logging
2023-04-06 20:42:07 +01:00
Adam Spitz ee50f06a3e
Sketching in "stateless mode". (#1495)
(Doesn't do anything yet.)
2023-03-13 14:18:30 -04:00
Jordan Hrycaj fe3a6d67c6
Prepare snap server client test scenario cont2 (#1487)
* Clean up some function prototypes

why:
  Simplify polymorphic prototype variances for easier maintenance.

* Fix fringe condition crash when importing bogus RLP node

why:
  Accessing non-list RLP entry as a list causes `Defect`

* Fix left boundary proof at range extractor

why:
  Was insufficient. The main problem was that there was no unit test for
  the validity of the generated left boundary.

* Handle incomplete left boundary proofs early

why:
  Attempt to do it later leads to overly complex code in order to prevent
  looping when the same peer repeats to send the same incomplete proof.

  Contrary, gaps in the leaf sequence can be handled gracefully with
  registering the gaps

* Implement a manual pivot setup mechanism for snap sync

why:
  For a test scenario it is convenient to set the pivot to something
  lower than the beacon header from the consensus layer. This does not
  need rely on any RPC mechanism.

details:
  The file containing the pivot specs is specified by the
  `--sync-ctrl-file` option. It is regularly parsed for updates.

* Fix calculation error

why:
  Prevent from calculating negative square root
2023-03-07 14:23:22 +00:00
Jordan Hrycaj f20f20f962
Prepare snap server client test scenario (#1483)
* Enable `snap/1` accounts range service

* Allow to change the garbage collector to `boehm` as a Makefile option.

why:
  There is still an unsolved memory corruption problem that might be
  related to the standard `gc`. It seemingly goes away if the `gc` is
  changed to `boehm`.

  Specifying another `gc` on the make level simplifies debugging and
  development.

* Code cosmetics

details:
* updated exception annotations
* extracted `worker_desc.nim` from `full/worker.nim`
* etc.

* Implement option to state a sync modifier file

why:
  This allows to specify extra sync type specific options which might
  change over time. This file is regularly checked for updates.

* Implement a threshold when to suspend full syncing

why:
  For a test scenario, a full sync beep may work as a local snap server.
  There is no need to download the full block chain.

details:
  The file containing the pivot specs is specified by the
  `--sync-ctrl-file` option. It is regularly parsed for updates.
2023-03-02 09:57:58 +00:00
Jordan Hrycaj df1217b7ca
Silence compiler gossip after nim upgrade cont3 (#1466)
* Removed some Windows specific unit test annoyances

details:
+ Short put()/get() cycles on persistent database have a race condition
  with vendor rocksdb. On a specific (and slow) qemu/win7 a 50ms `sleep()`
  in between will mostly do the job (i.e. unless heavy CPU load.) This
  issue was not observed on github/ci.
+ Removed annoyances when qemu/Win7 keeps the rocksdb database files
  locked even after closing the db. The problem is solved by strictly
  using fresh names for each test. No assumption made to be able to
  properly clean up. This issue was not observed on github/ci.

* Silence some compiler gossip -- part 7, misc/non(sync or graphql)

details:
  Adding some missing exception annotation
2023-02-14 20:27:17 +00:00
Kim De Mey 73e93f1f11
Reduce Nim 1.6 compiler warnings & hints
* Reduce Nim 1.6 compiler warnings/hints for Fluffy and Nimbus proxy

Mostly raises Defect removals, TaintedString removal and some
unnecessary imports.

Also updating the copyright years alongside.

* Further reduce Nim 1.6 compiler warnings/hints for Nimbus
2023-01-31 13:38:08 +01:00
Jordan Hrycaj 6b9f3c9ac5
Silence compiler gossip after nim upgrade cont1 (#1455)
* Silence some compiler gossip -- part 5, common

details:
  Mostly removing redundant imports and `Defect` tracer after switch
  to nim 1.6

* Silence some compiler gossip -- part 6, db, rpc, utils

details:
  Mostly removing redundant imports and `Defect` tracer after switch
  to nim 1.6

* Silence some compiler gossip -- part 7, randomly collected source files

details:
  Mostly removing redundant imports and `Defect` tracer after switch
  to nim 1.6

* Silence some compiler gossip -- part 8, assorted tests

details:
  Mostly removing redundant imports and `Defect` tracer after switch
  to nim 1.6

* Clique update

why:
  More impossible exceptions (undoes temporary fix from previous PR)
2023-01-31 01:32:17 +00:00
Jordan Hrycaj 6fb48517ba
Add snap protocol service stub (#1438)
* Cosmetics, update logger `topics`

* Clean up sync/start methods in nimbus

why:
* The `protocols` list selects served (as opposed to sync) protocols only.
* The `SyncMode.Default` object is allocated with the other possible sync
  mode objects.

* Add snap service stub to `nimbus`

* Provide full set of snap response handler stubs

* Bicarb for the latest CI hiccup

why:
  Might be a change in the CI engine for MacOS.
2023-01-20 15:01:29 +00:00
jangko e36d2f432a
fix default --key-store location if --data-dir is custom 2022-12-22 11:17:04 +07:00
jangko 94a94c5b65 implement better hardfork management 2022-12-02 13:51:42 +07:00
Jordan Hrycaj 7688148565
Snap sync can start on saved checkpoint (#1327)
* Stop negotiating pivot if peer repeatedly replies w/usesless answers

why:
  There is some fringe condition where a peer replies with legit but
  useless empty headers repetely. This goes on until somebody stops.
  We stop now.

* Rename `missingNodes` => `sickSubTries`

why:
  These (probably missing) nodes represent in reality fully or partially
  missing sub-tries. The top nodes may even exist, e.g. as a shallow
  sub-trie.

also:
  Keep track of account healing on/of by bool variable `accountsHealing`
  controlled in `pivot_helper.execSnapSyncAction()`

* Add `nimbus` option argument `snapCtx` for starting snap recovery (if any)

also:
+ Trigger the recovery (or similar) process from inside the global peer
  worker initialisation `worker.setup()` and not by the `snap.start()`
  function.
+ Have `runPool()` returned a `bool` code to indicate early stop to
  scheduler.

* Can import partial snap sync checkpoint at start

details:
 + Modified what is stored with the checkpoint in `snapdb_pivot.nim`
 + Will be loaded within `runDaemon()` if activated

* Forgot to import total coverage range

why:
  Only the top (or latest) pivot needs coverage but the total coverage
  is the list of all ranges for all pivots -- simply forgotten.
2022-11-25 14:56:42 +00:00
Kim De Mey 74a83c1229
Bump nim-eth and remove all now unneeded p2p related imports (#1273) 2022-10-20 10:34:59 +02:00
jangko c6f35142a8
simple peer manager to handle static peers reconnection
fix #618
2022-08-26 22:07:56 +07:00
jangko a5d4759bfd
enhance net-key command line option to accept random, hex, and path
- previously it only accept hex
- fix #587
2022-07-30 08:46:11 +07:00
Jordan Hrycaj 5d98f68c09
Sync update to work with sepolia reorgs (#1168)
* Error return in `persistBlocks()` on initial `VmState` roblem

why:
  previously threw an exception

* Updated sync mode option

why:
 using enum rather than bool => space for more

* Added sync mode `full`, re-factued legacy sync

also:
  rebased

* Fix typo (crashes `pesistBlocks()` otherwise)

also:
  rebase to master

* Reduce log ticker noise by suppressing duplicate messages

* Clarify staged queue overflow handling

why:
  backtrack/re-org mode in `stageItem()` should be detected by both,
  the global indicator or the work item where it might have moved into.

also:
  rebased
2022-07-21 13:14:41 +01:00
jangko 4721fc7a54
add CORS support for HTTP local services(json-rpc, engine-api, graphql)
There is also a stub for websocket CORS handler. We still need to add
non immediate response management to websock.
2022-07-19 16:12:36 +07:00
Jordan Hrycaj feda401e85
remove unused entry (#1151)
why:
  temporary value, `nim-eth` master did not compile cleanly
2022-07-04 12:47:15 +01:00
Jordan Hrycaj 7293a54e58
Added sepolia specs (#1148)
* Added sepolia specs

* temporarily avoid latest `master` branch from `nim-eth1`

why:
  Currently does not cleanly compile after the `bearssl` split api update.
2022-07-01 21:16:26 +01:00
jangko 00a43234d7
fix enable rpc logic in makeConfig 2022-06-17 07:54:13 +07:00
jangko a37f8b17e2
fix rpc and websocket server config if they share the same port with engine api
also fixes json-rpc-engine-api server and websocket-engine-api server shutdown code,
checking if they actually created or not at startup.

fix #1119
2022-06-13 19:32:19 +07:00
Jordan Hrycaj 62d31d6f1d
Normalise sync handler prototypes (#1087)
* Use type name eth and snap (rather than snap1)

* Prettified snap/eth handler trace messages

* Regrouped sync sources

details:
  Snap storage related sources are moved to common directory.
  Option --new-sync renamed to --snap-sync

also:
  Normalised logging for secondary/non-protocol handlers.

* Merge protocol wrapper files => protocol.nim

details:
  Merge wrapper sync/protocol_ethxx.nim and sync/protocol_snapxx.nim
  into single file snap/protocol.nim

* Comments cosmetics

* Similar start logic for blockchain_sync.nim and sync/snap.nim

* Renamed p2p/blockchain_sync.nim -> sync/fast.nim
2022-05-13 17:30:10 +01:00
Jordan Hrycaj 58e0543920
Squashed snap-sync-preview patch (#1076)
* Squashed snap-sync-preview patch

why:
  Providing end results makes it easier to have an overview.

  Collected patch set comments are available as nimbus/sync/ChangeLog.md
  in chronological order, oldest first.

* Removed some cruft and obsolete imports, normalised logging
2022-05-09 15:04:48 +01:00
jangko f2f204293e
first step into styleCheck fixes 2022-04-14 08:39:50 +07:00
Jordan Hrycaj 737236fd6e
Enable JWT authentication for websockets (#1039)
* Enable JWT authentication for websockets

details:
  Currently, this is optional and only enabled when the jwtsecret option
  is set.

  There is a default mechanism to generate a JWT secret if it is not
  explicitly stated. This mechanism is currently unused.

* Make JWT authentication compulsory for websockets

* Fix unit test entry point + cosmetics

* Update JSON-RPC link

* Improvements as suggested by Mamy
2022-04-06 15:11:13 +01:00
Ștefan Talpalaru 51bc1cf87f
dist: precompiled binaries and Docker images (#1015)
* dist: precompiled binaries and Docker images

The builds are reproducible, the binaries are portable and statically link librocksdb.

This took some patching. Upstream PR: https://github.com/facebook/rocksdb/pull/9752

32-bit ARM is missing as a target because two different GCC versions
fail with an ICE when trying to cross-compile RocksDB. Using Clang
instead is too much trouble for a platform that nobody should be using
anyway.

(Clang doesn't come with its own target headers and libraries, can't be
easily convinced to use the ones from GCC, so it needs an fs image from
a 32-bit ARM distro - at which point I stopped caring).

* CI: disable reproducibility test
2022-03-27 13:21:15 +02:00
jangko 9230e6a024
fixes engine api help message in config.nim
somehow a const string still need `$` to display the
content instead of the identifier itself.
2022-03-08 13:29:34 +07:00
jangko d1dd5d5cad
ttd from command-line takes precedence over ttd from config-file 2022-03-08 11:27:14 +07:00