Commit Graph

6692 Commits

Author SHA1 Message Date
Etan Kissling 79b230828d
bump `nim-unittest2` to `e96f3215030cbfa13abc2f5827069b6f8ba87e38` (#6063)
- release `v0.2.2`
2024-03-12 03:35:37 +01:00
tersec 707409b8d3
make EL light client disk usage table consistent with hardware table (#6064) 2024-03-12 02:34:40 +00:00
Etan Kissling 1a9cafd3dd
bump `nim-libp2p` to `28609597d104a9be880ed5e1648e1ce18ca9dc38` (#6062)
- fix: allFuturesThrowing compilation issue on daily
- feat: Yamux timeout
- Fix empty path crash issue for MultiAddresses `unix`, `ip6zone`, `dns***`.
- fix: move transport interop tests to nim-libp2p repo
- use chronos 4.0.0
- fix typo in `ProtoMessage.toString()`
- increase tolerance of `simple heartbeat` test
- remove unused `MultiBase.encode(..., Cid)` function
- move `allFutureThrowing` helper to tests
- use new Chronos `trackCounter` APIs for leaks checks in tests
- remove unused `LPStreamError` types
- change `SecioError` and `NoiseError` to descendants of `LPStreamError`
- do not log yamux buffers without sanitization (trace log level)
- don't forget closing the stream when final `{Fin}` fails in yamux
- use `race` instead of `or` to avoid lockup
- in yamux, do not write `{Rst}` packet to stream that's in use
- avoid `KeyError` in edge case of yamux handler
- avoid triple lookup in `m.flushed` yamux table
- catch `WebSocketError` in `wstransport`
- define proper parent error type for `YamuxError`
- document known `--mm:orc` crash
- add `{.async: (raises).}` to `libp2p/stream` modules
2024-03-12 03:28:51 +01:00
tersec fd8429da74
rm Web3Signer Bellatrix block support (#6061) 2024-03-11 22:09:31 +00:00
Jacek Sieka 7d06edb7c9
Mention blob retention policy in docs (#6058) 2024-03-11 20:07:41 +00:00
Eugene Kabanov f088e5f57b
Consensus block value calculation for produceBlockV3 API call. (#5873)
* allow specifying get_proposer_reward block root at state.slot

* Add consensus_block_value calculation.

* Address review comments.

* Post-rebase adjustments.

* Use proper state to calculate consensus block value.

* Revert "allow specifying get_proposer_reward block root at state.slot"

This reverts commit 9fef9a8199f63056060527ac2531acc3b0ed8dcb.

* Fix post-revert problems.
Return back to Gwei.

* Adding test which is not working.

* Do not use test suite if it does not have post-state.

* Add debug logging.

* Increase logging to track sources of balance changes.

* Fix sync committee rewards/penalties calculation.

* Revert "Increase logging to track sources of balance changes."

This reverts commit 32feb20f2fdb66521401710866cd59ecc9951ef8.

* Adopt new vision to block rewards.

* Add block produce logging to VC.

* Remove rewards.nim.

* Eliminate toWei changes.

* Improve UInt256 shortLog.

* Fix conversion procedure.

* Address review comments.

* Fix test.

* Revert "Fix test."

This reverts commit 4948b2c1ec.

---------

Co-authored-by: tersec <tersec@users.noreply.github.com>
Co-authored-by: Etan Kissling <etan@status.im>
2024-03-11 14:18:50 +00:00
Jacek Sieka 2d519ac715
remove some unused rest serialization code (#6056) 2024-03-09 11:38:24 +01:00
Jacek Sieka 65c47c7769
Reapply "ssz: bump (#5996)" (#6037) (#6051)
* Reapply "ssz: bump (#5996)" (#6037)

This reverts commit a299d17fe7.

Fixed in https://github.com/status-im/nim-faststreams/pull/51

* bump
2024-03-09 10:25:39 +01:00
tersec 55ba43592b
req/resp status message requires zero hash for genesis finalized epoch (#6053) 2024-03-08 21:33:59 +01:00
Kim De Mey 513ccc1e0d
Bump eth2-networks for fixed LH bootstrap nodes on mainnet (#6052) 2024-03-08 20:02:28 +01:00
Etan Kissling 1cdd3c62fd
extend `syncEth1Chain tick` log (#6049)
Provide additional context in the `syncEth1Chain tick` debug log to aid
with understanding of flow when debugging on a more precise basis than
just having the metrics.
2024-03-08 19:27:45 +01:00
Etan Kissling 984d5e4631
fix regression when fetching genesis data in trustedNodeSync mode (#6047)
Corrects a regression from #5998 that led to crashes in #6046.
In `trustedNodeSync` mode, the config does not contain genesis keys,
so attempting to load from them is a `Defect`.
2024-03-08 14:52:54 +01:00
Etan Kissling a0bc3fff86
fix `/eth/v1/beacon/deposit_snapshot` for EIP-4881 (#6038)
Fix the `/eth/v1/beacon/deposit_snapshot` API to produce proper EIP-4881
compatible `DepositTreeSnapshot` responses. The endpoint used to expose
a Nimbus-specific database internal format.

Also fix trusted node sync to consume properly formatted EIP-4881 data
with `--with-deposit-snapshot`, and `--finalized-deposit-tree-snapshot`
beacon node launch option to use the EIP-4881 data. Further ensure that
`ncli_testnet` produces EIP-4881 formatted data for interoperability.
2024-03-08 14:22:03 +01:00
Etan Kissling f0f63c2c53
bump `nim-kzg4844` to `9d4b32e3fbadd77526e7faefa50ab6f507fe8583` (#6043)
- bump `c-kzg-4844` to `v0.4.3`
2024-03-08 12:17:47 +00:00
Etan Kissling 6f7138e7d0
bump `nim-libp2p` to `aa4ebb0b3c58ef26561741e355467d248ed7bf0d` (#6045)
- fix(hole-punching-interop): update nim to 1.6.16
- fix: compilation issue when `libp2p_expensive_metrics` is enabled.
- fix: more metrics issues when libp2p_expensive_metrics is enabled
- tests(flaky): Increase check timeouts
- docs: add comments and improve yamux readability
- test(flaky): Log checkExpiring failure
- fix: yamux metrics
- improvement: enhanced checkExpiring macro with custom timeout
- docs(general): Improve docs
2024-03-08 03:58:36 +00:00
Etan Kissling 8be52de557
bump `nim-libbacktrace` to `a6746c4df8dd8550ae5e75f4740f4470d5ed9aad` (#6044)
- bump `libbacktrace` to `f41cf1fecbc01966e0a3c32a1a54ff9cb0a19bb9`
2024-03-08 03:28:53 +00:00
Etan Kissling d1935376e3
bump `nim-json-rpc` to `a20ba207d25a9ab496518e84ec0c28b2d8d25ad7` (#6042)
- Introduce custom application error with optional data object
2024-03-08 01:53:09 +00:00
Etan Kissling 6d91305a7b
bump `nim-chronos` to `17b7a76c7e40c89f31351cd1a5faf76f177b30ac` (#6041)
- fix conversion error with `or` on futures with `{.async: (raises: []).}`
- Fix `or` deadlock issue.
- Ensure that `OwnCancelSchedule` flag will not be removed from `wait()` and `withTimeout()`.
- Add missing parts of defaults buffer size increase.
- Avoid `ValueError` effect in varargs `race`/`one`
- fix circular reference in timer
- Ensure `transp.reader` is reset to `nil` on error
2024-03-08 01:42:19 +00:00
Etan Kissling 96ebc32f26
bump `nim-bearssl` to `86f212c6a5d76b52e20fad2e318cc5436d04fc26` (#6040)
- release `v0.2.2`
2024-03-08 01:56:25 +01:00
Etan Kissling 5f641d09ff
bump `goerli` to `e3c0766b1cf5ddc22fe017e1389c6f60f0f9a468` (#6039)
- Delete `genesis_deposit_contract_snapshot.ssz` file
2024-03-08 01:33:28 +01:00
tersec 816361ed8c
Bloom filter acceleration for deposit processing (#5982) 2024-03-07 23:38:24 +00:00
Etan Kissling a299d17fe7
Revert "ssz: bump (#5996)" (#6037)
This reverts commit 21601f59e2.

Reason: `/eth/v1/beacon/light_client/updates?start_period=0&count=128`
provides malformed data. Reverting fixes it locally.

Download JSON and SSZ, then use this script to validate (Holesky).
Need to export some symbols for parsing.

```nim
import
  stew/io2,
  ./beacon_chain/spec/[digest, forks, helpers, presets],
  ./beacon_chain/spec/eth2_apis/[eth2_rest_serialization, rest_light_client_calls]

let jsonBytes = io2.readAllBytes("/Users/etan/Downloads/json_updates.json").get()

let
  (cfg, unknowns) = readRuntimeConfig("/Users/etan/Documents/Repos/nimbus-eth2/vendor/holesky/custom_config_data/config.yaml")
  genesis_validators_root = Eth2Digest.fromHex("0x9143aa7c615a7f7115e2b6aac319c03529df8242ae705fba9df39b79c59fa8b1")
  forkDigests = newClone ForkDigests.init(cfg, genesis_validators_root)

let updates = seq[ForkedLightClientUpdate].decodeJsonLightClientObjects(jsonBytes, cfg, forkDigests)

var entries = newSeqOfCap[RestVersioned[ForkedLightClientUpdate]](updates.len)
for update in updates:
  let
    contextEpoch = withForkyUpdate(update):
      when lcDataFork > LightClientDataFork.None:
        forkyUpdate.contextEpoch
      else:
        continue
    contextFork = cfg.consensusForkAtEpoch(contextEpoch)
  entries.add RestVersioned[ForkedLightClientUpdate](
    data: update,
    jsonVersion: contextFork,
    sszContext: forkDigests[].atconsensusFork(contextFork))

let res =
  block:
    var default: seq[byte]
    try:
      var stream = memoryOutput()
      for e in entries:
        withForkyUpdate(e.data):
          when lcDataFork > LightClientDataFork.None:
            var cursor = stream.delayFixedSizeWrite(sizeof(uint64))
            let initPos = stream.pos
            stream.write e.sszContext.data
            var writer = SszWriter.init(stream)
            writer.writeValue forkyUpdate
            cursor.finalWrite (stream.pos - initPos).uint64.toBytesLE()
      stream.getOutput(seq[byte])
    except SerializationError:
      default
    except IOError:
      default

let sszBytes = io2.readAllBytes("/Users/etan/Downloads/ssz_updates.ssz").get()

let sszUpdates = seq[ForkedLightClientUpdate].decodeSszLightClientObjects(res, cfg, forkDigests)
doAssert updates.len == sszUpdates.len
for i in 0 ..< updates.len:
  doAssert updates[i].kind == sszUpdates[i].kind
  withForkyUpdate(updates[i]):
    when lcDataFork > LightClientDataFork.None:
      doAssert forkyUpdate == sszUpdates[i].forky(lcDataFork)

doAssert sszBytes == res
```
2024-03-07 22:37:03 +00:00
tersec 11d522dd86
use debug- prefix for Yamux experimental support (#6010) 2024-03-07 19:52:13 +01:00
Etan Kissling 50a43f397f
rename `DepositTreeSnapshot` -> `DepositContractSnapshot` (#6036)
EIP-4881 was never correctly implemented, the `DepositTreeSnapshot`
structure has nothing to do with its actual definition. Reflect that
by renaming the type to a Nimbus-specific `DepositContractSnapshot`,
so that an actual EIP-4881 implementation can use the correct names.

- https://eips.ethereum.org/EIPS/eip-4881#specification

Notably, `DepositTreeSnapshot` contains a compressed sequence in
`finalized`, only containing the minimally required intermediate roots.

That also explains the incorrect REST response reported in #5508.

The non-canonical representation was introduced in #4303 and is also
persisted in the database. We'll have to maintain it for a while.
2024-03-07 18:42:52 +01:00
Etan Kissling 3b4b1ccb89
metrics for proposer and sync duties (#6028)
To avoid restarting with scheduled proposals or sync duties, which are
rare but have high rewards, add metrics to simpify tracking them.
2024-03-07 15:08:48 +01:00
tersec e6253c0793
don't gate test execution on not defined(i386) (#6035) 2024-03-06 17:24:05 +00:00
tersec 39fafc1248
unconditionally run test_keystore_management (#6034) 2024-03-06 15:55:31 +01:00
tersec 9e4a70e0c4
unconditionally run test_keystore (#6033) 2024-03-06 13:14:52 +01:00
Etan Kissling b7c52c9537
consider proposal slot in `Slot end`'s `nextActionWait` log/metric (#6026)
`nextActionWait` currently shows `n/a` if only proposal is scheduled
but no attestation, e.g., attestation was already made for current
epoch and validator is exiting next epoch so doesn't have another
attestation lined up. It's an edge case but it's still more correct
to also log `nextActionWait` if only proposal is scheduled.
2024-03-06 12:20:53 +01:00
tersec 9fda162e99
unconditionally run test_blockchain_dag (#6032) 2024-03-06 06:10:34 +00:00
Etan Kissling 89d9dc24bd
allow `--external-beacon-api-url` to fallback to genesis if viable (#5998)
When using `--external-beacon-api-url`, one has to accompany it with
either `--trusted-block-root` or `--trusted-state-root`. If neither is
specified, we can fallback to a deeply finalized noncontroversial block
root. For networks that started post Altair, e.g., Holesky, the genesis
block root fulfills that requirement, as in, it is implicitly trusted.
Therefore, if only `--external-beacon-api-url` is provided without any
`--trusted-block-root` or `--trusted-state-root`, use genesis block root
if it is a viable starting point (post-Altair).

```
build/nimbus_beacon_node \
    --network=holesky \
    --data-dir="$HOME/Downloads/nimbus/data/holesky" \
    "--external-beacon-api-url=http://unstable.holesky.beacon-api.nimbus.team" \
    --tcp-port=9010 --udp-port=9010 \
    --rest --log-level=DEBUG \
    --no-el
```
2024-03-05 15:41:22 +01:00
Etan Kissling 2c924bcc8c
add test runner for LC data collection tests (#6021)
Introduce a test runner for upcoming EF test suites related to canonical
light client data collection.

- https://github.com/ethereum/consensus-specs/pull/3553
2024-03-05 15:41:09 +01:00
tersec 8222d62500
rm unused phase0 and altair remote signing test code (#6027) 2024-03-05 02:38:34 +01:00
Etan Kissling d158c0cb2f
bump `goerli` to `4cd051913c95afba2e702c51cab2a156d0dde5b8` (#6024)
- remove Nimbus bootstrap nodes
2024-03-04 21:19:06 +01:00
tersec 2a13c09615
add proposer reward accounting to block transitions (#6022)
* add proposer reward accounting to block transitions

* Update beacon_chain/spec/state_transition_block.nim

Co-authored-by: Etan Kissling <etan@status.im>

---------

Co-authored-by: Etan Kissling <etan@status.im>
2024-03-04 17:00:46 +00:00
Etan Kissling a18c396d9d
use correct Nim version in daily job, and `--mm:refc` on `devel` (#6023)
Daily CI uses an outdated Nim 1.6 because it uses `origin/version-1-6`
which is not maintained very regularly. Pull from `upstream/version-1-6`
instead, same as in `ci.yml`, and also make sure that `--mm:refc` is
turned on for `upstream/devel`.
2024-03-04 15:21:28 +01:00
Etan Kissling bb8c6cda18
bump `nim-web3` to `747001250a27278e0f0f1706fe1dec88ad3ede8d` (#6020)
- Remove error field from AccessListResult
- Add optional yParity field to TransactionObject
2024-03-03 17:17:06 +01:00
Etan Kissling 649f0165b8
bump `nim-eth` to `d66a29db7ca4372dba116928f979e92cb7f7661f` (#6017)
- avoid `XDeclaredButNotUsed` in `kvstore_sqlite3` with empty tuple
2024-03-03 14:40:57 +01:00
Etan Kissling 7527b3a152
bump `nim-results` to `193d3c6648bd0f7e834d4ebd6a1e1d5f93998197` (#6019)
- test refc in CI in Nim 2.0 and later
2024-03-03 02:44:39 +00:00
Etan Kissling 6788c08869
bump `nim-json-serialization` to `7516a92eb8339f8fc1de7dd16cfa3d77bce87cd2` (#6018)
- avoid `XDeclaredButNotUsed` when no field `isnot JsonVoid`
- Fix JsonValueRef.Object comparison
2024-03-03 02:49:57 +01:00
tersec 9d56f3920b
fix some styleCheck hints (#6016) 2024-03-03 02:04:45 +01:00
Etan Kissling 542854e743
use `ConsensusFork.init` for parsing lowercase version (#6015)
Instead of maintaining string to `ConsensusFork` version inline,
use shared helper that also comes with static assertion checks
for completion.
2024-03-02 23:45:20 +01:00
tersec b4db011daf
enable --styleCheck:error in ncli/ and research/ (#6014) 2024-03-02 06:25:17 +01:00
Etan Kissling d8b8aee7b7
avoid style check issue with `syncAggregate` (#6013)
Style check confuses `func syncAggregate` because it accesses some other
object's `sync_aggregate` member in the body. Rename the func to avoid.
2024-03-02 02:54:37 +01:00
tersec 8015c0bb47
rm spec/forks capella_mev usage which had been for remote signer (#6009) 2024-03-02 01:12:48 +01:00
tersec e12f5e0bba
rm more Capella builder API remote signer support (#6012) 2024-03-01 22:29:47 +00:00
Etan Kissling 45c283c256
bump `nim-sqlite3-abi` to `6d99f874d610d9ddac3c5b63ca5d0a341049c2ed` (#5976)
- validate SHA3 hash
- bump `sqlite-amalgamation` to `3.41.2`
- bump `sqlite-amalgamation` to `3.42.0`
- apply nimble update and git commit in `update.sh`
- bump `sqlite-amalgamation` to `3.43.0`
- bump `sqlite-amalgamation` to `3.43.1`
- bump `sqlite-amalgamation` to `3.43.2`
- bump `sqlite-amalgamation` to `3.44.0`
- bump `sqlite-amalgamation` to `3.44.1`
- bump `sqlite-amalgamation` to `3.44.2`
- bump `sqlite-amalgamation` to `3.45.0`
- bump `sqlite-amalgamation` to `3.45.1`
2024-03-01 18:08:53 +01:00
tersec 0d48535f9a
refactor get_flag_and_inactivity_deltas() loop body to use per-validator get_flag_and_inactivity_delta() (#6007) 2024-03-01 16:31:25 +00:00
Etan Kissling a802e68c9f
handle exceptions in `generate_makefile` tool (#5985)
Update `generate_makefile` to produce sensible error message in all
exceptional situations, instead of just some of them. These may surface
as one of the first things to someone getting started with the repo.
2024-03-01 16:15:23 +01:00
tersec 84034c0379
rm Capella builder API-related remote signer support (#6003) 2024-03-01 05:30:09 +00:00