status-im
/
nimbus-eth2
mirror of https://github.com/status-im/nimbus-eth2.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
nimbus-eth2/beacon_chain/sszdump.nim

69 lines
2.4 KiB
Nim
Raw Normal View History

EH cleanup (#2455) almost 100% raises in nimbus-eth2 now! * fix some rare exception-related crashes in json-rpc
2021-03-26 06:52:01 +00:00
# beacon_chain
# Copyright (c) 2018-2021 Status Research & Development GmbH
# Licensed and distributed under either of
# * MIT license (license terms in the root directory or at https://opensource.org/licenses/MIT).
# * Apache v2 license (license terms in the root directory or at https://www.apache.org/licenses/LICENSE-2.0).
# at your option. This file may not be copied, modified, or distributed except according to those terms.
Small fixes (#1165) * random fixes * create dump dir on startup * don't crash on failure to write dump * fix a few `uint64` instances being used when indexing arrays - this should be a compile error but isn't due to compiler bugs * fix standalone test_block_pool compilation * add signed block processing in ncli * reuse cache entry instead of allocating a new one * allow for small clock disparities when validating blocks
2020-06-12 16:43:20 +00:00
{.push raises: [Defect].}
Clean 20200205 (#729) * beacon node code cleanup * rudimentary error checking on mainnet monitor * start client even when sending deposit * work around missing block number exception * connect to testnet with web3 url * pretty-print digests in json
2020-02-07 07:13:38 +00:00
import
disentangle eth2 types from the ssz library (#2785) * reorganize ssz dependencies This PR continues the work in https://github.com/status-im/nimbus-eth2/pull/2646, https://github.com/status-im/nimbus-eth2/pull/2779 as well as past issues with serialization and type, to disentangle SSZ from eth2 and at the same time simplify imports and exports with a structured approach. The principal idea here is that when a library wants to introduce SSZ support, they do so via 3 files: * `ssz_codecs` which imports and reexports `codecs` - this covers the basic byte conversions and ensures no overloads get lost * `xxx_merkleization` imports and exports `merkleization` to specialize and get access to `hash_tree_root` and friends * `xxx_ssz_serialization` imports and exports `ssz_serialization` to specialize ssz for a specific library Those that need to interact with SSZ always import the `xxx_` versions of the modules and never `ssz` itself so as to keep imports simple and safe. This is similar to how the REST / JSON-RPC serializers are structured in that someone wanting to serialize spec types to REST-JSON will import `eth2_rest_serialization` and nothing else. * split up ssz into a core library that is independendent of eth2 types * rename `bytes_reader` to `codec` to highlight that it contains coding and decoding of bytes and native ssz types * remove tricky List init overload that causes compile issues * get rid of top-level ssz import * reenable merkleization tests * move some "standard" json serializers to spec * remove `ValidatorIndex` serialization for now * remove test_ssz_merkleization * add tests for over/underlong byte sequences * fix broken seq[byte] test - seq[byte] is not an SSZ type There are a few things this PR doesn't solve: * like #2646 this PR is weak on how to handle root and other dontSerialize fields that "sometimes" should be computed - the same problem appears in REST / JSON-RPC etc * Fix a build problem on macOS * Another way to fix the macOS builds Co-authored-by: Zahary Karadjov <zahary@gmail.com>
2021-08-18 18:57:58 +00:00
std/[os, strformat],
chronicles,
altair fork handling cleanups (#3050) * fix stack overflow crash in REST/debug/getStateV2 * introduce `ForkyXxx` for generic type matching of `Xxx` across branches (SomeHashedBeaconState -> ForkyHashedBeaconState et al) - `Some` is already used for other types of type classes * consolidate function naming in BeaconChainDB, use some generics * import `forks.nim` from other spec modules and move `Forked*` helpers around to resolve circular imports * remove `ForkedBeaconState`, use `ForkedHashedBeaconState` throughout (less data shuffling between the types) * fix several cases of states being stored on stack in tests, causing random failures on some platforms * remove reading json support from ncli - this should be ported to the rest json reading instead (doesn't currently work because stack sizes)
2021-11-05 07:34:34 +00:00
./spec/[eth2_ssz_serialization, eth2_merkleization, forks],
implement forked merge state/block support (#2890) * implement forked state/block support * merge support for containsOrphan; import cleanup; 80-column lines * add merge block header operations and slot sanity fixture * add epoch state transition tests; implement is_valid_gas_limit(), is_merge_block(), is_execution_enabled(), and compute_timestamp_at_slot() * implement process_execution_payload() and add merge deposit operations tests * add merge block sanity tests * add merge case to syncCommitteeParticipants * v1.1.0-beta.5 updates * reduce getTestStates-based memory usage; don't try to REST-serialize ExecutionPayload transactions without underlying support * add execution payload tests; switch var to let in tests/official/
2021-09-27 14:22:58 +00:00
./spec/datatypes/[phase0, altair, merge],
disentangle eth2 types from the ssz library (#2785) * reorganize ssz dependencies This PR continues the work in https://github.com/status-im/nimbus-eth2/pull/2646, https://github.com/status-im/nimbus-eth2/pull/2779 as well as past issues with serialization and type, to disentangle SSZ from eth2 and at the same time simplify imports and exports with a structured approach. The principal idea here is that when a library wants to introduce SSZ support, they do so via 3 files: * `ssz_codecs` which imports and reexports `codecs` - this covers the basic byte conversions and ensures no overloads get lost * `xxx_merkleization` imports and exports `merkleization` to specialize and get access to `hash_tree_root` and friends * `xxx_ssz_serialization` imports and exports `ssz_serialization` to specialize ssz for a specific library Those that need to interact with SSZ always import the `xxx_` versions of the modules and never `ssz` itself so as to keep imports simple and safe. This is similar to how the REST / JSON-RPC serializers are structured in that someone wanting to serialize spec types to REST-JSON will import `eth2_rest_serialization` and nothing else. * split up ssz into a core library that is independendent of eth2 types * rename `bytes_reader` to `codec` to highlight that it contains coding and decoding of bytes and native ssz types * remove tricky List init overload that causes compile issues * get rid of top-level ssz import * reenable merkleization tests * move some "standard" json serializers to spec * remove `ValidatorIndex` serialization for now * remove test_ssz_merkleization * add tests for over/underlong byte sequences * fix broken seq[byte] test - seq[byte] is not an SSZ type There are a few things this PR doesn't solve: * like #2646 this PR is weak on how to handle root and other dontSerialize fields that "sometimes" should be computed - the same problem appears in REST / JSON-RPC etc * Fix a build problem on macOS * Another way to fix the macOS builds Co-authored-by: Zahary Karadjov <zahary@gmail.com>
2021-08-18 18:57:58 +00:00
./consensus_object_pools/block_pools_types
Clean 20200205 (#729) * beacon node code cleanup * rudimentary error checking on mainnet monitor * start client even when sending deposit * work around missing block number exception * connect to testnet with web3 url * pretty-print digests in json
2020-02-07 07:13:38 +00:00
altair fork handling cleanups (#3050) * fix stack overflow crash in REST/debug/getStateV2 * introduce `ForkyXxx` for generic type matching of `Xxx` across branches (SomeHashedBeaconState -> ForkyHashedBeaconState et al) - `Some` is already used for other types of type classes * consolidate function naming in BeaconChainDB, use some generics * import `forks.nim` from other spec modules and move `Forked*` helpers around to resolve circular imports * remove `ForkedBeaconState`, use `ForkedHashedBeaconState` throughout (less data shuffling between the types) * fix several cases of states being stored on stack in tests, causing random failures on some platforms * remove reading json support from ncli - this should be ported to the rest json reading instead (doesn't currently work because stack sizes)
2021-11-05 07:34:34 +00:00
export
eth2_ssz_serialization, eth2_merkleization, forks, block_pools_types
Small fixes (#1165) * random fixes * create dump dir on startup * don't crash on failure to write dump * fix a few `uint64` instances being used when indexing arrays - this should be a compile error but isn't due to compiler bugs * fix standalone test_block_pool compilation * add signed block processing in ncli * reuse cache entry instead of allocating a new one * allow for small clock disparities when validating blocks
2020-06-12 16:43:20 +00:00
# Dump errors are generally not fatal where used currently - the code calling
# these functions, like most code, is not exception safe
template logErrors(body: untyped) =
try:
body
except CatchableError as err:
notice "Failed to write SSZ", dir, msg = err.msg
Clean 20200205 (#729) * beacon node code cleanup * rudimentary error checking on mainnet monitor * start client even when sending deposit * work around missing block number exception * connect to testnet with web3 url * pretty-print digests in json
2020-02-07 07:13:38 +00:00
proc dump*(dir: string, v: AttestationData, validator: ValidatorPubKey) =
Small fixes (#1165) * random fixes * create dump dir on startup * don't crash on failure to write dump * fix a few `uint64` instances being used when indexing arrays - this should be a compile error but isn't due to compiler bugs * fix standalone test_block_pool compilation * add signed block processing in ncli * reuse cache entry instead of allocating a new one * allow for small clock disparities when validating blocks
2020-06-12 16:43:20 +00:00
logErrors:
SSZ.saveFile(dir / &"att-{v.slot}-{v.index}-{shortLog(validator)}.ssz", v)
Clean 20200205 (#729) * beacon node code cleanup * rudimentary error checking on mainnet monitor * start client even when sending deposit * work around missing block number exception * connect to testnet with web3 url * pretty-print digests in json
2020-02-07 07:13:38 +00:00
Implement split preset/config support (#2710) * Implement split preset/config support This is the initial bulk refactor to introduce runtime config values in a number of places, somewhat replacing the existing mechanism of loading network metadata. It still needs more work, this is the initial refactor that introduces runtime configuration in some of the places that need it. The PR changes the way presets and constants work, to match the spec. In particular, a "preset" now refers to the compile-time configuration while a "cfg" or "RuntimeConfig" is the dynamic part. A single binary can support either mainnet or minimal, but not both. Support for other presets has been removed completely (can be readded, in case there's need). There's a number of outstanding tasks: * `SECONDS_PER_SLOT` still needs fixing * loading custom runtime configs needs redoing * checking constants against YAML file * yeerongpilly support `build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG` * load fork epoch from config * fix fork digest sent in status * nicer error string for request failures * fix tools * one more * fixup * fixup * fixup * use "standard" network definition folder in local testnet Files are loaded from their standard locations, including genesis etc, to conform to the format used in the `eth2-networks` repo. * fix launch scripts, allow unknown config values * fix base config of rest test * cleanups * bundle mainnet config using common loader * fix spec links and names * only include supported preset in binary * drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 13:01:38 +00:00
proc dump*(dir: string, v: phase0.TrustedSignedBeaconBlock) =
Small fixes (#1165) * random fixes * create dump dir on startup * don't crash on failure to write dump * fix a few `uint64` instances being used when indexing arrays - this should be a compile error but isn't due to compiler bugs * fix standalone test_block_pool compilation * add signed block processing in ncli * reuse cache entry instead of allocating a new one * allow for small clock disparities when validating blocks
2020-06-12 16:43:20 +00:00
logErrors:
cache block hash (#1329) hash_tree_root was turning up when running beacon_node, turns out to be repeated hash_tree_root invocations - this pr brings them back down to normal. this PR caches the root of a block in the SignedBeaconBlock object - this has the potential downside that even invalid blocks will be hashed (as part of deserialization) - later, one could imagine delaying this until checks have passed there's also some cleanup of the `cat=` logs which were applied randomly and haphazardly, and to a large degree are duplicated by other information in the log statements - in particular, topics fulfill the same role
2020-07-16 13:16:51 +00:00
SSZ.saveFile(dir / &"block-{v.message.slot}-{shortLog(v.root)}.ssz", v)
Syncv2 (#2723) * bump libp2p * altair sync v2 Use V2 sync requests after the altair fork has happened, according to the wall clock * Fix the behavior of the v1 req/resp calls after Altair Co-authored-by: Zahary Karadjov <zahary@gmail.com>
2021-07-15 19:01:07 +00:00
proc dump*(dir: string, v: altair.TrustedSignedBeaconBlock) =
logErrors:
SSZ.saveFile(dir / &"block-{v.message.slot}-{shortLog(v.root)}.ssz", v)
Implement split preset/config support (#2710) * Implement split preset/config support This is the initial bulk refactor to introduce runtime config values in a number of places, somewhat replacing the existing mechanism of loading network metadata. It still needs more work, this is the initial refactor that introduces runtime configuration in some of the places that need it. The PR changes the way presets and constants work, to match the spec. In particular, a "preset" now refers to the compile-time configuration while a "cfg" or "RuntimeConfig" is the dynamic part. A single binary can support either mainnet or minimal, but not both. Support for other presets has been removed completely (can be readded, in case there's need). There's a number of outstanding tasks: * `SECONDS_PER_SLOT` still needs fixing * loading custom runtime configs needs redoing * checking constants against YAML file * yeerongpilly support `build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG` * load fork epoch from config * fix fork digest sent in status * nicer error string for request failures * fix tools * one more * fixup * fixup * fixup * use "standard" network definition folder in local testnet Files are loaded from their standard locations, including genesis etc, to conform to the format used in the `eth2-networks` repo. * fix launch scripts, allow unknown config values * fix base config of rest test * cleanups * bundle mainnet config using common loader * fix spec links and names * only include supported preset in binary * drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 13:01:38 +00:00
proc dump*(dir: string, v: phase0.SignedBeaconBlock) =
Trusted blocks (#1227) * cleanups * fix ncli state root check flag * add block dump to ncli_db * limit ncli_db benchmark length * tone down finalization logs * introduce trusted blocks We only store blocks whose signature we've verified in the database - as such, there's no need to check it again, and most importantly, no need to deserialize the signature when loading from database. 50x startup time improvement, 200x block load time improvement. * fix rewinding when deposits have invalid signature * speed up ancestor iteration by avoiding copy * avoid deserializing signatures for trusted data * load blocks lazily when rewinding (less memory used) * chronicles workarounds * document trustedbeaconblock
2020-06-25 10:23:10 +00:00
logErrors:
cache block hash (#1329) hash_tree_root was turning up when running beacon_node, turns out to be repeated hash_tree_root invocations - this pr brings them back down to normal. this PR caches the root of a block in the SignedBeaconBlock object - this has the potential downside that even invalid blocks will be hashed (as part of deserialization) - later, one could imagine delaying this until checks have passed there's also some cleanup of the `cat=` logs which were applied randomly and haphazardly, and to a large degree are duplicated by other information in the log statements - in particular, topics fulfill the same role
2020-07-16 13:16:51 +00:00
SSZ.saveFile(dir / &"block-{v.message.slot}-{shortLog(v.root)}.ssz", v)
Clean 20200205 (#729) * beacon node code cleanup * rudimentary error checking on mainnet monitor * start client even when sending deposit * work around missing block number exception * connect to testnet with web3 url * pretty-print digests in json
2020-02-07 07:13:38 +00:00
Syncv2 (#2723) * bump libp2p * altair sync v2 Use V2 sync requests after the altair fork has happened, according to the wall clock * Fix the behavior of the v1 req/resp calls after Altair Co-authored-by: Zahary Karadjov <zahary@gmail.com>
2021-07-15 19:01:07 +00:00
proc dump*(dir: string, v: altair.SignedBeaconBlock) =
logErrors:
SSZ.saveFile(dir / &"block-{v.message.slot}-{shortLog(v.root)}.ssz", v)
implement forked merge state/block support (#2890) * implement forked state/block support * merge support for containsOrphan; import cleanup; 80-column lines * add merge block header operations and slot sanity fixture * add epoch state transition tests; implement is_valid_gas_limit(), is_merge_block(), is_execution_enabled(), and compute_timestamp_at_slot() * implement process_execution_payload() and add merge deposit operations tests * add merge block sanity tests * add merge case to syncCommitteeParticipants * v1.1.0-beta.5 updates * reduce getTestStates-based memory usage; don't try to REST-serialize ExecutionPayload transactions without underlying support * add execution payload tests; switch var to let in tests/official/
2021-09-27 14:22:58 +00:00
proc dump*(dir: string, v: merge.SignedBeaconBlock) =
logErrors:
SSZ.saveFile(dir / &"block-{v.message.slot}-{shortLog(v.root)}.ssz", v)
altair fork handling cleanups (#3050) * fix stack overflow crash in REST/debug/getStateV2 * introduce `ForkyXxx` for generic type matching of `Xxx` across branches (SomeHashedBeaconState -> ForkyHashedBeaconState et al) - `Some` is already used for other types of type classes * consolidate function naming in BeaconChainDB, use some generics * import `forks.nim` from other spec modules and move `Forked*` helpers around to resolve circular imports * remove `ForkedBeaconState`, use `ForkedHashedBeaconState` throughout (less data shuffling between the types) * fix several cases of states being stored on stack in tests, causing random failures on some platforms * remove reading json support from ncli - this should be ported to the rest json reading instead (doesn't currently work because stack sizes)
2021-11-05 07:34:34 +00:00
proc dump*(dir: string, v: ForkyHashedBeaconState, blck: BlockRef) =
Implement split preset/config support (#2710) * Implement split preset/config support This is the initial bulk refactor to introduce runtime config values in a number of places, somewhat replacing the existing mechanism of loading network metadata. It still needs more work, this is the initial refactor that introduces runtime configuration in some of the places that need it. The PR changes the way presets and constants work, to match the spec. In particular, a "preset" now refers to the compile-time configuration while a "cfg" or "RuntimeConfig" is the dynamic part. A single binary can support either mainnet or minimal, but not both. Support for other presets has been removed completely (can be readded, in case there's need). There's a number of outstanding tasks: * `SECONDS_PER_SLOT` still needs fixing * loading custom runtime configs needs redoing * checking constants against YAML file * yeerongpilly support `build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG` * load fork epoch from config * fix fork digest sent in status * nicer error string for request failures * fix tools * one more * fixup * fixup * fixup * use "standard" network definition folder in local testnet Files are loaded from their standard locations, including genesis etc, to conform to the format used in the `eth2-networks` repo. * fix launch scripts, allow unknown config values * fix base config of rest test * cleanups * bundle mainnet config using common loader * fix spec links and names * only include supported preset in binary * drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 13:01:38 +00:00
mixin saveFile
Small fixes (#1165) * random fixes * create dump dir on startup * don't crash on failure to write dump * fix a few `uint64` instances being used when indexing arrays - this should be a compile error but isn't due to compiler bugs * fix standalone test_block_pool compilation * add signed block processing in ncli * reuse cache entry instead of allocating a new one * allow for small clock disparities when validating blocks
2020-06-12 16:43:20 +00:00
logErrors:
SSZ.saveFile(
dir / &"state-{v.data.slot}-{shortLog(blck.root)}-{shortLog(v.root)}.ssz",
v.data)
sync fixes (#1005) * sync fixes * fix Status message finalized info * work around sync starting before initial status exchange * don't fail block on deposit signature check failure (fixes #989) * print ForkDigest and Version nicely * dump incoming blocks * fix crash when libp2p peer connection is closed * update chunk size to 16 to work around missing blocks when syncing * bump libp2p * bump libp2p * better deposit skip message
2020-05-11 18:08:52 +00:00
altair fork handling cleanups (#3050) * fix stack overflow crash in REST/debug/getStateV2 * introduce `ForkyXxx` for generic type matching of `Xxx` across branches (SomeHashedBeaconState -> ForkyHashedBeaconState et al) - `Some` is already used for other types of type classes * consolidate function naming in BeaconChainDB, use some generics * import `forks.nim` from other spec modules and move `Forked*` helpers around to resolve circular imports * remove `ForkedBeaconState`, use `ForkedHashedBeaconState` throughout (less data shuffling between the types) * fix several cases of states being stored on stack in tests, causing random failures on some platforms * remove reading json support from ncli - this should be ported to the rest json reading instead (doesn't currently work because stack sizes)
2021-11-05 07:34:34 +00:00
proc dump*(dir: string, v: ForkyHashedBeaconState) =
Implement split preset/config support (#2710) * Implement split preset/config support This is the initial bulk refactor to introduce runtime config values in a number of places, somewhat replacing the existing mechanism of loading network metadata. It still needs more work, this is the initial refactor that introduces runtime configuration in some of the places that need it. The PR changes the way presets and constants work, to match the spec. In particular, a "preset" now refers to the compile-time configuration while a "cfg" or "RuntimeConfig" is the dynamic part. A single binary can support either mainnet or minimal, but not both. Support for other presets has been removed completely (can be readded, in case there's need). There's a number of outstanding tasks: * `SECONDS_PER_SLOT` still needs fixing * loading custom runtime configs needs redoing * checking constants against YAML file * yeerongpilly support `build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG` * load fork epoch from config * fix fork digest sent in status * nicer error string for request failures * fix tools * one more * fixup * fixup * fixup * use "standard" network definition folder in local testnet Files are loaded from their standard locations, including genesis etc, to conform to the format used in the `eth2-networks` repo. * fix launch scripts, allow unknown config values * fix base config of rest test * cleanups * bundle mainnet config using common loader * fix spec links and names * only include supported preset in binary * drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 13:01:38 +00:00
mixin saveFile
Small fixes (#1165) * random fixes * create dump dir on startup * don't crash on failure to write dump * fix a few `uint64` instances being used when indexing arrays - this should be a compile error but isn't due to compiler bugs * fix standalone test_block_pool compilation * add signed block processing in ncli * reuse cache entry instead of allocating a new one * allow for small clock disparities when validating blocks
2020-06-12 16:43:20 +00:00
logErrors:
SSZ.saveFile(
dir / &"state-{v.data.slot}-{shortLog(v.root)}.ssz",
v.data)
some whole-file copies from altair branch (#2728) * some whole-file copies from altair branch * rpc/node_api and rpc/node_rest_api also need to be copied * remove new sync committee-related functionality
2021-07-19 11:58:30 +00:00
proc dump*(dir: string, v: SyncCommitteeMessage, validator: ValidatorPubKey) =
logErrors:
SSZ.saveFile(dir / &"sync-committee-msg-{v.slot}-{shortLog(validator)}.ssz", v)