status-im
/
nimbus-eth1
mirror of https://github.com/status-im/nimbus-eth1.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
nimbus-eth1/tests/test_sync_snap/test_pivot.nim

94 lines
3.1 KiB
Nim
Raw Normal View History

Declutter snap sync unit tests (#1444) * Extracted RocksDB timing unit tests into separate file why: make space for more in main module :) * Extracted `inspectionRunner()` unit tests into separate file why: make space for more in main module :) * Extracted `storagesRunner()` unit tests into separate file why: make space for more in main module :) * Extracted pivot checkpoint store/retrieval unit tests into separate file why: make space for more in main module :) * Extract helper functions into separate source file * Extracted account import unit tests into separate file why: make space for more in main module :) * Rename `test_decompose()` => `test_NodeRangeDecompose()` why: There will be more functions with `test_NodeRange` prefix.
2023-01-23 16:09:12 +00:00
# Nimbus - Types, data structures and shared utilities used in network sync
#
# Copyright (c) 2018-2021 Status Research & Development GmbH
# Licensed under either of
# * Apache License, version 2.0, ([LICENSE-APACHE](LICENSE-APACHE) or
# http://www.apache.org/licenses/LICENSE-2.0)
# * MIT license ([LICENSE-MIT](LICENSE-MIT) or
# http://opensource.org/licenses/MIT)
# at your option. This file may not be copied, modified, or
# distributed except according to those terms.
## Snap sync components tester and TDD environment
import
eth/[common, p2p],
unittest2,
Unified database frontend integration (#1670) * Nimbus folder environment update details: * Integrated `CoreDbRef` for the sources in the `nimbus` sub-folder. * The `nimbus` program does not compile yet as it needs the updates in the parallel `stateless` sub-folder. * Stateless environment update details: * Integrated `CoreDbRef` for the sources in the `stateless` sub-folder. * The `nimbus` program compiles now. * Premix environment update details: * Integrated `CoreDbRef` for the sources in the `premix` sub-folder. * Fluffy environment update details: * Integrated `CoreDbRef` for the sources in the `fluffy` sub-folder. * Tools environment update details: * Integrated `CoreDbRef` for the sources in the `tools` sub-folder. * Nodocker environment update details: * Integrated `CoreDbRef` for the sources in the `hive_integration/nodocker` sub-folder. * Tests environment update details: * Integrated `CoreDbRef` for the sources in the `tests` sub-folder. * The unit tests compile and run cleanly now. * Generalise `CoreDbRef` to any `select_backend` supported database why: Generalisation was just missed due to overcoming some compiler oddity which was tied to rocksdb for testing. * Suppress compiler warning for `newChainDB()` why: Warning was added to this function which must be wrapped so that any `CatchableError` is re-raised as `Defect`. * Split off persistent `CoreDbRef` constructor into separate file why: This allows to compile a memory only database version without linking the backend library. * Use memory `CoreDbRef` database by default detail: Persistent DB constructor needs to import `db/core_db/persistent why: Most tests use memory DB anyway. This avoids linking `-lrocksdb` or any other backend by default. * fix `toLegacyBackend()` availability check why: got garbled after memory/persistent split. * Clarify raw access to MPT for snap sync handler why: Logically, `kvt` is not the raw access for the hexary trie (although this holds for the legacy database)
2023-08-04 11:10:09 +00:00
../../nimbus/db/core_db,
Declutter snap sync unit tests (#1444) * Extracted RocksDB timing unit tests into separate file why: make space for more in main module :) * Extracted `inspectionRunner()` unit tests into separate file why: make space for more in main module :) * Extracted `storagesRunner()` unit tests into separate file why: make space for more in main module :) * Extracted pivot checkpoint store/retrieval unit tests into separate file why: make space for more in main module :) * Extract helper functions into separate source file * Extracted account import unit tests into separate file why: make space for more in main module :) * Rename `test_decompose()` => `test_NodeRangeDecompose()` why: There will be more functions with `test_NodeRange` prefix.
2023-01-23 16:09:12 +00:00
../../nimbus/sync/snap/range_desc,
../../nimbus/sync/snap/worker/db/[snapdb_desc, snapdb_pivot]
Silence compiler gossip after nim upgrade cont3 (#1466) * Removed some Windows specific unit test annoyances details: + Short put()/get() cycles on persistent database have a race condition with vendor rocksdb. On a specific (and slow) qemu/win7 a 50ms `sleep()` in between will mostly do the job (i.e. unless heavy CPU load.) This issue was not observed on github/ci. + Removed annoyances when qemu/Win7 keeps the rocksdb database files locked even after closing the db. The problem is solved by strictly using fresh names for each test. No assumption made to be able to properly clean up. This issue was not observed on github/ci. * Silence some compiler gossip -- part 7, misc/non(sync or graphql) details: Adding some missing exception annotation
2023-02-14 20:27:17 +00:00
when defined(windows):
import std/os
Declutter snap sync unit tests (#1444) * Extracted RocksDB timing unit tests into separate file why: make space for more in main module :) * Extracted `inspectionRunner()` unit tests into separate file why: make space for more in main module :) * Extracted `storagesRunner()` unit tests into separate file why: make space for more in main module :) * Extracted pivot checkpoint store/retrieval unit tests into separate file why: make space for more in main module :) * Extract helper functions into separate source file * Extracted account import unit tests into separate file why: make space for more in main module :) * Rename `test_decompose()` => `test_NodeRangeDecompose()` why: There will be more functions with `test_NodeRange` prefix.
2023-01-23 16:09:12 +00:00
# ------------------------------------------------------------------------------
# Public test function
# ------------------------------------------------------------------------------
proc test_pivotStoreRead*(
accKeys: seq[NodeKey];
Unified database frontend integration (#1670) * Nimbus folder environment update details: * Integrated `CoreDbRef` for the sources in the `nimbus` sub-folder. * The `nimbus` program does not compile yet as it needs the updates in the parallel `stateless` sub-folder. * Stateless environment update details: * Integrated `CoreDbRef` for the sources in the `stateless` sub-folder. * The `nimbus` program compiles now. * Premix environment update details: * Integrated `CoreDbRef` for the sources in the `premix` sub-folder. * Fluffy environment update details: * Integrated `CoreDbRef` for the sources in the `fluffy` sub-folder. * Tools environment update details: * Integrated `CoreDbRef` for the sources in the `tools` sub-folder. * Nodocker environment update details: * Integrated `CoreDbRef` for the sources in the `hive_integration/nodocker` sub-folder. * Tests environment update details: * Integrated `CoreDbRef` for the sources in the `tests` sub-folder. * The unit tests compile and run cleanly now. * Generalise `CoreDbRef` to any `select_backend` supported database why: Generalisation was just missed due to overcoming some compiler oddity which was tied to rocksdb for testing. * Suppress compiler warning for `newChainDB()` why: Warning was added to this function which must be wrapped so that any `CatchableError` is re-raised as `Defect`. * Split off persistent `CoreDbRef` constructor into separate file why: This allows to compile a memory only database version without linking the backend library. * Use memory `CoreDbRef` database by default detail: Persistent DB constructor needs to import `db/core_db/persistent why: Most tests use memory DB anyway. This avoids linking `-lrocksdb` or any other backend by default. * fix `toLegacyBackend()` availability check why: got garbled after memory/persistent split. * Clarify raw access to MPT for snap sync handler why: Logically, `kvt` is not the raw access for the hexary trie (although this holds for the legacy database)
2023-08-04 11:10:09 +00:00
cdb: CoreDbRef;
Declutter snap sync unit tests (#1444) * Extracted RocksDB timing unit tests into separate file why: make space for more in main module :) * Extracted `inspectionRunner()` unit tests into separate file why: make space for more in main module :) * Extracted `storagesRunner()` unit tests into separate file why: make space for more in main module :) * Extracted pivot checkpoint store/retrieval unit tests into separate file why: make space for more in main module :) * Extract helper functions into separate source file * Extracted account import unit tests into separate file why: make space for more in main module :) * Rename `test_decompose()` => `test_NodeRangeDecompose()` why: There will be more functions with `test_NodeRange` prefix.
2023-01-23 16:09:12 +00:00
) =
## Storing/retrieving items on persistent pivot/checkpoint registry
let
dbBase = SnapDbRef.init(cdb)
processed = @[(1.to(NodeTag),2.to(NodeTag)),
(4.to(NodeTag),5.to(NodeTag)),
(6.to(NodeTag),7.to(NodeTag))]
slotAccounts = seq[NodeKey].default
Snap sync interval range extractor (#1449) * Update comments and test noise * Fix boundary proofs why: Where neither used in production, nor unit tested. For production, other methods apply to test leaf range integrity directly based of the proof nodes. * Added `hexary_range()`: interval range + proof extractor details: + Will be used for `snap/1` protocol handler + Unit tests added (also for testing left boundary proof) todo: Need to verify completeness of proof nodes * Reduce some nim 1.6 compiler noise * Stop unit test gossip for ci tests
2023-01-30 17:50:58 +00:00
for n in 0 ..< accKeys.len:
let w = accKeys[n]
Prepare snap client for continuing with full sync (#1534) * Somewhat tighten error handling why: Zombie state is invoked when the current peer turns out to be useless for further communication. While there is a chance to further talk to a peer about another topic (aka healing) after some protocol failure, it makes no sense to do so after a network problem. The latter state is explained bu the `peerDegraded` flag that goes together with the `zombie` state flag. A degraded peer is dropped immediately. * Remove `--sync-mode=snapCtx` option, always start snap in recovery mode why: No need for a snap sync option without recovery mode, can be achieved by deleting the database. * Code cosmetics, typos, prettify logging, debugging helper, etc. * Split off snap sync sub-mode handler into separate modules details: The original `worker.nim` source has become a multiplexer for several snap sync sub-modes `full` and `snap`. The source modules of the incarnations of a particular sync sub-mode are places into the `worker/play` directory. * Update ticker for snap and full sync logging
2023-04-06 19:42:07 +00:00
check dbBase.pivotSaveDB(
Declutter snap sync unit tests (#1444) * Extracted RocksDB timing unit tests into separate file why: make space for more in main module :) * Extracted `inspectionRunner()` unit tests into separate file why: make space for more in main module :) * Extracted `storagesRunner()` unit tests into separate file why: make space for more in main module :) * Extracted pivot checkpoint store/retrieval unit tests into separate file why: make space for more in main module :) * Extract helper functions into separate source file * Extracted account import unit tests into separate file why: make space for more in main module :) * Rename `test_decompose()` => `test_NodeRangeDecompose()` why: There will be more functions with `test_NodeRange` prefix.
2023-01-23 16:09:12 +00:00
SnapDbPivotRegistry(
header: BlockHeader(stateRoot: w.to(Hash256)),
nAccounts: n.uint64,
nSlotLists: n.uint64,
processed: processed,
slotAccounts: slotAccounts)).isOk
Silence compiler gossip after nim upgrade cont3 (#1466) * Removed some Windows specific unit test annoyances details: + Short put()/get() cycles on persistent database have a race condition with vendor rocksdb. On a specific (and slow) qemu/win7 a 50ms `sleep()` in between will mostly do the job (i.e. unless heavy CPU load.) This issue was not observed on github/ci. + Removed annoyances when qemu/Win7 keeps the rocksdb database files locked even after closing the db. The problem is solved by strictly using fresh names for each test. No assumption made to be able to properly clean up. This issue was not observed on github/ci. * Silence some compiler gossip -- part 7, misc/non(sync or graphql) details: Adding some missing exception annotation
2023-02-14 20:27:17 +00:00
when defined(windows):
# There might be a race condition on Windows (seen on qemu/win7)
sleep(50)
Declutter snap sync unit tests (#1444) * Extracted RocksDB timing unit tests into separate file why: make space for more in main module :) * Extracted `inspectionRunner()` unit tests into separate file why: make space for more in main module :) * Extracted `storagesRunner()` unit tests into separate file why: make space for more in main module :) * Extracted pivot checkpoint store/retrieval unit tests into separate file why: make space for more in main module :) * Extract helper functions into separate source file * Extracted account import unit tests into separate file why: make space for more in main module :) * Rename `test_decompose()` => `test_NodeRangeDecompose()` why: There will be more functions with `test_NodeRange` prefix.
2023-01-23 16:09:12 +00:00
# verify latest state root
block:
Prepare snap client for continuing with full sync (#1534) * Somewhat tighten error handling why: Zombie state is invoked when the current peer turns out to be useless for further communication. While there is a chance to further talk to a peer about another topic (aka healing) after some protocol failure, it makes no sense to do so after a network problem. The latter state is explained bu the `peerDegraded` flag that goes together with the `zombie` state flag. A degraded peer is dropped immediately. * Remove `--sync-mode=snapCtx` option, always start snap in recovery mode why: No need for a snap sync option without recovery mode, can be achieved by deleting the database. * Code cosmetics, typos, prettify logging, debugging helper, etc. * Split off snap sync sub-mode handler into separate modules details: The original `worker.nim` source has become a multiplexer for several snap sync sub-modes `full` and `snap`. The source modules of the incarnations of a particular sync sub-mode are places into the `worker/play` directory. * Update ticker for snap and full sync logging
2023-04-06 19:42:07 +00:00
let rc = dbBase.pivotRecoverDB()
Declutter snap sync unit tests (#1444) * Extracted RocksDB timing unit tests into separate file why: make space for more in main module :) * Extracted `inspectionRunner()` unit tests into separate file why: make space for more in main module :) * Extracted `storagesRunner()` unit tests into separate file why: make space for more in main module :) * Extracted pivot checkpoint store/retrieval unit tests into separate file why: make space for more in main module :) * Extract helper functions into separate source file * Extracted account import unit tests into separate file why: make space for more in main module :) * Rename `test_decompose()` => `test_NodeRangeDecompose()` why: There will be more functions with `test_NodeRange` prefix.
2023-01-23 16:09:12 +00:00
check rc.isOk
if rc.isOk:
check rc.value.nAccounts == n.uint64
check rc.value.nSlotLists == n.uint64
check rc.value.processed == processed
Snap sync interval range extractor (#1449) * Update comments and test noise * Fix boundary proofs why: Where neither used in production, nor unit tested. For production, other methods apply to test leaf range integrity directly based of the proof nodes. * Added `hexary_range()`: interval range + proof extractor details: + Will be used for `snap/1` protocol handler + Unit tests added (also for testing left boundary proof) todo: Need to verify completeness of proof nodes * Reduce some nim 1.6 compiler noise * Stop unit test gossip for ci tests
2023-01-30 17:50:58 +00:00
# Stop gossiping (happens whith corrupted database)
if rc.value.nAccounts != n.uint64 or
rc.value.nSlotLists != n.uint64 or
rc.value.processed != processed:
return
for n in 0 ..< accKeys.len:
let w = accKeys[n]
Declutter snap sync unit tests (#1444) * Extracted RocksDB timing unit tests into separate file why: make space for more in main module :) * Extracted `inspectionRunner()` unit tests into separate file why: make space for more in main module :) * Extracted `storagesRunner()` unit tests into separate file why: make space for more in main module :) * Extracted pivot checkpoint store/retrieval unit tests into separate file why: make space for more in main module :) * Extract helper functions into separate source file * Extracted account import unit tests into separate file why: make space for more in main module :) * Rename `test_decompose()` => `test_NodeRangeDecompose()` why: There will be more functions with `test_NodeRange` prefix.
2023-01-23 16:09:12 +00:00
block:
Prepare snap client for continuing with full sync (#1534) * Somewhat tighten error handling why: Zombie state is invoked when the current peer turns out to be useless for further communication. While there is a chance to further talk to a peer about another topic (aka healing) after some protocol failure, it makes no sense to do so after a network problem. The latter state is explained bu the `peerDegraded` flag that goes together with the `zombie` state flag. A degraded peer is dropped immediately. * Remove `--sync-mode=snapCtx` option, always start snap in recovery mode why: No need for a snap sync option without recovery mode, can be achieved by deleting the database. * Code cosmetics, typos, prettify logging, debugging helper, etc. * Split off snap sync sub-mode handler into separate modules details: The original `worker.nim` source has become a multiplexer for several snap sync sub-modes `full` and `snap`. The source modules of the incarnations of a particular sync sub-mode are places into the `worker/play` directory. * Update ticker for snap and full sync logging
2023-04-06 19:42:07 +00:00
let rc = dbBase.pivotRecoverDB(w)
Declutter snap sync unit tests (#1444) * Extracted RocksDB timing unit tests into separate file why: make space for more in main module :) * Extracted `inspectionRunner()` unit tests into separate file why: make space for more in main module :) * Extracted `storagesRunner()` unit tests into separate file why: make space for more in main module :) * Extracted pivot checkpoint store/retrieval unit tests into separate file why: make space for more in main module :) * Extract helper functions into separate source file * Extracted account import unit tests into separate file why: make space for more in main module :) * Rename `test_decompose()` => `test_NodeRangeDecompose()` why: There will be more functions with `test_NodeRange` prefix.
2023-01-23 16:09:12 +00:00
check rc.isOk
if rc.isOk:
check rc.value.nAccounts == n.uint64
check rc.value.nSlotLists == n.uint64
# Update record in place
Prepare snap client for continuing with full sync (#1534) * Somewhat tighten error handling why: Zombie state is invoked when the current peer turns out to be useless for further communication. While there is a chance to further talk to a peer about another topic (aka healing) after some protocol failure, it makes no sense to do so after a network problem. The latter state is explained bu the `peerDegraded` flag that goes together with the `zombie` state flag. A degraded peer is dropped immediately. * Remove `--sync-mode=snapCtx` option, always start snap in recovery mode why: No need for a snap sync option without recovery mode, can be achieved by deleting the database. * Code cosmetics, typos, prettify logging, debugging helper, etc. * Split off snap sync sub-mode handler into separate modules details: The original `worker.nim` source has become a multiplexer for several snap sync sub-modes `full` and `snap`. The source modules of the incarnations of a particular sync sub-mode are places into the `worker/play` directory. * Update ticker for snap and full sync logging
2023-04-06 19:42:07 +00:00
check dbBase.pivotSaveDB(
Declutter snap sync unit tests (#1444) * Extracted RocksDB timing unit tests into separate file why: make space for more in main module :) * Extracted `inspectionRunner()` unit tests into separate file why: make space for more in main module :) * Extracted `storagesRunner()` unit tests into separate file why: make space for more in main module :) * Extracted pivot checkpoint store/retrieval unit tests into separate file why: make space for more in main module :) * Extract helper functions into separate source file * Extracted account import unit tests into separate file why: make space for more in main module :) * Rename `test_decompose()` => `test_NodeRangeDecompose()` why: There will be more functions with `test_NodeRange` prefix.
2023-01-23 16:09:12 +00:00
SnapDbPivotRegistry(
header: BlockHeader(stateRoot: w.to(Hash256)),
nAccounts: n.uint64,
nSlotLists: 0,
processed: @[],
slotAccounts: @[])).isOk
Silence compiler gossip after nim upgrade cont3 (#1466) * Removed some Windows specific unit test annoyances details: + Short put()/get() cycles on persistent database have a race condition with vendor rocksdb. On a specific (and slow) qemu/win7 a 50ms `sleep()` in between will mostly do the job (i.e. unless heavy CPU load.) This issue was not observed on github/ci. + Removed annoyances when qemu/Win7 keeps the rocksdb database files locked even after closing the db. The problem is solved by strictly using fresh names for each test. No assumption made to be able to properly clean up. This issue was not observed on github/ci. * Silence some compiler gossip -- part 7, misc/non(sync or graphql) details: Adding some missing exception annotation
2023-02-14 20:27:17 +00:00
when defined(windows):
# There might be a race condition on Windows (seen on qemu/win7)
sleep(50)
Declutter snap sync unit tests (#1444) * Extracted RocksDB timing unit tests into separate file why: make space for more in main module :) * Extracted `inspectionRunner()` unit tests into separate file why: make space for more in main module :) * Extracted `storagesRunner()` unit tests into separate file why: make space for more in main module :) * Extracted pivot checkpoint store/retrieval unit tests into separate file why: make space for more in main module :) * Extract helper functions into separate source file * Extracted account import unit tests into separate file why: make space for more in main module :) * Rename `test_decompose()` => `test_NodeRangeDecompose()` why: There will be more functions with `test_NodeRange` prefix.
2023-01-23 16:09:12 +00:00
block:
Prepare snap client for continuing with full sync (#1534) * Somewhat tighten error handling why: Zombie state is invoked when the current peer turns out to be useless for further communication. While there is a chance to further talk to a peer about another topic (aka healing) after some protocol failure, it makes no sense to do so after a network problem. The latter state is explained bu the `peerDegraded` flag that goes together with the `zombie` state flag. A degraded peer is dropped immediately. * Remove `--sync-mode=snapCtx` option, always start snap in recovery mode why: No need for a snap sync option without recovery mode, can be achieved by deleting the database. * Code cosmetics, typos, prettify logging, debugging helper, etc. * Split off snap sync sub-mode handler into separate modules details: The original `worker.nim` source has become a multiplexer for several snap sync sub-modes `full` and `snap`. The source modules of the incarnations of a particular sync sub-mode are places into the `worker/play` directory. * Update ticker for snap and full sync logging
2023-04-06 19:42:07 +00:00
let rc = dbBase.pivotRecoverDB(w)
Declutter snap sync unit tests (#1444) * Extracted RocksDB timing unit tests into separate file why: make space for more in main module :) * Extracted `inspectionRunner()` unit tests into separate file why: make space for more in main module :) * Extracted `storagesRunner()` unit tests into separate file why: make space for more in main module :) * Extracted pivot checkpoint store/retrieval unit tests into separate file why: make space for more in main module :) * Extract helper functions into separate source file * Extracted account import unit tests into separate file why: make space for more in main module :) * Rename `test_decompose()` => `test_NodeRangeDecompose()` why: There will be more functions with `test_NodeRange` prefix.
2023-01-23 16:09:12 +00:00
check rc.isOk
if rc.isOk:
check rc.value.nAccounts == n.uint64
check rc.value.nSlotLists == 0
check rc.value.processed == seq[(NodeTag,NodeTag)].default
# ------------------------------------------------------------------------------
# End
# ------------------------------------------------------------------------------