nimbus-eth1/premix/hunter.nim

155 lines
4.5 KiB
Nim
Raw Normal View History

2019-01-15 08:55:39 +00:00
import
std/[json, tables, hashes],
2022-12-02 04:39:12 +00:00
eth/trie/trie_defs,
stint, stew/byteutils, chronicles,
2022-12-02 04:39:12 +00:00
../nimbus/[vm_state, vm_types],
../nimbus/utils/utils,
../nimbus/tracer,
Unified database frontend integration (#1670) * Nimbus folder environment update details: * Integrated `CoreDbRef` for the sources in the `nimbus` sub-folder. * The `nimbus` program does not compile yet as it needs the updates in the parallel `stateless` sub-folder. * Stateless environment update details: * Integrated `CoreDbRef` for the sources in the `stateless` sub-folder. * The `nimbus` program compiles now. * Premix environment update details: * Integrated `CoreDbRef` for the sources in the `premix` sub-folder. * Fluffy environment update details: * Integrated `CoreDbRef` for the sources in the `fluffy` sub-folder. * Tools environment update details: * Integrated `CoreDbRef` for the sources in the `tools` sub-folder. * Nodocker environment update details: * Integrated `CoreDbRef` for the sources in the `hive_integration/nodocker` sub-folder. * Tests environment update details: * Integrated `CoreDbRef` for the sources in the `tests` sub-folder. * The unit tests compile and run cleanly now. * Generalise `CoreDbRef` to any `select_backend` supported database why: Generalisation was just missed due to overcoming some compiler oddity which was tied to rocksdb for testing. * Suppress compiler warning for `newChainDB()` why: Warning was added to this function which must be wrapped so that any `CatchableError` is re-raised as `Defect`. * Split off persistent `CoreDbRef` constructor into separate file why: This allows to compile a memory only database version without linking the backend library. * Use memory `CoreDbRef` database by default detail: Persistent DB constructor needs to import `db/core_db/persistent why: Most tests use memory DB anyway. This avoids linking `-lrocksdb` or any other backend by default. * fix `toLegacyBackend()` availability check why: got garbled after memory/persistent split. * Clarify raw access to MPT for snap sync handler why: Logically, `kvt` is not the raw access for the hexary trie (although this holds for the legacy database)
2023-08-04 11:10:09 +00:00
../nimbus/db/[core_db, state_db],
2022-12-02 04:39:12 +00:00
../nimbus/core/executor,
../nimbus/common/common,
"."/[configuration, downloader, parser, premixcore]
2019-07-07 10:12:01 +00:00
2019-01-15 08:55:39 +00:00
const
emptyCodeHash = blankStringHash
Unified database frontend integration (#1670) * Nimbus folder environment update details: * Integrated `CoreDbRef` for the sources in the `nimbus` sub-folder. * The `nimbus` program does not compile yet as it needs the updates in the parallel `stateless` sub-folder. * Stateless environment update details: * Integrated `CoreDbRef` for the sources in the `stateless` sub-folder. * The `nimbus` program compiles now. * Premix environment update details: * Integrated `CoreDbRef` for the sources in the `premix` sub-folder. * Fluffy environment update details: * Integrated `CoreDbRef` for the sources in the `fluffy` sub-folder. * Tools environment update details: * Integrated `CoreDbRef` for the sources in the `tools` sub-folder. * Nodocker environment update details: * Integrated `CoreDbRef` for the sources in the `hive_integration/nodocker` sub-folder. * Tests environment update details: * Integrated `CoreDbRef` for the sources in the `tests` sub-folder. * The unit tests compile and run cleanly now. * Generalise `CoreDbRef` to any `select_backend` supported database why: Generalisation was just missed due to overcoming some compiler oddity which was tied to rocksdb for testing. * Suppress compiler warning for `newChainDB()` why: Warning was added to this function which must be wrapped so that any `CatchableError` is re-raised as `Defect`. * Split off persistent `CoreDbRef` constructor into separate file why: This allows to compile a memory only database version without linking the backend library. * Use memory `CoreDbRef` database by default detail: Persistent DB constructor needs to import `db/core_db/persistent why: Most tests use memory DB anyway. This avoids linking `-lrocksdb` or any other backend by default. * fix `toLegacyBackend()` availability check why: got garbled after memory/persistent split. * Clarify raw access to MPT for snap sync handler why: Logically, `kvt` is not the raw access for the hexary trie (although this holds for the legacy database)
2023-08-04 11:10:09 +00:00
proc store(memoryDB: CoreDbRef, branch: JsonNode) =
2019-01-15 08:55:39 +00:00
for p in branch:
let rlp = hexToSeqByte(p.getStr)
2019-03-07 15:53:09 +00:00
let hash = keccakHash(rlp)
Unified database frontend integration (#1670) * Nimbus folder environment update details: * Integrated `CoreDbRef` for the sources in the `nimbus` sub-folder. * The `nimbus` program does not compile yet as it needs the updates in the parallel `stateless` sub-folder. * Stateless environment update details: * Integrated `CoreDbRef` for the sources in the `stateless` sub-folder. * The `nimbus` program compiles now. * Premix environment update details: * Integrated `CoreDbRef` for the sources in the `premix` sub-folder. * Fluffy environment update details: * Integrated `CoreDbRef` for the sources in the `fluffy` sub-folder. * Tools environment update details: * Integrated `CoreDbRef` for the sources in the `tools` sub-folder. * Nodocker environment update details: * Integrated `CoreDbRef` for the sources in the `hive_integration/nodocker` sub-folder. * Tests environment update details: * Integrated `CoreDbRef` for the sources in the `tests` sub-folder. * The unit tests compile and run cleanly now. * Generalise `CoreDbRef` to any `select_backend` supported database why: Generalisation was just missed due to overcoming some compiler oddity which was tied to rocksdb for testing. * Suppress compiler warning for `newChainDB()` why: Warning was added to this function which must be wrapped so that any `CatchableError` is re-raised as `Defect`. * Split off persistent `CoreDbRef` constructor into separate file why: This allows to compile a memory only database version without linking the backend library. * Use memory `CoreDbRef` database by default detail: Persistent DB constructor needs to import `db/core_db/persistent why: Most tests use memory DB anyway. This avoids linking `-lrocksdb` or any other backend by default. * fix `toLegacyBackend()` availability check why: got garbled after memory/persistent split. * Clarify raw access to MPT for snap sync handler why: Logically, `kvt` is not the raw access for the hexary trie (although this holds for the legacy database)
2023-08-04 11:10:09 +00:00
memoryDB.kvt.put(hash.data, rlp)
2019-01-15 08:55:39 +00:00
proc parseAddress(address: string): EthAddress =
hexToByteArray(address, result)
2022-04-08 04:54:11 +00:00
proc parseU256(val: string): UInt256 =
2019-01-15 08:55:39 +00:00
UInt256.fromHex(val)
Unified database frontend integration (#1670) * Nimbus folder environment update details: * Integrated `CoreDbRef` for the sources in the `nimbus` sub-folder. * The `nimbus` program does not compile yet as it needs the updates in the parallel `stateless` sub-folder. * Stateless environment update details: * Integrated `CoreDbRef` for the sources in the `stateless` sub-folder. * The `nimbus` program compiles now. * Premix environment update details: * Integrated `CoreDbRef` for the sources in the `premix` sub-folder. * Fluffy environment update details: * Integrated `CoreDbRef` for the sources in the `fluffy` sub-folder. * Tools environment update details: * Integrated `CoreDbRef` for the sources in the `tools` sub-folder. * Nodocker environment update details: * Integrated `CoreDbRef` for the sources in the `hive_integration/nodocker` sub-folder. * Tests environment update details: * Integrated `CoreDbRef` for the sources in the `tests` sub-folder. * The unit tests compile and run cleanly now. * Generalise `CoreDbRef` to any `select_backend` supported database why: Generalisation was just missed due to overcoming some compiler oddity which was tied to rocksdb for testing. * Suppress compiler warning for `newChainDB()` why: Warning was added to this function which must be wrapped so that any `CatchableError` is re-raised as `Defect`. * Split off persistent `CoreDbRef` constructor into separate file why: This allows to compile a memory only database version without linking the backend library. * Use memory `CoreDbRef` database by default detail: Persistent DB constructor needs to import `db/core_db/persistent why: Most tests use memory DB anyway. This avoids linking `-lrocksdb` or any other backend by default. * fix `toLegacyBackend()` availability check why: got garbled after memory/persistent split. * Clarify raw access to MPT for snap sync handler why: Logically, `kvt` is not the raw access for the hexary trie (although this holds for the legacy database)
2023-08-04 11:10:09 +00:00
proc prepareBlockEnv(parent: BlockHeader, thisBlock: Block): CoreDbRef =
2019-01-15 08:55:39 +00:00
var
accounts = requestPostState(thisBlock)
Unified database frontend integration (#1670) * Nimbus folder environment update details: * Integrated `CoreDbRef` for the sources in the `nimbus` sub-folder. * The `nimbus` program does not compile yet as it needs the updates in the parallel `stateless` sub-folder. * Stateless environment update details: * Integrated `CoreDbRef` for the sources in the `stateless` sub-folder. * The `nimbus` program compiles now. * Premix environment update details: * Integrated `CoreDbRef` for the sources in the `premix` sub-folder. * Fluffy environment update details: * Integrated `CoreDbRef` for the sources in the `fluffy` sub-folder. * Tools environment update details: * Integrated `CoreDbRef` for the sources in the `tools` sub-folder. * Nodocker environment update details: * Integrated `CoreDbRef` for the sources in the `hive_integration/nodocker` sub-folder. * Tests environment update details: * Integrated `CoreDbRef` for the sources in the `tests` sub-folder. * The unit tests compile and run cleanly now. * Generalise `CoreDbRef` to any `select_backend` supported database why: Generalisation was just missed due to overcoming some compiler oddity which was tied to rocksdb for testing. * Suppress compiler warning for `newChainDB()` why: Warning was added to this function which must be wrapped so that any `CatchableError` is re-raised as `Defect`. * Split off persistent `CoreDbRef` constructor into separate file why: This allows to compile a memory only database version without linking the backend library. * Use memory `CoreDbRef` database by default detail: Persistent DB constructor needs to import `db/core_db/persistent why: Most tests use memory DB anyway. This avoids linking `-lrocksdb` or any other backend by default. * fix `toLegacyBackend()` availability check why: got garbled after memory/persistent split. * Clarify raw access to MPT for snap sync handler why: Logically, `kvt` is not the raw access for the hexary trie (although this holds for the legacy database)
2023-08-04 11:10:09 +00:00
memoryDB = newCoreDbRef LegacyDbMemory
2019-01-15 08:55:39 +00:00
accountDB = newAccountStateDB(memoryDB, parent.stateRoot, false)
parentNumber = %(parent.blockNumber.prefixHex)
for address, account in accounts:
updateAccount(address, account, parent.blockNumber)
let
accountProof = account["accountProof"]
storageProof = account["storageProof"]
address = parseAddress(address)
acc = parseAccount(account)
memoryDB.store(accountProof)
accountDB.setAccount(address, acc)
for storage in storageProof:
let
key = parseU256(storage["key"].getStr)
val = parseU256(storage["value"].getStr)
proof = storage["proof"]
memoryDB.store(proof)
accountDB.setStorage(address, key, val)
if acc.codeHash != emptyCodeHash:
let codeStr = request("eth_getCode", %[%address.prefixHex, parentNumber])
let code = hexToSeqByte(codeStr.getStr)
2019-01-15 08:55:39 +00:00
accountDB.setCode(address, code)
accountDB.setAccount(address, acc)
2019-01-15 08:55:39 +00:00
result = memoryDB
2019-02-22 08:52:53 +00:00
type
HunterVMState = ref object of BaseVMState
headers: Table[BlockNumber, BlockHeader]
2022-04-08 04:54:11 +00:00
proc hash*(x: UInt256): Hash =
2019-02-22 08:52:53 +00:00
result = hash(x.toByteArrayBE)
2022-12-02 04:39:12 +00:00
proc new(T: type HunterVMState; parent, header: BlockHeader, com: CommonRef): T =
2019-02-22 08:52:53 +00:00
new result
2022-12-02 04:39:12 +00:00
result.init(parent, header, com)
2019-02-22 08:52:53 +00:00
result.headers = initTable[BlockNumber, BlockHeader]()
method getAncestorHash*(vmState: HunterVMState, blockNumber: BlockNumber): Hash256 {.gcsafe.} =
if blockNumber in vmState.headers:
result = vmState.headers[blockNumber].hash
else:
let data = requestHeader(blockNumber)
let header = parseBlockHeader(data)
result = header.hash
vmState.headers[blockNumber] = header
Unified database frontend integration (#1670) * Nimbus folder environment update details: * Integrated `CoreDbRef` for the sources in the `nimbus` sub-folder. * The `nimbus` program does not compile yet as it needs the updates in the parallel `stateless` sub-folder. * Stateless environment update details: * Integrated `CoreDbRef` for the sources in the `stateless` sub-folder. * The `nimbus` program compiles now. * Premix environment update details: * Integrated `CoreDbRef` for the sources in the `premix` sub-folder. * Fluffy environment update details: * Integrated `CoreDbRef` for the sources in the `fluffy` sub-folder. * Tools environment update details: * Integrated `CoreDbRef` for the sources in the `tools` sub-folder. * Nodocker environment update details: * Integrated `CoreDbRef` for the sources in the `hive_integration/nodocker` sub-folder. * Tests environment update details: * Integrated `CoreDbRef` for the sources in the `tests` sub-folder. * The unit tests compile and run cleanly now. * Generalise `CoreDbRef` to any `select_backend` supported database why: Generalisation was just missed due to overcoming some compiler oddity which was tied to rocksdb for testing. * Suppress compiler warning for `newChainDB()` why: Warning was added to this function which must be wrapped so that any `CatchableError` is re-raised as `Defect`. * Split off persistent `CoreDbRef` constructor into separate file why: This allows to compile a memory only database version without linking the backend library. * Use memory `CoreDbRef` database by default detail: Persistent DB constructor needs to import `db/core_db/persistent why: Most tests use memory DB anyway. This avoids linking `-lrocksdb` or any other backend by default. * fix `toLegacyBackend()` availability check why: got garbled after memory/persistent split. * Clarify raw access to MPT for snap sync handler why: Logically, `kvt` is not the raw access for the hexary trie (although this holds for the legacy database)
2023-08-04 11:10:09 +00:00
proc putAncestorsIntoDB(vmState: HunterVMState, db: CoreDbRef) =
2019-02-22 08:52:53 +00:00
for header in vmState.headers.values:
db.addBlockNumberToHashLookup(header)
2022-04-08 04:54:11 +00:00
proc huntProblematicBlock(blockNumber: UInt256): ValidationResult =
2019-01-15 08:55:39 +00:00
let
# prepare needed state from previous block
parentNumber = blockNumber - 1
thisBlock = requestBlock(blockNumber)
parentBlock = requestBlock(parentNumber)
memoryDB = prepareBlockEnv(parentBlock.header, thisBlock)
# try to execute current block
2022-12-02 04:39:12 +00:00
com = CommonRef.new(memoryDB, false)
2019-01-15 08:55:39 +00:00
2022-12-02 04:39:12 +00:00
discard com.db.setHead(parentBlock.header, true)
2019-01-15 08:55:39 +00:00
let transaction = memoryDB.beginTransaction()
defer: transaction.dispose()
2019-01-15 08:55:39 +00:00
let
2022-12-02 04:39:12 +00:00
vmState = HunterVMState.new(parentBlock.header, thisBlock.header, com)
Fearture/poa clique tuning (#765) * Provide API details: API is bundled via clique.nim. * Set extraValidation as default for PoA chains why: This triggers consensus verification and an update of the list of authorised signers. These signers are integral part of the PoA block chain. todo: Option argument to control validation for the nimbus binary. * Fix snapshot state block number why: Using sub-sequence here, so the len() function was wrong. * Optional start where block verification begins why: Can speed up time building loading initial parts of block chain. For PoA, this allows to prove & test that authorised signers can be (correctly) calculated starting at any point on the block chain. todo: On Goerli around blocks #193537..#197568, processing time increases disproportionally -- needs to be understand * For Clique test, get old grouping back (7 transactions per log entry) why: Forgot to change back after troubleshooting * Fix field/function/module-name misunderstanding why: Make compilation work * Use eth_types.blockHash() rather than utils.hash() in Clique modules why: Prefer lib module * Dissolve snapshot_misc.nim details: .. into clique_verify.nim (the other source file clique_unused.nim is inactive) * Hide unused AsyncLock in Clique descriptor details: Unused here but was part of the Go reference implementation * Remove fakeDiff flag from Clique descriptor details: This flag was a kludge in the Go reference implementation used for the canonical tests. The tests have been adapted so there is no need for the fakeDiff flag and its implementation. * Not observing minimum distance from epoch sync point why: For compiling PoA state, the go implementation will walk back to the epoch header with at least 90000 blocks apart from the current header in the absence of other synchronisation points. Here just the nearest epoch header is used. The assumption is that all the checkpoints before have been vetted already regardless of the current branch. details: The behaviour of using the nearest vs the minimum distance epoch is controlled by a flag and can be changed at run time. * Analysing processing time (patch adds some debugging/visualisation support) why: At the first half million blocks of the Goerli replay, blocks on the interval #194854..#196224 take exceptionally long to process, but not due to PoA processing. details: It turns out that much time is spent in p2p/excecutor.processBlock() where the elapsed transaction execution time is significantly greater for many of these blocks. Between the 1371 blocks #194854..#196224 there are 223 blocks with more than 1/2 seconds execution time whereas there are only 4 such blocks before and 13 such after this range up to #504192. * fix debugging symbol in clique_desc (causes CI failing) * Fixing canonical reference tests why: Two errors were introduced earlier but ovelooked: 1. "Remove fakeDiff flag .." patch was incomplete 2. "Not observing minimum distance .." introduced problem w/tests 23/24 details: Fixing 2. needed to revert the behaviour by setting the applySnapsMinBacklog flag for the Clique descriptor. Also a new test was added to lock the new behaviour. * Remove cruft why: Clique/PoA processing was intended to take place somewhere in executor/process_block.processBlock() but was decided later to run from chain/persist_block.persistBlock() instead. * Update API comment * ditto
2021-07-30 14:06:51 +00:00
validationResult = vmState.processBlockNotPoA(thisBlock.header, thisBlock.body)
2019-01-15 08:55:39 +00:00
if validationResult != ValidationResult.OK:
transaction.rollback()
2022-12-02 04:39:12 +00:00
putAncestorsIntoDB(vmState, com.db)
dumpDebuggingMetaData(com, thisBlock.header, thisBlock.body, vmState, false)
2019-01-15 08:55:39 +00:00
result = validationResult
2020-07-21 06:15:06 +00:00
proc main() {.used.} =
2019-01-15 08:55:39 +00:00
let conf = getConfiguration()
if conf.head == 0.u256:
echo "please specify the starting block with `--head:blockNumber`"
quit(QuitFailure)
if conf.maxBlocks == 0:
echo "please specify the number of problematic blocks you want to hunt with `--maxBlocks:number`"
quit(QuitFailure)
var
2022-04-08 04:54:11 +00:00
problematicBlocks = newSeq[UInt256]()
2019-01-15 08:55:39 +00:00
blockNumber = conf.head
while true:
echo blockNumber
if huntProblematicBlock(blockNumber) != ValidationResult.OK:
echo "shot down problematic block: ", blockNumber
problematicBlocks.add blockNumber
blockNumber = blockNumber + 1
if problematicBlocks.len >= conf.maxBlocks:
echo "Problematic blocks: ", problematicBlocks
break
when isMainModule:
var message: string
## Processing command line arguments
if processArguments(message) != Success:
echo message
quit(QuitFailure)
else:
if len(message) > 0:
echo message
quit(QuitSuccess)
try:
main()
except:
echo getCurrentExceptionMsg()