Jacek Sieka 7bbb0f4421
Stream blocks during import (#2937)
When running the import, currently blocks are loaded in batches into a
`seq` then passed to the importer as such.

In reality, blocks are still processed one by one, so the batching does
not offer any performance advantage. It does however require that the
client wastes memory, up to several GB, on the block sequence while
they're waiting to be processed.

This PR introduces a persister that accepts these potentially large
blocks one by one and at the same time removes a number of redundant /
unnecessary copies, assignments and resets that were slowing down the
import process in general.
2024-12-18 13:21:20 +01:00

36 lines
826 B
Nim

# Nimbus
# Copyright (c) 2021-2024 Status Research & Development GmbH
# Licensed under either of
# * Apache License, version 2.0, ([LICENSE-APACHE](LICENSE-APACHE))
# * MIT license ([LICENSE-MIT](LICENSE-MIT))
# at your option.
# This file may not be copied, modified, or distributed except according to
# those terms.
import
std/[times],
chronos,
taskpools,
"."/[rpc_tests, test_env],
../sim_utils
proc runRpcTest() =
var stat: SimStat
let taskPool = Taskpool.new()
let start = getTime()
for x in testList:
try:
let env = setupEnv(taskPool)
let status = waitFor x.run(env)
env.stopEnv()
stat.inc(x.name, status)
except ValueError as ex:
stat.inc(x.name, TestStatus.Failed)
echo ex.msg
let elpd = getTime() - start
print(stat, elpd, "rpc")
runRpcTest()