refactor(integration_bench)!: pivot to docker-compose via TestContext, share one node per run

BREAKING CHANGE:
- crate renamed e2e_bench → integration_bench. Run via `cargo run -p integration_bench`.
- env vars removed: LEZ_BEDROCK_BIN, LEZ_BEDROCK_CONFIG_DIR, LEZ_BEDROCK_PORT. Replaced by a docker prerequisite (docker-compose Bedrock via test_fixtures::TestContext).
- output filenames: target/e2e_bench_{dev,prove}.json → target/integration_bench_{dev,prove}.json.
- JSON schema: per-scenario `setup_s` field removed; replaced by run-level `shared_setup_s` (one TestContext is shared across all scenarios in a run).
- internal: bedrock_handle.rs and bench_context.rs deleted; placeholder-string config (PLACEHOLDER_CHAIN_START_TIME) gone.
This commit is contained in:
moudyellaz 2026-05-20 11:04:06 +02:00
parent 563a9ce0f7
commit 0119b38c1b
17 changed files with 135 additions and 537 deletions

42
Cargo.lock generated
View File

@ -2378,29 +2378,6 @@ version = "1.0.20"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d0881ea181b1df73ff77ffaaf9c7544ecc11e82fba9b5f27b262a3c73a332555"
[[package]]
name = "e2e_bench"
version = "0.1.0"
dependencies = [
"anyhow",
"borsh",
"chrono",
"clap",
"common",
"indexer_service",
"indexer_service_rpc",
"jsonrpsee",
"nssa",
"sequencer_service",
"sequencer_service_rpc",
"serde",
"serde_json",
"tempfile",
"test_fixtures",
"tokio",
"wallet",
]
[[package]]
name = "ecdsa"
version = "0.16.9"
@ -3989,6 +3966,25 @@ dependencies = [
"hybrid-array",
]
[[package]]
name = "integration_bench"
version = "0.1.0"
dependencies = [
"anyhow",
"borsh",
"clap",
"common",
"indexer_service_rpc",
"jsonrpsee",
"nssa",
"sequencer_service_rpc",
"serde",
"serde_json",
"test_fixtures",
"tokio",
"wallet",
]
[[package]]
name = "integration_tests"
version = "0.1.0"

View File

@ -44,7 +44,7 @@ members = [
"test_fixtures",
"tools/cycle_bench",
"tools/crypto_primitives_bench",
"tools/e2e_bench",
"tools/integration_bench",
]
[workspace.dependencies]

View File

@ -6,6 +6,6 @@ Bench tools live under `tools/` with READMEs for how to run each one. This direc
|---|---|
| cycle_bench | [cycle_bench.md](cycle_bench.md) |
| crypto_primitives_bench | [crypto_primitives_bench.md](crypto_primitives_bench.md) |
| e2e_bench | [e2e_bench.md](e2e_bench.md) |
| integration_bench | [integration_bench.md](integration_bench.md) |
All numbers are from a single M2 Pro dev box unless noted otherwise.

View File

@ -1,6 +1,6 @@
# e2e_bench
# integration_bench
End-to-end LEZ scenarios driven through the wallet against an in-process sequencer + indexer wired to an external Bedrock node. Times each step and records borsh sizes per block, split by tx variant.
End-to-end LEZ scenarios driven through the wallet against a docker-compose Bedrock node + in-process sequencer + indexer (via `test_fixtures::TestContext`). Times each step and records borsh sizes per block, split by tx variant.
No numeric tables here yet. Absolute wall time and block sizes depend heavily on the bedrock config (block cadence and confirmation depth) and on dev-mode vs real proving; re-run the bench locally to get numbers for your own setup. Canonical numbers will be added once the bench runs against the standard configuration.
@ -31,30 +31,29 @@ Numbers are intentionally omitted in this document until the canonical run lands
## Methodology
Per scenario, every produced block is fetched via `getBlock(BlockId)` and serialized with `borsh::to_vec(&Block)`. Each transaction is serialized individually and counted by variant. Empty clock-only ticks give the per-block fixed-cost baseline. Wall time is captured per step (submit + inclusion + wallet sync) and per scenario (setup + steps + closing bedrock finality wait).
Per scenario, every produced block is fetched via `getBlock(BlockId)` and serialized with `borsh::to_vec(&Block)`. Each transaction is serialized individually and counted by variant. Empty clock-only ticks give the per-block fixed-cost baseline. Wall time is captured per step (submit + inclusion + wallet sync) and aggregated to the per-scenario `total_s`. The one-time stack-setup cost (`shared_setup_s` at the run level) and the closing bedrock finality wait (`bedrock_finality_s` per scenario) are reported separately, not folded into `total_s`.
## Reproduce
Prerequisite: a running local Docker daemon (the `bedrock/docker-compose.yml` is brought up by the bench).
```sh
export LEZ_BEDROCK_BIN=/path/to/logos-blockchain/target/release/logos-blockchain-node
export LEZ_BEDROCK_CONFIG_DIR=/path/to/bedrock/configs
# Dev-mode sweep (fast)
RISC0_DEV_MODE=1 cargo run --release -p integration_bench -- --scenario all
# Dev-mode sweep (fast, ~16 min for all five scenarios)
RISC0_DEV_MODE=1 cargo run --release -p e2e_bench -- --scenario all
# Real-proving for representative private flow
cargo run --release -p integration_bench -- --scenario private
# Real-proving for representative private flow (~6 min on M2 Pro CPU)
cargo run --release -p e2e_bench -- --scenario private
# Real-proving for representative public flow (~3 min)
cargo run --release -p e2e_bench -- --scenario amm
# Real-proving for representative public flow
cargo run --release -p integration_bench -- --scenario amm
```
JSON output: `target/e2e_bench_dev.json` / `target/e2e_bench_prove.json` (suffix toggled by `RISC0_DEV_MODE`).
JSON output: `target/integration_bench_dev.json` / `target/integration_bench_prove.json` (suffix toggled by `RISC0_DEV_MODE`).
## Caveats
- Dev-mode `ppe_tx_bytes` and PPE-step latencies are not representative of production; use real-proving numbers for any fee-model input that touches the storage or prover-cost components.
- Single-host run, no GPU acceleration. Real-proving on production prover hardware will move per-step latencies by orders of magnitude; byte counts will not change.
- Bedrock running locally; no real network latency between sequencer and Bedrock.
- Bedrock L1 finality (`bedrock_finality_s`) is set by the bedrock config in `LEZ_BEDROCK_CONFIG_DIR` (block cadence × confirmation depth). Different configs will shift `bedrock_finality_s` materially.
- Some scenarios share account state via the same wallet; this is intentional (mirrors `integration_tests::TestContext`) and not a realistic multi-wallet workload.
- Bedrock running locally via docker-compose; no real network latency between sequencer and Bedrock.
- Bedrock L1 finality (`bedrock_finality_s`) is set by the bedrock config in `bedrock/docker-compose.yml` (block cadence × confirmation depth). Different configs will shift `bedrock_finality_s` materially.
- All scenarios share a single TestContext for the run (one bedrock + sequencer + indexer + wallet for the whole run, chain state accumulating across scenarios), which matches how the node runs in production.

View File

@ -1,33 +0,0 @@
# e2e_bench
End-to-end LEZ scenarios driven through the wallet against an in-process sequencer + indexer wired to an external Bedrock node. Times each step (submit, inclusion, wallet sync) and records borsh sizes for every block produced, split into per-tx-variant counts.
## Run
Required env vars (no defaults):
```sh
export LEZ_BEDROCK_BIN=/path/to/logos-blockchain/target/release/logos-blockchain-node
export LEZ_BEDROCK_CONFIG_DIR=/path/to/bedrock/configs
# optional: LEZ_BEDROCK_PORT (default 18080)
```
The config dir must contain `node-config.yaml` and a `deployment-settings.yaml` template with the literal string `PLACEHOLDER_CHAIN_START_TIME` (rewritten per launch).
```sh
# All scenarios, dev-mode proving (fast)
RISC0_DEV_MODE=1 cargo run --release -p e2e_bench -- --scenario all
# One scenario, real proving (slow)
cargo run --release -p e2e_bench -- --scenario amm
```
Scenarios: `token`, `amm`, `fanout`, `private`, `parallel`, `all`.
## What you'll see
Per scenario: a step table (`submit_s`, `inclusion_s`, `sync_s`, `total_s`) and a size summary covering every block captured during the scenario (block_bytes total/mean/min/max; per-tx-variant sizes for public, PPE, and program-deployment transactions).
The fanout, parallel, and private scenarios are the most representative for L1-payload-size measurements since they put multiple txs per block.
JSON output is written to `target/e2e_bench.json`.

View File

@ -1,152 +0,0 @@
//! Manages an external `logos-blockchain-node` process as a child of the bench.
//! Launches a fresh Bedrock instance per scenario so the indexer never has to
//! catch up a large finalization backlog.
//!
//! Required env vars (no defaults, path layouts differ per developer):
//! - `LEZ_BEDROCK_BIN` absolute path to the `logos-blockchain-node` binary.
//! - `LEZ_BEDROCK_CONFIG_DIR` directory containing `node-config.yaml` and
//! `deployment-settings.yaml` (template with `PLACEHOLDER_CHAIN_START_TIME`).
//!
//! Optional:
//! - `LEZ_BEDROCK_PORT` (default: 18080)
#![allow(
clippy::let_underscore_must_use,
reason = "file is deleted in the docker-compose pivot; teardown ignores child kill/wait results by design"
)]
use std::{
env,
net::SocketAddr,
path::PathBuf,
process::{Child, Command, Stdio},
time::{Duration, Instant},
};
use anyhow::{Context as _, Result, bail};
pub struct BedrockHandle {
child: Option<Child>,
addr: SocketAddr,
workdir: PathBuf,
}
impl BedrockHandle {
/// Launch a fresh Bedrock node. Cleans `state/` in the working dir, rewrites
/// `deployment-settings.yaml` with the current UTC `chain_start_time`, spawns
/// the binary, and polls the HTTP port until ready.
pub async fn launch_fresh() -> Result<Self> {
let bin = env::var("LEZ_BEDROCK_BIN").map_err(|err| {
anyhow::anyhow!(
"LEZ_BEDROCK_BIN is required ({err}). Set it to the absolute path of the \
logos-blockchain-node binary (e.g. \
`export LEZ_BEDROCK_BIN=/path/to/logos-blockchain/target/release/logos-blockchain-node`)."
)
})?;
let config_dir = env::var("LEZ_BEDROCK_CONFIG_DIR").map_err(|err| {
anyhow::anyhow!(
"LEZ_BEDROCK_CONFIG_DIR is required ({err}). Set it to the directory containing \
node-config.yaml and deployment-settings.yaml \
(see tools/e2e_bench/README.md for the expected layout)."
)
})?;
let port: u16 = env::var("LEZ_BEDROCK_PORT")
.ok()
.and_then(|p| p.parse().ok())
.unwrap_or(18080);
let bin_path = PathBuf::from(&bin);
if !bin_path.is_file() {
bail!(
"LEZ_BEDROCK_BIN does not point at a file: {bin}. Build it via \
`cargo build -p logos-blockchain-node --release` in logos-blockchain."
);
}
let config_dir = PathBuf::from(config_dir);
let node_config = config_dir.join("node-config.yaml");
let dep_template = config_dir.join("deployment-settings.yaml");
if !node_config.is_file() || !dep_template.is_file() {
bail!(
"LEZ_BEDROCK_CONFIG_DIR is missing node-config.yaml or \
deployment-settings.yaml at {}",
config_dir.display()
);
}
let workdir = tempfile::tempdir()
.context("create bedrock workdir")?
.keep();
let dep_runtime = workdir.join("deployment-settings.yaml");
let raw = std::fs::read_to_string(&dep_template).context("read deployment template")?;
let timestamp = chrono_now_utc_string();
let filled = raw.replace("PLACEHOLDER_CHAIN_START_TIME", &timestamp);
std::fs::write(&dep_runtime, filled).context("write deployment-settings runtime")?;
let log_path = workdir.join("bedrock.log");
let log_file = std::fs::File::create(&log_path).context("create bedrock log")?;
let log_err = log_file.try_clone().context("clone bedrock log")?;
eprintln!(
"BedrockHandle: launching {} (workdir {})",
bin,
workdir.display()
);
let child = Command::new(&bin_path)
.current_dir(&workdir)
.arg("--deployment")
.arg(&dep_runtime)
.arg(&node_config)
.env("POL_PROOF_DEV_MODE", "true")
.stdout(Stdio::from(log_file))
.stderr(Stdio::from(log_err))
.spawn()
.context("spawn logos-blockchain-node")?;
let addr = SocketAddr::from(([127, 0, 0, 1], port));
wait_for_http(addr, Duration::from_secs(60))
.await
.context("bedrock HTTP did not come up in 60s")?;
eprintln!("BedrockHandle: stdout/stderr at {}", log_path.display());
Ok(Self {
child: Some(child),
addr,
workdir,
})
}
pub const fn addr(&self) -> SocketAddr {
self.addr
}
}
impl Drop for BedrockHandle {
fn drop(&mut self) {
if let Some(mut child) = self.child.take() {
eprintln!("BedrockHandle: stopping bedrock pid {}", child.id());
let _ = child.kill();
let _ = child.wait();
}
let _ = std::fs::remove_dir_all(&self.workdir);
}
}
async fn wait_for_http(addr: SocketAddr, timeout: Duration) -> Result<()> {
let deadline = Instant::now() + timeout;
while Instant::now() < deadline {
if tokio::net::TcpStream::connect(addr).await.is_ok() {
// TCP accepts; give Bedrock a moment to finish chain bootstrap.
tokio::time::sleep(Duration::from_secs(2)).await;
return Ok(());
}
tokio::time::sleep(Duration::from_millis(250)).await;
}
bail!("Bedrock at {addr} did not accept TCP within {timeout:?}");
}
fn chrono_now_utc_string() -> String {
// Format: YYYY-MM-DD HH:MM:SS.000000 +00:00:00 (matches the deployment-settings template).
chrono::Utc::now()
.format("%Y-%m-%d %H:%M:%S%.6f +00:00:00")
.to_string()
}

View File

@ -1,210 +0,0 @@
//! `BenchContext`: wires sequencer + indexer + wallet in-process against an
//! externally-running Bedrock node. Mirrors the surface of
//! `integration_tests::TestContext` for the methods the scenarios need
//! (`wallet_mut()`, `sequencer_client()`), but skips the docker setup.
//!
//! The external Bedrock URL defaults to 127.0.0.1:18080 and can be overridden
//! with the `LEZ_BEDROCK_ADDR` env var.
#![allow(
clippy::arbitrary_source_item_ordering,
reason = "file is deleted in the docker-compose pivot; ordering churn is wasted work"
)]
use std::{env, net::SocketAddr, path::Path};
use anyhow::{Context as _, Result};
use indexer_service::IndexerHandle;
use test_fixtures::config::{
SequencerPartialConfig, UrlProtocol, addr_to_url, default_private_accounts_for_wallet,
default_public_accounts_for_wallet, genesis_from_accounts, indexer_config, sequencer_config,
wallet_config,
};
use sequencer_service::SequencerHandle;
use sequencer_service_rpc::{SequencerClient, SequencerClientBuilder};
use serde::Serialize;
use tempfile::TempDir;
use wallet::{WalletCore, config::WalletConfigOverrides};
const DEFAULT_BEDROCK_ADDR: &str = "127.0.0.1:18080";
#[expect(
clippy::partial_pub_fields,
reason = "Internal TempDirs are kept alive via private fields for RAII; \
client and wallet are public for scenarios to drive."
)]
pub struct BenchContext {
pub sequencer_client: SequencerClient,
pub wallet: WalletCore,
#[expect(
dead_code,
reason = "Retained for parity with TestContext; may be needed later."
)]
pub wallet_password: String,
sequencer_handle: Option<SequencerHandle>,
indexer_handle: IndexerHandle,
temp_indexer_dir: TempDir,
temp_sequencer_dir: TempDir,
temp_wallet_dir: TempDir,
}
impl BenchContext {
pub async fn new() -> Result<Self> {
let bedrock_addr_str =
env::var("LEZ_BEDROCK_ADDR").unwrap_or_else(|_| DEFAULT_BEDROCK_ADDR.to_owned());
let bedrock_addr: SocketAddr = bedrock_addr_str
.parse()
.with_context(|| format!("invalid LEZ_BEDROCK_ADDR `{bedrock_addr_str}`"))?;
eprintln!("BenchContext: using external bedrock at {bedrock_addr}");
let initial_public_accounts = default_public_accounts_for_wallet();
let initial_private_accounts = default_private_accounts_for_wallet();
let genesis_transactions =
genesis_from_accounts(&initial_public_accounts, &initial_private_accounts);
let sequencer_partial = SequencerPartialConfig::default();
let temp_indexer_dir = tempfile::tempdir().context("indexer temp dir")?;
let indexer_cfg = indexer_config(bedrock_addr, temp_indexer_dir.path().to_owned())
.context("indexer config")?;
let indexer_handle = indexer_service::run_server(indexer_cfg, 0)
.await
.context("indexer run_server")?;
let temp_sequencer_dir = tempfile::tempdir().context("sequencer temp dir")?;
let sequencer_cfg = sequencer_config(
sequencer_partial,
temp_sequencer_dir.path().to_owned(),
bedrock_addr,
genesis_transactions,
)
.context("sequencer config")?;
let sequencer_handle = sequencer_service::run(sequencer_cfg, 0)
.await
.context("sequencer run")?;
let temp_wallet_dir = tempfile::tempdir().context("wallet temp dir")?;
let mut wallet_cfg = wallet_config(sequencer_handle.addr()).context("wallet config")?;
// The default 30s poll interval is far too slow for a measurement run;
// shrink so the wallet sees new blocks within ~1s.
wallet_cfg.seq_poll_timeout = std::time::Duration::from_secs(1);
let wallet_cfg_str =
serde_json::to_string_pretty(&wallet_cfg).context("serialize wallet config")?;
let wallet_cfg_path = temp_wallet_dir.path().join("wallet_config.json");
std::fs::write(&wallet_cfg_path, wallet_cfg_str).context("write wallet config")?;
let storage_path = temp_wallet_dir.path().join("storage.json");
let password = "bench_pass".to_owned();
let (mut wallet, _mnemonic) = WalletCore::new_init_storage(
wallet_cfg_path,
storage_path,
Some(WalletConfigOverrides::default()),
&password,
)
.context("wallet init")?;
// Mirror integration_tests::setup_wallet: import the initial accounts
// produced above so the wallet can reference them by AccountId in scenarios.
for (private_key, _balance) in &initial_public_accounts {
wallet
.storage_mut()
.key_chain_mut()
.add_imported_public_account(private_key.clone());
}
for private_account in &initial_private_accounts {
wallet
.storage_mut()
.key_chain_mut()
.add_imported_private_account(
private_account.key_chain.clone(),
None,
private_account.identifier,
nssa::Account::default(),
);
}
wallet
.store_persistent_data()
.context("wallet store persistent")?;
let sequencer_url =
addr_to_url(UrlProtocol::Http, sequencer_handle.addr()).context("sequencer url")?;
let sequencer_client = SequencerClientBuilder::default()
.build(sequencer_url)
.context("build sequencer client")?;
Ok(Self {
sequencer_client,
wallet,
wallet_password: password,
sequencer_handle: Some(sequencer_handle),
indexer_handle,
temp_indexer_dir,
temp_sequencer_dir,
temp_wallet_dir,
})
}
pub const fn wallet_mut(&mut self) -> &mut WalletCore {
&mut self.wallet
}
pub const fn sequencer_client(&self) -> &SequencerClient {
&self.sequencer_client
}
pub const fn indexer_addr(&self) -> SocketAddr {
self.indexer_handle.addr()
}
/// Recursively-sized bytes on disk for sequencer + indexer + wallet tempdirs.
pub fn disk_sizes(&self) -> DiskSizes {
DiskSizes {
sequencer_bytes: dir_size_bytes(self.temp_sequencer_dir.path()),
indexer_bytes: dir_size_bytes(self.temp_indexer_dir.path()),
wallet_bytes: dir_size_bytes(self.temp_wallet_dir.path()),
}
}
}
#[derive(Debug, Clone, Copy, Default, Serialize)]
#[expect(
clippy::struct_field_names,
reason = "The `_bytes` suffix carries the unit and is preserved verbatim in JSON output."
)]
pub struct DiskSizes {
pub sequencer_bytes: u64,
pub indexer_bytes: u64,
pub wallet_bytes: u64,
}
fn dir_size_bytes(path: &Path) -> u64 {
let mut total = 0_u64;
let Ok(entries) = std::fs::read_dir(path) else {
return 0;
};
for entry in entries.flatten() {
let Ok(metadata) = entry.metadata() else {
continue;
};
if metadata.is_file() {
total = total.saturating_add(metadata.len());
} else if metadata.is_dir() {
total = total.saturating_add(dir_size_bytes(&entry.path()));
} else {
// Sockets, FIFOs, block/char devices: ignore. Symlinks are
// already followed by `is_file()` / `is_dir()`.
}
}
total
}
impl Drop for BenchContext {
fn drop(&mut self) {
if let Some(handle) = self.sequencer_handle.take()
&& !handle.is_healthy()
{
eprintln!("BenchContext drop: sequencer handle was unhealthy");
}
if !self.indexer_handle.is_healthy() {
eprintln!("BenchContext drop: indexer handle was unhealthy");
}
}
}

View File

@ -1,5 +1,5 @@
[package]
name = "e2e_bench"
name = "integration_bench"
version = "0.1.0"
edition = "2024"
license = { workspace = true }
@ -10,20 +10,16 @@ workspace = true
[dependencies]
common.workspace = true
indexer_service.workspace = true
indexer_service_rpc = { workspace = true, features = ["client"] }
nssa.workspace = true
sequencer_service.workspace = true
sequencer_service_rpc = { workspace = true, features = ["client"] }
test_fixtures.workspace = true
wallet.workspace = true
anyhow.workspace = true
borsh.workspace = true
chrono.workspace = true
clap.workspace = true
jsonrpsee = { workspace = true, features = ["ws-client"] }
serde.workspace = true
serde_json.workspace = true
tempfile.workspace = true
tokio = { workspace = true, features = ["macros", "rt-multi-thread", "time"] }

View File

@ -0,0 +1,27 @@
# integration_bench
End-to-end LEZ scenarios driven through the wallet against a docker-compose Bedrock node + in-process sequencer + indexer (via `test_fixtures::TestContext`). Times each step (submit, inclusion, wallet sync) and records borsh sizes for every block produced, split into per-tx-variant counts.
## Run
Prerequisite: a running local Docker daemon. The Bedrock service comes up via the same `bedrock/docker-compose.yml` that integration tests use, so no host-side binary or env vars are required.
```sh
# All scenarios, dev-mode proving (fast)
RISC0_DEV_MODE=1 cargo run --release -p integration_bench -- --scenario all
# One scenario, real proving (slow)
cargo run --release -p integration_bench -- --scenario amm
```
Scenarios: `token`, `amm`, `fanout`, `private`, `parallel`, `all`.
All scenarios share a single TestContext for the run (one Bedrock + sequencer + indexer + wallet across the whole run, chain state accumulating), which matches how the node runs in production.
## What you'll see
Per scenario: a step table (`submit_s`, `inclusion_s`, `sync_s`, `total_s`) and a size summary covering every block captured during the scenario (block_bytes total/mean/min/max; per-tx-variant sizes for public, PPE, and program-deployment transactions).
The fanout, parallel, and private scenarios are the most representative for L1-payload-size measurements since they put multiple txs per block.
JSON output is written to `target/integration_bench_dev.json` (dev mode) or `target/integration_bench_prove.json` (real proving).

View File

@ -11,10 +11,9 @@ use anyhow::{Result, bail};
use common::transaction::NSSATransaction;
use sequencer_service_rpc::RpcClient as _;
use serde::{Serialize, Serializer};
use test_fixtures::{DiskSizes, TestContext};
use wallet::cli::SubcommandReturnValue;
use crate::bench_context::BenchContext;
const TX_INCLUSION_POLL_INTERVAL: Duration = Duration::from_millis(250);
const TX_INCLUSION_TIMEOUT: Duration = Duration::from_secs(120);
@ -51,15 +50,13 @@ pub struct StepResult {
#[derive(Debug, Serialize, Default)]
pub struct ScenarioOutput {
pub name: String,
#[serde(serialize_with = "ser_duration_secs", rename = "setup_s")]
pub setup: Duration,
pub steps: Vec<StepResult>,
#[serde(serialize_with = "ser_duration_secs", rename = "total_s")]
pub total: Duration,
/// Disk sizes (sequencer / indexer / wallet tempdirs) sampled at scenario start.
pub disk_before: Option<crate::bench_context::DiskSizes>,
pub disk_before: Option<DiskSizes>,
/// Disk sizes sampled at scenario end.
pub disk_after: Option<crate::bench_context::DiskSizes>,
pub disk_after: Option<DiskSizes>,
/// Bedrock-finality latency: time from final-step inclusion to the indexer
/// reporting the sequencer tip as L1-finalised. Effectively measures the
/// sequencer→Bedrock posting + Bedrock finalisation + indexer L1 ingest path.
@ -85,7 +82,7 @@ impl ScenarioOutput {
/// Begin a timed step. Capture this *before* submitting the wallet operation
/// so we can later subtract it from the post-submit block height to detect
/// when the chain has advanced past the tx's block.
pub async fn begin_step(ctx: &BenchContext) -> Result<u64> {
pub async fn begin_step(ctx: &TestContext) -> Result<u64> {
Ok(ctx.sequencer_client().get_last_block_id().await?)
}
@ -105,7 +102,7 @@ pub async fn finalize_step(
started: Instant,
pre_block_id: u64,
ret: &SubcommandReturnValue,
ctx: &mut BenchContext,
ctx: &mut TestContext,
) -> Result<StepResult> {
let label = label.into();
let submit = started.elapsed();
@ -174,7 +171,7 @@ pub async fn finalize_step(
/// Wait for `get_last_block_id` to advance by at least `min_blocks` from `from_block_id`.
pub async fn wait_for_chain_advance(
ctx: &BenchContext,
ctx: &TestContext,
from_block_id: u64,
min_blocks: u64,
) -> Result<()> {
@ -197,7 +194,7 @@ pub async fn wait_for_chain_advance(
}
}
async fn sync_wallet_to_tip(ctx: &mut BenchContext) -> Result<()> {
async fn sync_wallet_to_tip(ctx: &mut TestContext) -> Result<()> {
let last_block = ctx.sequencer_client().get_last_block_id().await?;
ctx.wallet_mut().sync_to_block(last_block).await?;
Ok(())
@ -213,9 +210,8 @@ pub fn print_table(output: &ScenarioOutput) {
.max("step".len());
println!(
"\nScenario: {} (setup {:.2}s, total {:.2}s)",
"\nScenario: {} (total {:.2}s)",
output.name,
output.setup.as_secs_f64(),
output.total.as_secs_f64(),
);
println!(

View File

@ -1,18 +1,18 @@
//! End-to-end LEZ scenario bench.
//!
//! Spins up the full stack (native Bedrock node launched per-scenario via
//! `BedrockHandle` + in-process sequencer + indexer + wallet via
//! `BenchContext`) and drives the wallet through configurable scenarios that
//! mirror real user flows. Times each step and records borsh-serialized
//! block + tx sizes per scenario.
//! Spins up the full stack via `test_fixtures::TestContext` (docker-compose
//! Bedrock + in-process sequencer + indexer + wallet) once for the whole run,
//! then drives the wallet through each requested scenario against that single
//! shared stack. Times each step and records borsh-serialized block + tx sizes
//! per scenario.
//!
//! Required env vars (no defaults; see `tools/e2e_bench/README.md`):
//! `LEZ_BEDROCK_BIN` absolute path to logos-blockchain-node.
//! `LEZ_BEDROCK_CONFIG_DIR` directory with node-config.yaml + deployment template.
//! Prerequisite: a working local Docker daemon. The Bedrock service is brought
//! up via the same `bedrock/docker-compose.yml` the integration tests use, so
//! no host-side binary or env vars are required.
//!
//! Run examples:
//! `RISC0_DEV_MODE=1` `cargo run --release -p e2e_bench -- --scenario all`.
//! `cargo run --release -p e2e_bench -- --scenario amm`.
//! `RISC0_DEV_MODE=1 cargo run --release -p integration_bench -- --scenario all`.
//! `cargo run --release -p integration_bench -- --scenario amm`.
//!
//! `RISC0_DEV_MODE=1` skips proving and produces latency-only numbers in
//! ~minutes; omitting it produces realistic proving-inclusive numbers but
@ -31,14 +31,11 @@
use std::{path::PathBuf, time::Duration};
use anyhow::{Context as _, Result};
use bedrock_handle::BedrockHandle;
use bench_context::BenchContext;
use clap::{Parser, ValueEnum};
use harness::ScenarioOutput;
use serde::Serialize;
use test_fixtures::TestContext;
mod bedrock_handle;
mod bench_context;
mod harness;
mod scenarios;
@ -59,7 +56,7 @@ struct Cli {
#[arg(long, value_enum, default_value_t = ScenarioName::All)]
scenario: ScenarioName,
/// Optional JSON output path. Defaults to `<workspace>/target/e2e_bench.json`.
/// Optional JSON output path. Defaults to `<workspace>/target/integration_bench.json`.
#[arg(long)]
json_out: Option<PathBuf>,
}
@ -67,20 +64,24 @@ struct Cli {
#[derive(Debug, Serialize)]
struct BenchRunReport {
risc0_dev_mode: bool,
/// Time to bring up the shared `TestContext` (docker-compose Bedrock +
/// sequencer + indexer + wallet). Paid once per run regardless of how many
/// scenarios are exercised.
shared_setup_s: f64,
scenarios: Vec<ScenarioOutput>,
total_wall_s: f64,
}
#[tokio::main(flavor = "multi_thread")]
async fn main() -> Result<()> {
// integration_tests initializes env_logger via a LazyLock, so we leave logger
// test_fixtures initializes env_logger via a LazyLock, so we leave logger
// setup to it. Set RUST_LOG=info before running to see logs.
let cli = Cli::parse();
let risc0_dev_mode = std::env::var("RISC0_DEV_MODE").is_ok_and(|v| !v.is_empty() && v != "0");
eprintln!(
"e2e_bench: scenario={:?}, RISC0_DEV_MODE={}",
"integration_bench: scenario={:?}, RISC0_DEV_MODE={}",
cli.scenario,
if risc0_dev_mode { "1" } else { "unset/0" }
);
@ -97,43 +98,28 @@ async fn main() -> Result<()> {
};
let overall_started = std::time::Instant::now();
// One shared stack for the entire run: docker-compose Bedrock + sequencer +
// indexer + wallet. Scenarios share chain state, which matches how the node
// runs in production (long-lived, accumulating).
let setup_started = std::time::Instant::now();
let mut ctx = TestContext::new()
.await
.context("failed to setup TestContext")?;
let shared_setup = setup_started.elapsed();
eprintln!("setup: {:.2}s", shared_setup.as_secs_f64());
let mut all_outputs = Vec::with_capacity(to_run.len());
for name in to_run {
eprintln!("\n=== running scenario: {name:?} ===");
{
let setup_started = std::time::Instant::now();
// Spawn a fresh Bedrock node for this scenario. Each scenario therefore
// starts with an empty chain so the indexer never has a backlog from a
// prior scenario.
let bedrock = BedrockHandle::launch_fresh()
.await
.with_context(|| format!("failed to spawn Bedrock for scenario {name:?}"))?;
let bedrock_addr_string = format!("{}", bedrock.addr());
// SAFETY: env::set_var happens before any threaded setup that reads env.
unsafe {
std::env::set_var("LEZ_BEDROCK_ADDR", &bedrock_addr_string);
}
let mut ctx = BenchContext::new()
.await
.with_context(|| format!("failed to setup BenchContext for scenario {name:?}"))?;
let setup = setup_started.elapsed();
eprintln!("setup: {:.2}s", setup.as_secs_f64());
let disk_before = ctx.disk_sizes();
let mut output = run_scenario(name, setup, &mut ctx).await?;
output.disk_before = Some(disk_before);
output.disk_after = Some(ctx.disk_sizes());
output.bedrock_finality = Some(measure_bedrock_finality(&ctx).await?);
harness::print_table(&output);
all_outputs.push(output);
// ctx and bedrock drop here at end of scope, killing the bedrock child
// before we sleep so the next iteration can rebind the port.
}
// Give Bedrock a moment to shut down before the next scenario.
tokio::time::sleep(Duration::from_secs(2)).await;
let disk_before = ctx.disk_sizes();
let mut output = run_scenario(name, &mut ctx).await?;
output.disk_before = Some(disk_before);
output.disk_after = Some(ctx.disk_sizes());
output.bedrock_finality = Some(measure_bedrock_finality(&ctx).await?);
harness::print_table(&output);
all_outputs.push(output);
}
let total_wall_s = overall_started.elapsed().as_secs_f64();
@ -141,6 +127,7 @@ async fn main() -> Result<()> {
let report = BenchRunReport {
risc0_dev_mode,
shared_setup_s: shared_setup.as_secs_f64(),
scenarios: all_outputs,
total_wall_s,
};
@ -155,7 +142,7 @@ async fn main() -> Result<()> {
let suffix = if risc0_dev_mode { "dev" } else { "prove" };
workspace_root
.join("target")
.join(format!("e2e_bench_{suffix}.json"))
.join(format!("integration_bench_{suffix}.json"))
};
if let Some(parent) = out_path.parent() {
std::fs::create_dir_all(parent)?;
@ -166,26 +153,21 @@ async fn main() -> Result<()> {
Ok(())
}
async fn run_scenario(
name: ScenarioName,
setup: Duration,
ctx: &mut BenchContext,
) -> Result<ScenarioOutput> {
let output = match name {
ScenarioName::Token => scenarios::token::run(ctx).await?,
ScenarioName::Amm => scenarios::amm::run(ctx).await?,
ScenarioName::Fanout => scenarios::fanout::run(ctx).await?,
ScenarioName::Private => scenarios::private::run(ctx).await?,
ScenarioName::Parallel => scenarios::parallel::run(ctx).await?,
async fn run_scenario(name: ScenarioName, ctx: &mut TestContext) -> Result<ScenarioOutput> {
match name {
ScenarioName::Token => scenarios::token::run(ctx).await,
ScenarioName::Amm => scenarios::amm::run(ctx).await,
ScenarioName::Fanout => scenarios::fanout::run(ctx).await,
ScenarioName::Private => scenarios::private::run(ctx).await,
ScenarioName::Parallel => scenarios::parallel::run(ctx).await,
ScenarioName::All => unreachable!("dispatched above"),
};
Ok(ScenarioOutput { setup, ..output })
}
}
/// Poll the indexer's L1-finalised block id until it catches up with the
/// sequencer's last block id. This is effectively the sequencer→Bedrock posting
/// plus Bedrock finalisation plus indexer ingest latency.
async fn measure_bedrock_finality(ctx: &BenchContext) -> Result<Duration> {
async fn measure_bedrock_finality(ctx: &TestContext) -> Result<Duration> {
use indexer_service_rpc::RpcClient as _;
use jsonrpsee::ws_client::WsClientBuilder;
use sequencer_service_rpc::RpcClient as _;

View File

@ -12,7 +12,7 @@ use wallet::cli::{
use crate::harness::{ScenarioOutput, finalize_step};
pub async fn run(ctx: &mut crate::bench_context::BenchContext) -> Result<ScenarioOutput> {
pub async fn run(ctx: &mut test_fixtures::TestContext) -> Result<ScenarioOutput> {
let mut output = ScenarioOutput::new("amm_swap_flow");
let def_a = new_public_account(ctx, &mut output, "create_acc_def_a").await?;
@ -125,7 +125,7 @@ pub async fn run(ctx: &mut crate::bench_context::BenchContext) -> Result<Scenari
}
async fn new_public_account(
ctx: &mut crate::bench_context::BenchContext,
ctx: &mut test_fixtures::TestContext,
output: &mut ScenarioOutput,
label: &str,
) -> Result<nssa::AccountId> {
@ -148,7 +148,7 @@ async fn new_public_account(
}
async fn timed_token_new(
ctx: &mut crate::bench_context::BenchContext,
ctx: &mut test_fixtures::TestContext,
output: &mut ScenarioOutput,
label: &str,
def_id: nssa::AccountId,
@ -173,7 +173,7 @@ async fn timed_token_new(
}
async fn timed_token_send(
ctx: &mut crate::bench_context::BenchContext,
ctx: &mut test_fixtures::TestContext,
output: &mut ScenarioOutput,
label: &str,
from_id: nssa::AccountId,

View File

@ -15,7 +15,7 @@ use crate::harness::{ScenarioOutput, finalize_step};
const FANOUT_COUNT: usize = 10;
const AMOUNT_PER_TRANSFER: u128 = 100;
pub async fn run(ctx: &mut crate::bench_context::BenchContext) -> Result<ScenarioOutput> {
pub async fn run(ctx: &mut test_fixtures::TestContext) -> Result<ScenarioOutput> {
let mut output = ScenarioOutput::new("multi_recipient_fanout");
let def_id = new_public_account(ctx, &mut output, "create_acc_def").await?;
@ -67,7 +67,7 @@ pub async fn run(ctx: &mut crate::bench_context::BenchContext) -> Result<Scenari
}
async fn new_public_account(
ctx: &mut crate::bench_context::BenchContext,
ctx: &mut test_fixtures::TestContext,
output: &mut ScenarioOutput,
label: &str,
) -> Result<nssa::AccountId> {

View File

@ -7,23 +7,20 @@ use std::time::Instant;
use anyhow::{Result, bail};
use common::transaction::NSSATransaction;
use test_fixtures::public_mention;
use sequencer_service_rpc::RpcClient as _;
use test_fixtures::{TestContext, public_mention};
use wallet::cli::{
Command, SubcommandReturnValue,
account::{AccountSubcommand, NewSubcommand},
programs::token::TokenProgramAgnosticSubcommand,
};
use crate::{
bench_context::BenchContext,
harness::{BlockSize, ScenarioOutput, StepResult, finalize_step},
};
use crate::harness::{BlockSize, ScenarioOutput, StepResult, finalize_step};
const PARALLEL_FANOUT_N: usize = 10;
const AMOUNT_PER_TRANSFER: u128 = 100;
pub async fn run(ctx: &mut BenchContext) -> Result<ScenarioOutput> {
pub async fn run(ctx: &mut TestContext) -> Result<ScenarioOutput> {
let mut output = ScenarioOutput::new("parallel_fanout");
// Setup: definition, master supply, N parallel supplies, N recipients.
@ -168,7 +165,7 @@ pub async fn run(ctx: &mut BenchContext) -> Result<ScenarioOutput> {
}
async fn new_public_account(
ctx: &mut BenchContext,
ctx: &mut TestContext,
output: &mut ScenarioOutput,
label: &str,
) -> Result<nssa::AccountId> {

View File

@ -12,7 +12,7 @@ use wallet::cli::{
use crate::harness::{ScenarioOutput, finalize_step};
pub async fn run(ctx: &mut crate::bench_context::BenchContext) -> Result<ScenarioOutput> {
pub async fn run(ctx: &mut test_fixtures::TestContext) -> Result<ScenarioOutput> {
let mut output = ScenarioOutput::new("private_chained_flow");
let def_id = new_public_account(ctx, &mut output, "create_acc_def").await?;
@ -104,7 +104,7 @@ pub async fn run(ctx: &mut crate::bench_context::BenchContext) -> Result<Scenari
}
async fn new_public_account(
ctx: &mut crate::bench_context::BenchContext,
ctx: &mut test_fixtures::TestContext,
output: &mut ScenarioOutput,
label: &str,
) -> Result<nssa::AccountId> {
@ -127,7 +127,7 @@ async fn new_public_account(
}
async fn new_private_account(
ctx: &mut crate::bench_context::BenchContext,
ctx: &mut test_fixtures::TestContext,
output: &mut ScenarioOutput,
label: &str,
) -> Result<nssa::AccountId> {

View File

@ -12,7 +12,7 @@ use wallet::cli::{
use crate::harness::{ScenarioOutput, finalize_step};
pub async fn run(ctx: &mut crate::bench_context::BenchContext) -> Result<ScenarioOutput> {
pub async fn run(ctx: &mut test_fixtures::TestContext) -> Result<ScenarioOutput> {
let mut output = ScenarioOutput::new("token_onboarding");
let definition_id = new_public_account(ctx, &mut output, "create_pub_definition").await?;
@ -81,7 +81,7 @@ pub async fn run(ctx: &mut crate::bench_context::BenchContext) -> Result<Scenari
}
async fn new_public_account(
ctx: &mut crate::bench_context::BenchContext,
ctx: &mut test_fixtures::TestContext,
output: &mut ScenarioOutput,
label: &str,
) -> Result<nssa::AccountId> {
@ -104,7 +104,7 @@ async fn new_public_account(
}
async fn new_private_account(
ctx: &mut crate::bench_context::BenchContext,
ctx: &mut test_fixtures::TestContext,
output: &mut ScenarioOutput,
label: &str,
) -> Result<nssa::AccountId> {