diff --git a/simulations/mix_lez_chat/README.md b/simulations/mix_lez_chat/README.md index b82ecba..1622be1 100644 --- a/simulations/mix_lez_chat/README.md +++ b/simulations/mix_lez_chat/README.md @@ -70,9 +70,11 @@ Override defaults via environment: | `SIM_LOG_LEVEL` | `INFO` | Node log level (TRACE, DEBUG, INFO, WARN, ERROR) | | `SIM_CHAT_RECV_PORT` | `60010` | Chat receiver TCP port | | `SIM_CHAT_SEND_PORT` | `60011` | Chat sender TCP port | -| `SIM_KADEMLIA_MIN_WAIT` | `30` | Minimum seconds to wait for kademlia propagation | -| `SIM_RECEIVER_MIN_WAIT` | `15` | Minimum seconds to wait for receiver to join mix | -| `SIM_DELIVERY_TIMEOUT` | `120` | Max seconds to wait for message delivery | +| `SIM_KADEMLIA_MIN_WAIT` | `30` (local) / `120` (testnet) | Minimum seconds to wait for kademlia propagation | +| `SIM_RECEIVER_MIN_WAIT` | `15` (local) / `60` (testnet) | Minimum seconds to wait for receiver to join mix | +| `SIM_DELIVERY_TIMEOUT` | `120` (local) / `300` (testnet) | Max seconds to wait for message delivery | +| `SIM_NODE_STARTUP_SLEEP` | `10` (local) / `30` (testnet) | Seconds between launching each mix node | +| `SIM_NETWORK` | `local` | `local` runs against a sequencer on `127.0.0.1:3040`; `testnet` runs against `https://testnet.lez.logos.co/` | Example — fast iteration with verbose logging: @@ -81,17 +83,52 @@ SIM_LOG_LEVEL=TRACE SIM_KADEMLIA_MIN_WAIT=10 SIM_RECEIVER_MIN_WAIT=5 \ bash simulations/mix_lez_chat/run_simulation.sh --fresh ``` +## Running against the public testnet + +```bash +SIM_NETWORK=testnet bash simulations/mix_lez_chat/run_simulation.sh --fresh +``` + +Effect of `SIM_NETWORK=testnet`: +- Phase 1 (local sequencer launch) is skipped; the script does a one-shot reachability check against `https://testnet.lez.logos.co/` and dies up front if unreachable. +- Wallet config is picked from `vendor/logos-lez-rln/testnet/` instead of `dev/`. The wallet's `storage.json` and the on-chain registration accounts persist across runs. +- `run_setup` deploys + initializes on first run, then short-circuits via `is_initialized` on every run after — see "Config account: …" in the setup output either way. +- Timing floors (`SIM_KADEMLIA_MIN_WAIT`, `SIM_RECEIVER_MIN_WAIT`, `SIM_DELIVERY_TIMEOUT`, `SIM_NODE_STARTUP_SLEEP`) and `LEZ_RLN_BLOCK_SEAL_SECS` default higher to match ~60s testnet block times. + +Prerequisites: +- The gifter mix node's payment account must be funded on testnet. The first successful `SIM_NETWORK=testnet … --fresh` run populates `~/.logos-lez-rln/payment_account_.txt` automatically; subsequent runs reuse it. +- Expected wall-clock runtime: ~20–25 minutes (first run) / ~15 minutes (subsequent runs), vs. ~3 minutes locally. +- Only one developer at a time — concurrent testnet sim runs share the gifter wallet and will collide. + +### Reproducibility on a fresh clone + +The canonical testnet deployment (RLN tree + minted supply) is shared across developers. On first run, the script seeds two artifacts from the submodule so `run_setup` can short-circuit to `create_funded_user`: + +- `vendor/logos-lez-rln/testnet/storage.json.seed` → copied to `vendor/logos-lez-rln/testnet/storage.json` if absent. Contains only the supply holding account + its signing key. +- `vendor/logos-lez-rln/testnet/supply_holding.txt` → copied to `~/.logos-lez-rln/supply_holding_.txt` if absent. Contains the supply `AccountId`. + +What stays shared vs. fresh: + +| Artifact | Shared | Per-dev fresh | +|---|---|---| +| `TREE_ID`, sequencer URL, deployed program IDs (on-chain), gifter EIP-191 auth keys, mix node identity keys | ✓ | | +| Supply holding account + signing key (seeded from submodule) | ✓ | | +| Per-run payment account (`~/.logos-lez-rln/payment_account_.txt`) | | ✓ | +| Working-copy `testnet/storage.json` (gitignored; accumulates payment accounts) | | ✓ | +| Mix + chat RLN credentials (in `.sim_state/rln_keystore_*.json`) | | ✓ | + +**Security:** the supply signing key being in the repo is acceptable only because testnet does not charge gas and the tokens are test tokens with no real value. + ## `--fresh` behavior When `--fresh` is passed: - Kills all existing `logos_host` processes - Cleans `/tmp/logos_*` Qt RemoteObjects sockets - Removes `.sim_state/` directory -- Removes sequencer state (`rocksdb/`, `bedrock_signing_key`) -- Rebuilds and restarts the sequencer -- Redeploys RLN programs via `run_setup` +- On `SIM_NETWORK=local` (default): removes sequencer state (`rocksdb/`, `bedrock_signing_key`), rebuilds and restarts the sequencer, redeploys RLN programs via `run_setup` +- On `SIM_NETWORK=testnet`: leaves on-chain state and the persistent wallet under `vendor/logos-lez-rln/testnet/` intact; `run_setup` short-circuits to `create_funded_user` -Without `--fresh`, reuses existing sequencer if port 3040 is already bound. +Without `--fresh`, on `SIM_NETWORK=local` it reuses an existing sequencer if port 3040 is already bound. ## Troubleshooting diff --git a/simulations/mix_lez_chat/run_simulation.sh b/simulations/mix_lez_chat/run_simulation.sh index b44f788..e4db098 100755 --- a/simulations/mix_lez_chat/run_simulation.sh +++ b/simulations/mix_lez_chat/run_simulation.sh @@ -31,11 +31,25 @@ DELIVERY_DIR="$DELIVERY_MODULE_DIR/vendor/logos-delivery" export RISC0_DEV_MODE=1 export TMPDIR=/tmp -export LOGOS_EVENT_STDERR=1 # Enable EVENT: stderr output for sim script event observation +export LOGOS_EVENT_STDERR=1 # Mirror EVENT: lines to stderr so the sim can grep them. die() { echo " FATAL: $*" >&2; exit 1; } log() { echo "[$(date '+%H:%M:%S')] $*"; } +# Poll a logoscore log until it shows >= $expected "Method call successful" +# lines, or until $timeout iterations (sleeping $sleep_sec each) have elapsed. +# Sets global $N to the last observed count so callers can branch on it. +wait_method_calls() { + local logfile="$1" expected="$2" timeout="$3" sleep_sec="${4:-1}" + local t + for t in $(seq 1 "$timeout"); do + N=$(grep -c '^Method call successful' "$logfile" 2>/dev/null || true); N=${N:-0} + [ "$N" -ge "$expected" ] && return 0 + sleep "$sleep_sec" + done + return 1 +} + # --- Node identity constants (4 mix nodes) --- NODEKEYS=( "f98e3fba96c32e8d1967d460f1b79457380e1a895f7971cecc8528abe733781a" @@ -72,9 +86,28 @@ TEST_MESSAGE_PREFIX="chatmixtest" LOG_LEVEL=${SIM_LOG_LEVEL:-INFO} CHAT_RECV_PORT=${SIM_CHAT_RECV_PORT:-60010} CHAT_SEND_PORT=${SIM_CHAT_SEND_PORT:-60011} -KADEMLIA_MIN_WAIT=${SIM_KADEMLIA_MIN_WAIT:-30} -RECEIVER_MIN_WAIT=${SIM_RECEIVER_MIN_WAIT:-15} -DELIVERY_TIMEOUT=${SIM_DELIVERY_TIMEOUT:-120} +SIM_NETWORK=${SIM_NETWORK:-local} +case "$SIM_NETWORK" in + local|testnet) ;; + *) die "SIM_NETWORK must be 'local' or 'testnet', got: $SIM_NETWORK";; +esac + +# Timing floors: local sequencer ~15s blocks vs testnet ~60s + more variance. +if [ "$SIM_NETWORK" = testnet ]; then + KADEMLIA_MIN_WAIT=${SIM_KADEMLIA_MIN_WAIT:-120} + KADEMLIA_HARD_CAP=${SIM_KADEMLIA_HARD_CAP:-600} + RECEIVER_MIN_WAIT=${SIM_RECEIVER_MIN_WAIT:-60} + DELIVERY_TIMEOUT=${SIM_DELIVERY_TIMEOUT:-300} + NODE_STARTUP_SLEEP=${SIM_NODE_STARTUP_SLEEP:-30} + export LEZ_RLN_BLOCK_SEAL_SECS="${LEZ_RLN_BLOCK_SEAL_SECS:-90}" +else + KADEMLIA_MIN_WAIT=${SIM_KADEMLIA_MIN_WAIT:-30} + KADEMLIA_HARD_CAP=${SIM_KADEMLIA_HARD_CAP:-180} + RECEIVER_MIN_WAIT=${SIM_RECEIVER_MIN_WAIT:-15} + DELIVERY_TIMEOUT=${SIM_DELIVERY_TIMEOUT:-120} + NODE_STARTUP_SLEEP=${SIM_NODE_STARTUP_SLEEP:-10} +fi +TESTNET_RPC_URL="https://testnet.lez.logos.co/" case "$(uname -s)-$(uname -m)" in Darwin-arm64) PLATFORM="darwin-arm64-dev"; EXT="dylib";; @@ -111,32 +144,43 @@ cleanup() { trap cleanup EXIT echo "=== Mix + LEZ RLN Chat Simulation ($NUM_NODES nodes) ===" +echo " Network: $SIM_NETWORK" +[ "$SIM_NETWORK" = testnet ] && echo " Testnet RPC: $TESTNET_RPC_URL" echo " LEZ repo: $LEZ_RLN_DIR" echo " Chat module: $CHAT_MODULE_DIR" echo " Logos-chat: $LOGOS_CHAT_DIR" echo "" pkill -f 'logos_host' 2>/dev/null || true; sleep 1 -# Clean stale QtRO LocalServer sockets from prior runs (can confuse capability_module lookups) +# Stale QtRO LocalServer sockets confuse capability_module lookups. rm -f /tmp/logos_* 2>/dev/null || true # ---------- Phase 1: Sequencer ---------- echo "[1/6] Sequencer..." -if nc -z 127.0.0.1 3040 2>/dev/null && [ "$FRESH" -eq 0 ]; then +if [ "$SIM_NETWORK" = testnet ]; then + # Fail fast if testnet RPC is unreachable before 10+ min of setup work. + BLOCK_RESP=$(curl -sS -m 10 -X POST -H 'Content-Type: application/json' \ + --data '{"jsonrpc":"2.0","method":"getLastBlockId","params":[],"id":1}' \ + "$TESTNET_RPC_URL" 2>&1) + case "$BLOCK_RESP" in + *'"result"'*) log " Testnet reachable: $(echo "$BLOCK_RESP" | grep -oE '"result":[0-9]+' | head -1)";; + *) die "Testnet unreachable at $TESTNET_RPC_URL: $BLOCK_RESP";; + esac +elif nc -z 127.0.0.1 3040 2>/dev/null && [ "$FRESH" -eq 0 ]; then SEQUENCER_PID=$(lsof -ti tcp:3040 2>/dev/null || true) echo " Already running (PID $SEQUENCER_PID)" else [ "$(nc -z 127.0.0.1 3040 2>/dev/null; echo $?)" = "0" ] && kill "$(lsof -ti tcp:3040 2>/dev/null)" 2>/dev/null || true; sleep 1 rm -rf "$LEZ_RLN_DIR/lssa/rocksdb" "$LEZ_RLN_DIR/lssa/sequencer/service/bedrock_signing_key" - # Skip sequencer build if binary already exists (pre-built in Docker image). - # Also skip lssa auto-sync (requires full git history, fails on shallow clones). + # Pre-built binary path skips both the cargo build and the lssa auto-sync + # (auto-sync needs full git history; shallow Docker clones break it). if [ -x "$LEZ_RLN_DIR/lssa/target/debug/sequencer_service" ]; then log " Using pre-built sequencer" SEQ_BIN="./target/debug/sequencer_service"; SEQ_CFG="sequencer/service/configs/debug/sequencer_config.json" else - # Sequencer guest binaries must match the lssa rev that lez-rln's host-side - # client pins. Mismatch → DeserializeUnexpectedEnd (wire format divergence). + # lssa rev must match what lez-rln's host-side client pins, else + # DeserializeUnexpectedEnd from wire-format divergence. LSSA_REV=$(grep -oE '(rev|tag)\s*=\s*"[^"]+"' "$LEZ_RLN_DIR/lez-rln/Cargo.toml" | head -1 | sed 's/.*"\([^"]*\)"/\1/') [ -z "$LSSA_REV" ] && die "Could not extract lssa rev from lez-rln/Cargo.toml" if ! git -C "$LEZ_RLN_DIR/lssa" merge-base --is-ancestor "$LSSA_REV" HEAD 2>/dev/null; then @@ -160,27 +204,78 @@ fi # ---------- Phase 2: Deploy programs ---------- echo "[2/6] Deploying programs..." -export NSSA_WALLET_HOME_DIR="$LEZ_RLN_DIR/dev" +# `local` -> dev/ (re-init each run); `testnet` -> testnet/ (persistent +# wallet + on-chain state). wallet_config.json under each picks the sequencer. +WALLET_HOME_SUBDIR=$([ "$SIM_NETWORK" = testnet ] && echo testnet || echo dev) +export NSSA_WALLET_HOME_DIR="$LEZ_RLN_DIR/$WALLET_HOME_SUBDIR" export WALLET_CONFIG="$NSSA_WALLET_HOME_DIR/wallet_config.json" export WALLET_STORAGE="$NSSA_WALLET_HOME_DIR/storage.json" -TREE_ID_HEX="000102030405060708090a0b0c0d0e0f10111213141516170000000000000000" +TREE_ID_HEX="000102030405060708090a0b0c0d0e0f10111213141516171a05100200000000" GIFTER_ACCOUNT_FILE="$HOME/.logos-lez-rln/payment_account_${TREE_ID_HEX}.txt" -rm -f "$WALLET_CONFIG" "$WALLET_STORAGE" -# Use pre-built binary if available, otherwise cargo run -if [ -x "$LEZ_RLN_DIR/lez-rln/target/debug/run_setup" ]; then - SETUP_OUTPUT=$(cd "$LEZ_RLN_DIR/lez-rln" && ./target/debug/run_setup 2>&1) || die "run_setup failed" -else - SETUP_OUTPUT=$(cd "$LEZ_RLN_DIR/lez-rln" && cargo run --bin run_setup 2>&1) || die "run_setup failed" +# Local: clean wallet each run so run_setup re-deploys. +# Testnet: persist wallet + on-chain state; run_setup short-circuits via +# is_initialized() into create_funded_user. +if [ "$SIM_NETWORK" = local ] && [ "${SIM_PERSIST_LOCAL:-0}" != "1" ]; then + rm -f "$WALLET_CONFIG" "$WALLET_STORAGE" fi -echo "$SETUP_OUTPUT" | tail -4 -CONFIG_ACCOUNT=$(echo "$SETUP_OUTPUT" | grep -oE 'Config account:\s+\S+' | awk '{print $NF}' || true) -[ -z "$CONFIG_ACCOUNT" ] && die "Failed to parse config account" -GIFTER_ACCOUNT=$(cat "$GIFTER_ACCOUNT_FILE" 2>/dev/null || true) -[ -z "$GIFTER_ACCOUNT" ] && die "Gifter account not found at $GIFTER_ACCOUNT_FILE" -# EIP-191 auth fixtures: per-process secp256k1 keys and the allowlist installed -# on the gifter (mix node 0). Keys are committed test fixtures — not for prod. +# Testnet bootstrap: seed wallet + supply-holding sidecar from the +# submodule-shipped artifacts on first run. After that, create_funded_user +# draws fresh per-dev payment accounts from the shared supply. +if [ "$SIM_NETWORK" = testnet ]; then + # Copy shipped seed -> runtime location iff runtime is missing. + seed_copy() { + local label="$1" src="$2" dst="$3" + [ -f "$dst" ] && return 0 + [ -f "$src" ] || return 0 + log " Seeding $label -> $dst" + mkdir -p "$(dirname "$dst")" + cp "$src" "$dst" + } + seed_copy "testnet wallet" \ + "$LEZ_RLN_DIR/testnet/storage.json.seed" "$WALLET_STORAGE" + seed_copy "supply holding sidecar" \ + "$LEZ_RLN_DIR/testnet/supply_holding.txt" \ + "$HOME/.logos-lez-rln/supply_holding_${TREE_ID_HEX}.txt" + seed_copy "payment account sidecar" \ + "$LEZ_RLN_DIR/testnet/payment_account.txt" \ + "$HOME/.logos-lez-rln/payment_account_${TREE_ID_HEX}.txt" +fi + +# Slim mode (SIM_SLIM=1, testnet only): skip run_setup when the shipped +# config_account + cached payment_account are both present — lets fresh +# clones avoid building lez-rln/run_setup. The shared payment_account is +# signed in storage.json.seed and has enough RLNTOK for ~1M Register txs. +# Experimental; the default run_setup path is better-tested. +CONFIG_ACCOUNT_SEED="$LEZ_RLN_DIR/testnet/config_account.txt" +SLIM=0 +if [ "$SIM_NETWORK" = testnet ] && [ "${SIM_SLIM:-0}" = "1" ] \ + && [ -f "$CONFIG_ACCOUNT_SEED" ] && [ -f "$GIFTER_ACCOUNT_FILE" ]; then + SLIM=1 +fi +if [ "$SLIM" = "1" ]; then + log " Slim mode: skipping run_setup (using shipped config_account + cached payment_account)" + CONFIG_ACCOUNT=$(tr -d '\n\r' < "$CONFIG_ACCOUNT_SEED") + GIFTER_ACCOUNT=$(cat "$GIFTER_ACCOUNT_FILE") +else + if [ -x "$LEZ_RLN_DIR/lez-rln/target/debug/run_setup" ]; then + SETUP_OUTPUT=$(cd "$LEZ_RLN_DIR/lez-rln" && ./target/debug/run_setup 2>&1) || die "run_setup failed" + else + SETUP_OUTPUT=$(cd "$LEZ_RLN_DIR/lez-rln" && cargo run --bin run_setup 2>&1) || die "run_setup failed" + fi + echo "$SETUP_OUTPUT" | tail -4 + # Both deploy + already-initialized branches print "Config account:". + CONFIG_ACCOUNT=$(echo "$SETUP_OUTPUT" | grep -oE 'Config account:\s+\S+' | awk '{print $NF}' || true) + [ -z "$CONFIG_ACCOUNT" ] && die "Failed to parse config account" + GIFTER_ACCOUNT=$(cat "$GIFTER_ACCOUNT_FILE" 2>/dev/null || true) + [ -z "$GIFTER_ACCOUNT" ] && die "Gifter account not found at $GIFTER_ACCOUNT_FILE" +fi +echo " CONFIG_ACCOUNT=$CONFIG_ACCOUNT" +echo " GIFTER_ACCOUNT=$GIFTER_ACCOUNT" + +# EIP-191 auth fixtures: secp256k1 keys + gifter (mix node 0) allowlist. +# Committed test fixtures only — do NOT reuse in prod. GIFTER_AUTH_DIR="$SCRIPT_DIR/fixtures/gifter_auth" # shellcheck disable=SC1091 source "$GIFTER_AUTH_DIR/keys.env" @@ -202,8 +297,8 @@ else DELIVERY_PLUGIN="$DELIVERY_MODULE_DIR/result/lib/delivery_module_plugin.$EXT" fi -# Chat module plugin (for sender/receiver) -# Prefer locally-built liblogoschat over nix result (uses vendored Nim toolchain) +# Chat module plugin (sender/receiver). Prefer locally-built liblogoschat +# (uses vendored Nim toolchain) over the nix result. CHAT_MODULE_RESULT="$CHAT_MODULE_DIR/result" if [ -f "$LOGOS_CHAT_DIR/build/liblogoschat.$EXT" ]; then CHAT_LIB="$LOGOS_CHAT_DIR/build/liblogoschat.$EXT" @@ -229,9 +324,8 @@ LOAD_ORDER="liblogos_execution_zone_wallet_module,liblogos_rln_module,delivery_m WALLET_CALL="liblogos_execution_zone_wallet_module.open($WALLET_CONFIG,$WALLET_STORAGE)" BOOTSTRAP_PEER="/ip4/127.0.0.1/tcp/$BASE_TCP_PORT/p2p/${PEER_IDS[0]}" -# NOTE: Off-chain credential generation (setup_credentials) and pre-registration -# (register_commitments) are not used. All nodes and chat clients register their -# RLN memberships at runtime via the gifter protocol on node 0. +# All RLN memberships are issued at runtime by the gifter on node 0 — no +# off-chain setup_credentials / pre-registration step. for i in $(seq 0 $((NUM_NODES - 1))); do TCP_PORT=$((BASE_TCP_PORT + i)); DISC_PORT=$((BASE_DISC_PORT + i)) @@ -250,7 +344,7 @@ for i in $(seq 0 $((NUM_NODES - 1))); do if [ "$i" -eq 0 ]; then GIFTER_FIELDS="\"mixGifterService\": true, \"mixGifterWalletAccount\": \"$GIFTER_ACCOUNT\", \"mixGifterAllowlist\": \"$GIFTER_ALLOWLIST\"," else - # KEY_MIX1..3 align with non-gifter nodes 1..3. + # KEY_MIX{1..3} align with non-gifter nodes 1..3. AUTH_KEY_VAR="KEY_MIX$i" GIFTER_FIELDS="\"mixGifterNode\": \"$BOOTSTRAP_PEER\", \"mixGifterWalletAccount\": \"$GIFTER_ACCOUNT\", \"mixGifterAuthKey\": \"${!AUTH_KEY_VAR}\"," fi @@ -284,17 +378,14 @@ for i in $(seq 0 $((NUM_NODES - 1))); do EOF MDIR=$(mktemp -d); MODULES_DIRS+=("$MDIR") - # Stage wallet module mkdir -p "$MDIR/liblogos_execution_zone_wallet_module" cp -L "$WALLET_MODULE/liblogos_execution_zone_wallet_module.$EXT" "$MDIR/liblogos_execution_zone_wallet_module/" [ -f "$WALLET_MODULE/libwallet_ffi.$EXT" ] && cp -L "$WALLET_MODULE/libwallet_ffi.$EXT" "$MDIR/liblogos_execution_zone_wallet_module/" echo "{\"name\":\"liblogos_execution_zone_wallet_module\",\"version\":\"1.0.0\",\"type\":\"core\",\"main\":{\"$PLATFORM\":\"liblogos_execution_zone_wallet_module.$EXT\"},\"dependencies\":[],\"capabilities\":[]}" > "$MDIR/liblogos_execution_zone_wallet_module/manifest.json" - # Stage RLN module mkdir -p "$MDIR/liblogos_rln_module" cp -L "$RLN_MODULE/liblogos_rln_module.$EXT" "$MDIR/liblogos_rln_module/" cp -L "$RLN_MODULE/liblez_rln_ffi.$EXT" "$MDIR/liblogos_rln_module/" 2>/dev/null || true echo "{\"name\":\"liblogos_rln_module\",\"version\":\"1.0.0\",\"type\":\"core\",\"main\":{\"$PLATFORM\":\"liblogos_rln_module.$EXT\"},\"dependencies\":[\"liblogos_execution_zone_wallet_module\"],\"capabilities\":[]}" > "$MDIR/liblogos_rln_module/manifest.json" - # Stage delivery module mkdir -p "$MDIR/delivery_module" cp -L "$DELIVERY_PLUGIN" "$MDIR/delivery_module/" if [ -f "$DELIVERY_DIR/build/liblogosdelivery.$EXT" ]; then @@ -315,63 +406,58 @@ EOF "$LOG_FILE" 2>&1) & EXPECTED_CALLS=5 NODE_PID=$!; INSTANCE_PIDS+=($NODE_PID) - WAIT_TIMEOUT=90 - for t in $(seq 1 $WAIT_TIMEOUT); do - N=$(grep -c '^Method call successful' "$LOG_FILE" 2>/dev/null || true); N=${N:-0} - [ "$N" -ge "$EXPECTED_CALLS" ] && break; sleep 1 - done + wait_method_calls "$LOG_FILE" "$EXPECTED_CALLS" 90 1 || true if [ "${N:-0}" -ge "$EXPECTED_CALLS" ]; then log " Node $i ready ($N/$EXPECTED_CALLS calls) PID: $NODE_PID" else echo " WARNING: Node $i: $N/$EXPECTED_CALLS calls" fi - sleep 10 + sleep "$NODE_STARTUP_SLEEP" done echo "" # ---------- Phase 5: Chat module sender/receiver ---------- echo "[5/6] Starting chat module instances..." -# Wait for all nodes to be fully ready (gifter registrations finalize during startup) +# Wait for all nodes ready (gifter registrations finalize during startup). for i in $(seq 0 $((NUM_NODES - 1))); do - LOG_FILE="$STATE_DIR/node${i}.log" - EC=5 - for t in $(seq 1 120); do - N=$(grep -c '^Method call successful' "$LOG_FILE" 2>/dev/null || true); N=${N:-0} - [ "$N" -ge "$EC" ] && break; sleep 2 - done + wait_method_calls "$STATE_DIR/node${i}.log" 5 120 2 || true done -# Wait for mix mesh + RLN root convergence. Requires: 3 gifter registrations on -# node0, LEZ root polling events across all nodes, AND min 30s floor for mix -# protocol handshakes to complete (no single log line signals this cleanly). +# Wait for RLN root convergence. Mix pool is seeded from each node's +# mixNodes config (processBootNodes), so kademlia isn't required for routing — +# the kad-peer count is logged only for diagnostics. echo " Waiting for kademlia propagation + RLN convergence..." KADEMLIA_T0=$SECONDS while true; do ELAPSED=$((SECONDS - KADEMLIA_T0)) GR=$(sed 's/\x1b\[[0-9;]*m//g' "$STATE_DIR/node0.log" 2>/dev/null | grep -c "RLN gifter registration succeeded" || true); GR=${GR:-0} LR=0 + MIX_PEERS_PER_NODE="" for i in $(seq 0 $((NUM_NODES - 1))); do - L=$(sed 's/\x1b\[[0-9;]*m//g' "$STATE_DIR/node${i}.log" 2>/dev/null | grep -c "Polled valid roots\|Fetched roots from\|valid_roots\|OnchainLEZGroupManager initialized\|Wired LEZ callbacks" || true) + LOG="$STATE_DIR/node${i}.log" + L=$(sed 's/\x1b\[[0-9;]*m//g' "$LOG" 2>/dev/null | grep -c "Polled valid roots\|Fetched roots from\|valid_roots\|OnchainLEZGroupManager initialized\|Wired LEZ callbacks" || true) LR=$((LR + L)) + MP=$(sed 's/\x1b\[[0-9;]*m//g' "$LOG" 2>/dev/null | grep -c "mix peer added via kademlia lookup" || true); MP=${MP:-0} + MIX_PEERS_PER_NODE="${MIX_PEERS_PER_NODE}n${i}=${MP} " done if [ "$ELAPSED" -ge "$KADEMLIA_MIN_WAIT" ] && [ "$GR" -ge 3 ] && [ "$LR" -ge 40 ]; then break; fi - [ "$ELAPSED" -ge 120 ] && break + [ "$ELAPSED" -ge "$KADEMLIA_HARD_CAP" ] && break sleep 1 done -log " Kademlia ready after $((SECONDS - KADEMLIA_T0))s ($GR gifter regs, $LR LEZ root events)" +log " Kademlia ready after $((SECONDS - KADEMLIA_T0))s ($GR gifter regs, $LR LEZ root events, kad mix peers: $MIX_PEERS_PER_NODE)" RECEIVER_LOG="$STATE_DIR/chat_receiver.log" SENDER_LOG="$STATE_DIR/chat_sender.log" -# Build mix node list for chat config (multiaddr:mixPubKey format) +# Build mix node list (multiaddr:mixPubKey) for chat config. MIXNODE_LIST="" for j in $(seq 0 $((NUM_NODES - 1))); do [ -n "$MIXNODE_LIST" ] && MIXNODE_LIST="$MIXNODE_LIST," MIXNODE_LIST="$MIXNODE_LIST\"/ip4/127.0.0.1/tcp/$((BASE_TCP_PORT + j))/p2p/${PEER_IDS[$j]}:${MIX_PUBKEYS[$j]}\"" done -# Helper: stage chat_module for a logoscore instance +# Stage wallet + RLN + chat modules for a logoscore instance. stage_chat_module() { local MDIR=$1 mkdir -p "$MDIR/liblogos_execution_zone_wallet_module" @@ -397,7 +483,7 @@ RECV_MDIR=$(mktemp -d); MODULES_DIRS+=("$RECV_MDIR") stage_chat_module "$RECV_MDIR" RECV_CONFIG="$STATE_DIR/chat_receiver_config.json" -# Build static peer list (ENR or multiaddr) for chat nodes to join relay mesh +# Static peers so chat nodes join the relay mesh. CHAT_STATIC_PEERS="" for j in $(seq 0 $((NUM_NODES - 1))); do [ -n "$CHAT_STATIC_PEERS" ] && CHAT_STATIC_PEERS="$CHAT_STATIC_PEERS," @@ -433,16 +519,11 @@ log " Starting receiver..." RECEIVER_PID=$!; INSTANCE_PIDS+=($RECEIVER_PID) log " Receiver PID: $RECEIVER_PID" -# Wait for receiver to complete all method calls (6 calls now including createIntroBundle) RECV_EXPECTED=6 -for t in $(seq 1 180); do - N=$(grep -c '^Method call successful' "$RECEIVER_LOG" 2>/dev/null || true); N=${N:-0} - [ "$N" -ge "$RECV_EXPECTED" ] && break; sleep 2 -done -N=$(grep -c '^Method call successful' "$RECEIVER_LOG" 2>/dev/null || true); N=${N:-0} +wait_method_calls "$RECEIVER_LOG" "$RECV_EXPECTED" 180 2 || true log " Receiver method calls: $N/$RECV_EXPECTED" -# Extract intro bundle from receiver log (event callback delivers it as chatCreateIntroBundleResult) +# Extract intro bundle (emitted as chatCreateIntroBundleResult). INTRO_BUNDLE="" for t in $(seq 1 30); do INTRO_BUNDLE=$(grep -oE 'logos_chatintro_[A-Za-z0-9_-]+' "$RECEIVER_LOG" 2>/dev/null | head -1 || true) @@ -454,8 +535,8 @@ else log " WARNING: Could not extract intro bundle from receiver log" fi -# Wait for receiver's async startChat to finish (Waku client started) AND give -# filter subscription a 15s floor to propagate through the relay mesh. +# Wait for async startChat (Waku client started) + a floor for the filter +# subscription to propagate through the relay mesh. echo " Waiting for receiver to join mix network..." JOIN_T0=$SECONDS while true; do @@ -488,13 +569,13 @@ cat > "$SEND_CONFIG" </dev/null || true); N=${N:-0} - [ "$N" -ge "$SEND_EXPECTED" ] && break; sleep 2 -done +wait_method_calls "$SENDER_LOG" "$SEND_EXPECTED" 180 2 || true -# On slower systems (Docker/ARM), the sender's async gifter registration may not -# complete before newPrivateConversation runs. Wait for gifter + RLN readiness -# before checking for message delivery. If newPrivateConversation initially failed, -# the Nim async code will retry once credentials are valid. +# On slower systems (Docker/ARM) the sender's async gifter registration may +# trail newPrivateConversation. Wait for RLN readiness; the Nim async code +# retries newPrivateConversation once credentials land. echo " Waiting for sender RLN readiness..." SENDER_RLN_T0=$SECONDS for t in $(seq 1 60); do @@ -530,7 +606,7 @@ log " Sender RLN ready after $((SECONDS - SENDER_RLN_T0))s" N=$(grep -c '^Method call successful' "$SENDER_LOG" 2>/dev/null || true); N=${N:-0} log " Sender method calls: $N/$SEND_EXPECTED" -# Poll receiver log for incoming message(s) instead of waiting a fixed 120s. +# Poll receiver log for delivery. echo " Waiting for message delivery via mix..." DELIVERY_T0=$SECONDS for t in $(seq 1 $DELIVERY_TIMEOUT); do