mirror of
https://github.com/logos-blockchain/logos-blockchain-testing.git
synced 2026-01-02 13:23:13 +00:00
docs: comprehensive documentation improvements
- Rename to 'Logos Blockchain Testing Framework Book' - Rebrand protocol references from Nomos to Logos - Add narrative improvements (Core Concept, learning paths, callouts) - Expand best-practices and what-you-will-learn pages - Add maintenance guide (README.md) with doc-snippets documentation - Add Notion documentation links - Fix code example imports and API signatures - Remove all icons/emojis
This commit is contained in:
andrussal
parent
64dfe398e0
commit
f355ead47e
533
book/README.md
Normal file
533
book/README.md
Normal file
@ -0,0 +1,533 @@
|
||||
# Documentation Maintenance Guide
|
||||
|
||||
This guide helps maintainers keep the book synchronized with code changes. Use these checklists when modifying the framework.
|
||||
|
||||
**Key Tool:** The `examples/doc-snippets/` crate contains compilable versions of code examples from the book. Always run `cargo build -p doc-snippets` after API changes to catch broken examples early.
|
||||
|
||||
## Quick Reference: What to Update When
|
||||
|
||||
| Change Type | Pages to Check | Estimated Time |
|
||||
|-------------|----------------|----------------|
|
||||
| API method renamed/changed | [API Changes](#api-changes) | 1-2 hours |
|
||||
| New workload/expectation added | [New Features](#new-features) | 30 minutes |
|
||||
| Environment variable added/changed | [Environment Variables](#environment-variables) | 15 minutes |
|
||||
| Script path/interface changed | [Scripts & Tools](#scripts--tools) | 30 minutes |
|
||||
| New runner/deployer added | [New Runner](#new-runner) | 2-3 hours |
|
||||
| Trait signature changed | [Trait Changes](#trait-changes) | 1-2 hours |
|
||||
|
||||
---
|
||||
|
||||
## Detailed Checklists
|
||||
|
||||
### API Changes
|
||||
|
||||
**When:** Builder API methods, trait methods, or core types change
|
||||
|
||||
**Examples:**
|
||||
- Rename: `.transactions_with()` → `.with_transactions()`
|
||||
- New method: `.with_timeout()`
|
||||
- Parameter change: `.validators(3)` → `.validators(count, config)`
|
||||
|
||||
**Update these pages:**
|
||||
|
||||
```bash
|
||||
# 1. Search for affected API usage
|
||||
rg "old_method_name" book/src/
|
||||
|
||||
# 2. Update these files:
|
||||
- [ ] src/dsl-cheat-sheet.md # Builder API reference (highest priority)
|
||||
- [ ] src/quickstart.md # First example users see
|
||||
- [ ] src/examples.md # 4 complete scenarios
|
||||
- [ ] src/examples-advanced.md # 3 advanced scenarios
|
||||
- [ ] src/introduction.md # "A Scenario in 20 Lines" example
|
||||
- [ ] src/project-context-primer.md # Quick example section
|
||||
- [ ] src/authoring-scenarios.md # Scenario patterns
|
||||
- [ ] src/best-practices.md # Code organization examples
|
||||
- [ ] src/custom-workload-example.md # Complete implementation
|
||||
- [ ] src/extending.md # Trait implementation examples
|
||||
```
|
||||
|
||||
**Verification:**
|
||||
```bash
|
||||
# Compile doc-snippets to catch API breakage
|
||||
cargo build -p doc-snippets
|
||||
|
||||
# Check if book links are valid
|
||||
mdbook test
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### New Features
|
||||
|
||||
#### New Workload or Expectation
|
||||
|
||||
**When:** Adding a new traffic generator or success criterion
|
||||
|
||||
**Examples:**
|
||||
- New workload: `MemoryPressureWorkload`
|
||||
- New expectation: `ExpectZeroDroppedTransactions`
|
||||
|
||||
**Update these pages:**
|
||||
|
||||
```bash
|
||||
- [ ] src/workloads.md # Add to built-in workloads section
|
||||
- [ ] src/dsl-cheat-sheet.md # Add DSL helper if provided
|
||||
- [ ] src/examples-advanced.md # Consider adding example usage
|
||||
- [ ] src/glossary.md # Add term definition
|
||||
- [ ] src/internal-crate-reference.md # Document crate location
|
||||
```
|
||||
|
||||
**Optional (if significant feature):**
|
||||
```bash
|
||||
- [ ] src/what-you-will-learn.md # Add to learning outcomes
|
||||
- [ ] src/best-practices.md # Add usage guidance
|
||||
```
|
||||
|
||||
#### New Runner/Deployer
|
||||
|
||||
**When:** Adding support for a new deployment target (e.g., AWS ECS)
|
||||
|
||||
**Update these pages:**
|
||||
|
||||
```bash
|
||||
# Core documentation
|
||||
- [ ] src/runners.md # Add to comparison table and decision guide
|
||||
- [ ] src/operations-overview.md # Update runner-agnostic matrix
|
||||
- [ ] src/architecture-overview.md # Update deployer list and diagram
|
||||
- [ ] src/running-examples.md # Add runner-specific section
|
||||
|
||||
# Reference pages
|
||||
- [ ] src/dsl-cheat-sheet.md # Add deployer import/usage
|
||||
- [ ] src/internal-crate-reference.md # Document new crate
|
||||
- [ ] src/glossary.md # Add runner type definition
|
||||
|
||||
# Potentially affected
|
||||
- [ ] src/ci-integration.md # Add CI example if applicable
|
||||
- [ ] src/troubleshooting.md # Add common issues
|
||||
- [ ] src/faq.md # Add FAQ entries
|
||||
```
|
||||
|
||||
#### New Topology Helper
|
||||
|
||||
**When:** Adding topology generation helpers (e.g., `.network_mesh()`)
|
||||
|
||||
**Update these pages:**
|
||||
|
||||
```bash
|
||||
- [ ] src/dsl-cheat-sheet.md # Add to topology section
|
||||
- [ ] src/authoring-scenarios.md # Add usage pattern
|
||||
- [ ] src/topology-chaos.md # Add topology description
|
||||
- [ ] src/examples.md # Consider adding example
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Trait Changes
|
||||
|
||||
**When:** Core trait signatures change (breaking changes)
|
||||
|
||||
**Examples:**
|
||||
- `Workload::init()` adds new parameter
|
||||
- `Expectation::evaluate()` changes return type
|
||||
- `Deployer::deploy()` signature update
|
||||
|
||||
**Update these pages:**
|
||||
|
||||
```bash
|
||||
# Critical - these show full trait definitions
|
||||
- [ ] src/extending.md # Complete trait outlines (6+ examples)
|
||||
- [ ] src/custom-workload-example.md # Full implementation example
|
||||
- [ ] src/scenario-model.md # Core model documentation
|
||||
|
||||
# Important - these reference traits
|
||||
- [ ] src/api-levels.md # Trait usage patterns
|
||||
- [ ] src/architecture-overview.md # Extension points diagram
|
||||
- [ ] src/internal-crate-reference.md # Trait locations
|
||||
```
|
||||
|
||||
**Verification:**
|
||||
```bash
|
||||
# Ensure trait examples would compile
|
||||
cargo doc --no-deps --document-private-items
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Environment Variables
|
||||
|
||||
**When:** New environment variable added, changed, or removed
|
||||
|
||||
**Examples:**
|
||||
- New: `NOMOS_NEW_FEATURE_ENABLED`
|
||||
- Changed: `NOMOS_LOG_LEVEL` accepts new values
|
||||
- Deprecated: `OLD_FEATURE_FLAG`
|
||||
|
||||
**Update these pages:**
|
||||
|
||||
```bash
|
||||
# Primary location (single source of truth)
|
||||
- [ ] src/environment-variables.md # Add to appropriate category table
|
||||
|
||||
# Secondary mentions
|
||||
- [ ] src/prerequisites.md # If affects setup
|
||||
- [ ] src/running-examples.md # If affects runner usage
|
||||
- [ ] src/troubleshooting.md # If commonly misconfigured
|
||||
- [ ] src/glossary.md # If significant/commonly referenced
|
||||
```
|
||||
|
||||
**Environment Variables Table Location:**
|
||||
```
|
||||
src/environment-variables.md
|
||||
├─ Runner Configuration
|
||||
├─ Node Binary & Paths
|
||||
├─ Circuit Assets
|
||||
├─ Logging & Tracing
|
||||
├─ Observability & Metrics
|
||||
├─ Proof System
|
||||
├─ Docker & Images
|
||||
├─ Testing Behavior
|
||||
└─ CI/CD
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Scripts & Tools
|
||||
|
||||
**When:** Helper scripts move, rename, or change interface
|
||||
|
||||
**Examples:**
|
||||
- Script moved: `scripts/run-examples.sh` → `scripts/run/run-examples.sh`
|
||||
- New script: `scripts/clean-all.sh`
|
||||
- Interface change: `run-examples.sh` adds new required flag
|
||||
|
||||
**Update these pages:**
|
||||
|
||||
```bash
|
||||
# High impact
|
||||
- [ ] src/quickstart.md # Uses run-examples.sh prominently
|
||||
- [ ] src/running-examples.md # Documents all scripts
|
||||
- [ ] src/prerequisites.md # References setup scripts
|
||||
- [ ] src/examples.md # Script recommendations
|
||||
- [ ] src/examples-advanced.md # Script recommendations
|
||||
|
||||
# Moderate impact
|
||||
- [ ] src/ci-integration.md # May reference scripts in workflows
|
||||
- [ ] src/troubleshooting.md # Cleanup scripts
|
||||
- [ ] src/architecture-overview.md # Asset preparation scripts
|
||||
```
|
||||
|
||||
**Find all script references:**
|
||||
```bash
|
||||
rg "scripts/" book/src/ --no-heading
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Operational Changes
|
||||
|
||||
#### Docker Image Changes
|
||||
|
||||
**When:** Image build process, tag names, or embedded assets change
|
||||
|
||||
**Update these pages:**
|
||||
|
||||
```bash
|
||||
- [ ] src/prerequisites.md # Image build instructions
|
||||
- [ ] src/runners.md # Compose/K8s prerequisites
|
||||
- [ ] src/environment-variables.md # NOMOS_TESTNET_IMAGE, NOMOS_BINARIES_TAR
|
||||
- [ ] src/architecture-overview.md # Assets and Images section
|
||||
```
|
||||
|
||||
#### Observability Stack Changes
|
||||
|
||||
**When:** Prometheus, Grafana, OTLP, or metrics configuration changes
|
||||
|
||||
**Update these pages:**
|
||||
|
||||
```bash
|
||||
- [ ] src/logging-observability.md # Primary documentation
|
||||
- [ ] src/environment-variables.md # NOMOS_METRICS_*, NOMOS_OTLP_*
|
||||
- [ ] src/architecture-overview.md # Observability section
|
||||
- [ ] src/runners.md # Runner observability support
|
||||
```
|
||||
|
||||
#### CI/CD Changes
|
||||
|
||||
**When:** CI workflow changes, new actions, or integration patterns
|
||||
|
||||
**Update these pages:**
|
||||
|
||||
```bash
|
||||
- [ ] src/ci-integration.md # Complete workflow examples
|
||||
- [ ] src/best-practices.md # CI recommendations
|
||||
- [ ] src/operations-overview.md # CI mentioned in runner matrix
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Node Protocol Changes
|
||||
|
||||
**When:** Changes to Logos blockchain protocol or node behavior
|
||||
|
||||
**Examples:**
|
||||
- New consensus parameter
|
||||
- DA protocol change
|
||||
- Network layer update
|
||||
|
||||
**Update these pages:**
|
||||
|
||||
```bash
|
||||
# Context pages (high-level only)
|
||||
- [ ] src/project-context-primer.md # Protocol overview
|
||||
- [ ] src/glossary.md # Protocol terms
|
||||
- [ ] src/faq.md # May need protocol updates
|
||||
|
||||
# Usually NOT affected (framework is protocol-agnostic)
|
||||
- Testing framework abstracts protocol details
|
||||
- Only update if change affects testing methodology
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Crate Structure Changes
|
||||
|
||||
**When:** Crate reorganization, renames, or new crates added
|
||||
|
||||
**Examples:**
|
||||
- New crate: `testing-framework-metrics`
|
||||
- Crate rename: `runner-examples` → `examples`
|
||||
- Module moved: `core::scenario` → `core::model`
|
||||
|
||||
**Update these pages:**
|
||||
|
||||
```bash
|
||||
# Critical
|
||||
- [ ] src/internal-crate-reference.md # Complete crate listing
|
||||
- [ ] src/architecture-overview.md # Crate dependency diagram
|
||||
- [ ] src/workspace-layout.md # Directory structure
|
||||
- [ ] src/annotated-tree.md # File tree with annotations
|
||||
|
||||
# Code examples (update imports)
|
||||
- [ ] src/dsl-cheat-sheet.md # Import statements
|
||||
- [ ] src/extending.md # use statements in examples
|
||||
- [ ] src/custom-workload-example.md # Full imports
|
||||
```
|
||||
|
||||
**Find all import statements:**
|
||||
```bash
|
||||
rg "^use testing_framework" book/src/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing Documentation Changes
|
||||
|
||||
### Build the Book
|
||||
|
||||
```bash
|
||||
cd book
|
||||
mdbook build
|
||||
|
||||
# Output: ../target/book/
|
||||
```
|
||||
|
||||
### Test Documentation
|
||||
|
||||
```bash
|
||||
# Check for broken links
|
||||
mdbook test
|
||||
|
||||
# Preview locally
|
||||
mdbook serve
|
||||
# Open http://localhost:3000
|
||||
```
|
||||
|
||||
### Test Code Examples (Doc Snippets)
|
||||
|
||||
**The `examples/doc-snippets/` crate contains compilable versions of code examples from the book.**
|
||||
|
||||
This ensures examples stay synchronized with the actual API and don't break when code changes.
|
||||
|
||||
**Why doc-snippets exist:**
|
||||
- Code examples in the book (73 blocks across 18 files) can drift from reality
|
||||
- Compilation failures catch API breakage immediately
|
||||
- Single source of truth for code examples
|
||||
|
||||
**Current coverage:** 40+ snippet files corresponding to examples in:
|
||||
- `quickstart.md` (7 snippets)
|
||||
- `examples.md` (4 scenarios)
|
||||
- `examples-advanced.md` (3 scenarios)
|
||||
- `dsl-cheat-sheet.md` (11 DSL examples)
|
||||
- `custom-workload-example.md` (2 trait implementations)
|
||||
- `internal-crate-reference.md` (6 extension examples)
|
||||
- And more...
|
||||
|
||||
**Testing snippets:**
|
||||
|
||||
```bash
|
||||
# Compile all doc snippets
|
||||
cargo build -p doc-snippets
|
||||
|
||||
# Run with full warnings
|
||||
cargo build -p doc-snippets --all-features
|
||||
|
||||
# Check during CI
|
||||
cargo check -p doc-snippets
|
||||
```
|
||||
|
||||
**When to update snippets:**
|
||||
|
||||
1. **API method changed** → Update corresponding snippet file
|
||||
```bash
|
||||
# Example: If .transactions_with() signature changes
|
||||
# Update: examples/doc-snippets/src/examples_transaction_workload.rs
|
||||
```
|
||||
|
||||
2. **New code example added to book** → Create new snippet file
|
||||
```bash
|
||||
# Example: Adding new topology pattern
|
||||
# Create: examples/doc-snippets/src/topology_mesh_example.rs
|
||||
```
|
||||
|
||||
3. **Trait signature changed** → Update trait implementation snippets
|
||||
```bash
|
||||
# Update: custom_workload_example_*.rs
|
||||
# Update: internal_crate_reference_add_*.rs
|
||||
```
|
||||
|
||||
**Snippet naming convention:**
|
||||
```
|
||||
book/src/examples.md → examples_*.rs
|
||||
book/src/quickstart.md → quickstart_*.rs
|
||||
book/src/dsl-cheat-sheet.md → dsl_cheat_sheet_*.rs
|
||||
```
|
||||
|
||||
**Best practice:**
|
||||
When updating code examples in markdown, update the corresponding snippet file first, verify it compiles, then copy to the book. This ensures examples are always valid.
|
||||
|
||||
### Check for Common Issues
|
||||
|
||||
```bash
|
||||
# Find outdated API references
|
||||
rg "old_deprecated_api" src/
|
||||
|
||||
# Find broken GitHub links
|
||||
rg "github.com.*404" src/
|
||||
|
||||
# Find TODO/FIXME markers
|
||||
rg "(TODO|FIXME|XXX)" src/
|
||||
|
||||
# Check for inconsistent terminology
|
||||
rg "(Nomos node|nomos blockchain)" src/ # Should be "Logos"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Maintenance Schedule
|
||||
|
||||
### On Every PR
|
||||
|
||||
- [ ] Check if changes affect documented APIs
|
||||
- [ ] Update relevant pages per checklist above
|
||||
- [ ] Update corresponding doc-snippets if code examples changed
|
||||
- [ ] Run `cargo build -p doc-snippets` to verify examples compile
|
||||
- [ ] Build book to verify no broken links
|
||||
- [ ] Verify code examples still make sense
|
||||
|
||||
### Monthly
|
||||
|
||||
- [ ] Review recent PRs for documentation impact
|
||||
- [ ] Update environment variables table
|
||||
- [ ] Check script references are current
|
||||
- [ ] Verify GitHub source links are not 404
|
||||
|
||||
### Quarterly
|
||||
|
||||
- [ ] Full audit of code examples against latest API
|
||||
- [ ] Verify all doc-snippets still compile with latest dependencies
|
||||
- [ ] Check for code examples in book that don't have corresponding snippets
|
||||
- [ ] Review troubleshooting for new patterns
|
||||
- [ ] Update FAQ with common questions
|
||||
- [ ] Check all Mermaid diagrams render correctly
|
||||
|
||||
### Major Release
|
||||
|
||||
- [ ] Complete review of all technical content
|
||||
- [ ] Verify all version-specific references
|
||||
- [ ] Update "What You Will Learn" outcomes
|
||||
- [ ] Add release notes for documentation changes
|
||||
|
||||
---
|
||||
|
||||
## Content Organization
|
||||
|
||||
### Stability Tiers (Change Frequency)
|
||||
|
||||
**Stable (Rarely Change)**
|
||||
- Part I — Foundations (philosophy, architecture, design rationale)
|
||||
- Part VI — Appendix (glossary, FAQ, troubleshooting symptoms)
|
||||
- Front matter (project context, introduction)
|
||||
|
||||
**Semi-Stable (Occasional Changes)**
|
||||
- Part II — User Guide (usage patterns, best practices, examples)
|
||||
- Part V — Operations (prerequisites, CI, logging)
|
||||
|
||||
**High Volatility (Frequent Changes)**
|
||||
- API references (dsl-cheat-sheet.md, extending.md)
|
||||
- Code examples (73 blocks across 18 files)
|
||||
- Environment variables (50+ documented)
|
||||
- Runner comparisons (features evolve)
|
||||
|
||||
### Page Dependency Map
|
||||
|
||||
**Core pages** (many other pages reference these):
|
||||
- `dsl-cheat-sheet.md` ← Referenced by examples, quickstart, authoring
|
||||
- `environment-variables.md` ← Referenced by operations, troubleshooting, runners
|
||||
- `runners.md` ← Referenced by operations, quickstart, examples
|
||||
- `glossary.md` ← Referenced throughout the book
|
||||
|
||||
**When updating core pages, check for broken cross-references.**
|
||||
|
||||
---
|
||||
|
||||
## Common Patterns
|
||||
|
||||
### Adding a Code Example
|
||||
|
||||
```markdown
|
||||
# 1. Add the code block
|
||||
```rust
|
||||
use testing_framework_core::scenario::ScenarioBuilder;
|
||||
// ... example code
|
||||
```
|
||||
|
||||
# 2. Add context
|
||||
**When to use:** [explain use case]
|
||||
|
||||
# 3. Link to complete source (if applicable)
|
||||
[View in source](https://github.com/logos-blockchain/logos-blockchain-testing/blob/master/examples/src/bin/example.rs)
|
||||
```
|
||||
|
||||
### Adding a Cross-Reference
|
||||
|
||||
```markdown
|
||||
See [Environment Variables](environment-variables.md) for complete configuration reference.
|
||||
```
|
||||
|
||||
### Adding a "When to Read" Callout
|
||||
|
||||
```markdown
|
||||
> **When should I read this?** [guidance on when this content is relevant]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Contact & Questions
|
||||
|
||||
When in doubt:
|
||||
1. Check this README for guidance
|
||||
2. Review recent similar changes in git history
|
||||
3. Ask the team in technical documentation discussions
|
||||
|
||||
**Remember:** Documentation quality directly impacts framework adoption and user success. Taking time to update docs properly is an investment in the project's future.
|
||||
@ -2,7 +2,7 @@
|
||||
authors = ["Nomos Testing"]
|
||||
language = "en"
|
||||
src = "src"
|
||||
title = "Nomos Testing Book"
|
||||
title = "Logos Blockchain Testing Framework Book"
|
||||
|
||||
[build]
|
||||
# Keep book output in target/ to avoid polluting the workspace root.
|
||||
|
||||
@ -92,14 +92,15 @@ Mixing is common: use the DSL for built-ins, and direct instantiation for custom
|
||||
use std::time::Duration;
|
||||
|
||||
use testing_framework_core::scenario::ScenarioBuilder;
|
||||
use testing_framework_workflows::ScenarioBuilderExt;
|
||||
use testing_framework_workflows::{ScenarioBuilderExt, workloads::transaction};
|
||||
|
||||
let custom_workload = MyCustomWorkload::new(config);
|
||||
let tx_workload = transaction::Workload::with_rate(5)
|
||||
.expect("transaction rate must be non-zero");
|
||||
|
||||
let plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(2))
|
||||
.transactions_with(|txs| txs.rate(5).users(3)) // DSL
|
||||
.with_workload(custom_workload) // direct
|
||||
.expect_consensus_liveness() // DSL
|
||||
.wallets(5)
|
||||
.with_workload(tx_workload) // direct instantiation
|
||||
.expect_consensus_liveness() // DSL
|
||||
.with_run_duration(Duration::from_secs(60))
|
||||
.build();
|
||||
```
|
||||
@ -108,15 +109,11 @@ let plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(3).exe
|
||||
|
||||
The DSL methods are thin wrappers. For example:
|
||||
|
||||
```rust
|
||||
builder.transactions_with(|txs| txs.rate(5).users(3))
|
||||
```
|
||||
`builder.transactions_with(|txs| txs.rate(5).users(3))`
|
||||
|
||||
is roughly equivalent to:
|
||||
|
||||
```rust
|
||||
builder.transactions().rate(5).users(3).apply()
|
||||
```
|
||||
`builder.transactions().rate(5).users(3).apply()`
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
|
||||
@ -1,17 +1,248 @@
|
||||
# Best Practices
|
||||
|
||||
- **State your intent**: document the goal of each scenario (throughput, DA
|
||||
validation, resilience) so expectation choices are obvious.
|
||||
- **Keep runs meaningful**: choose durations that allow multiple blocks and make
|
||||
timing-based assertions trustworthy.
|
||||
- **Separate concerns**: start with deterministic workloads for functional
|
||||
checks; add chaos in dedicated resilience scenarios to avoid noisy failures.
|
||||
- **Reuse patterns**: standardize on shared topology and workload presets so
|
||||
results are comparable across environments and teams.
|
||||
- **Observe first, tune second**: rely on liveness and inclusion signals to
|
||||
interpret outcomes before tweaking rates or topology.
|
||||
- **Environment fit**: pick runners that match the feedback loop you need—local
|
||||
for speed (including fast CI smoke tests), compose for reproducible stacks
|
||||
(recommended for CI), k8s for cluster-grade fidelity.
|
||||
- **Minimal surprises**: seed only necessary wallets and keep configuration
|
||||
deltas explicit when moving between CI and developer machines.
|
||||
This page collects proven patterns for authoring, running, and maintaining test scenarios that are reliable, maintainable, and actionable.
|
||||
|
||||
## Scenario Design
|
||||
|
||||
**State your intent**
|
||||
- Document the goal of each scenario (throughput, DA validation, resilience) so expectation choices are obvious
|
||||
- Use descriptive variable names that explain topology purpose (e.g., `star_topology_3val_2exec` vs `topology`)
|
||||
- Add comments explaining why specific rates or durations were chosen
|
||||
|
||||
**Keep runs meaningful**
|
||||
- Choose durations that allow multiple blocks and make timing-based assertions trustworthy
|
||||
- Use [FAQ: Run Duration Calculator](faq.md#how-long-should-a-scenario-run) to estimate minimum duration
|
||||
- Avoid runs shorter than 30 seconds unless testing startup behavior specifically
|
||||
|
||||
**Separate concerns**
|
||||
- Start with deterministic workloads for functional checks
|
||||
- Add chaos in dedicated resilience scenarios to avoid noisy failures
|
||||
- Don't mix high transaction load with aggressive chaos in the same test (hard to debug)
|
||||
|
||||
**Start small, scale up**
|
||||
- Begin with minimal topology (1-2 validators) to validate scenario logic
|
||||
- Gradually increase topology size and workload rates
|
||||
- Use Host runner for fast iteration, then validate on Compose before production
|
||||
|
||||
## Code Organization
|
||||
|
||||
**Reuse patterns**
|
||||
- Standardize on shared topology and workload presets so results are comparable across environments and teams
|
||||
- Extract common topology builders into helper functions
|
||||
- Create workspace-level constants for standard rates and durations
|
||||
|
||||
**Example: Topology preset**
|
||||
|
||||
```rust
|
||||
pub fn standard_da_topology() -> GeneratedTopology {
|
||||
TopologyBuilder::new()
|
||||
.network_star()
|
||||
.validators(3)
|
||||
.executors(2)
|
||||
.generate()
|
||||
}
|
||||
```
|
||||
|
||||
**Example: Shared constants**
|
||||
|
||||
```rust
|
||||
pub const STANDARD_TX_RATE: f64 = 10.0;
|
||||
pub const STANDARD_DA_CHANNEL_RATE: f64 = 2.0;
|
||||
pub const SHORT_RUN_DURATION: Duration = Duration::from_secs(60);
|
||||
pub const LONG_RUN_DURATION: Duration = Duration::from_secs(300);
|
||||
```
|
||||
|
||||
## Debugging & Observability
|
||||
|
||||
**Observe first, tune second**
|
||||
- Rely on liveness and inclusion signals to interpret outcomes before tweaking rates or topology
|
||||
- Enable detailed logging (`RUST_LOG=debug`, `NOMOS_LOG_LEVEL=debug`) only after initial failure
|
||||
- Use `NOMOS_TESTS_KEEP_LOGS=1` to persist logs when debugging failures
|
||||
|
||||
**Use BlockFeed effectively**
|
||||
- Subscribe to BlockFeed in expectations for real-time block monitoring
|
||||
- Track block production rate to detect liveness issues early
|
||||
- Use block statistics (`block_feed.stats().total_transactions()`) to verify inclusion
|
||||
|
||||
**Collect metrics**
|
||||
- Set up Prometheus/Grafana via `scripts/observability/compose/up.sh` for visualizing node behavior
|
||||
- Use metrics to identify bottlenecks before adding more load
|
||||
- Monitor mempool size, block size, and consensus timing
|
||||
|
||||
## Environment & Runner Selection
|
||||
|
||||
**Environment fit**
|
||||
- Pick runners that match the feedback loop you need:
|
||||
- **Host**: Fast iteration during development, quick CI smoke tests
|
||||
- **Compose**: Reproducible environments (recommended for CI), chaos testing
|
||||
- **K8s**: Production-like fidelity, large topologies (10+ nodes)
|
||||
|
||||
**Runner-specific considerations**
|
||||
|
||||
| Runner | When to Use | When to Avoid |
|
||||
|--------|-------------|---------------|
|
||||
| Host | Development iteration, fast feedback | Chaos testing, container-specific issues |
|
||||
| Compose | CI pipelines, chaos tests, reproducibility | Very large topologies (>10 nodes) |
|
||||
| K8s | Production-like testing, cluster behaviors | Local development, fast iteration |
|
||||
|
||||
**Minimal surprises**
|
||||
- Seed only necessary wallets and keep configuration deltas explicit when moving between CI and developer machines
|
||||
- Use `versions.env` to pin node versions consistently across environments
|
||||
- Document non-default environment variables in scenario comments or README
|
||||
|
||||
## CI/CD Integration
|
||||
|
||||
**Use matrix builds**
|
||||
|
||||
```yaml
|
||||
strategy:
|
||||
matrix:
|
||||
runner: [host, compose]
|
||||
topology: [small, medium]
|
||||
```
|
||||
|
||||
**Cache aggressively**
|
||||
- Cache Rust build artifacts (`target/`)
|
||||
- Cache circuit parameters (`assets/stack/kzgrs_test_params/`)
|
||||
- Cache Docker layers (use BuildKit cache)
|
||||
|
||||
**Collect logs on failure**
|
||||
|
||||
```yaml
|
||||
- name: Collect logs on failure
|
||||
if: failure()
|
||||
run: |
|
||||
mkdir -p test-logs
|
||||
find /tmp -name "nomos-*.log" -exec cp {} test-logs/ \;
|
||||
- uses: actions/upload-artifact@v3
|
||||
if: failure()
|
||||
with:
|
||||
name: test-logs-${{ matrix.runner }}
|
||||
path: test-logs/
|
||||
```
|
||||
|
||||
**Time limits**
|
||||
- Set job timeout to prevent hung runs: `timeout-minutes: 30`
|
||||
- Use shorter durations in CI (60s) vs local testing (300s)
|
||||
- Run expensive tests (k8s, large topologies) only on main branch or release tags
|
||||
|
||||
**See also:** [CI Integration](ci-integration.md) for complete workflow examples
|
||||
|
||||
## Anti-Patterns to Avoid
|
||||
|
||||
**DON'T: Run without POL_PROOF_DEV_MODE**
|
||||
```bash
|
||||
# BAD: Will hang/timeout on proof generation
|
||||
cargo run -p runner-examples --bin local_runner
|
||||
|
||||
# GOOD: Fast mode for testing
|
||||
POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin local_runner
|
||||
```
|
||||
|
||||
**DON'T: Use tiny durations**
|
||||
```rust
|
||||
// BAD: Not enough time for blocks to propagate
|
||||
.with_run_duration(Duration::from_secs(5))
|
||||
|
||||
// GOOD: Allow multiple consensus rounds
|
||||
.with_run_duration(Duration::from_secs(60))
|
||||
```
|
||||
|
||||
**DON'T: Ignore cleanup failures**
|
||||
```rust
|
||||
// BAD: Next run inherits leaked state
|
||||
runner.run(&mut scenario).await?;
|
||||
// forgot to call cleanup or use CleanupGuard
|
||||
|
||||
// GOOD: Cleanup via guard (automatic on panic)
|
||||
let _cleanup = CleanupGuard::new(runner.clone());
|
||||
runner.run(&mut scenario).await?;
|
||||
```
|
||||
|
||||
**DON'T: Mix concerns in one scenario**
|
||||
```rust
|
||||
// BAD: Hard to debug when it fails
|
||||
.transactions_with(|tx| tx.rate(50).users(100)) // high load
|
||||
.chaos_with(|c| c.restart().min_delay(...)) // AND chaos
|
||||
.da_with(|da| da.channel_rate(10).blob_rate(20)) // AND DA stress
|
||||
|
||||
// GOOD: Separate tests for each concern
|
||||
// Test 1: High transaction load only
|
||||
// Test 2: Chaos resilience only
|
||||
// Test 3: DA stress only
|
||||
```
|
||||
|
||||
**DON'T: Hardcode paths or ports**
|
||||
```rust
|
||||
// BAD: Breaks on different machines
|
||||
let path = PathBuf::from("/home/user/circuits/kzgrs_test_params");
|
||||
let port = 9000; // might conflict
|
||||
|
||||
// GOOD: Use env vars and dynamic allocation
|
||||
let path = std::env::var("NOMOS_KZGRS_PARAMS_PATH")
|
||||
.unwrap_or_else(|_| "assets/stack/kzgrs_test_params/kzgrs_test_params".to_string());
|
||||
let port = get_available_tcp_port();
|
||||
```
|
||||
|
||||
**DON'T: Ignore resource limits**
|
||||
```bash
|
||||
# BAD: Large topology without checking resources
|
||||
scripts/run/run-examples.sh -v 20 -e 10 compose
|
||||
# (might OOM or exhaust ulimits)
|
||||
|
||||
# GOOD: Scale gradually and monitor resources
|
||||
scripts/run/run-examples.sh -v 3 -e 2 compose # start small
|
||||
docker stats # monitor resource usage
|
||||
# then increase if resources allow
|
||||
```
|
||||
|
||||
## Scenario Design Heuristics
|
||||
|
||||
**Minimal viable topology**
|
||||
- Consensus: 3 validators (minimum for Byzantine fault tolerance)
|
||||
- DA: 2+ executors (test dispersal and sampling)
|
||||
- Network: Star topology (simplest for debugging)
|
||||
|
||||
**Workload rate selection**
|
||||
- Start with 1-5 tx/s per user, then increase
|
||||
- DA: 1-2 channels, 1-3 blobs/channel initially
|
||||
- Chaos: 30s+ intervals between restarts (allow recovery)
|
||||
|
||||
**Duration guidelines**
|
||||
|
||||
| Test Type | Minimum Duration | Typical Duration |
|
||||
|-----------|------------------|------------------|
|
||||
| Smoke test | 30s | 60s |
|
||||
| Integration test | 60s | 120s |
|
||||
| Load test | 120s | 300s |
|
||||
| Resilience test | 120s | 300s |
|
||||
| Soak test | 600s (10m) | 3600s (1h) |
|
||||
|
||||
**Expectation selection**
|
||||
|
||||
| Test Goal | Expectations |
|
||||
|-----------|--------------|
|
||||
| Basic functionality | `expect_consensus_liveness()` |
|
||||
| Transaction handling | `expect_consensus_liveness()` + custom inclusion check |
|
||||
| DA correctness | `expect_consensus_liveness()` + DA dispersal/sampling checks |
|
||||
| Resilience | `expect_consensus_liveness()` + recovery time measurement |
|
||||
|
||||
## Testing the Tests
|
||||
|
||||
**Validate scenarios before committing**
|
||||
1. Run on Host runner first (fast feedback)
|
||||
2. Run on Compose runner (reproducibility check)
|
||||
3. Check logs for warnings or errors
|
||||
4. Verify cleanup (no leaked processes/containers)
|
||||
5. Run 2-3 times to check for flakiness
|
||||
|
||||
**Handling flaky tests**
|
||||
- Increase run duration (timing-sensitive assertions need longer runs)
|
||||
- Reduce workload rates (might be saturating nodes)
|
||||
- Check resource limits (CPU/RAM/ulimits)
|
||||
- Add debugging output to identify race conditions
|
||||
- Consider if test is over-specified (too strict expectations)
|
||||
|
||||
**See also:**
|
||||
- [Troubleshooting](troubleshooting.md) for common failure patterns
|
||||
- [FAQ](faq.md) for design decisions and gotchas
|
||||
|
||||
@ -1,5 +1,7 @@
|
||||
# Chaos Workloads
|
||||
|
||||
> **When should I read this?** You don't need chaos testing to be productive with the framework. Focus on basic scenarios first—chaos is for resilience validation and operational readiness drills once your core tests are stable.
|
||||
|
||||
Chaos in the framework uses node control to introduce failures and validate
|
||||
recovery. The built-in restart workload lives in
|
||||
`testing_framework_workflows::workloads::chaos::RandomRestartWorkload`.
|
||||
|
||||
@ -1,5 +1,7 @@
|
||||
# Advanced Examples
|
||||
|
||||
> **When should I read this?** Skim now to see what's possible, revisit later when you need load testing, chaos scenarios, or custom extensions. Start with [basic examples](examples.md) first.
|
||||
|
||||
Realistic advanced scenarios demonstrating framework capabilities for production testing.
|
||||
|
||||
**Adapt from Complete Source:**
|
||||
|
||||
@ -20,6 +20,19 @@ use testing_framework_core::scenario::{
|
||||
};
|
||||
use testing_framework_core::topology::generation::GeneratedTopology;
|
||||
|
||||
struct MyExpectation;
|
||||
|
||||
#[async_trait]
|
||||
impl Expectation for MyExpectation {
|
||||
fn name(&self) -> &str {
|
||||
"my_expectation"
|
||||
}
|
||||
|
||||
async fn evaluate(&mut self, _ctx: &RunContext) -> Result<(), DynError> {
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
pub struct MyWorkload {
|
||||
// Configuration fields
|
||||
target_rate: u64,
|
||||
@ -39,7 +52,7 @@ impl Workload for MyWorkload {
|
||||
|
||||
fn expectations(&self) -> Vec<Box<dyn Expectation>> {
|
||||
// Return bundled expectations that should run with this workload
|
||||
vec![Box::new(MyExpectation::new(self.target_rate))]
|
||||
vec![Box::new(MyExpectation)]
|
||||
}
|
||||
|
||||
fn init(
|
||||
@ -60,7 +73,7 @@ impl Workload for MyWorkload {
|
||||
|
||||
for client in clients {
|
||||
let info = client.consensus_info().await?;
|
||||
tracing::info!(?info, "workload queried node");
|
||||
tracing::info!(height = info.height, "workload queried node");
|
||||
}
|
||||
|
||||
Ok(())
|
||||
@ -117,7 +130,7 @@ impl Expectation for MyExpectation {
|
||||
.ok_or("no validators")?;
|
||||
|
||||
let info = client.consensus_info().await?;
|
||||
self.captured_baseline = Some(info.current_block_id.slot);
|
||||
self.captured_baseline = Some(info.height);
|
||||
|
||||
tracing::info!(baseline = self.captured_baseline, "captured baseline");
|
||||
Ok(())
|
||||
@ -129,10 +142,10 @@ impl Expectation for MyExpectation {
|
||||
.ok_or("no validators")?;
|
||||
|
||||
let info = client.consensus_info().await?;
|
||||
let final_slot = info.current_block_id.slot;
|
||||
let final_height = info.height;
|
||||
|
||||
let baseline = self.captured_baseline.unwrap_or(0);
|
||||
let delta = final_slot.saturating_sub(baseline);
|
||||
let delta = final_height.saturating_sub(baseline);
|
||||
|
||||
if delta < self.expected_value {
|
||||
return Err(format!(
|
||||
@ -198,7 +211,11 @@ impl Deployer<()> for MyDeployer {
|
||||
let topology: Option<Topology> = None; // Some(topology) if you spawned one
|
||||
let node_clients = NodeClients::default(); // Or NodeClients::from_topology(...)
|
||||
|
||||
let (block_feed, block_feed_guard) = spawn_block_feed(&node_clients).await?;
|
||||
let client = node_clients
|
||||
.any_client()
|
||||
.ok_or("no api clients available")?
|
||||
.clone();
|
||||
let (block_feed, block_feed_guard) = spawn_block_feed(client).await?;
|
||||
|
||||
let telemetry = Metrics::empty(); // or Metrics::from_prometheus(...)
|
||||
let node_control = None; // or Some(Arc<dyn NodeControlHandle>)
|
||||
@ -237,28 +254,18 @@ impl Deployer<()> for MyDeployer {
|
||||
**Example:**
|
||||
|
||||
```rust
|
||||
use testing_framework_core::topology::config::TopologyBuilder;
|
||||
use testing_framework_core::topology::{
|
||||
config::TopologyBuilder,
|
||||
configs::network::Libp2pNetworkLayout,
|
||||
};
|
||||
|
||||
impl TopologyBuilder {
|
||||
/// Creates a "ring" topology where each node connects to its neighbors
|
||||
pub fn network_ring(&mut self) -> &mut Self {
|
||||
// Configure peer connections in a ring layout
|
||||
self.with_network_layout(|layout| {
|
||||
// Implement ring connection logic
|
||||
layout.ring_peers()
|
||||
});
|
||||
self
|
||||
}
|
||||
pub trait TopologyBuilderExt {
|
||||
fn network_full(self) -> Self;
|
||||
}
|
||||
|
||||
/// Preset for high-throughput DA configuration
|
||||
pub fn da_high_throughput(&mut self) -> &mut Self {
|
||||
self.with_da_params(|params| {
|
||||
params
|
||||
.dispersal_factor(8)
|
||||
.replication_factor(16)
|
||||
.chunk_size(4096)
|
||||
});
|
||||
self
|
||||
impl TopologyBuilderExt for TopologyBuilder {
|
||||
fn network_full(self) -> Self {
|
||||
self.with_network_layout(Libp2pNetworkLayout::Full)
|
||||
}
|
||||
}
|
||||
```
|
||||
@ -273,7 +280,49 @@ impl TopologyBuilder {
|
||||
To expose your custom workload through the high-level DSL, add a trait extension:
|
||||
|
||||
```rust
|
||||
use testing_framework_core::scenario::Builder as ScenarioBuilder;
|
||||
use async_trait::async_trait;
|
||||
use testing_framework_core::scenario::{DynError, RunContext, ScenarioBuilder, Workload};
|
||||
|
||||
#[derive(Default)]
|
||||
pub struct MyWorkloadBuilder {
|
||||
target_rate: u64,
|
||||
some_option: bool,
|
||||
}
|
||||
|
||||
impl MyWorkloadBuilder {
|
||||
pub const fn target_rate(mut self, target_rate: u64) -> Self {
|
||||
self.target_rate = target_rate;
|
||||
self
|
||||
}
|
||||
|
||||
pub const fn some_option(mut self, some_option: bool) -> Self {
|
||||
self.some_option = some_option;
|
||||
self
|
||||
}
|
||||
|
||||
pub const fn build(self) -> MyWorkload {
|
||||
MyWorkload {
|
||||
target_rate: self.target_rate,
|
||||
some_option: self.some_option,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub struct MyWorkload {
|
||||
target_rate: u64,
|
||||
some_option: bool,
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl Workload for MyWorkload {
|
||||
fn name(&self) -> &str {
|
||||
"my_workload"
|
||||
}
|
||||
|
||||
async fn start(&self, _ctx: &RunContext) -> Result<(), DynError> {
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
pub trait MyWorkloadDsl {
|
||||
fn my_workload_with(
|
||||
@ -282,7 +331,7 @@ pub trait MyWorkloadDsl {
|
||||
) -> Self;
|
||||
}
|
||||
|
||||
impl<Caps> MyWorkloadDsl for ScenarioBuilder<Caps> {
|
||||
impl MyWorkloadDsl for ScenarioBuilder {
|
||||
fn my_workload_with(
|
||||
self,
|
||||
f: impl FnOnce(MyWorkloadBuilder) -> MyWorkloadBuilder,
|
||||
@ -296,7 +345,7 @@ impl<Caps> MyWorkloadDsl for ScenarioBuilder<Caps> {
|
||||
Users can then call:
|
||||
|
||||
```rust
|
||||
ScenarioBuilder::topology_with(|t| { /* ... */ })
|
||||
ScenarioBuilder::topology_with(|t| t.network_star().validators(1).executors(1))
|
||||
.my_workload_with(|w| {
|
||||
w.target_rate(10)
|
||||
.some_option(true)
|
||||
|
||||
@ -42,7 +42,7 @@
|
||||
- **State assertion**: expectation that verifies specific values in the system
|
||||
state (e.g., wallet balances, UTXO sets) rather than just progress signals.
|
||||
Also called "correctness expectations."
|
||||
- **Mantle transaction**: transaction type in Nomos that can contain UTXO transfers
|
||||
- **Mantle transaction**: transaction type in Logos that can contain UTXO transfers
|
||||
(LedgerTx) and operations (Op), including channel data (ChannelBlob).
|
||||
- **Channel**: logical grouping for DA blobs; each blob belongs to a channel and
|
||||
references a parent blob in the same channel, creating a chain of related data.
|
||||
@ -50,3 +50,9 @@
|
||||
proof generation for leader election. **Required for all runners** (local, compose, k8s)
|
||||
for practical testing—without it, proof generation causes timeouts. Should never be
|
||||
used in production environments.
|
||||
|
||||
---
|
||||
|
||||
## External Resources
|
||||
|
||||
- **[Nomos Project Documentation](https://nomos-tech.notion.site/project)** — Protocol specifications, node internals, and architecture details
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
# Introduction
|
||||
|
||||
The Nomos Testing Framework is a purpose-built toolkit for exercising Nomos in
|
||||
The Nomos Testing Framework is a purpose-built toolkit for exercising Logos in
|
||||
realistic, multi-node environments. It solves the gap between small, isolated
|
||||
tests and full-system validation by letting teams describe a cluster layout,
|
||||
drive meaningful traffic, and assert the outcomes in one coherent plan.
|
||||
@ -9,7 +9,38 @@ It is for protocol engineers, infrastructure operators, and QA teams who need
|
||||
repeatable confidence that validators, executors, and data-availability
|
||||
components work together under network and timing constraints.
|
||||
|
||||
Multi-node integration testing is required because many Nomos behaviors—block
|
||||
Multi-node integration testing is required because many Logos behaviors—block
|
||||
progress, data availability, liveness under churn—only emerge when several
|
||||
roles interact over real networking and time. This framework makes those checks
|
||||
declarative, observable, and portable across environments.
|
||||
|
||||
## A Scenario in 20 Lines
|
||||
|
||||
Here's the conceptual shape of every test you'll write:
|
||||
|
||||
```rust
|
||||
// 1. Define the cluster
|
||||
let scenario = ScenarioBuilder::topology_with(|t| {
|
||||
t.network_star()
|
||||
.validators(3)
|
||||
.executors(2)
|
||||
})
|
||||
// 2. Add workloads (traffic)
|
||||
.transactions_with(|tx| tx.rate(10).users(5))
|
||||
.da_with(|da| da.channel_rate(2).blob_rate(2))
|
||||
|
||||
// 3. Define success criteria
|
||||
.expect_consensus_liveness()
|
||||
|
||||
// 4. Set experiment duration
|
||||
.with_run_duration(Duration::from_secs(60))
|
||||
.build();
|
||||
|
||||
// 5. Deploy and run
|
||||
let runner = deployer.deploy(&scenario).await?;
|
||||
runner.run(&mut scenario).await?;
|
||||
```
|
||||
|
||||
This pattern—topology, workloads, expectations, duration—repeats across all scenarios in this book.
|
||||
|
||||
**Learn more:** For protocol-level documentation and node internals, see the [Nomos Project Documentation](https://nomos-tech.notion.site/project).
|
||||
|
||||
@ -291,13 +291,14 @@ POL_PROOF_DEV_MODE=true scripts/run/run-examples.sh -t 60 -v 3 -e 1 compose
|
||||
**Example usage in expectations:**
|
||||
|
||||
```rust
|
||||
async fn evaluate(&self, ctx: &RunContext) -> Result<(), DynError> {
|
||||
let clients = ctx.node_clients().validator_clients();
|
||||
let client = &clients[0];
|
||||
|
||||
use testing_framework_core::scenario::{DynError, RunContext};
|
||||
|
||||
async fn evaluate(ctx: &RunContext) -> Result<(), DynError> {
|
||||
let client = &ctx.node_clients().validator_clients()[0];
|
||||
|
||||
let info = client.consensus_info().await?;
|
||||
tracing::info!(?info, "consensus info from validator 0");
|
||||
|
||||
tracing::info!(height = info.height, "consensus info from validator 0");
|
||||
|
||||
Ok(())
|
||||
}
|
||||
```
|
||||
@ -362,4 +363,3 @@ kubectl logs -n nomos-debug -l nomos/logical-role=validator
|
||||
- [Troubleshooting](troubleshooting.md) — Log-related debugging (see "Where to Find Logs")
|
||||
- [Running Examples](running-examples.md) — Runner-specific logging details
|
||||
- [Prerequisites & Setup](prerequisites.md) — Setup before running
|
||||
|
||||
|
||||
@ -43,10 +43,13 @@ Expectations typically use BlockFeed to verify block production and inclusion of
|
||||
**Example: Counting blocks during a run**
|
||||
|
||||
```rust
|
||||
use std::sync::{Arc, atomic::{AtomicU64, Ordering}};
|
||||
use std::sync::{
|
||||
Arc,
|
||||
atomic::{AtomicU64, Ordering},
|
||||
};
|
||||
|
||||
use async_trait::async_trait;
|
||||
use testing_framework_core::scenario::{DynError, Expectation, RunContext};
|
||||
use tokio::{spawn, time::sleep, select, pin};
|
||||
|
||||
struct MinimumBlocksExpectation {
|
||||
min_blocks: u64,
|
||||
@ -67,7 +70,7 @@ impl Expectation for MinimumBlocksExpectation {
|
||||
let mut receiver = ctx.block_feed().subscribe();
|
||||
|
||||
// Spawn a task to count blocks
|
||||
spawn(async move {
|
||||
tokio::spawn(async move {
|
||||
loop {
|
||||
match receiver.recv().await {
|
||||
Ok(_record) => {
|
||||
@ -110,12 +113,12 @@ impl Expectation for MinimumBlocksExpectation {
|
||||
**Example: Inspecting block contents**
|
||||
|
||||
```rust
|
||||
use testing_framework_core::scenario::{BlockRecord, RunContext};
|
||||
use testing_framework_core::scenario::{DynError, RunContext};
|
||||
|
||||
async fn start_capture(&mut self, ctx: &RunContext) -> Result<(), DynError> {
|
||||
async fn start_capture(ctx: &RunContext) -> Result<(), DynError> {
|
||||
let mut receiver = ctx.block_feed().subscribe();
|
||||
|
||||
spawn(async move {
|
||||
tokio::spawn(async move {
|
||||
loop {
|
||||
match receiver.recv().await {
|
||||
Ok(record) => {
|
||||
@ -123,8 +126,7 @@ async fn start_capture(&mut self, ctx: &RunContext) -> Result<(), DynError> {
|
||||
let header_id = &record.header;
|
||||
|
||||
// Access full block
|
||||
let block = &record.block;
|
||||
let tx_count = block.transactions().len();
|
||||
let tx_count = record.block.transactions().len();
|
||||
|
||||
tracing::debug!(
|
||||
?header_id,
|
||||
@ -195,26 +197,32 @@ impl Workload for DelayedWorkload {
|
||||
**Example: Rate limiting based on block production**
|
||||
|
||||
```rust
|
||||
async fn start(&self, ctx: &RunContext) -> Result<(), DynError> {
|
||||
use testing_framework_core::scenario::{DynError, RunContext};
|
||||
|
||||
async fn generate_request() -> Option<()> {
|
||||
None
|
||||
}
|
||||
|
||||
async fn start(ctx: &RunContext) -> Result<(), DynError> {
|
||||
let clients = ctx.node_clients().validator_clients();
|
||||
let mut receiver = ctx.block_feed().subscribe();
|
||||
let mut pending_txs = Vec::new();
|
||||
|
||||
let mut pending_requests: Vec<()> = Vec::new();
|
||||
|
||||
loop {
|
||||
tokio::select! {
|
||||
// Send batch on new block
|
||||
// Issue a batch on each new block.
|
||||
Ok(_record) = receiver.recv() => {
|
||||
if !pending_txs.is_empty() {
|
||||
tracing::debug!(count = pending_txs.len(), "sending batch on new block");
|
||||
for tx in pending_txs.drain(..) {
|
||||
clients[0].send_transaction(tx).await?;
|
||||
if !pending_requests.is_empty() {
|
||||
tracing::debug!(count = pending_requests.len(), "issuing requests on new block");
|
||||
for _req in pending_requests.drain(..) {
|
||||
let _info = clients[0].consensus_info().await?;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Generate transactions continuously
|
||||
Some(tx) = generate_transaction() => {
|
||||
pending_txs.push(tx);
|
||||
|
||||
// Generate work continuously.
|
||||
Some(req) = generate_request() => {
|
||||
pending_requests.push(req);
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -238,7 +246,9 @@ async fn start(&self, ctx: &RunContext) -> Result<(), DynError> {
|
||||
Example direct polling in expectations:
|
||||
|
||||
```rust
|
||||
async fn evaluate(&mut self, ctx: &RunContext) -> Result<(), DynError> {
|
||||
use testing_framework_core::scenario::{DynError, RunContext};
|
||||
|
||||
async fn evaluate(ctx: &RunContext) -> Result<(), DynError> {
|
||||
let client = &ctx.node_clients().validator_clients()[0];
|
||||
|
||||
// Poll current height once
|
||||
@ -255,16 +265,18 @@ async fn evaluate(&mut self, ctx: &RunContext) -> Result<(), DynError> {
|
||||
Access aggregated statistics without subscribing to the feed:
|
||||
|
||||
```rust
|
||||
async fn evaluate(&mut self, ctx: &RunContext) -> Result<(), DynError> {
|
||||
use testing_framework_core::scenario::{DynError, RunContext};
|
||||
|
||||
async fn evaluate(ctx: &RunContext, expected_min: u64) -> Result<(), DynError> {
|
||||
let stats = ctx.block_feed().stats();
|
||||
let total_txs = stats.total_transactions();
|
||||
|
||||
tracing::info!(total_txs, "transactions observed across all blocks");
|
||||
|
||||
if total_txs < self.expected_min {
|
||||
if total_txs < expected_min {
|
||||
return Err(format!(
|
||||
"expected at least {} transactions, observed {}",
|
||||
self.expected_min, total_txs
|
||||
expected_min, total_txs
|
||||
).into());
|
||||
}
|
||||
|
||||
|
||||
@ -32,6 +32,24 @@ Operational readiness focuses on prerequisites, environment fit, and clear signa
|
||||
- Health checks prevent premature workload execution
|
||||
- Consensus liveness expectations validate basic operation
|
||||
|
||||
## Runner-Agnostic Design
|
||||
|
||||
The framework is intentionally **runner-agnostic**: the same scenario plan runs across all deployment targets. Understanding which operational concerns apply to each runner helps you choose the right fit.
|
||||
|
||||
| Concern | Host | Compose | Kubernetes |
|
||||
|---------|------|---------|------------|
|
||||
| **Topology** | Full support | Full support | Full support |
|
||||
| **Workloads** | All workloads | All workloads | All workloads |
|
||||
| **Expectations** | All expectations | All expectations | All expectations |
|
||||
| **Chaos / Node Control** | Not supported | Supported | Not yet |
|
||||
| **Metrics / Observability** | Manual setup | External stack | Cluster-wide |
|
||||
| **Log Collection** | Temp files | Container logs | Pod logs |
|
||||
| **Isolation** | Process-level | Container | Pod + namespace |
|
||||
| **Setup Time** | < 1 min | 2-5 min | 5-10 min |
|
||||
| **CI Recommended?** | Smoke tests | Primary | Large-scale only |
|
||||
|
||||
**Key insight:** Operational concerns (prerequisites, environment variables) are largely **consistent** across runners, while deployment-specific concerns (isolation, chaos support) vary by backend.
|
||||
|
||||
## Operational Workflow
|
||||
|
||||
```mermaid
|
||||
|
||||
@ -8,6 +8,20 @@ The Nomos Testing Framework enables you to test consensus, data availability, an
|
||||
|
||||
---
|
||||
|
||||
## Core Concept
|
||||
|
||||
**Everything in this framework is a Scenario.**
|
||||
|
||||
A Scenario is a controlled experiment over time, composed of:
|
||||
- **Topology** — The cluster shape (validators, executors, network layout)
|
||||
- **Workloads** — Traffic and conditions that exercise the system (transactions, DA, chaos)
|
||||
- **Expectations** — Success criteria verified after execution (liveness, inclusion, recovery)
|
||||
- **Duration** — The time window for the experiment
|
||||
|
||||
This single abstraction makes tests declarative, portable, and composable.
|
||||
|
||||
---
|
||||
|
||||
## How It Works
|
||||
|
||||
```mermaid
|
||||
@ -56,7 +70,10 @@ flowchart LR
|
||||
## Quick Example
|
||||
|
||||
```rust
|
||||
use std::time::Duration;
|
||||
|
||||
use testing_framework_core::scenario::ScenarioBuilder;
|
||||
use testing_framework_core::scenario::Deployer as _;
|
||||
use testing_framework_runner_local::LocalDeployer;
|
||||
use testing_framework_workflows::ScenarioBuilderExt;
|
||||
|
||||
@ -67,7 +84,7 @@ async fn main() -> anyhow::Result<()> {
|
||||
.validators(3)
|
||||
.executors(1)
|
||||
})
|
||||
.transactions_with(|tx| tx.rate(10.0).users(5))
|
||||
.transactions_with(|tx| tx.rate(10).users(5))
|
||||
.expect_consensus_liveness()
|
||||
.with_run_duration(Duration::from_secs(60))
|
||||
.build();
|
||||
@ -116,6 +133,8 @@ These roles interact tightly, which is why meaningful testing must be performed
|
||||
|
||||
The Nomos Testing Framework provides the infrastructure to orchestrate these multi-node scenarios reliably across development, CI, and production-like environments.
|
||||
|
||||
**Learn more about the protocol:** [Nomos Project Documentation](https://nomos-tech.notion.site/project)
|
||||
|
||||
---
|
||||
|
||||
## Documentation Structure
|
||||
|
||||
@ -1,5 +1,7 @@
|
||||
# Core Content: ScenarioBuilderExt Patterns
|
||||
|
||||
> **When should I read this?** After writing 2-3 scenarios. This page documents patterns that emerge from real usage—come back when you're refactoring or standardizing your test suite.
|
||||
|
||||
Patterns that keep scenarios readable and reusable:
|
||||
|
||||
- **Topology-first**: start by shaping the cluster (counts, layout) so later
|
||||
|
||||
@ -334,15 +334,15 @@ thread 'main' panicked at 'workload init failed: insufficient wallets'
|
||||
**Fix:**
|
||||
|
||||
```rust
|
||||
// In your scenario:
|
||||
let scenario = Scenario::builder("my_test")
|
||||
.topology(
|
||||
Topology::preset_3v1e()
|
||||
.wallets(20) // ← Increase wallet count
|
||||
)
|
||||
.workload(TransactionWorkload::new()
|
||||
.users(10) // ← Must be ≤ wallets(20)
|
||||
.rate(5.0))
|
||||
use testing_framework_core::scenario::ScenarioBuilder;
|
||||
use testing_framework_workflows::ScenarioBuilderExt;
|
||||
|
||||
let scenario = ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(1))
|
||||
.wallets(20) // ← Increase wallet count
|
||||
.transactions_with(|tx| {
|
||||
tx.users(10) // ← Must be ≤ wallets(20)
|
||||
.rate(5)
|
||||
})
|
||||
.build();
|
||||
```
|
||||
|
||||
@ -453,13 +453,15 @@ thread 'main' panicked at 'expectations failed'
|
||||
**Fix:**
|
||||
|
||||
```rust
|
||||
// Increase run duration to allow more blocks
|
||||
let scenario = Scenario::builder("my_test")
|
||||
.topology(Topology::preset_3v1e())
|
||||
.with_run_duration(Duration::from_secs(120)) // ← Give more time
|
||||
.expectation(ConsensusLiveness::new()
|
||||
.min_blocks(5) // ← Adjust expectation to match duration
|
||||
)
|
||||
use std::time::Duration;
|
||||
|
||||
use testing_framework_core::scenario::ScenarioBuilder;
|
||||
use testing_framework_workflows::ScenarioBuilderExt;
|
||||
|
||||
// Increase run duration to allow more blocks.
|
||||
let scenario = ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(1))
|
||||
.expect_consensus_liveness()
|
||||
.with_run_duration(Duration::from_secs(120)) // ← Give more time
|
||||
.build();
|
||||
```
|
||||
|
||||
|
||||
@ -1,6 +1,63 @@
|
||||
# What You Will Learn
|
||||
|
||||
This book gives you a clear mental model for Nomos multi-node testing, shows how
|
||||
This book gives you a clear mental model for Logos multi-node testing, shows how
|
||||
to author scenarios that pair realistic workloads with explicit expectations,
|
||||
and guides you to run them across local, containerized, and cluster environments
|
||||
without changing the plan.
|
||||
|
||||
## By the End of This Book, You Will Be Able To:
|
||||
|
||||
**Understand the Framework**
|
||||
- Explain the six-phase scenario lifecycle (Build, Deploy, Capture, Execute, Evaluate, Cleanup)
|
||||
- Describe how Deployers, Runners, Workloads, and Expectations work together
|
||||
- Navigate the crate architecture and identify extension points
|
||||
- Understand when to use each runner (Host, Compose, Kubernetes)
|
||||
|
||||
**Author and Run Scenarios**
|
||||
- Define multi-node topologies with validators and executors
|
||||
- Configure transaction and DA workloads with appropriate rates
|
||||
- Add consensus liveness and inclusion expectations
|
||||
- Run scenarios across all three deployment modes
|
||||
- Use BlockFeed to monitor block production in real-time
|
||||
- Implement chaos testing with node restarts
|
||||
|
||||
**Operate in Production**
|
||||
- Set up prerequisites and dependencies correctly
|
||||
- Configure environment variables for different runners
|
||||
- Integrate tests into CI/CD pipelines (GitHub Actions)
|
||||
- Troubleshoot common failure scenarios
|
||||
- Collect and analyze logs from multi-node runs
|
||||
- Optimize test durations and resource usage
|
||||
|
||||
**Extend the Framework**
|
||||
- Implement custom Workload traits for new traffic patterns
|
||||
- Create custom Expectation traits for domain-specific checks
|
||||
- Add new Deployer implementations for different backends
|
||||
- Contribute topology helpers and DSL extensions
|
||||
|
||||
## Learning Path
|
||||
|
||||
**Beginner** (0-2 hours)
|
||||
- Read [Quickstart](quickstart.md) and run your first scenario
|
||||
- Review [Examples](examples.md) to see common patterns
|
||||
- Understand [Scenario Lifecycle](scenario-lifecycle.md) phases
|
||||
|
||||
**Intermediate** (2-8 hours)
|
||||
- Study [Runners](runners.md) comparison and choose appropriate mode
|
||||
- Learn [Workloads & Expectations](workloads.md) in depth
|
||||
- Review [Prerequisites & Setup](prerequisites.md) for your environment
|
||||
- Practice with [Advanced Examples](examples-advanced.md)
|
||||
|
||||
**Advanced** (8+ hours)
|
||||
- Master [Environment Variables](environment-variables.md) configuration
|
||||
- Implement [Custom Workloads](extending.md) for your use cases
|
||||
- Set up [CI Integration](ci-integration.md) for automated testing
|
||||
- Explore [Internal Crate Reference](internal-crate-reference.md) for deep dives
|
||||
|
||||
## What This Book Does NOT Cover
|
||||
|
||||
- **Nomos node internals** — This book focuses on testing infrastructure, not the blockchain protocol implementation. See the Nomos node repository for protocol documentation.
|
||||
- **Consensus algorithm theory** — We assume familiarity with basic blockchain concepts (validators, blocks, transactions, data availability).
|
||||
- **Rust language basics** — Examples use Rust, but we don't teach the language. See [The Rust Book](https://doc.rust-lang.org/book/) if you're new to Rust.
|
||||
- **Kubernetes administration** — We show how to use the K8s runner, but don't cover cluster setup, networking, or operations.
|
||||
- **Docker fundamentals** — We assume basic Docker/Compose knowledge for the Compose runner.
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user