Commit Graph

231 Commits

Author SHA1 Message Date
Roman Zajic 516d11b50a
chore: Data dispersal storage tests (#921)
* test: send blob with closed receiver

* fix: add tracing-subscriber dep

* test: unwrap on not existing blob_col_idx

* fix: add third node

* test: write orphan blob info into mempool

* test: indexer filtering with app_id

* fix: assert on result returned from indexer

* fix: remove custom logs from verifier service

* fix: remove newline from verifier service
2024-11-28 21:39:56 +08:00
Youngjoon Lee 63c472e172
Mix: Integrate cover traffic to mix service (#920) 2024-11-28 10:36:43 +09:00
Youngjoon Lee dc286a967a
Mix: Don't panic on SendError in libp2p mix backend (#919) 2024-11-28 10:36:30 +09:00
Youngjoon Lee 7b984a4e29
Just use priv/pubkey as Node ID in the mock mix message encoding (#912)
* Just use priv/pubkey as Node ID in the mock mix message encoding

* do not convert private key to public key in mix service
2024-11-08 10:14:34 +09:00
gusto a5243c6af2
Tracing: Otlp metrics (#909)
* Metrics layer configuration in nomos-tracing

* Add metrics layer to tracing service

* Use http/proto for prometheus

* Add metrics layer to integration tests

* Use metrics layer in cfgsync

* Plug metric in node and tests

* Remove old metrics

* Use otlp metrics in mempool
2024-11-07 04:25:20 +02:00
Daniel Sanchez 9b29c17e2f
Mix: time abstract from streams (#906)
* Extract temporal async stream

* Make stuff public

* Extract persistent transmission scheduler

* Fix tests

* Push temporal scheduler one layer up

* fix compile errors

* return Poll::Pending instead of Poll::Ready(None) when queue is empty

---------

Co-authored-by: Youngjoon Lee <5462944+youngjoon-lee@users.noreply.github.com>
2024-11-06 14:08:52 +09:00
Youngjoon Lee c84a29db31
Use public keys for the mock mix message encoding (#905)
* add node IDs in message

* integrate

* handle variable-lenght messages in nomos-mix-network

* add trait MixMessage

* make tests work

* clippy happy

* remove println!

* remove unnecssary dependency

* remove unnecessary trait bounds

* remove leftover
2024-11-06 10:49:38 +09:00
Daniel Sanchez d0eafb7635
Abstract blend rng (#904) 2024-11-05 11:24:41 +01:00
Daniel Sanchez 23f93dcc28
Abstract persistent transmission rng (#903) 2024-11-05 09:00:31 +01:00
Daniel Sanchez c237333791
Mix: Use streams in message blend and mix service (#896)
* Use streams in message blend and mix service

* Refactor temporal stream method naming

* Clippy happy

* Clippy test happy

* Undo coupling

* Remove bypassing from blend

* Clippy happy

* Un-entangle tiers

* Send local messages to persistent transmission

Co-authored-by: Youngjoon Lee <5462944+youngjoon-lee@users.noreply.github.com>

---------

Co-authored-by: Youngjoon Lee <5462944+youngjoon-lee@users.noreply.github.com>
2024-11-02 22:57:10 +01:00
Giacomo Pasini 9b5d4d329e
Move POL definition into nomos-core (#837) 2024-10-30 11:46:51 +01:00
Giacomo Pasini 3213453c43
[CL-1] Add proofs to CL txs (#850)
* Add missing tx proofs to complete bundle validation

* address review comments
2024-10-30 10:07:55 +01:00
Youngjoon Lee 7aea30132d
Mix: Implement PersistsentTransmission & Temporal/CryptoProcessor and use them in `MixService` (#844)
* forward msgs immediately without any processing

* Mix: Offload transmission rate and message processing from libp2p behaviour/handler

* Mix: Core skeleton used in `MixService`

* rename Processor to MessageBlend

* Mix: Implement Persistent Transmission (Tier 1) (#845)

* Mix: Add Persistent Transmission (Tier 1)

* add test

* define Coin struct with Uniform distribution for fast repeated sampling

* use ChaCha12Rng for Coin

* improve comment

* Mix: Implement Temporal Processor (Tier 2) (#846)

* Mix: Add Persistent Transmission (Tier 1)

* Mix: Implement TemporalProcessor

* use pub(crate)

* Mix: Use TemporalProcessor in MessageBlend (#847)

* Mix: Add Persistent Transmission (Tier 1)

* Mix: Implement TemporalProcessor

* Mix: Use TemporalProcessor in MessageBlend

* remove duplicate members in Cargo.toml
2024-10-28 17:32:39 +09:00
gusto 5a2c41574a
Tests: Local debugging setup using grafana (#855)
* Local debug setup for integration tests

* Tests debugging setup readme

* Clippy too smart
2024-10-25 13:39:07 +03:00
gusto 938bcd7926
Nomos tracing crate (#849)
* tmp grafana stack

* Testnet configure otlp tracing

* tmp: Configurable loki and tempo in cfgsync

* Rename nomos-log to nomos-tracing-service

* nomos-tracing crate for tracing layer creation

* Use multiple layers in nomos-tracing-service

* Tracing in tests and testnet

* Tempo volume instead of local host dir
2024-10-24 19:33:04 +03:00
Daniel Sanchez 733f50b521
Abort tasks for da networks backends (#853)
* Abort tasks for da networks backends

* fmt
2024-10-24 18:17:33 +02:00
Youngjoon Lee ce24a03a23
Mix: Offload transmission rate and message processing from libp2p behaviour/handler (#843) 2024-10-24 21:00:26 +09:00
Daniel Sanchez 5a85281ec0
Use tokio 1 in deps (#842) 2024-10-22 16:59:42 +02:00
Giacomo Pasini ad88d2bc0c
Add first tx defition (#835)
* Add first tx defition

* address review comments

* review comments
2024-10-22 15:51:54 +02:00
gusto 328398ca68
DA: Use executor in tests (#818)
* Use executor in tests instead of nomos node

* Executor config from node config

* Bring generics to the testing game

* Fill in missing gaps in test

* Implement testnode wrapper

* Use sleep on dispersal service instead

* Fix cfgsync

* Clippy happy

* Clippy happy tests

* Mixnet config in tests for validator

* Tests: General config and multiple nodes (#832)

* Use executor in tests instead of nomos node

* Bring generics to the testing game

* Fill in missing gaps in test

* Clippy happy

* Mixnet config in tests for validator

* Derive different types of configs from general in tests

* Validator and executor in cfgsync

---------

Co-authored-by: danielSanchezQ <3danimanimal@gmail.com>

* Tests executor node mix config (#834)

* Merge branch 'master' into tests-executor-node-mix-config

* add mix configs

---------

Co-authored-by: danielSanchezQ <3danimanimal@gmail.com>
Co-authored-by: Youngjoon Lee <5462944+youngjoon-lee@users.noreply.github.com>
2024-10-21 15:38:56 +03:00
Youngjoon Lee 83d9ef7738
Mix: Integrate mix service to the system (#824)
* Mix: remove all of the previous mixnet stuff

* Mix: Add mix service

* Integrate mix service to the node

* Add mock mix service for DA integration tests

* clippy happy

* use explicit function name and add TODOs

* remove the redundant use of tokio::sync::broadcast channel in mix adapter

* add mix service to nomos-executor binary

* removed mock mix backend and related adapters

Use the real mix backend for DA integration tests

* fix compile error generated when merging master

* rename all `MixNetwork*` to `Mix*` for further changes

* use explicit name for adapters

* Mix: Perform broadcasting within the mix service (#829)

* rename relay variables properly
2024-10-21 18:12:47 +09:00
Daniel Sanchez ba726bad84
Remove task from consensus network adapter (#828) 2024-10-18 16:25:08 +02:00
Youngjoon Lee 8dbcf560f9
Mix: Add mix service (#823)
* Mix: remove all of the previous mixnet stuff

* Mix: Add mix service

* refactor extract_peer_id and add comments
2024-10-17 18:12:26 +09:00
Youngjoon Lee cb86528a4a
Mix: remove all of the previous mixnet stuff (#822) 2024-10-17 00:19:20 +09:00
Daniel Sanchez 228d4b2147
Da: Implement verifier adapter for executor network backend (#816)
* Implement adapter for executor
Refactored into common macro

* Executor and node indexer types

* Sampling service generic over network adapter

* Verifier changes tmp

---------

Co-authored-by: Gusto <bacvinka@gmail.com>
2024-10-10 08:16:25 +00:00
Daniel Sanchez 461a9cacdf
DA: Executor sampling adapter (#815)
* Add macro to share adapter implementation for both validator and executor

* Fix imports in binaries

* Adapt cryptarchia types in binaries

* Fix tests build
2024-10-09 10:32:00 +00:00
gusto 0d6e14c773
Executor: Dispersal http endpoint (#814)
* Data dispersal endpoint in executor

* Use path consts in openapi defs
2024-10-09 10:52:41 +03:00
gusto 07c9096924
Integrate dispersal service into executor node (#812) 2024-10-08 17:03:38 +03:00
gusto 22dfb51eba
Executor: Dispersal service mempool adapter (#810)
* Pass metadata when dispersing

* Sketch of executor dispersal mempool adapter
2024-10-08 11:03:34 +03:00
Daniel Sanchez 98e9cde66d
DA: Executor dispersal service (#806)
* Added basic dispersal structure

* Expand impl

* Pull events from network service

* Implement network adapter
Tie types together in service

* Fill up service processing

* Tie up types in kzgrs backend implementation

* Implement disperse

* Pipe encode and dispersal

* Create mempool adapter

* Add mempool adapter

* Tiny comment
2024-10-04 13:06:42 +00:00
Daniel Sanchez 9ecd738d6e
DA: Executor network service (#795)
* Refactor common things in backends

* Further extract to common

* Extract dial_peers

* Pipe out executor events

* Add wrapper settings

* Dial up dispersal peers

* Clippy happy

* dial_peers -> dial_validator_subnetwork_peers

* Add members list to membership trait

* Implement peers selection and dialing up from specification

* Fix tests

* Fix tiny doc
2024-10-01 12:44:13 +00:00
gusto 915fe3f098
DA: Replication integration (#764)
* Handle replication messages

* Multiple indexes for kzgrs verifier

* Add rocksdb storage to node config

* Configurable delay before sending blob info to mempool

* Dial peers in same subnet for replication

* Update da integration tests

* Update nomos node tests

* Load kzgrs config from file

* SLOW_ENV set true for macos CI

* Assert retrieved blobs in dissemination test

* Kzgrs params deserialize unchecked
2024-09-25 11:27:39 +03:00
gusto 9f4f139771
DA: Integration tests for sampling and dispersal (#733)
* Use common test node in da tests

* Pick ranged subnetids when sampling

* Da network settings for integration tests

* Predefined node keys and deterministic rng

* Disperse kzgrs encoded blob

* Cli swarm fixes

* Increase indexer integration test timeout

* Check dispersal responses in cli da adapter

* DA membership configuration in node tests

* Nomos Cli act as a node in tests

* Increase timeout for dispersal tests

* Different node configurations for da tests

* Collect unique ids for sampling
2024-09-18 16:22:36 +03:00
Youngjoon Lee d2db454765
Add missing error handling: `LedgerError::ParentNotFound` (#630) 2024-09-13 15:07:47 +02:00
Giacomo Pasini f2dfd468ac
Integrate PoL into consensus (#721)
* Add PoL crate

* promote cl crate to nomos-node repo

* Integrate PoL into consensus

* remove metal feature

* install risc0 on ci

* more risc0 install

* fix da tests

* pass gh token

* fix tests and run in release mode

* clippy allow

* do not run in release mode

* drop windows in CI

* execute heavy tests nightly or accelerated in macos

* fmt

* configurable slot time

* use xlarge workers

* fix workflows

* remove unstable test

* use xlarge runners

* separate integrations tests

* fix features

* add token to ci

* downgrade runners

* fix integration tests

* revert testing changes and use risc0 dev mode

* fix coverage risc0 install

* increase slot time

* use cargo binstall

* clean before coverage
2024-09-11 01:59:09 +02:00
holisticode 49ac81b950
Implemented get_blob on sampling rocksdb adapter (#732)
* Implemented get_blob on sampling rocksdb adapter

* addressed PR comments
2024-09-06 13:31:51 -05:00
gusto 9835d09b8c
Column index in the sample requests (#731) 2024-09-06 18:39:50 +03:00
gusto f8a73529cf
DA: Incoming sampling request handling (#730)
* WIP: Sample message to behaviour massage translation

* Sampling request handling

* Sampling storage in node

* Sampling rocksdb scaffold
2024-09-06 16:47:38 +03:00
holisticode b31e867f7f
Trigger sampling from DA mempool (#722)
* first iteration, does not compile due to incompatible types

* Trigger sampling in node (#725)

* Sampling service in node

* Sampling in da integration tests

---------

Co-authored-by: gusto <bacv@users.noreply.github.com>
2024-09-04 12:49:29 -05:00
gusto 93509bc09f
DA network in verifier libp2p adapter (#715)
* Use da network in verifier libp2p adapter

* VerifierAdapter in da tests

* Simplify verifier adapter

* Start da_network in da tests
2024-09-03 19:21:30 +03:00
Daniel Sanchez a13f8611ee
DA: Consensus sampling (#708)
* Add sampling relay to consensus and massage all generics

* Pipe in sampling filtering of blob info

* Add mark in block

* Pipe validate block

* Refactor mark_in_block -> mark_complete

* Fix generics on tests

* Fix generics on tests

* Fix rebase

* Cargo fmt after rebase

* Sampling service configuration

* Sampling service config in indexer integration tests

---------

Co-authored-by: Gusto <bacvinka@gmail.com>
2024-09-03 15:02:49 +00:00
holisticode efff80de67
Sampling backend implementation (#709)
* implemented backend; correct start of monitoring thread needs to be finished

* Fix kzgrs backend (#710)

* abstract rng

* fix rng abstraction

* replaced subnets vector with HashSet, fixed bugs, added tests

* addressed PR comments

* fix clippy warnings

* Rename TrackingState -> SamplingState

* Short circuit failure on init error

---------

Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com>
Co-authored-by: danielSanchezQ <3danimanimal@gmail.com>
2024-09-03 08:25:12 +00:00
gusto 7dc2111341
DA: Verifier and Indexer in Node (#713)
* Verifier and Indexer attached to node

* Serialization for kzgrs verifier settings
2024-08-30 20:16:21 +03:00
gusto 4ad9ebec46
DA: Nomos Cli dispersal tests (#711)
* Dissemination app in tests module

* Expose remaining executor settings

* Configure peer ids and addresses for cli tests

* Add BlobInfo to mempool via api

* Expose columns number args for nomos cli

* Spawn all nodes in nodes config
2024-08-30 13:02:02 +03:00
holisticode 0cb039d806
DA: Sampling service (#705)
* initial

* first iteration, lots of lifetime and trait issues, does not compile

* Daniel/sampling service (#706)

* Implement sampling service

* Implement libp2p adapter listen_to_sampling_messages

* temporary empty backend mod; implement start_sampling

* addressed PR comments; moved random sampling subnet list generation to backend (todo impl)

* addressed PR nitpicks

* remove empty mod which was failing CI due to cargo fmt

* removed unused code

---------

Co-authored-by: holisticode <holistic.computing@gmail.com>
Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com>
2024-08-27 13:42:52 -05:00
gusto d5ceceff9e
DA Network in node (#704)
* DA Network in node

* Use ed25519 curve for peerIds in libp2p
2024-08-27 16:47:14 +03:00
Daniel Sanchez c611319622
Da: Network addresses (#703)
* Crate address book

* Add addressbook on necesary places
Incorporate to sampling test

* fmt happy
2024-08-27 10:15:45 +00:00
gusto 37c22d23ce
DA: Network service executor (#701)
* Nomos cli executor swarm

* Executor swarm and backend in nomos-cli

* Libp2p executor adapter in nomos cli

* Update nomos-cli/src/da/network/backend.rs

Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com>

* Use FillFromNodeList with executor backend

* Integrate Executor backend into the cli

* Use subnetwork id where appropriate

* Common clone_deserialize_error

* Use thiserror

* Minimal dialing in cli swarm

* Send dispersal messages async

* Remove unused connection id

---------

Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com>
2024-08-26 11:00:34 +03:00
Daniel Sanchez 5f5b394017
Put swarm to listen at the specified address (#699) 2024-08-23 07:27:39 +00:00
Daniel Sanchez 66de571bfa
Da: network service backend (#695)
* Sketch up types and backend skeleton

* Pipe up sampling events

* Pipe up validation blobs events

* Added docs

* Cargo fmt

* Clippy happy

* Debug log events in validator swarm run method
2024-08-22 10:14:42 +00:00