Commit Graph

499 Commits

Author SHA1 Message Date
Youngjoon Lee 28ae50ea58
Mix: Packet header encryption 2024-10-30 19:26:53 +09:00
Roman Zajic 07b215e885
fix: Codecov workflow output directory cleanup (#866)
* fix: self-hosted runner needs clean up

* fix: remove on push trigger
2024-10-30 17:15:27 +08:00
Giacomo Pasini 3213453c43
[CL-1] Add proofs to CL txs (#850)
* Add missing tx proofs to complete bundle validation

* address review comments
2024-10-30 10:07:55 +01:00
Roman Zajic 6567a2d890
fix: Codecov workflow conflicting output directory (#862)
* fix: change conflicting output directory name

* test: try to remove unwanted software

* test: run on self-hosted

* fix: remove dispatch on push
2024-10-29 20:27:05 +08:00
Youngjoon Lee 0c0aae0712
Mix: Packet algorithms (#859)
* add keyset

* add packet

* add comments

* add comments

* use functions in sphinx
2024-10-29 17:51:52 +09:00
Daniel Sanchez 9d52297cdf
Mix: temporal as stream extension (#861)
* forward msgs immediately without any processing

* Mix: Offload transmission rate and message processing from libp2p behaviour/handler

* Mix: Core skeleton used in `MixService`

* rename Processor to MessageBlend

* Mix: Implement Persistent Transmission (Tier 1) (#845)

* Mix: Add Persistent Transmission (Tier 1)

* add test

* define Coin struct with Uniform distribution for fast repeated sampling

* use ChaCha12Rng for Coin

* improve comment

* Mix: Implement Temporal Processor (Tier 2) (#846)

* Mix: Add Persistent Transmission (Tier 1)

* Mix: Implement TemporalProcessor

* use pub(crate)

* Mix: Use TemporalProcessor in MessageBlend (#847)

* Mix: Add Persistent Transmission (Tier 1)

* Mix: Implement TemporalProcessor

* Mix: Use TemporalProcessor in MessageBlend

* remove duplicate members in Cargo.toml

* Implement persistent transmission as stream trait extension

* Replicate test for stream version

* Temporal processor as stream extension

---------

Co-authored-by: Youngjoon Lee <5462944+youngjoon-lee@users.noreply.github.com>
2024-10-28 17:12:50 +01:00
Daniel Sanchez e207743f69
Mix: Persistent rate as stream + Extension (#860)
* forward msgs immediately without any processing

* Mix: Offload transmission rate and message processing from libp2p behaviour/handler

* Mix: Core skeleton used in `MixService`

* rename Processor to MessageBlend

* Mix: Implement Persistent Transmission (Tier 1) (#845)

* Mix: Add Persistent Transmission (Tier 1)

* add test

* define Coin struct with Uniform distribution for fast repeated sampling

* use ChaCha12Rng for Coin

* improve comment

* Mix: Implement Temporal Processor (Tier 2) (#846)

* Mix: Add Persistent Transmission (Tier 1)

* Mix: Implement TemporalProcessor

* use pub(crate)

* Mix: Use TemporalProcessor in MessageBlend (#847)

* Mix: Add Persistent Transmission (Tier 1)

* Mix: Implement TemporalProcessor

* Mix: Use TemporalProcessor in MessageBlend

* remove duplicate members in Cargo.toml

* Implement persistent transmission as stream trait extension

* Replicate test for stream version

---------

Co-authored-by: Youngjoon Lee <5462944+youngjoon-lee@users.noreply.github.com>
2024-10-28 15:24:40 +01:00
Youngjoon Lee 7aea30132d
Mix: Implement PersistsentTransmission & Temporal/CryptoProcessor and use them in `MixService` (#844)
* forward msgs immediately without any processing

* Mix: Offload transmission rate and message processing from libp2p behaviour/handler

* Mix: Core skeleton used in `MixService`

* rename Processor to MessageBlend

* Mix: Implement Persistent Transmission (Tier 1) (#845)

* Mix: Add Persistent Transmission (Tier 1)

* add test

* define Coin struct with Uniform distribution for fast repeated sampling

* use ChaCha12Rng for Coin

* improve comment

* Mix: Implement Temporal Processor (Tier 2) (#846)

* Mix: Add Persistent Transmission (Tier 1)

* Mix: Implement TemporalProcessor

* use pub(crate)

* Mix: Use TemporalProcessor in MessageBlend (#847)

* Mix: Add Persistent Transmission (Tier 1)

* Mix: Implement TemporalProcessor

* Mix: Use TemporalProcessor in MessageBlend

* remove duplicate members in Cargo.toml
2024-10-28 17:32:39 +09:00
gusto bebb15f921
Cli: Deserialization and runtime fixes (#854)
* Deserialization and runtime fixes for cli

* Disable dissemination test
2024-10-25 14:56:37 +03:00
gusto 461a7beaf8
Ignore reconstruction test in CI (#856) 2024-10-25 13:48:13 +03:00
gusto 5a2c41574a
Tests: Local debugging setup using grafana (#855)
* Local debug setup for integration tests

* Tests debugging setup readme

* Clippy too smart
2024-10-25 13:39:07 +03:00
gusto 938bcd7926
Nomos tracing crate (#849)
* tmp grafana stack

* Testnet configure otlp tracing

* tmp: Configurable loki and tempo in cfgsync

* Rename nomos-log to nomos-tracing-service

* nomos-tracing crate for tracing layer creation

* Use multiple layers in nomos-tracing-service

* Tracing in tests and testnet

* Tempo volume instead of local host dir
2024-10-24 19:33:04 +03:00
Daniel Sanchez 733f50b521
Abort tasks for da networks backends (#853)
* Abort tasks for da networks backends

* fmt
2024-10-24 18:17:33 +02:00
gusto 193ff82980
Cli: Dissemination and retrieval (#851)
* Remove chat and dissemination from nomos-cli

* Dissemination using executor http client in nomos-cli

* Blob retrieval from indexer using executor http client

* Range reqeust type from nomos-node

* Split executor and validator commands
2024-10-24 16:21:27 +03:00
Youngjoon Lee ce24a03a23
Mix: Offload transmission rate and message processing from libp2p behaviour/handler (#843) 2024-10-24 21:00:26 +09:00
gusto 97330a763f
Reconstruct from incomplete blobs (#852) 2024-10-24 12:01:19 +03:00
Youngjoon Lee 929fb1b7b1
forward msgs immediately without any processing (#841) 2024-10-24 11:44:35 +09:00
Roman Zajic ca786fd264
fix: Coverage workflow to run on ubuntu-22.04 (#836)
* fix: remove Remove unwanted software step

* fix: return Remove unwanted software step
- change to run ubuntu-latest to not occupy self-hosted runner

* fix: remove redundant condition

* fix: remove Checkout submodules step

* test: run on ubuntu-22.04

* test: without Remove unwanted software

* fix: remove on push trigger
2024-10-23 11:12:01 +08:00
Daniel Sanchez 5a85281ec0
Use tokio 1 in deps (#842) 2024-10-22 16:59:42 +02:00
Giacomo Pasini ad88d2bc0c
Add first tx defition (#835)
* Add first tx defition

* address review comments

* review comments
2024-10-22 15:51:54 +02:00
Deep Mondal b0142d7173
Fix: membership comments to documentation (#838)
* rebased

* doc comment add

* fmt
2024-10-22 13:13:29 +02:00
gusto 03854f4c23
Executor in testnet (#839) 2024-10-22 12:36:06 +03:00
gusto 328398ca68
DA: Use executor in tests (#818)
* Use executor in tests instead of nomos node

* Executor config from node config

* Bring generics to the testing game

* Fill in missing gaps in test

* Implement testnode wrapper

* Use sleep on dispersal service instead

* Fix cfgsync

* Clippy happy

* Clippy happy tests

* Mixnet config in tests for validator

* Tests: General config and multiple nodes (#832)

* Use executor in tests instead of nomos node

* Bring generics to the testing game

* Fill in missing gaps in test

* Clippy happy

* Mixnet config in tests for validator

* Derive different types of configs from general in tests

* Validator and executor in cfgsync

---------

Co-authored-by: danielSanchezQ <3danimanimal@gmail.com>

* Tests executor node mix config (#834)

* Merge branch 'master' into tests-executor-node-mix-config

* add mix configs

---------

Co-authored-by: danielSanchezQ <3danimanimal@gmail.com>
Co-authored-by: Youngjoon Lee <5462944+youngjoon-lee@users.noreply.github.com>
2024-10-21 15:38:56 +03:00
Youngjoon Lee 83d9ef7738
Mix: Integrate mix service to the system (#824)
* Mix: remove all of the previous mixnet stuff

* Mix: Add mix service

* Integrate mix service to the node

* Add mock mix service for DA integration tests

* clippy happy

* use explicit function name and add TODOs

* remove the redundant use of tokio::sync::broadcast channel in mix adapter

* add mix service to nomos-executor binary

* removed mock mix backend and related adapters

Use the real mix backend for DA integration tests

* fix compile error generated when merging master

* rename all `MixNetwork*` to `Mix*` for further changes

* use explicit name for adapters

* Mix: Perform broadcasting within the mix service (#829)

* rename relay variables properly
2024-10-21 18:12:47 +09:00
Youngjoon Lee 163cc2ce64
chore: clippy: remove irrefutable `let...else` patterns in DA (#830) 2024-10-20 01:49:48 +09:00
Youngjoon Lee 26ae6c0a5d
CI: Update Rust 1.82 (#831) 2024-10-20 00:17:22 +09:00
Daniel Sanchez ba726bad84
Remove task from consensus network adapter (#828) 2024-10-18 16:25:08 +02:00
Daniel Sanchez 41a9387b4b
DA: Executor dispersal auto stream (#826)
* Re-structure connection handling properly

* Handle event based connections

* Fix test

* Handling auto reconnection for dispersal behaviour

* Ignore cli test
2024-10-17 14:10:28 +00:00
Youngjoon Lee 8dbcf560f9
Mix: Add mix service (#823)
* Mix: remove all of the previous mixnet stuff

* Mix: Add mix service

* refactor extract_peer_id and add comments
2024-10-17 18:12:26 +09:00
Youngjoon Lee cb86528a4a
Mix: remove all of the previous mixnet stuff (#822) 2024-10-17 00:19:20 +09:00
Daniel Sanchez 228d4b2147
Da: Implement verifier adapter for executor network backend (#816)
* Implement adapter for executor
Refactored into common macro

* Executor and node indexer types

* Sampling service generic over network adapter

* Verifier changes tmp

---------

Co-authored-by: Gusto <bacvinka@gmail.com>
2024-10-10 08:16:25 +00:00
Daniel Sanchez 461a9cacdf
DA: Executor sampling adapter (#815)
* Add macro to share adapter implementation for both validator and executor

* Fix imports in binaries

* Adapt cryptarchia types in binaries

* Fix tests build
2024-10-09 10:32:00 +00:00
gusto 0d6e14c773
Executor: Dispersal http endpoint (#814)
* Data dispersal endpoint in executor

* Use path consts in openapi defs
2024-10-09 10:52:41 +03:00
Daniel Sanchez 11ddb23795
Ci: kzg bench update (#813)
* Update kzg benches

* Add bench check to CI
2024-10-08 16:39:38 +00:00
Daniel Sanchez c983eb2260
Add Executor client (#811)
* Refactor paths into common modules

* Kickoff http client

* Add blob to send blob

* Refactor publish method. Will reuse older endpoint in further prs

* Use paths in tests

* Fix more tests
2024-10-08 15:43:58 +00:00
gusto 07c9096924
Integrate dispersal service into executor node (#812) 2024-10-08 17:03:38 +03:00
gusto 22dfb51eba
Executor: Dispersal service mempool adapter (#810)
* Pass metadata when dispersing

* Sketch of executor dispersal mempool adapter
2024-10-08 11:03:34 +03:00
Daniel Sanchez a3cea9a09c
DA: Implement data reconstruction from original columns (#808)
* Implement data reconstruction from original columns

* Make reconstruction public
2024-10-08 07:15:41 +00:00
Alexis c03dec4b3d
ci: build on self-hosted runner (#799)
Signed-off-by: Alexis Pentori <alexis@status.im>
2024-10-07 15:16:02 +03:00
Daniel Sanchez 98e9cde66d
DA: Executor dispersal service (#806)
* Added basic dispersal structure

* Expand impl

* Pull events from network service

* Implement network adapter
Tie types together in service

* Fill up service processing

* Tie up types in kzgrs backend implementation

* Implement disperse

* Pipe encode and dispersal

* Create mempool adapter

* Add mempool adapter

* Tiny comment
2024-10-04 13:06:42 +00:00
gusto d17afcbe4d
Replication behavior membership test (#805) 2024-10-03 20:37:13 +03:00
gusto d7fe88f30b
Remove graylog and related containers (#804) 2024-10-03 14:36:36 +03:00
Caio 93de3afaf9
chose: fk20 benches clippy happy (#800) 2024-10-03 04:43:46 +00:00
Deep Mondal 563bf6f948
DA: kzg main testing parallelize proofs checks (#803) 2024-10-03 04:37:53 +00:00
gusto 66fc046091
Testnet: Kzgrs config (#797)
* Remove out of place wait_online test param

* Cfgsync crate for testnet startup

* Config dispersal to node hosts

* Remove etcd dep and obsolete bash scripts

* Kzgrs test params in testnet

* Nodes listen on all interfaces
2024-10-02 14:26:39 +03:00
gusto b01c4dd8c7
DA: Executor http api (#801)
* Node api handlers and backend modules

* Executor axum api backend

* Expose config functions from node

* Descriptive generics names in http api

* Nomos node metrics feature
2024-10-02 12:18:59 +03:00
Daniel Sanchez 9ecd738d6e
DA: Executor network service (#795)
* Refactor common things in backends

* Further extract to common

* Extract dial_peers

* Pipe out executor events

* Add wrapper settings

* Dial up dispersal peers

* Clippy happy

* dial_peers -> dial_validator_subnetwork_peers

* Add members list to membership trait

* Implement peers selection and dialing up from specification

* Fix tests

* Fix tiny doc
2024-10-01 12:44:13 +00:00
Roman Zajic 2672b7bde4
fix: Coverage and PR workflows - no space left on device (#798)
* test: remove LLVM profiling

* fix: add on push

* test: use release mode

* test: remove Android support from the runner

* test: return back LLVM profiling
- remove dotnet

* fix: remove on push condition

* fix: prune-vm action added

* fix: move actions-checkout

* fix: PR workflow

* fix: remove on push condition from codecov
2024-09-30 23:46:18 +08:00
gusto a30a809ea1
CI: Jenkins risczero setup (#770)
* Remove dep

* Use cargo as jenkins user

* Ensure that toolchain directory is setup before installation

* Debian 12 and gcc deps

* Env variables for risc zero dev mode

* Update testnet and base dockerfiles

* Env vars comments
2024-09-26 18:00:47 +03:00
Daniel Sanchez d83380cf02
Add default template (#769) 2024-09-26 12:41:26 +02:00