Commit Graph

73 Commits

Author SHA1 Message Date
Youngjoon Lee 83d9ef7738
Mix: Integrate mix service to the system (#824)
* Mix: remove all of the previous mixnet stuff

* Mix: Add mix service

* Integrate mix service to the node

* Add mock mix service for DA integration tests

* clippy happy

* use explicit function name and add TODOs

* remove the redundant use of tokio::sync::broadcast channel in mix adapter

* add mix service to nomos-executor binary

* removed mock mix backend and related adapters

Use the real mix backend for DA integration tests

* fix compile error generated when merging master

* rename all `MixNetwork*` to `Mix*` for further changes

* use explicit name for adapters

* Mix: Perform broadcasting within the mix service (#829)

* rename relay variables properly
2024-10-21 18:12:47 +09:00
Youngjoon Lee cb86528a4a
Mix: remove all of the previous mixnet stuff (#822) 2024-10-17 00:19:20 +09:00
gusto 0d6e14c773
Executor: Dispersal http endpoint (#814)
* Data dispersal endpoint in executor

* Use path consts in openapi defs
2024-10-09 10:52:41 +03:00
Daniel Sanchez c983eb2260
Add Executor client (#811)
* Refactor paths into common modules

* Kickoff http client

* Add blob to send blob

* Refactor publish method. Will reuse older endpoint in further prs

* Use paths in tests

* Fix more tests
2024-10-08 15:43:58 +00:00
gusto 66fc046091
Testnet: Kzgrs config (#797)
* Remove out of place wait_online test param

* Cfgsync crate for testnet startup

* Config dispersal to node hosts

* Remove etcd dep and obsolete bash scripts

* Kzgrs test params in testnet

* Nodes listen on all interfaces
2024-10-02 14:26:39 +03:00
gusto b01c4dd8c7
DA: Executor http api (#801)
* Node api handlers and backend modules

* Executor axum api backend

* Expose config functions from node

* Descriptive generics names in http api

* Nomos node metrics feature
2024-10-02 12:18:59 +03:00
Daniel Sanchez 9ecd738d6e
DA: Executor network service (#795)
* Refactor common things in backends

* Further extract to common

* Extract dial_peers

* Pipe out executor events

* Add wrapper settings

* Dial up dispersal peers

* Clippy happy

* dial_peers -> dial_validator_subnetwork_peers

* Add members list to membership trait

* Implement peers selection and dialing up from specification

* Fix tests

* Fix tiny doc
2024-10-01 12:44:13 +00:00
gusto 915fe3f098
DA: Replication integration (#764)
* Handle replication messages

* Multiple indexes for kzgrs verifier

* Add rocksdb storage to node config

* Configurable delay before sending blob info to mempool

* Dial peers in same subnet for replication

* Update da integration tests

* Update nomos node tests

* Load kzgrs config from file

* SLOW_ENV set true for macos CI

* Assert retrieved blobs in dissemination test

* Kzgrs params deserialize unchecked
2024-09-25 11:27:39 +03:00
gusto 9f4f139771
DA: Integration tests for sampling and dispersal (#733)
* Use common test node in da tests

* Pick ranged subnetids when sampling

* Da network settings for integration tests

* Predefined node keys and deterministic rng

* Disperse kzgrs encoded blob

* Cli swarm fixes

* Increase indexer integration test timeout

* Check dispersal responses in cli da adapter

* DA membership configuration in node tests

* Nomos Cli act as a node in tests

* Increase timeout for dispersal tests

* Different node configurations for da tests

* Collect unique ids for sampling
2024-09-18 16:22:36 +03:00
Giacomo Pasini f2dfd468ac
Integrate PoL into consensus (#721)
* Add PoL crate

* promote cl crate to nomos-node repo

* Integrate PoL into consensus

* remove metal feature

* install risc0 on ci

* more risc0 install

* fix da tests

* pass gh token

* fix tests and run in release mode

* clippy allow

* do not run in release mode

* drop windows in CI

* execute heavy tests nightly or accelerated in macos

* fmt

* configurable slot time

* use xlarge workers

* fix workflows

* remove unstable test

* use xlarge runners

* separate integrations tests

* fix features

* add token to ci

* downgrade runners

* fix integration tests

* revert testing changes and use risc0 dev mode

* fix coverage risc0 install

* increase slot time

* use cargo binstall

* clean before coverage
2024-09-11 01:59:09 +02:00
gusto f8a73529cf
DA: Incoming sampling request handling (#730)
* WIP: Sample message to behaviour massage translation

* Sampling request handling

* Sampling storage in node

* Sampling rocksdb scaffold
2024-09-06 16:47:38 +03:00
Daniel Sanchez a13f8611ee
DA: Consensus sampling (#708)
* Add sampling relay to consensus and massage all generics

* Pipe in sampling filtering of blob info

* Add mark in block

* Pipe validate block

* Refactor mark_in_block -> mark_complete

* Fix generics on tests

* Fix generics on tests

* Fix rebase

* Cargo fmt after rebase

* Sampling service configuration

* Sampling service config in indexer integration tests

---------

Co-authored-by: Gusto <bacvinka@gmail.com>
2024-09-03 15:02:49 +00:00
gusto 7dc2111341
DA: Verifier and Indexer in Node (#713)
* Verifier and Indexer attached to node

* Serialization for kzgrs verifier settings
2024-08-30 20:16:21 +03:00
gusto 4ad9ebec46
DA: Nomos Cli dispersal tests (#711)
* Dissemination app in tests module

* Expose remaining executor settings

* Configure peer ids and addresses for cli tests

* Add BlobInfo to mempool via api

* Expose columns number args for nomos cli

* Spawn all nodes in nodes config
2024-08-30 13:02:02 +03:00
gusto d5ceceff9e
DA Network in node (#704)
* DA Network in node

* Use ed25519 curve for peerIds in libp2p
2024-08-27 16:47:14 +03:00
gusto 9ad4ee7a3c
DA: DispersedBlobInfo trait in nomos-core and kzgrs-backend (#686)
* DispersedBlobData in nomos-core

* Implement BlobInfo in kzgrs-backend

* Fullreplication BlobInfo

* Use block info in da mempool

* Blob info in consensus

* Blob info in da-verifier

* BlobInfo in da indexer

* BlobInfo in nomos-api

* Verifier and Indexer integration tests with BlobInfo

* Blob info in nomos node

* Import DaEncoder trait
2024-08-02 20:01:18 +03:00
gusto c9c94c4e55
DA Http Endpoints (#670)
* Api endpoint to add blob and receive attestation

* Nomos DA Certificate serialization

* Kzgrs verification params provider

* Indexer http api using kzgrs

* Plug da http endpoints into nomos api
2024-07-03 12:37:23 +03:00
gusto b4b24e4678
Rust 1.79 stable (#661)
* Rust 1.79 clippy fix

* Rust 1.79 stable in ci and docker

* Allow unused_var in nomos tests

Clippy struggle to recognize the usage of enumeration variable inside one particular testcase.
2024-06-18 10:53:56 +03:00
Daniel Sanchez 7b9643f17e
DA Protocol V1 (#626)
* Base cleaning of da to new traits/structure
Added new da protocols and types

* DA: KZG+RS core (#632)

* Removed old kzg rs modules

* Added new kzgrs core module

* Implemented bytes_to_polynomial and tests

* Use coefficient form

* Refactor evaluations into method

* Use domain elements instead of roots of unity in tests

* Fix encoding and test

* Clippy happy

* Add comments

* Implement polynomial commitment

* Implement proof generation

* Sketch fn signature for verification

* implement proof verification

* Implemented verification and tests

* Return evaluations from bytes_to_polynomial as well

* Use modular le bytes

* Implement rs encode/decode

* Implement decoding tests

* Implement decode using lagrange

* Cleanup imports

* Da: v1 encoder (#633)

* Added new kzgrs core module

* Implemented bytes_to_polynomial and tests

* Use coefficient form

* Refactor evaluations into method

* Use domain elements instead of roots of unity in tests

* Fix encoding and test

* Clippy happy

* Add comments

* Implement polynomial commitment

* Implement proof generation

* Sketch fn signature for verification

* implement proof verification

* Implemented verification and tests

* Return evaluations from bytes_to_polynomial as well

* Use modular le bytes

* Implement rs encode/decode

* Implement decoding tests

* Implement decode using lagrange

* Added chunksmatrix

* Implement encoder with chunkify

* Added missing files

* Implement commit row commitments

* Implement compute elements (row) proofs

* Fix lib and types exposures

* Implement encoder main methods

* Implement encode method

* Implement chunkify test
Fix related bugs

* Implement compute row kzg commitments
Fix related bugs

* Implement rs encode rows test
Fix related bugs
Refactored API

* Implement row proofs tests
Fix fieldelement encoding/decoding bug

* Implement aggregated commitment test
Implement aggregated column proofs test

* Cleanup

* Fix deps

* Fix tests

* Fix chunk too big test

* Da: v1 verifier (#635)

* Fix encoding and test

* Implement commit row commitments

* Implemented dablob

* Implement verifier new
Implement verifier check column

* Clippy cleanup

* Implement verifier

* Implemented verify column test

* Implemented full verify test

* DA API Payload to Item in mempool  (#634)

* Base cleaning of da to new traits/structure
Added new da protocols and types

* DA: KZG+RS core (#632)

* Removed old kzg rs modules

* Added new kzgrs core module

* Implemented bytes_to_polynomial and tests

* Use coefficient form

* Refactor evaluations into method

* Use domain elements instead of roots of unity in tests

* Fix encoding and test

* Clippy happy

* Add comments

* Implement polynomial commitment

* Implement proof generation

* Sketch fn signature for verification

* implement proof verification

* Implemented verification and tests

* Return evaluations from bytes_to_polynomial as well

* Use modular le bytes

* Implement rs encode/decode

* Implement decoding tests

* Implement decode using lagrange

* Cleanup imports

* Reduce abstraction for certificate and vid metadata

* Allow payload to mempool as long as it converts into item

* Da Certificate verifier

* Add mock certificate for core tests

* Mempool payload verification

* Integrate mock verifiers for tx and certs

* Detach verification from cert and tx

* Seperate payload and item in mempools

* Mempools in integration tests

* Remove old cert verifier

* Network payload to item constraints in da mempool

* Update nomos-da/full-replication/src/lib.rs

Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com>

* Sort attestations for cert signature

* Update nomos-da/full-replication/src/lib.rs

Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com>

---------

Co-authored-by: danielsanchezq <sanchez.quiros.daniel@gmail.com>

* DA API Certificate verification  (#641)

* Redo certificate verification in mempool

* FullReplication verifier params provider

* Integrate da params provider into the node

* DA API Indexer service (#643)

* Base cleaning of da to new traits/structure
Added new da protocols and types

* Remove da availability crate completely

* Scaffold for da storage service

* Indexer service responsible for storage and blockchain subscription

* Handle index related ops only

* Acquire storage and consensus relays

* Indexer trait

* wip: storage adapter

* Use storage adapter instead of storage

* Add consensus adapter trait for block subscriptions

* Consensus block subscriber adapter

* Use metadata from da core in indexer

* Update nomos-services/data-availability/indexer/src/lib.rs

Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com>

* Update nomos-services/data-availability/indexer/src/lib.rs

Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com>

* Use std::ops::Range for da queries

* Return certificate metadata

* Da storage adapter methods

---------

Co-authored-by: danielsanchezq <sanchez.quiros.daniel@gmail.com>

* Reuse evaluations when computing proofs (#647)

* Reuse precomputed evaluations instead of evaluation polynomial for each proof

* kzgrs benchmarks

* Clippy happy

* DA API indexer implementation (#644)

* Rocksb adapter in da indexer

* Handle da service messages

* Remove indexer trait, use storage directly in the indexer service

* Return unordered indexes range

* Load blob by vid from file

* Use VID in consensus

* Change full replication index type to byte array

* Change payload to cert and item to vid where required

* Service integration tests for indexer

* Feature gate rocksdb backend

* Propagate range response send error

* FRIndex to Index

* VID to VidCertificate

* Pass blobs file dir via settings

* Da v1 multiple proofs bench (#648)

* Parallel proof generation bench

* Added bench comment

* Modify domain to fit exact sizes

* Fix domain in benches

* Force parallelization features in lib

* DA: Implement base structure for verifier service (#627)

* Base cleaning of da to new traits/structure
Added new da protocols and types

* Implement base structure for verifier service

* Added comments and todo!

* Cleanup imports

* Size of VidCert in full replication

* Nomos Da Verifier service crate

* Extension replaced with metadata

* Fix DaIndexer service name

* Storage adapter trait in verifier

* Manage lifecycle and messages in verifier

* Blob trait in core

* Common nomos da storage crate

* Use updated nomos da storage in indexer

* Verifier storage adapter

* Libp2p adaper for verifier

* Kzgrs backend in verifier service

* Fix fmt

* Clippy happy

---------

Co-authored-by: Gusto <bacvinka@gmail.com>

* DA Verifier service integration tests (#650)

* Base cleaning of da to new traits/structure
Added new da protocols and types

* Implement base structure for verifier service

* Added comments and todo!

* Cleanup imports

* Size of VidCert in full replication

* Nomos Da Verifier service crate

* Extension replaced with metadata

* Storage adapter trait in verifier

* Manage lifecycle and messages in verifier

* Common nomos da storage crate

* Use updated nomos da storage in indexer

* Verifier storage adapter

* Libp2p adaper for verifier

* Kzgrs backend in verifier service

* Fix fmt

* Data availability tests module

* Return attestation in service msg response

* Common definitions for da tests

* Serde for kzgrs proofs and commitments

* Da verifier integration test

* WIP nomos-core in kzgrs backend

* Kzgrs blob to common module

* Add client zone to verifier test and check if attestations are created

* Cleanup and comments

* Use libp2p only for verifier and indexer service tests

* Lint in da tests

* Simplify da blob serialization

* Remove attester from nomos-core attestation

* Verifier backend error

---------

Co-authored-by: danielsanchezq <sanchez.quiros.daniel@gmail.com>

* DA Kzgrs Backend Certificate implementation (#651)

* Kzgrs backend certificate definition

* Encoded data to certificate test

* Nomos da domain specific tag

* Handle errors in da certificate creation

* Add nomos core traits to da cert

* Derive ordering traits for da index

* Add failure test cases to kzgrs certificate

* Da v1 benches expand (#658)

* Update benches with more cases

* Expand benches

* Added parallel feature

* Fix test comment

Co-authored-by: Youngjoon Lee <5462944+youngjoon-lee@users.noreply.github.com>

* Remove outdated comment

---------

Co-authored-by: Gusto <bacvinka@gmail.com>
Co-authored-by: gusto <bacv@users.noreply.github.com>
Co-authored-by: Youngjoon Lee <5462944+youngjoon-lee@users.noreply.github.com>
2024-06-13 11:23:11 +03:00
Giacomo Pasini 7e4d00cc78
Switch from Carnot to Cryptarchia (#624)
* Switch from Carnot to Cryptarchia

* Do not delete old coins

* Update tests/src/lib.rs

Co-authored-by: davidrusu <davidrusu.me@gmail.com>

* clippy happy

---------

Co-authored-by: davidrusu <davidrusu.me@gmail.com>
2024-03-28 12:56:04 +01:00
gusto b341a11eb2
Metrics to nomos services (#623)
* Remove unused metrics crate from services

* Move prometheus metrics to nomos-services
2024-03-22 14:03:35 +02:00
Youngjoon Lee d449114044
Replace mixnet backend with mixnet addon (#615) 2024-03-19 19:15:09 +09:00
Youngjoon Lee dbda061f04
Added `mixnet` compilation feature (#610) 2024-03-14 09:53:41 +09:00
Giacomo Pasini 50cff241fe
Refactor block (#609)
* Refactor Block/Header definition

Refactor block/header definition so that it's now responsibility
of the nomos-core crate. This removes definitions in ledger/consensus
crates since there's no need at that level to have an understanding
of the block format.

The new header format supports both carnot and cryptarchia.
2024-03-13 18:46:10 +01:00
Youngjoon Lee e7d591b7bc
Mixnet v1 (#569)
* base

* Remove mixnet client from libp2p network backend (#572)

* Mixnet v1: Remove all mixnet legacies: mixnet crate, mixnode binary, tests, and docker (#573)

* Mixnet v1: Skeleton (#570)

* Use QUIC for libp2p (#580)

* Add Poisson interval function for Mixnet (#575)

* Mixnet network backend skeleton (#586)

* Libp2p stream read/write (#587)

* Emitting packets from mixclient using libp2p stream (#588)

* Handle outputs from mixnode using libp2p stream/gossipsub (#589)

* Refactor poisson (#590)

* Mix client Poisson emission (#591)

* Mix node packet handling (#592)

* Mix Packet / Fragment logic (#593)

* Move FisherYates to `nomos-utils` (#594)

* Mixnet topology (#595)

* Mix client/node unit tests (#596)

* change multiaddr from tcp to udp with quic-v1 (#607)

---------

Co-authored-by: Al Liu <scygliu1@gmail.com>
2024-03-12 15:47:35 +09:00
Youngjoon Lee 42d6816b1b
Write mixnode logs to files in integration tests (#510) 2024-01-24 16:15:11 +09:00
gusto f741590315
Disseminate app file input (#565)
* Accept file option in dissemination app

* File dissemination tests
2024-01-23 18:17:05 +02:00
gusto aa06080baa
Docker compose metrics containers (#564)
* Prometheus container and configuration for testnet

* Add graphana related configuration and container

* Use metrics feature in testnet nodes

* Grafana configuration files and datasources
2024-01-23 18:16:33 +02:00
Giacomo Pasini 16c97c07ec
Add test for big blob dissemination (#563)
* Add tests for dissemination of big blob

* Warn if a requested blob was not returned
2024-01-19 19:08:13 +01:00
Daniel Sanchez c3b5dc98e4
Rename consensus service to carnot consensus (#556) 2024-01-09 11:08:24 +01:00
Daniel Sanchez 6287e554ae
consensus-engine to carnot-engine. (#551)
* Refactor: consensus-engine to carnot-engine.
Move under consensus/carnot-engine

* Fuzztests update with carnot-engine

* Update missing refactor in tests

---------

Co-authored-by: Gusto <bacvinka@gmail.com>
2024-01-04 15:09:43 +01:00
Youngjoon Lee 2e6365bb95
Use 2x timeout for integration tests in Codecov CI (#533) 2023-11-28 09:02:24 +09:00
Daniel Sanchez cc6b6d1cc9
Refactor nomos api: (#521)
* Extracted common service into services
* Moved api implementation to nomos node crate
2023-11-10 09:28:10 +01:00
Al Liu c3422c196c
New http api to nomos-node integration (#490)
* Integrate new http api to nomos-node
2023-11-08 16:55:47 +08:00
Youngjoon Lee a58d3fb63c
Store logs from failed CI as artifacts (#508) 2023-11-07 09:38:10 +09:00
Giacomo Pasini 56e8506704
Split long consensus response in separate APIs (#502)
* Split long consensus response in separate APIs

Consensus info was returning the full list of blocks even though
that can get quite large with time. Instead, this commit change
that API to return a constant size message and adds a new one to
return a chain of blocks with user specified endings.

* Update nomos-services/consensus/src/lib.rs

Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>

* Fix test

---------

Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>
2023-11-06 12:43:48 +01:00
Youngjoon Lee 1c9528e38f
Fix occasional port conflict in CLI integration test (#504) 2023-11-06 08:47:15 +09:00
Youngjoon Lee c3478cf6a6
Set voter to DA attestation (#498) 2023-11-02 18:44:31 +09:00
GoodDaisy ff8fc577a0
Fix typos (#481)
Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com>
2023-10-31 14:38:42 +01:00
Youngjoon Lee ea1dcb82fd
Increase the timeout of happy integration tests (#494) 2023-10-31 20:32:07 +09:00
Youngjoon Lee 5e520ae194
Set consensus timeout conservatively for happy-path integration tests (#492) 2023-10-31 00:39:02 +09:00
Giacomo Pasini a487d8c7a0
Add DA utilities to nomos-cli (#493)
* Move da disseminate command into separate module

* Add utilities to retrieve block contents

* fix tests

* fix typo
2023-10-30 16:19:25 +01:00
Giacomo Pasini 2f9ebbd32f
Storage api (#473)
* Change impl of StorageReceiver to Option<Bytes>

Load and remove messages return Option<Bytes> and not Bytes, so
let's change the implementation to work around that.

* Add storage/block http api to retrieve blocks from storage

* add tests for storage/block api

* debug tests

* tweak test node online condition
2023-10-25 12:46:26 +02:00
Daniel Sanchez 75b36020c2
Lifecycle update and implementations (#457)
* Update deps

* Implement base lifecycle handling in network service

* Implement base lifecycle handling in storage service

* Use methods instead of functions

* Pipe lifecycle in metrics service

* Pipe lifecycle in mempool service

* Pipe lifecycle in log service

* Pipe lifecycle in da service

* Pipe lifecycle in consensus service

* Refactor handling of lifecycle message to should_stop_service

* Update overwatch version to fixed run_all one
2023-10-25 12:10:21 +02:00
Giacomo Pasini d528ebd2ea
Extract nomos-cli da stuff into shared module (#461)
This will be useful for other commands that might want to interact
with the network to perform tasks related to DA. One example is
the upcoming testnet demo.
2023-10-23 11:34:20 +02:00
Gusto 8f442005a5 Mixnet client address config 2023-10-18 16:14:24 +03:00
Youngjoon Lee 81c34036f7
test(unhappy): mask `sender` when comparing timeout_qcs (#458) 2023-10-17 00:20:28 +09:00
Daniel Sanchez f04c4a6492
Use specific revision on overwatch (#455) 2023-10-09 09:18:56 +02:00
Al Liu dbac7d7597
fix clippy and check warnings (#452) 2023-10-06 15:08:52 +08:00
gusto 99d77f7e1c
Nomos node tree overlay (#415)
* Use tree overlay in nomos node

* Use tree overlay in tests module

* Handle unexpected consensus vote stream end in tally

* Spawn the next leader first (#425)

* Report unhappy blocks in happy path test (#430)

* report unhappy blocks in the happy path test

* Make threshold configurable for TreeOverlay (#426)

* Modified test, so that all nodes don’t all connect to the first node only (#442)

* merge fix

---------

Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>
Co-authored-by: Al Liu <scygliu1@gmail.com>
2023-10-05 11:56:24 +03:00