Commit Graph

53 Commits

Author SHA1 Message Date
gusto b341a11eb2
Metrics to nomos services (#623)
* Remove unused metrics crate from services

* Move prometheus metrics to nomos-services
2024-03-22 14:03:35 +02:00
Youngjoon Lee d449114044
Replace mixnet backend with mixnet addon (#615) 2024-03-19 19:15:09 +09:00
Youngjoon Lee dbda061f04
Added `mixnet` compilation feature (#610) 2024-03-14 09:53:41 +09:00
Giacomo Pasini 50cff241fe
Refactor block (#609)
* Refactor Block/Header definition

Refactor block/header definition so that it's now responsibility
of the nomos-core crate. This removes definitions in ledger/consensus
crates since there's no need at that level to have an understanding
of the block format.

The new header format supports both carnot and cryptarchia.
2024-03-13 18:46:10 +01:00
Youngjoon Lee e7d591b7bc
Mixnet v1 (#569)
* base

* Remove mixnet client from libp2p network backend (#572)

* Mixnet v1: Remove all mixnet legacies: mixnet crate, mixnode binary, tests, and docker (#573)

* Mixnet v1: Skeleton (#570)

* Use QUIC for libp2p (#580)

* Add Poisson interval function for Mixnet (#575)

* Mixnet network backend skeleton (#586)

* Libp2p stream read/write (#587)

* Emitting packets from mixclient using libp2p stream (#588)

* Handle outputs from mixnode using libp2p stream/gossipsub (#589)

* Refactor poisson (#590)

* Mix client Poisson emission (#591)

* Mix node packet handling (#592)

* Mix Packet / Fragment logic (#593)

* Move FisherYates to `nomos-utils` (#594)

* Mixnet topology (#595)

* Mix client/node unit tests (#596)

* change multiaddr from tcp to udp with quic-v1 (#607)

---------

Co-authored-by: Al Liu <scygliu1@gmail.com>
2024-03-12 15:47:35 +09:00
Youngjoon Lee 42d6816b1b
Write mixnode logs to files in integration tests (#510) 2024-01-24 16:15:11 +09:00
gusto f741590315
Disseminate app file input (#565)
* Accept file option in dissemination app

* File dissemination tests
2024-01-23 18:17:05 +02:00
gusto aa06080baa
Docker compose metrics containers (#564)
* Prometheus container and configuration for testnet

* Add graphana related configuration and container

* Use metrics feature in testnet nodes

* Grafana configuration files and datasources
2024-01-23 18:16:33 +02:00
Giacomo Pasini 16c97c07ec
Add test for big blob dissemination (#563)
* Add tests for dissemination of big blob

* Warn if a requested blob was not returned
2024-01-19 19:08:13 +01:00
Daniel Sanchez c3b5dc98e4
Rename consensus service to carnot consensus (#556) 2024-01-09 11:08:24 +01:00
Daniel Sanchez 6287e554ae
consensus-engine to carnot-engine. (#551)
* Refactor: consensus-engine to carnot-engine.
Move under consensus/carnot-engine

* Fuzztests update with carnot-engine

* Update missing refactor in tests

---------

Co-authored-by: Gusto <bacvinka@gmail.com>
2024-01-04 15:09:43 +01:00
Youngjoon Lee 2e6365bb95
Use 2x timeout for integration tests in Codecov CI (#533) 2023-11-28 09:02:24 +09:00
Daniel Sanchez cc6b6d1cc9
Refactor nomos api: (#521)
* Extracted common service into services
* Moved api implementation to nomos node crate
2023-11-10 09:28:10 +01:00
Al Liu c3422c196c
New http api to nomos-node integration (#490)
* Integrate new http api to nomos-node
2023-11-08 16:55:47 +08:00
Youngjoon Lee a58d3fb63c
Store logs from failed CI as artifacts (#508) 2023-11-07 09:38:10 +09:00
Giacomo Pasini 56e8506704
Split long consensus response in separate APIs (#502)
* Split long consensus response in separate APIs

Consensus info was returning the full list of blocks even though
that can get quite large with time. Instead, this commit change
that API to return a constant size message and adds a new one to
return a chain of blocks with user specified endings.

* Update nomos-services/consensus/src/lib.rs

Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>

* Fix test

---------

Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>
2023-11-06 12:43:48 +01:00
Youngjoon Lee 1c9528e38f
Fix occasional port conflict in CLI integration test (#504) 2023-11-06 08:47:15 +09:00
Youngjoon Lee c3478cf6a6
Set voter to DA attestation (#498) 2023-11-02 18:44:31 +09:00
GoodDaisy ff8fc577a0
Fix typos (#481)
Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com>
2023-10-31 14:38:42 +01:00
Youngjoon Lee ea1dcb82fd
Increase the timeout of happy integration tests (#494) 2023-10-31 20:32:07 +09:00
Youngjoon Lee 5e520ae194
Set consensus timeout conservatively for happy-path integration tests (#492) 2023-10-31 00:39:02 +09:00
Giacomo Pasini a487d8c7a0
Add DA utilities to nomos-cli (#493)
* Move da disseminate command into separate module

* Add utilities to retrieve block contents

* fix tests

* fix typo
2023-10-30 16:19:25 +01:00
Giacomo Pasini 2f9ebbd32f
Storage api (#473)
* Change impl of StorageReceiver to Option<Bytes>

Load and remove messages return Option<Bytes> and not Bytes, so
let's change the implementation to work around that.

* Add storage/block http api to retrieve blocks from storage

* add tests for storage/block api

* debug tests

* tweak test node online condition
2023-10-25 12:46:26 +02:00
Daniel Sanchez 75b36020c2
Lifecycle update and implementations (#457)
* Update deps

* Implement base lifecycle handling in network service

* Implement base lifecycle handling in storage service

* Use methods instead of functions

* Pipe lifecycle in metrics service

* Pipe lifecycle in mempool service

* Pipe lifecycle in log service

* Pipe lifecycle in da service

* Pipe lifecycle in consensus service

* Refactor handling of lifecycle message to should_stop_service

* Update overwatch version to fixed run_all one
2023-10-25 12:10:21 +02:00
Giacomo Pasini d528ebd2ea
Extract nomos-cli da stuff into shared module (#461)
This will be useful for other commands that might want to interact
with the network to perform tasks related to DA. One example is
the upcoming testnet demo.
2023-10-23 11:34:20 +02:00
Gusto 8f442005a5 Mixnet client address config 2023-10-18 16:14:24 +03:00
Youngjoon Lee 81c34036f7
test(unhappy): mask `sender` when comparing timeout_qcs (#458) 2023-10-17 00:20:28 +09:00
Daniel Sanchez f04c4a6492
Use specific revision on overwatch (#455) 2023-10-09 09:18:56 +02:00
Al Liu dbac7d7597
fix clippy and check warnings (#452) 2023-10-06 15:08:52 +08:00
gusto 99d77f7e1c
Nomos node tree overlay (#415)
* Use tree overlay in nomos node

* Use tree overlay in tests module

* Handle unexpected consensus vote stream end in tally

* Spawn the next leader first (#425)

* Report unhappy blocks in happy path test (#430)

* report unhappy blocks in the happy path test

* Make threshold configurable for TreeOverlay (#426)

* Modified test, so that all nodes don’t all connect to the first node only (#442)

* merge fix

---------

Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>
Co-authored-by: Al Liu <scygliu1@gmail.com>
2023-10-05 11:56:24 +03:00
Giacomo Pasini 58686b2a04
Send certificate to node after dissemination + tests (#450)
* Send DA certificate to node after dissemination

* rename mempool endpoints

* Check certificate inclusion in tests

* rename endpoint

* Rename addcert and addtx to add

* tweak test condition

* add option to save certificate to file

* move thread join

* remove fancy prints
2023-10-04 13:37:13 +02:00
Al Liu 4cf57eee74
remove waku from the codebase (#446) 2023-10-02 16:41:08 +08:00
Youngjoon Lee f307e03ff4
Fix unhappy test: Handle the case where TARGET_VIEW is reached by timeout_qc (#438) 2023-09-27 10:31:52 +09:00
Al Liu 8ea341dbaf
Mixnet packet handle (#435)
* finish mixnet packet sending fan-in model
2023-09-26 18:23:34 +08:00
Giacomo Pasini 3b64cd6b3a
Add tests for blob dissemination (#416)
* Make nomos-cli config public

Split nomos-cli into lib and bin and make config public
so that it can be reused in tests

* Add tests for nomos-cli disseminate

* fmt

* feature gate whole file instead of function

* Exit unsuccessfully when failing dissemination
2023-09-25 11:52:45 +02:00
Youngjoon Lee 285804e1d5
Remove unnecessary dependencies (#433) 2023-09-25 17:26:21 +09:00
Daniel Sanchez e71c359171
Remove waku support in nomos-node binary (#429) 2023-09-20 16:32:43 +02:00
Daniel Sanchez 500683b0da
Pipe propose block (#422)
* Add new method to tx selector

* Add new method to blob selector

* Docs typo

* Added missing txs and blobs methods to block builder

* Extend selector generics

* Pipe proposing selector types

* Remove leftover

* Added missing types on waku feature
2023-09-20 11:01:04 +02:00
Al Liu 2429893ac2
Add mixnet retry mechanism (#386)
Imiplement mixnet retry
2023-09-18 18:26:54 +08:00
Giacomo Pasini 5e194922c6
Add da service to nomos node (#404)
* Make  the data availability service work with multiple protocols

* Add a generic way to instantiate DaProtocol

Add settings type and a new `new(Self::Settings)` method to
build a new DaProtocol instance

* Add data availability service to node

* fix tests

* fix imports
2023-09-18 11:43:24 +02:00
Youngjoon Lee 82946b8637
etry connecting with initial peers (#410) 2023-09-16 00:23:21 +09:00
Daniel Sanchez 04d07038dc
Remove unused fountain codes (#407)
* Remove unused fountain codes

* Refactor ProposalChunk to Proposal
2023-09-15 09:39:53 +02:00
Youngjoon Lee 8449c81d0f
Mixnet PoC base branch (#316)
* Add `mixnode` and `mixnet-client` crate (#302)

* Add `mixnode` binary (#317)

* Integrate mixnet with libp2p network backend (#318)

* Fix #312: proper delays (#321)

* proper delays

* add missing duration param

* tiny fix: compilation error caused by `rand` 0.8 -> 0.7

* use `get_available_port()` for mixnet integration tests (#333)

* add missing comments

* Overwatch mixnet node (#339)

* Add mixnet service and overwatch app

* remove #[tokio::main]

---------

Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>

* fix tests for the overwatch mixnode (#342)

* fix panic when corner case happen in RandomDelayIter (#335)

* Use `log` service for `mixnode` bin (#341)

* Use `wire` for MixnetMessage in libp2p (#347)

* Prevent tmixnet tests from running forever (#363)

* Use random delay when sending msgs to mixnet (#362)

* fix a minor compilation error caused by the latest master

* Fix run output fd (#343)

* add a connection pool

* Exp backoff (#332)

* move mixnet listening into separate task

* add exponential retry for insufficient peers in libp2p

* fix logging

* Fix MutexGuard across await (#373)

* Fix MutexGuard across await

Holding a MutexGuard across an await point is not a good idea.
Removing that solves the issues we had with the mixnet test

* Make mixnode handle bodies coming from the same source concurrently (#372)

---------

Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>

* Move wait at network startup (#338)

We now wait after the call to 'subscribe' to give the network
the time to register peers in the mesh before starting to
publish messages

* Remove unused functions from mixnet connpool (#374)

* Mixnet benchmark (#375)

* merge fixes

* add `connection_pool_size` field to `config.yaml`

* Simplify mixnet topology (#393)

* Simplify bytes and duration range ser/de (#394)

* optimize bytes serde and duration serde

---------

Co-authored-by: Al Liu <scygliu1@gmail.com>
Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com>
Co-authored-by: Giacomo Pasini <Zeegomo@users.noreply.github.com>
2023-09-14 17:38:47 +09:00
Youngjoon Lee 115d45d0f0
Revert/improve `get_available_port()` in tests (#334) 2023-08-29 20:25:05 +09:00
Youngjoon Lee ebbdf14406
Pick ports more randomly for integration tests (#315) 2023-08-21 22:13:52 +09:00
Al Liu 87f49c0fe9
Optimize serialization/deserialization for `[u8; 32]` (#306)
* optimize simulation app serialization
2023-08-17 23:46:07 +08:00
Giacomo Pasini 78c6566d8a
Add libp2p backend to nomos-node (#285)
* Add support for libp2p backend in integration tests

* Add support for libp2p in nomos-node

* change default to waku

* add mutually exclusive features warning

* disable default features to avoid unification

* disable default features

* remove leftover cargo build

* Make sure we are subscribed to libp2p topic at startup

* unify imports

* typo in ci config

* Sequential build and test steps for features

* Add RandomBeaconState to libp2p carnot variant

---------

Co-authored-by: Gusto <bacvinka@gmail.com>
2023-08-09 07:42:08 +02:00
gusto 94e384f609
Tree settings for sim overlay (#257)
* Tree settings for sim overlay

* WIP dyn node simulation

* Allow generic settings and states for dispatched nodes

* Update tests
2023-07-18 14:44:23 +03:00
Al Liu 9467351c10
Finish `View` wrapper (#254)
* finish View wrapper
2023-07-12 21:30:22 +08:00
Al Liu 2135676606
Finish `NodeId` type wrapper (#252)
* add NodeId wrapper
2023-07-11 23:16:49 +08:00