Commit Graph

54 Commits

Author SHA1 Message Date
Giacomo Pasini 2f9ebbd32f
Storage api (#473)
* Change impl of StorageReceiver to Option<Bytes>

Load and remove messages return Option<Bytes> and not Bytes, so
let's change the implementation to work around that.

* Add storage/block http api to retrieve blocks from storage

* add tests for storage/block api

* debug tests

* tweak test node online condition
2023-10-25 12:46:26 +02:00
Daniel Sanchez 75b36020c2
Lifecycle update and implementations (#457)
* Update deps

* Implement base lifecycle handling in network service

* Implement base lifecycle handling in storage service

* Use methods instead of functions

* Pipe lifecycle in metrics service

* Pipe lifecycle in mempool service

* Pipe lifecycle in log service

* Pipe lifecycle in da service

* Pipe lifecycle in consensus service

* Refactor handling of lifecycle message to should_stop_service

* Update overwatch version to fixed run_all one
2023-10-25 12:10:21 +02:00
Giacomo Pasini 89b8e27612
Add API to revieve DA blobs (#477)
* Use Vec instead of HashMap in DaMsg::Get

* Add http API to retrieve DA blobs
2023-10-24 15:38:33 +02:00
Gusto f14df0c479 Add libp2p feature flag for da dep 2023-10-17 14:46:41 +03:00
Giacomo Pasini f90704f1d7
Save safe blocks contents to storage (#464) 2023-10-17 11:04:47 +02:00
Daniel Sanchez f04c4a6492
Use specific revision on overwatch (#455) 2023-10-09 09:18:56 +02:00
gusto 309e5a29e9
Nomos node config (#423)
* Use tree overlay in nomos node

* Use tree overlay in tests module

* Handle unexpected consensus vote stream end in tally

* Add defaults for retry and delay in mixnet client

* Add missing entries in config.yaml

* Spawn the next leader first (#425)

---------

Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>
Co-authored-by: Al Liu <scygliu1@gmail.com>
2023-10-05 15:16:40 +03:00
gusto 99d77f7e1c
Nomos node tree overlay (#415)
* Use tree overlay in nomos node

* Use tree overlay in tests module

* Handle unexpected consensus vote stream end in tally

* Spawn the next leader first (#425)

* Report unhappy blocks in happy path test (#430)

* report unhappy blocks in the happy path test

* Make threshold configurable for TreeOverlay (#426)

* Modified test, so that all nodes don’t all connect to the first node only (#442)

* merge fix

---------

Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>
Co-authored-by: Al Liu <scygliu1@gmail.com>
2023-10-05 11:56:24 +03:00
Giacomo Pasini 58686b2a04
Send certificate to node after dissemination + tests (#450)
* Send DA certificate to node after dissemination

* rename mempool endpoints

* Check certificate inclusion in tests

* rename endpoint

* Rename addcert and addtx to add

* tweak test condition

* add option to save certificate to file

* move thread join

* remove fancy prints
2023-10-04 13:37:13 +02:00
Giacomo Pasini 29b963fa29
Add mempool status API (#449)
* Add status service API

* Add status API to node
2023-10-02 19:06:12 +02:00
Giacomo Pasini f9c446e48c
Add a canonical way to get an id for Attestation and Certificate (#448)
* add hash() method on Attestation and Certificate

* clippy
2023-10-02 15:59:57 +02:00
Al Liu 4cf57eee74
remove waku from the codebase (#446) 2023-10-02 16:41:08 +08:00
Giacomo Pasini d67d08e81d
Add a second mempool for DA certificates (#443)
* Use more descriptive names for generic parameters

We're starting to have tens on generic parameters, if we don't use
descriptive names it'll be pretty hard to understand what they do.
This commit changes the names of the mempool parameters in
preparation for the insertion of the da certificates mempool to
distinguish it from cl transactions.

* Add mempool for da certificates

* Add separate certificates mempool to binary

* ignore clippy lints
2023-10-02 10:38:05 +02:00
Giacomo Pasini 36b3ccc043
Add `send` method to mempool network adapter (#439)
* Add `send` method to mempool network adapter

Centralize responsabilities for mempool-network interface in the
adapter trait.

* Update nomos-services/mempool/src/lib.rs

Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>

---------

Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>
2023-09-27 11:58:42 +02:00
Giacomo Pasini dda4a1365c
Add discriminant type to allow multiple mempool services (#441)
Overwatch requires all services to have a different service id.
Unfortunately, such service id can't depend on generic parameters,
which means that we can't have two instances of the mempool
service even if they are instantiated with different types.
This commit circuments this limitation by adding another
type parameter.
2023-09-27 10:55:15 +02:00
Giacomo Pasini 03973cd422
Make mempool generic (#428)
* Make mempool item generic

Make the mempool generic with respect to the item and remove
mentions of specific transaction formats/traits. This will allow
us to reuse the same code for both coordination layer transactions
and certificates, or in general, whatever items need to be included
in a block.

* Add mempool network adapter settings

Allow for greater customization of the mempool network adapter by
adding a settings field.

* update node after mempool changes

* fix waku mempool adapter

* fmt

* fix tests

* fmt
2023-09-26 11:14:44 +02:00
Daniel Sanchez 95618c0a72
Use selection for blob certificates (#427)
* Use selection for blob certificates

* Fix bin imports

* Fix rebase

* Missing blobs -> certificates refactor

* Fix attestation and certificate as_bytes

* More naming refactors
2023-09-25 15:34:05 +02:00
Daniel Sanchez e71c359171
Remove waku support in nomos-node binary (#429) 2023-09-20 16:32:43 +02:00
Daniel Sanchez 500683b0da
Pipe propose block (#422)
* Add new method to tx selector

* Add new method to blob selector

* Docs typo

* Added missing txs and blobs methods to block builder

* Extend selector generics

* Pipe proposing selector types

* Remove leftover

* Added missing types on waku feature
2023-09-20 11:01:04 +02:00
Giacomo Pasini 5e194922c6
Add da service to nomos node (#404)
* Make  the data availability service work with multiple protocols

* Add a generic way to instantiate DaProtocol

Add settings type and a new `new(Self::Settings)` method to
build a new DaProtocol instance

* Add data availability service to node

* fix tests

* fix imports
2023-09-18 11:43:24 +02:00
gusto 8d9a1516be
ci: fix cargo clippy deprecated warnings (#412) 2023-09-18 11:24:49 +03:00
Youngjoon Lee 82946b8637
etry connecting with initial peers (#410) 2023-09-16 00:23:21 +09:00
gusto 97c653efe3
Docker compose for small libp2p node network (#364)
* Update rust build image version

* Docker compose file for testnet

* Wrap tcp into dns transport in order to resolve hostnames (#346)

* Docker compose for small libp2p node network

* Install etcdctl to node containers

* Register libp2p nodes on etcd

* Register IP address in KV store

* Interconnect libp2p nodes

* Use delimiter in cli and env variables

* Use docker compose initial env config

* Leave main Dockerfile as is, use new ones in testnet dir

* Remove etcd installation script

* run_mixnet.sh placeholder

* Use .env file for docker compose config

* Ignore local .env file

* Wrap sh envvars used in strings

* Remove mixnode placeholders

* Use default values for envconfig

* Update labels in Dockerfiles

* Sanitize scripts via shellcheck

* Export env for nomos node

* Updated to latest libp2p config

* Pass config to bootstrap node
2023-09-15 17:27:38 +03:00
Youngjoon Lee 8c4d1577df
Add mixnet usage docs (#408) 2023-09-15 17:36:28 +09:00
Giacomo Pasini 3121a41b60
Change default network backend to libp2p (#403)
We've been using it for a while and we don't plan to switch back
to Waku anytime soon.
2023-09-15 09:58:26 +02:00
Daniel Sanchez 04d07038dc
Remove unused fountain codes (#407)
* Remove unused fountain codes

* Refactor ProposalChunk to Proposal
2023-09-15 09:39:53 +02:00
Youngjoon Lee 8449c81d0f
Mixnet PoC base branch (#316)
* Add `mixnode` and `mixnet-client` crate (#302)

* Add `mixnode` binary (#317)

* Integrate mixnet with libp2p network backend (#318)

* Fix #312: proper delays (#321)

* proper delays

* add missing duration param

* tiny fix: compilation error caused by `rand` 0.8 -> 0.7

* use `get_available_port()` for mixnet integration tests (#333)

* add missing comments

* Overwatch mixnet node (#339)

* Add mixnet service and overwatch app

* remove #[tokio::main]

---------

Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>

* fix tests for the overwatch mixnode (#342)

* fix panic when corner case happen in RandomDelayIter (#335)

* Use `log` service for `mixnode` bin (#341)

* Use `wire` for MixnetMessage in libp2p (#347)

* Prevent tmixnet tests from running forever (#363)

* Use random delay when sending msgs to mixnet (#362)

* fix a minor compilation error caused by the latest master

* Fix run output fd (#343)

* add a connection pool

* Exp backoff (#332)

* move mixnet listening into separate task

* add exponential retry for insufficient peers in libp2p

* fix logging

* Fix MutexGuard across await (#373)

* Fix MutexGuard across await

Holding a MutexGuard across an await point is not a good idea.
Removing that solves the issues we had with the mixnet test

* Make mixnode handle bodies coming from the same source concurrently (#372)

---------

Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>

* Move wait at network startup (#338)

We now wait after the call to 'subscribe' to give the network
the time to register peers in the mesh before starting to
publish messages

* Remove unused functions from mixnet connpool (#374)

* Mixnet benchmark (#375)

* merge fixes

* add `connection_pool_size` field to `config.yaml`

* Simplify mixnet topology (#393)

* Simplify bytes and duration range ser/de (#394)

* optimize bytes serde and duration serde

---------

Co-authored-by: Al Liu <scygliu1@gmail.com>
Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com>
Co-authored-by: Giacomo Pasini <Zeegomo@users.noreply.github.com>
2023-09-14 17:38:47 +09:00
gusto 8e0937beb7
Node envconfig (#382)
* Move config to a new location

* Log config override with env variables

* Network config override with env variables

* Http env vars

* Consensus env vars
2023-09-11 14:56:35 +03:00
Daniel Sanchez 57746f5e76
Block refactor (#368)
* Add sidecars to block

* Use cl and bl prefixes
2023-09-05 13:43:36 +02:00
gusto e9b752aa4d
Set node_key from cli args (#361) 2023-08-31 14:48:49 +03:00
Daniel Sanchez 26a3910566
Pipe addtx endpoint for libp2p (#345)
* Pipe add tx for both backends

* Cleanup imports
2023-08-31 09:13:35 +02:00
Giacomo Pasini 78c6566d8a
Add libp2p backend to nomos-node (#285)
* Add support for libp2p backend in integration tests

* Add support for libp2p in nomos-node

* change default to waku

* add mutually exclusive features warning

* disable default features to avoid unification

* disable default features

* remove leftover cargo build

* Make sure we are subscribed to libp2p topic at startup

* unify imports

* typo in ci config

* Sequential build and test steps for features

* Add RandomBeaconState to libp2p carnot variant

---------

Co-authored-by: Gusto <bacvinka@gmail.com>
2023-08-09 07:42:08 +02:00
Daniel Sanchez 4bdc3ed15a
Add update committees structure for overlay (#295)
* Move roundrobin to leadership module

* Use references in leader selection

* Add membership traits to overlay and create membership module

* Implement committee membership for random beacon state

* Update flat overlay

* Create updateable membership traits and impls

* Update tree overlay

* Update overlay on consensus service

* Update overlay on simulations nomos node

* Update types on tests and modules

* Use chacha for shuffling

* Change to mut slice instead of inner cloning

* Use fisher yates shuffle from scratch

* Stylish and clippy happy
2023-08-08 10:34:02 +02:00
Youngjoon Lee 2b9769b5b7
feat: add `nomos-libp2p` crate (for nomos-network backend) (#237)
* feat: add libp2p network backend skeleton

* use tokio runtime managed by Overwatch

* feat: add nomos-libp2p crate

* remove gossipsub_message_id_fn

* clippy

* use next() instead of select_next_some()

* rename send_command to execute_command

* const timeout

* disable authn / msg signing to start from a clean slate

* rename CommandSender to CommandResultSender

* add comments

* move node machinery to networkbackend

* fmt

* logs more network events

---------

Co-authored-by: Giacomo Pasini <g.pasini98@gmail.com>
2023-07-11 10:33:57 +02:00
Giacomo Pasini 09370dcef8
Integrations tests (#221)
* make config struct fields public

* add basic integration tests

* Add libssl-dev dependency

* fix comments

---------

Co-authored-by: Gusto <bacvinka@gmail.com>
2023-06-27 13:05:09 +02:00
Giacomo Pasini 8fad13b0cc
Separate nomos node into bin and lib sections (#209) 2023-06-22 16:58:47 +02:00
Alberto Soutullo 53a89c33a8
Fixed ident in initial_peers and added relayTopics that was missing (#213) 2023-06-21 14:16:45 +02:00
Giacomo Pasini 2fc10f94f8
Rename mockpool-node to nomos-node (#203) 2023-06-20 17:04:52 +02:00
Youngjoon Lee f02234f9d1
deps: bump waku-bindings to handle `ephemeral` field (#160) 2023-06-20 20:58:22 +09:00
Giacomo Pasini 2c23bda702
Carnot info api (#177)
* add missing accessors

* add api to return info from consensus service

* move processing into its own function

* feat: add HTTP GET `/carnot/info` API (#178)

---------

Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>
2023-06-15 11:29:50 +02:00
Giacomo Pasini 9c81b72711
Random beacon (#167)
* move overlay to consensus engine

* Integrate random beacon in overlay

This commit integrates the random beacon as specified in the nomos
spec into the consensus engine library as part of the overlay.
In addition, it separates the overlay part responsible for leader
selection as an independent trait LeaderSelection, so as to share
the overall overlay structure among most constructions.

Furthermore, a leader proof has been added to a block to verify
that the proposer had valid rights to do so.

The current implementation hardcodes the leader selection update
in the consensus service, but we probably want to abstract it away
through something like the adapter pattern we use of other services
in the node.

* Move leader selection update to separate function

* Add generic support for leader selection in consensus service (#170)

* Add generic support for leader selection in consensus service

* fix

* use settings struct instead of tuple

* fix tests
2023-06-12 15:14:49 +02:00
Youngjoon Lee 2d60ce9921
fix: remove duplicate `tracing_subscriber` init for `nomos-node` bin (#157) 2023-05-29 22:40:22 +09:00
Daniel Sanchez bc453b686f
Consensus orchestrator (#128)
* Add happy-path consensus engine

* tmp

* Fit types from spec (#124)

* Match types to spec

* Remove Output import

* Consensus engine rework (#126)

* rework

* fix test

* clippy happy

---------

Co-authored-by: Giacomo Pasini <Zeegomo@users.noreply.github.com>

* Adapt carnot network adapter interfaces and implementations

* Fix errors

* Update network with engine types

* Fit types yet again

* Remove leadership and old overlay
Create carnot event builder
Added some adjustments

* Add view to vote

* Fix serde derive in consensus-engine

* Add serde feature for engine in core

* Use view in tally

* Move carnot tally to consensus service

* Add new view msg

* Fit engine types in adapter

* Missing serde feature in consensus service

* Implement carnot event builder

* Implement even builder run main tasks

* Fill up view resolver

* Fix errors on network adapter implementations

* Clippy happy

* Extract event handling to independent methods in View

* Fix test

* Refactor carnot event builder (#135)

* refactor

* format

* Discriminate proposal messages (#136)

* Derive block id from wire format (#139)

* Derive block id from wire format

* Derive id on block creation

* Use compile time hash size

* Add leader role (#138)

* add leadership stub

* fix gather_new_views

* fmt

* actually build qc

* remove redundant fields

* add flat overlay (#143)

* add flat overlay

* fix

* sort imports

* fix tests

* Unhappy tally (#137)

* Refactor tally module

* Implement tally for new view messages

* Assess pr comments

* Fix rebase

* simplify tally

---------

Co-authored-by: Giacomo Pasini <g.pasini98@gmail.com>

* Working node (#149)

* fix gather_new_views

* working node

* fix unhappy path

* remove leftover

* fix comments

* update waku (#146)

* update waku

* Fix waku update

---------

Co-authored-by: danielsanchezq <sanchez.quiros.daniel@gmail.com>

* little fixes

* Consensus tasks cancellation (#147)

* fix

* Create view cancel and cancel cache

* Attach cancellation to consensus tasks

* Fix view binds

---------

Co-authored-by: Giacomo Pasini <g.pasini98@gmail.com>

---------

Co-authored-by: danielsanchezq <sanchez.quiros.daniel@gmail.com>

* Remove clones on consts

---------

Co-authored-by: Al Liu <scygliu1@gmail.com>
Co-authored-by: Giacomo Pasini <g.pasini98@gmail.com>
Co-authored-by: Giacomo Pasini <Zeegomo@users.noreply.github.com>
2023-05-22 14:56:56 +02:00
Giacomo Pasini fdc22111d3
use nanosecond precision in waku timestamp (#152) 2023-05-18 15:19:36 +02:00
Giacomo Pasini bbd313f70a
update waku (#146)
* update waku

* Fix waku update

---------

Co-authored-by: danielsanchezq <sanchez.quiros.daniel@gmail.com>
2023-05-15 16:21:49 +02:00
Daniel Sanchez 91ce4e6fa1
Make a transaction trait (#98)
* Impl Transaction trait

* Impl Transaction for MockTransaction

* <ake carnot transaction a module

* Refactor consensus to use Transaction

* Fix tests

* Constrain Transaction::Hash

* Refactor redundant Carnot in CarnotTx
2023-03-16 22:42:56 -07:00
gusto 4981c724af
Http status codes (#88)
* Include status codes in http error responses

* Mockpool bridges error handling

* Last TX in milliseconds in mempool metrics

* u64 for last tx metrics
2023-03-06 15:19:27 +02:00
gusto 42ea8f9be3
Update waku-bindings to 0.1.0-rc.2 (#86) 2023-03-06 15:18:27 +02:00
Daniel Sanchez 55aece674b
Update to waku rc 1 (#80) 2023-02-22 15:55:59 +01:00
Daniel Sanchez 320755d19d
Detach tx broadcast from mempool (#69)
* Remove send tx method from mempool network adapter

* Add error reporting to add_tx operation in mempool
Delegate broadcasting to external caller
2023-02-08 11:07:09 +01:00