Commit Graph

451 Commits

Author SHA1 Message Date
Giacomo Pasini 29b963fa29
Add mempool status API (#449)
* Add status service API

* Add status API to node
2023-10-02 19:06:12 +02:00
Giacomo Pasini f9c446e48c
Add a canonical way to get an id for Attestation and Certificate (#448)
* add hash() method on Attestation and Certificate

* clippy
2023-10-02 15:59:57 +02:00
Al Liu 4cf57eee74
remove waku from the codebase (#446) 2023-10-02 16:41:08 +08:00
Giacomo Pasini d67d08e81d
Add a second mempool for DA certificates (#443)
* Use more descriptive names for generic parameters

We're starting to have tens on generic parameters, if we don't use
descriptive names it'll be pretty hard to understand what they do.
This commit changes the names of the mempool parameters in
preparation for the insertion of the da certificates mempool to
distinguish it from cl transactions.

* Add mempool for da certificates

* Add separate certificates mempool to binary

* ignore clippy lints
2023-10-02 10:38:05 +02:00
Giacomo Pasini 36b3ccc043
Add `send` method to mempool network adapter (#439)
* Add `send` method to mempool network adapter

Centralize responsabilities for mempool-network interface in the
adapter trait.

* Update nomos-services/mempool/src/lib.rs

Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>

---------

Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>
2023-09-27 11:58:42 +02:00
Giacomo Pasini dda4a1365c
Add discriminant type to allow multiple mempool services (#441)
Overwatch requires all services to have a different service id.
Unfortunately, such service id can't depend on generic parameters,
which means that we can't have two instances of the mempool
service even if they are instantiated with different types.
This commit circuments this limitation by adding another
type parameter.
2023-09-27 10:55:15 +02:00
Youngjoon Lee f307e03ff4
Fix unhappy test: Handle the case where TARGET_VIEW is reached by timeout_qc (#438) 2023-09-27 10:31:52 +09:00
Al Liu 8ea341dbaf
Mixnet packet handle (#435)
* finish mixnet packet sending fan-in model
2023-09-26 18:23:34 +08:00
Giacomo Pasini 03973cd422
Make mempool generic (#428)
* Make mempool item generic

Make the mempool generic with respect to the item and remove
mentions of specific transaction formats/traits. This will allow
us to reuse the same code for both coordination layer transactions
and certificates, or in general, whatever items need to be included
in a block.

* Add mempool network adapter settings

Allow for greater customization of the mempool network adapter by
adding a settings field.

* update node after mempool changes

* fix waku mempool adapter

* fmt

* fix tests

* fmt
2023-09-26 11:14:44 +02:00
Daniel Sanchez 95618c0a72
Use selection for blob certificates (#427)
* Use selection for blob certificates

* Fix bin imports

* Fix rebase

* Missing blobs -> certificates refactor

* Fix attestation and certificate as_bytes

* More naming refactors
2023-09-25 15:34:05 +02:00
Youngjoon Lee 4da06a9740
Handle mixnet invalid fragment error (#436) 2023-09-25 21:12:48 +09:00
Giacomo Pasini 3b64cd6b3a
Add tests for blob dissemination (#416)
* Make nomos-cli config public

Split nomos-cli into lib and bin and make config public
so that it can be reused in tests

* Add tests for nomos-cli disseminate

* fmt

* feature gate whole file instead of function

* Exit unsuccessfully when failing dissemination
2023-09-25 11:52:45 +02:00
Youngjoon Lee 285804e1d5
Remove unnecessary dependencies (#433) 2023-09-25 17:26:21 +09:00
Al Liu cb343156b7
Add concrete Error implementation for mixnet (#405)
Add concrete Error implementation
2023-09-25 14:21:07 +08:00
gusto 46c7e25f6f
Use single workflow for master and pr checks (#434) 2023-09-22 11:19:56 +03:00
gusto 0eae715f8d
GH: Waku windows build configuration (#432)
* Update binutils for windows

* Create mingw-env action and move configuration there

* Use cc and cxx vars for gowaku

* Use go 1.20 for waku
2023-09-22 09:31:31 +03:00
Daniel Sanchez 2ceba073c0
Refactor blob -> blob_certificate in Block (#424) 2023-09-20 16:40:04 +02:00
Daniel Sanchez e71c359171
Remove waku support in nomos-node binary (#429) 2023-09-20 16:32:43 +02:00
Daniel Sanchez 500683b0da
Pipe propose block (#422)
* Add new method to tx selector

* Add new method to blob selector

* Docs typo

* Added missing txs and blobs methods to block builder

* Extend selector generics

* Pipe proposing selector types

* Remove leftover

* Added missing types on waku feature
2023-09-20 11:01:04 +02:00
Youngjoon Lee e0cd35dcdc
Refactor libp2p network backend into submodules (#417) 2023-09-19 22:16:27 +09:00
gusto 21fac726dd
Use build all before integration tests (#419) 2023-09-19 12:00:51 +03:00
gusto 5f040d5be7
Run integration tests in all gh actions (#418) 2023-09-19 12:00:27 +03:00
gusto c860b632dd
Add missing mixnet test to integration runs (#414) 2023-09-18 17:10:05 +03:00
Giacomo Pasini d672be3bb0
[nomos-cli] Add command to disseminate data (#400)
* Add nomos-cli command to disseminate data across the network

Add nomos-cli to hold various utilities for the node.
To start, this commit adds a command to disseminate some
data through the network and build a certificate of correct
dispersal for inclusion in a block

* fmt

* reorder loop instructions

* Send blobs concurrently
2023-09-18 15:35:20 +02:00
Al Liu 2429893ac2
Add mixnet retry mechanism (#386)
Imiplement mixnet retry
2023-09-18 18:26:54 +08:00
Giacomo Pasini 5e194922c6
Add da service to nomos node (#404)
* Make  the data availability service work with multiple protocols

* Add a generic way to instantiate DaProtocol

Add settings type and a new `new(Self::Settings)` method to
build a new DaProtocol instance

* Add data availability service to node

* fix tests

* fix imports
2023-09-18 11:43:24 +02:00
gusto 633b5d6cbf
CI: integrations tests libp2p (#411)
* Update rust version for ci linux host

* Add mixnode build step
2023-09-18 11:25:27 +03:00
gusto 8d9a1516be
ci: fix cargo clippy deprecated warnings (#412) 2023-09-18 11:24:49 +03:00
Youngjoon Lee 82946b8637
etry connecting with initial peers (#410) 2023-09-16 00:23:21 +09:00
gusto 97c653efe3
Docker compose for small libp2p node network (#364)
* Update rust build image version

* Docker compose file for testnet

* Wrap tcp into dns transport in order to resolve hostnames (#346)

* Docker compose for small libp2p node network

* Install etcdctl to node containers

* Register libp2p nodes on etcd

* Register IP address in KV store

* Interconnect libp2p nodes

* Use delimiter in cli and env variables

* Use docker compose initial env config

* Leave main Dockerfile as is, use new ones in testnet dir

* Remove etcd installation script

* run_mixnet.sh placeholder

* Use .env file for docker compose config

* Ignore local .env file

* Wrap sh envvars used in strings

* Remove mixnode placeholders

* Use default values for envconfig

* Update labels in Dockerfiles

* Sanitize scripts via shellcheck

* Export env for nomos node

* Updated to latest libp2p config

* Pass config to bootstrap node
2023-09-15 17:27:38 +03:00
gusto 90529dada3
Disable codecov annotations (#409)
* Add codecov config and disable gh annotations

* Allow some deviation in test coverage
2023-09-15 15:11:50 +03:00
Youngjoon Lee 8c4d1577df
Add mixnet usage docs (#408) 2023-09-15 17:36:28 +09:00
Giacomo Pasini 3121a41b60
Change default network backend to libp2p (#403)
We've been using it for a while and we don't plan to switch back
to Waku anytime soon.
2023-09-15 09:58:26 +02:00
Daniel Sanchez 04d07038dc
Remove unused fountain codes (#407)
* Remove unused fountain codes

* Refactor ProposalChunk to Proposal
2023-09-15 09:39:53 +02:00
Daniel Sanchez e1e2b84921
Block building core (#401)
* Move core block to folder base module

* Added blob select trait

* Added tx select trait

* Implement blob and tx selection filling size

* Added blobs to block constructor
Added block builder module

* Implement block builder

* Remove unnnecesary phantoms

* Add default impl for selections

* Added example

* Fix typos

Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>

* tx to item on generic function

Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>

---------

Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>
2023-09-15 09:39:16 +02:00
gusto 70dc4f3ae6
Simulation branch with one committee fix (#402)
* Tree and branch overlay comparison tests

* Update leader super majority fn in branch overlay

* Cleanup tests
2023-09-15 09:10:46 +03:00
Youngjoon Lee 8449c81d0f
Mixnet PoC base branch (#316)
* Add `mixnode` and `mixnet-client` crate (#302)

* Add `mixnode` binary (#317)

* Integrate mixnet with libp2p network backend (#318)

* Fix #312: proper delays (#321)

* proper delays

* add missing duration param

* tiny fix: compilation error caused by `rand` 0.8 -> 0.7

* use `get_available_port()` for mixnet integration tests (#333)

* add missing comments

* Overwatch mixnet node (#339)

* Add mixnet service and overwatch app

* remove #[tokio::main]

---------

Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>

* fix tests for the overwatch mixnode (#342)

* fix panic when corner case happen in RandomDelayIter (#335)

* Use `log` service for `mixnode` bin (#341)

* Use `wire` for MixnetMessage in libp2p (#347)

* Prevent tmixnet tests from running forever (#363)

* Use random delay when sending msgs to mixnet (#362)

* fix a minor compilation error caused by the latest master

* Fix run output fd (#343)

* add a connection pool

* Exp backoff (#332)

* move mixnet listening into separate task

* add exponential retry for insufficient peers in libp2p

* fix logging

* Fix MutexGuard across await (#373)

* Fix MutexGuard across await

Holding a MutexGuard across an await point is not a good idea.
Removing that solves the issues we had with the mixnet test

* Make mixnode handle bodies coming from the same source concurrently (#372)

---------

Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>

* Move wait at network startup (#338)

We now wait after the call to 'subscribe' to give the network
the time to register peers in the mesh before starting to
publish messages

* Remove unused functions from mixnet connpool (#374)

* Mixnet benchmark (#375)

* merge fixes

* add `connection_pool_size` field to `config.yaml`

* Simplify mixnet topology (#393)

* Simplify bytes and duration range ser/de (#394)

* optimize bytes serde and duration serde

---------

Co-authored-by: Al Liu <scygliu1@gmail.com>
Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com>
Co-authored-by: Giacomo Pasini <Zeegomo@users.noreply.github.com>
2023-09-14 17:38:47 +09:00
Giacomo Pasini 5b7c17e450
Remove failed deserialization error message (#399)
Firstly, a failure in deserialization for a network message is not
an error especially since we're using a public channel.
Secondly, that same channel is shared by different kind of messages
so trying to interpret one as the other will surely lead to a
unsuccessfull attempt.
2023-09-13 12:13:18 +02:00
Giacomo Pasini d72e54f9be
Add full replication implementation for DA (#396)
Add an initial simple but functional implementation for a data
availability protocol.
Full replication simply encodes bytes in a single blob which is
replicated in all nodes.
2023-09-13 11:01:20 +02:00
Giacomo Pasini 1eeeeb1834
Adopt a sans-IO style for DaProtocol (#395)
* Adopt a sans-IO style for DaProtocol

Remove async streams from the DaProtocol trait to avoid creating
dependencies on actual I/O. The protocol it's now transformed into
a state machine with an input and output buffer.

In addition, a method to create an attestation was added to the
trait.

* add forgotten &mut to certify_dispersal

* add forgotten &self param to validate certificate
2023-09-12 16:34:13 +02:00
Daniel Sanchez 7cabddc71f
Fix tracing error import in da service (#392) 2023-09-11 19:21:27 +02:00
Daniel Sanchez 8da13f7012
Da nomos core (#390)
* Refactor da modules
Include da core module in nomos-core

* Include attestation trait

* Added initial approach for Da protocol

* Added empty certificate trait

* Added certificate dispersal method

* Rename validate method to validate attestation

* Clippy happy

* Add validate certificate method
2023-09-11 17:44:26 +02:00
Giacomo Pasini efabf66994
Add default values for libp2p config (#388) 2023-09-11 15:26:01 +02:00
gusto 8e0937beb7
Node envconfig (#382)
* Move config to a new location

* Log config override with env variables

* Network config override with env variables

* Http env vars

* Consensus env vars
2023-09-11 14:56:35 +03:00
gusto ea64b74aa6
Remove codecov token, not required for public repos (#389) 2023-09-11 14:54:21 +03:00
gusto 03b9f69a01
Simulations: make polars dependancy optional (#385)
* Remove explicit target

* Use default profile for windows build

* Use stable-gnu for tests

* Make polars optional feature in simulations

* Remove unnecessary ci debug steps
2023-09-11 14:29:54 +03:00
Daniel Sanchez 3b90cb786c
Da service network (#384)
* Make da backend async

* Added remove blob

* Added send_blob method to network adapter trait

* Added libp2p backend
Implemented blob stream

* Implement attestation stream

* Implement send methods
2023-09-11 12:50:25 +02:00
Daniel Sanchez 96e3c2d499
Da service backend (#381)
* Add basic da module and traits

* Pipe new blobs and internal message handling

* Add and pipe send attestation method

* Add blob trait

* Make da backend async

* Implement mocka backend

* Bound blob in da backend to blob trait

* Added remove blob

* Rename reply to attestation
2023-09-08 11:27:52 +02:00
Giacomo Pasini a79d6c52e6
Add derive feature to serde (#380) 2023-09-07 10:46:13 +02:00
Daniel Sanchez bca27bd27f
Initial DA service sketch (#376)
* Add basic da module and traits

* Pipe new blobs and internal message handling

* Add and pipe send attestation method
2023-09-06 15:11:15 +02:00