Commit Graph

245 Commits

Author SHA1 Message Date
Youngjoon Lee f307e03ff4
Fix unhappy test: Handle the case where TARGET_VIEW is reached by timeout_qc (#438) 2023-09-27 10:31:52 +09:00
Al Liu 8ea341dbaf
Mixnet packet handle (#435)
* finish mixnet packet sending fan-in model
2023-09-26 18:23:34 +08:00
Giacomo Pasini 03973cd422
Make mempool generic (#428)
* Make mempool item generic

Make the mempool generic with respect to the item and remove
mentions of specific transaction formats/traits. This will allow
us to reuse the same code for both coordination layer transactions
and certificates, or in general, whatever items need to be included
in a block.

* Add mempool network adapter settings

Allow for greater customization of the mempool network adapter by
adding a settings field.

* update node after mempool changes

* fix waku mempool adapter

* fmt

* fix tests

* fmt
2023-09-26 11:14:44 +02:00
Daniel Sanchez 95618c0a72
Use selection for blob certificates (#427)
* Use selection for blob certificates

* Fix bin imports

* Fix rebase

* Missing blobs -> certificates refactor

* Fix attestation and certificate as_bytes

* More naming refactors
2023-09-25 15:34:05 +02:00
Youngjoon Lee 4da06a9740
Handle mixnet invalid fragment error (#436) 2023-09-25 21:12:48 +09:00
Giacomo Pasini 3b64cd6b3a
Add tests for blob dissemination (#416)
* Make nomos-cli config public

Split nomos-cli into lib and bin and make config public
so that it can be reused in tests

* Add tests for nomos-cli disseminate

* fmt

* feature gate whole file instead of function

* Exit unsuccessfully when failing dissemination
2023-09-25 11:52:45 +02:00
Youngjoon Lee 285804e1d5
Remove unnecessary dependencies (#433) 2023-09-25 17:26:21 +09:00
Al Liu cb343156b7
Add concrete Error implementation for mixnet (#405)
Add concrete Error implementation
2023-09-25 14:21:07 +08:00
gusto 46c7e25f6f
Use single workflow for master and pr checks (#434) 2023-09-22 11:19:56 +03:00
gusto 0eae715f8d
GH: Waku windows build configuration (#432)
* Update binutils for windows

* Create mingw-env action and move configuration there

* Use cc and cxx vars for gowaku

* Use go 1.20 for waku
2023-09-22 09:31:31 +03:00
Daniel Sanchez 2ceba073c0
Refactor blob -> blob_certificate in Block (#424) 2023-09-20 16:40:04 +02:00
Daniel Sanchez e71c359171
Remove waku support in nomos-node binary (#429) 2023-09-20 16:32:43 +02:00
Daniel Sanchez 500683b0da
Pipe propose block (#422)
* Add new method to tx selector

* Add new method to blob selector

* Docs typo

* Added missing txs and blobs methods to block builder

* Extend selector generics

* Pipe proposing selector types

* Remove leftover

* Added missing types on waku feature
2023-09-20 11:01:04 +02:00
Youngjoon Lee e0cd35dcdc
Refactor libp2p network backend into submodules (#417) 2023-09-19 22:16:27 +09:00
gusto 21fac726dd
Use build all before integration tests (#419) 2023-09-19 12:00:51 +03:00
gusto 5f040d5be7
Run integration tests in all gh actions (#418) 2023-09-19 12:00:27 +03:00
gusto c860b632dd
Add missing mixnet test to integration runs (#414) 2023-09-18 17:10:05 +03:00
Giacomo Pasini d672be3bb0
[nomos-cli] Add command to disseminate data (#400)
* Add nomos-cli command to disseminate data across the network

Add nomos-cli to hold various utilities for the node.
To start, this commit adds a command to disseminate some
data through the network and build a certificate of correct
dispersal for inclusion in a block

* fmt

* reorder loop instructions

* Send blobs concurrently
2023-09-18 15:35:20 +02:00
Al Liu 2429893ac2
Add mixnet retry mechanism (#386)
Imiplement mixnet retry
2023-09-18 18:26:54 +08:00
Giacomo Pasini 5e194922c6
Add da service to nomos node (#404)
* Make  the data availability service work with multiple protocols

* Add a generic way to instantiate DaProtocol

Add settings type and a new `new(Self::Settings)` method to
build a new DaProtocol instance

* Add data availability service to node

* fix tests

* fix imports
2023-09-18 11:43:24 +02:00
gusto 633b5d6cbf
CI: integrations tests libp2p (#411)
* Update rust version for ci linux host

* Add mixnode build step
2023-09-18 11:25:27 +03:00
gusto 8d9a1516be
ci: fix cargo clippy deprecated warnings (#412) 2023-09-18 11:24:49 +03:00
Youngjoon Lee 82946b8637
etry connecting with initial peers (#410) 2023-09-16 00:23:21 +09:00
gusto 97c653efe3
Docker compose for small libp2p node network (#364)
* Update rust build image version

* Docker compose file for testnet

* Wrap tcp into dns transport in order to resolve hostnames (#346)

* Docker compose for small libp2p node network

* Install etcdctl to node containers

* Register libp2p nodes on etcd

* Register IP address in KV store

* Interconnect libp2p nodes

* Use delimiter in cli and env variables

* Use docker compose initial env config

* Leave main Dockerfile as is, use new ones in testnet dir

* Remove etcd installation script

* run_mixnet.sh placeholder

* Use .env file for docker compose config

* Ignore local .env file

* Wrap sh envvars used in strings

* Remove mixnode placeholders

* Use default values for envconfig

* Update labels in Dockerfiles

* Sanitize scripts via shellcheck

* Export env for nomos node

* Updated to latest libp2p config

* Pass config to bootstrap node
2023-09-15 17:27:38 +03:00
gusto 90529dada3
Disable codecov annotations (#409)
* Add codecov config and disable gh annotations

* Allow some deviation in test coverage
2023-09-15 15:11:50 +03:00
Youngjoon Lee 8c4d1577df
Add mixnet usage docs (#408) 2023-09-15 17:36:28 +09:00
Giacomo Pasini 3121a41b60
Change default network backend to libp2p (#403)
We've been using it for a while and we don't plan to switch back
to Waku anytime soon.
2023-09-15 09:58:26 +02:00
Daniel Sanchez 04d07038dc
Remove unused fountain codes (#407)
* Remove unused fountain codes

* Refactor ProposalChunk to Proposal
2023-09-15 09:39:53 +02:00
Daniel Sanchez e1e2b84921
Block building core (#401)
* Move core block to folder base module

* Added blob select trait

* Added tx select trait

* Implement blob and tx selection filling size

* Added blobs to block constructor
Added block builder module

* Implement block builder

* Remove unnnecesary phantoms

* Add default impl for selections

* Added example

* Fix typos

Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>

* tx to item on generic function

Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>

---------

Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>
2023-09-15 09:39:16 +02:00
gusto 70dc4f3ae6
Simulation branch with one committee fix (#402)
* Tree and branch overlay comparison tests

* Update leader super majority fn in branch overlay

* Cleanup tests
2023-09-15 09:10:46 +03:00
Youngjoon Lee 8449c81d0f
Mixnet PoC base branch (#316)
* Add `mixnode` and `mixnet-client` crate (#302)

* Add `mixnode` binary (#317)

* Integrate mixnet with libp2p network backend (#318)

* Fix #312: proper delays (#321)

* proper delays

* add missing duration param

* tiny fix: compilation error caused by `rand` 0.8 -> 0.7

* use `get_available_port()` for mixnet integration tests (#333)

* add missing comments

* Overwatch mixnet node (#339)

* Add mixnet service and overwatch app

* remove #[tokio::main]

---------

Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>

* fix tests for the overwatch mixnode (#342)

* fix panic when corner case happen in RandomDelayIter (#335)

* Use `log` service for `mixnode` bin (#341)

* Use `wire` for MixnetMessage in libp2p (#347)

* Prevent tmixnet tests from running forever (#363)

* Use random delay when sending msgs to mixnet (#362)

* fix a minor compilation error caused by the latest master

* Fix run output fd (#343)

* add a connection pool

* Exp backoff (#332)

* move mixnet listening into separate task

* add exponential retry for insufficient peers in libp2p

* fix logging

* Fix MutexGuard across await (#373)

* Fix MutexGuard across await

Holding a MutexGuard across an await point is not a good idea.
Removing that solves the issues we had with the mixnet test

* Make mixnode handle bodies coming from the same source concurrently (#372)

---------

Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>

* Move wait at network startup (#338)

We now wait after the call to 'subscribe' to give the network
the time to register peers in the mesh before starting to
publish messages

* Remove unused functions from mixnet connpool (#374)

* Mixnet benchmark (#375)

* merge fixes

* add `connection_pool_size` field to `config.yaml`

* Simplify mixnet topology (#393)

* Simplify bytes and duration range ser/de (#394)

* optimize bytes serde and duration serde

---------

Co-authored-by: Al Liu <scygliu1@gmail.com>
Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com>
Co-authored-by: Giacomo Pasini <Zeegomo@users.noreply.github.com>
2023-09-14 17:38:47 +09:00
Giacomo Pasini 5b7c17e450
Remove failed deserialization error message (#399)
Firstly, a failure in deserialization for a network message is not
an error especially since we're using a public channel.
Secondly, that same channel is shared by different kind of messages
so trying to interpret one as the other will surely lead to a
unsuccessfull attempt.
2023-09-13 12:13:18 +02:00
Giacomo Pasini d72e54f9be
Add full replication implementation for DA (#396)
Add an initial simple but functional implementation for a data
availability protocol.
Full replication simply encodes bytes in a single blob which is
replicated in all nodes.
2023-09-13 11:01:20 +02:00
Giacomo Pasini 1eeeeb1834
Adopt a sans-IO style for DaProtocol (#395)
* Adopt a sans-IO style for DaProtocol

Remove async streams from the DaProtocol trait to avoid creating
dependencies on actual I/O. The protocol it's now transformed into
a state machine with an input and output buffer.

In addition, a method to create an attestation was added to the
trait.

* add forgotten &mut to certify_dispersal

* add forgotten &self param to validate certificate
2023-09-12 16:34:13 +02:00
Daniel Sanchez 7cabddc71f
Fix tracing error import in da service (#392) 2023-09-11 19:21:27 +02:00
Daniel Sanchez 8da13f7012
Da nomos core (#390)
* Refactor da modules
Include da core module in nomos-core

* Include attestation trait

* Added initial approach for Da protocol

* Added empty certificate trait

* Added certificate dispersal method

* Rename validate method to validate attestation

* Clippy happy

* Add validate certificate method
2023-09-11 17:44:26 +02:00
Giacomo Pasini efabf66994
Add default values for libp2p config (#388) 2023-09-11 15:26:01 +02:00
gusto 8e0937beb7
Node envconfig (#382)
* Move config to a new location

* Log config override with env variables

* Network config override with env variables

* Http env vars

* Consensus env vars
2023-09-11 14:56:35 +03:00
gusto ea64b74aa6
Remove codecov token, not required for public repos (#389) 2023-09-11 14:54:21 +03:00
gusto 03b9f69a01
Simulations: make polars dependancy optional (#385)
* Remove explicit target

* Use default profile for windows build

* Use stable-gnu for tests

* Make polars optional feature in simulations

* Remove unnecessary ci debug steps
2023-09-11 14:29:54 +03:00
Daniel Sanchez 3b90cb786c
Da service network (#384)
* Make da backend async

* Added remove blob

* Added send_blob method to network adapter trait

* Added libp2p backend
Implemented blob stream

* Implement attestation stream

* Implement send methods
2023-09-11 12:50:25 +02:00
Daniel Sanchez 96e3c2d499
Da service backend (#381)
* Add basic da module and traits

* Pipe new blobs and internal message handling

* Add and pipe send attestation method

* Add blob trait

* Make da backend async

* Implement mocka backend

* Bound blob in da backend to blob trait

* Added remove blob

* Rename reply to attestation
2023-09-08 11:27:52 +02:00
Giacomo Pasini a79d6c52e6
Add derive feature to serde (#380) 2023-09-07 10:46:13 +02:00
Daniel Sanchez bca27bd27f
Initial DA service sketch (#376)
* Add basic da module and traits

* Pipe new blobs and internal message handling

* Add and pipe send attestation method
2023-09-06 15:11:15 +02:00
Daniel Sanchez 57746f5e76
Block refactor (#368)
* Add sidecars to block

* Use cl and bl prefixes
2023-09-05 13:43:36 +02:00
gusto b9ef37e89e
Add missing strategy and target (#369) 2023-09-04 13:28:07 +03:00
gusto 1cbdbf0bfe
GH Actions: Codecov (#366)
* Add codecov gh action

* Skip integration tests
2023-09-04 10:27:44 +03:00
gusto 83d3d6f56c
GH Actions: Fix Windows build (#367)
* Remove continue-on-error cases

* Skip integration tests for PR and master builds

* Use gnu toolchain on windows

* Use toolchain instead of target
2023-09-04 10:27:13 +03:00
gusto 3be0d1a8a0
Update GH actions for master and prs (#365) 2023-09-01 10:45:53 +03:00
gusto e9b752aa4d
Set node_key from cli args (#361) 2023-08-31 14:48:49 +03:00