* Make the data availability service work with multiple protocols
* Add a generic way to instantiate DaProtocol
Add settings type and a new `new(Self::Settings)` method to
build a new DaProtocol instance
* Add data availability service to node
* fix tests
* fix imports
* Add `mixnode` and `mixnet-client` crate (#302)
* Add `mixnode` binary (#317)
* Integrate mixnet with libp2p network backend (#318)
* Fix#312: proper delays (#321)
* proper delays
* add missing duration param
* tiny fix: compilation error caused by `rand` 0.8 -> 0.7
* use `get_available_port()` for mixnet integration tests (#333)
* add missing comments
* Overwatch mixnet node (#339)
* Add mixnet service and overwatch app
* remove #[tokio::main]
---------
Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>
* fix tests for the overwatch mixnode (#342)
* fix panic when corner case happen in RandomDelayIter (#335)
* Use `log` service for `mixnode` bin (#341)
* Use `wire` for MixnetMessage in libp2p (#347)
* Prevent tmixnet tests from running forever (#363)
* Use random delay when sending msgs to mixnet (#362)
* fix a minor compilation error caused by the latest master
* Fix run output fd (#343)
* add a connection pool
* Exp backoff (#332)
* move mixnet listening into separate task
* add exponential retry for insufficient peers in libp2p
* fix logging
* Fix MutexGuard across await (#373)
* Fix MutexGuard across await
Holding a MutexGuard across an await point is not a good idea.
Removing that solves the issues we had with the mixnet test
* Make mixnode handle bodies coming from the same source concurrently (#372)
---------
Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>
* Move wait at network startup (#338)
We now wait after the call to 'subscribe' to give the network
the time to register peers in the mesh before starting to
publish messages
* Remove unused functions from mixnet connpool (#374)
* Mixnet benchmark (#375)
* merge fixes
* add `connection_pool_size` field to `config.yaml`
* Simplify mixnet topology (#393)
* Simplify bytes and duration range ser/de (#394)
* optimize bytes serde and duration serde
---------
Co-authored-by: Al Liu <scygliu1@gmail.com>
Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com>
Co-authored-by: Giacomo Pasini <Zeegomo@users.noreply.github.com>
Firstly, a failure in deserialization for a network message is not
an error especially since we're using a public channel.
Secondly, that same channel is shared by different kind of messages
so trying to interpret one as the other will surely lead to a
unsuccessfull attempt.
* Add basic da module and traits
* Pipe new blobs and internal message handling
* Add and pipe send attestation method
* Add blob trait
* Make da backend async
* Implement mocka backend
* Bound blob in da backend to blob trait
* Added remove blob
* Rename reply to attestation
* Add support for libp2p backend in integration tests
* Add support for libp2p in nomos-node
* change default to waku
* add mutually exclusive features warning
* disable default features to avoid unification
* disable default features
* remove leftover cargo build
* Make sure we are subscribed to libp2p topic at startup
* unify imports
* typo in ci config
* Sequential build and test steps for features
* Add RandomBeaconState to libp2p carnot variant
---------
Co-authored-by: Gusto <bacvinka@gmail.com>
* Move roundrobin to leadership module
* Use references in leader selection
* Add membership traits to overlay and create membership module
* Implement committee membership for random beacon state
* Update flat overlay
* Create updateable membership traits and impls
* Update tree overlay
* Update overlay on consensus service
* Update overlay on simulations nomos node
* Update types on tests and modules
* Use chacha for shuffling
* Change to mut slice instead of inner cloning
* Use fisher yates shuffle from scratch
* Stylish and clippy happy
* Generate network events for self messages
Waku does that and it's kind of convenient not to handle ourselves
in a different way from the rest.
* Use bigger buffer + fmt
When receiving messages coming from libp2p IWANT requests, it's
common to receive a burst of packest which can cause subscribers
to lag. To account for that, let's increase the buffer in the
broadcast channel.
* Check if topic is being subscribed before self-notification (#292)
* fmt
---------
Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>
* Add initial implementation of libp2p consensus adapter
Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>
* fix
* Handle all message types received via gossipsub (#283)
* remove todo
Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>
---------
Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>
* Rework NetworkAdapter API
The NetworkAdapter API failed to isolate the internals by
providing a way to send a message to a user-provided channel while
the stream listeners expected specific formats.
Unify network messages under the same enum and simplify sending/
broadcasting messages.
* remove redundant inlines
* use committee.id()
* fmt
* Make timeout configurable
Add a way to configure the consensus timeout at startup.
* Make leader threshold and timeout configurable in tests
* Add tests for the unhappy path
Add a test for the unhappy path by stopping a node.
The rest of the peers are sufficient to reach a quorum but the
offline node will fail to produce a block when it's its turn as a
leader, thus triggering the recovery procedure twice before the
test is considered complete.
* ignore clippy warning
* CommitteeId type wrapper
* rewrite committee id logic
* remove unused convert
* use correct way to get committe id by member id
* cleanup and add comment
* Simplify child committees method
---------
Co-authored-by: danielsanchezq <sanchez.quiros.daniel@gmail.com>
* Add unhappy path handlers for first view
Nodes resolving the first view were not instructed to handle
possible failures. In particular, no timeouts were configured
and nodes were not listening for timeouts from other nodes.
This commit fixes this by making them use the
same path as every other view.
* Endless timeouts
Keep signaling timeout until consensus is achieved.
This could be helpful in network partitioning where messages could
be lost.
* Ensure consistent committee hash
Sort committee members before hashing to ensure all nodes in the
network obtain a consistent hash
* Fix timeout_qc filtering condition
'>' was used instead of '=='
* Fix new view topic name
'votes' was used instead of 'new-view'
* Fix filtering condition for unhappy path tally
Rogue '!' at the beginning
* Add timeout tally
Filter timeouts to allow only timeouts for the current view
coming from root committee members.
We might want to try to unigy this with happy and unhappy tally.
* Add debug logs
* clippy happy
* add basic integration tests
* add a way to configure overlay threshold
* Save logs to file in case of failure
* Increase number of test nodes to 10
* fix tests
* use fraction instead of tuple
* fmt
* Fix order of network streams
When fetching a message from the network, we need to first listen
for incoming messages and then look at the storage. If we do this
in the opposite order, there's a brief moment where we've freezed
our view of stored messages and are not yet listening for incoming
ones, thus risking loosing messages.
* Wait for waku db spurious delays
Sometimes waku takes some time (e.g. a few seconds) before making
a received message available through the archive query. Let's
account for this by making repeated calls if the first one is not
successful.
* Add initial network wait
We've observed in testing that even if waku reports that some
peers are connected it can't really deliver a message. To overcome
this limitation, we add a wait at the network service startup.
We know this is not ideal, but Waku will eventually be replaced and
we're looking for a quick fix.
* fmt
* add missing accessors
* add api to return info from consensus service
* move processing into its own function
* feat: add HTTP GET `/carnot/info` API (#178)
---------
Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>
The consensus service was becoming a bit messy and difficult to maintain.
This PR moves handling invididual events to their own function and refactors the cancelabe task system in a separate struct.
* move overlay to consensus engine
* Integrate random beacon in overlay
This commit integrates the random beacon as specified in the nomos
spec into the consensus engine library as part of the overlay.
In addition, it separates the overlay part responsible for leader
selection as an independent trait LeaderSelection, so as to share
the overall overlay structure among most constructions.
Furthermore, a leader proof has been added to a block to verify
that the proposer had valid rights to do so.
The current implementation hardcodes the leader selection update
in the consensus service, but we probably want to abstract it away
through something like the adapter pattern we use of other services
in the node.
* Move leader selection update to separate function
* Add generic support for leader selection in consensus service (#170)
* Add generic support for leader selection in consensus service
* fix
* use settings struct instead of tuple
* fix tests
* Add handling of root timeout
* Use StandardQc instead of Qc for high_qc
Since high_qc is guaranteed to be a standard qc, let's just use
StandardQc as a type in all messages that use it.
* Added Qc type to Tally trait
* Add generic Qc to block header
* Use blockid instead of unnecessary header
* Expand Qc over generics
* Build up block with proper qc header
* Include status codes in http error responses
* Mockpool bridges error handling
* Last TX in milliseconds in mempool metrics
* u64 for last tx metrics
* Added waku archive message to waku network backend
* Use cached streams in consensus waku adapter
* Fix mock test
* Add missing import
* Join requests tasks
* Use waku-bindings beta4
* Get stream from archive query method
* Set store protocol active for waku backend
* Implement local query stream response
* Add missing linking flags for new waku-bindings version
* Cleanup unbounded sender fuse/unwrap
* Clippy happy
* Add enpoint to make a new peer conn for mockpool node
* Pass multiple addresses in connection request
* Join async reqs to network service, cleanup
* Collect and join reqs
* Broadcast transaction when validated in mempool
* Clippy happy
* Use standard bincode config for tx message decoding
* add send_transaction for mock (#64)
* add send_transaction for mock
* Use wire instead of direct bincode
* Use wire instead of direct bincode on deserialization
---------
Co-authored-by: Al Liu <scygliu1@gmail.com>
* Add committee and view information to network adapter
* Use committee and view on waku adapter
* Add committee and view info to flat view implementation.
* Clippy happy
* Rename flat -> root
* Split broadcast block iterator into l/r childs
* Extract topics to builder functions
* Cargo http-service folder
* WIP: http server
* Revert comments in network service
* Router service and axum implementation structure
* Move bin contents to examples dir
* Add http service and server traits
* HttpMsg definition
* WIP: axum backend
* fix example in Cargo.toml
* Shared axum router and router modification methods
* Http example with axum and metrics service
* make project compile and add Error associated type
* Axum backend shared router fixes
* Dummy service implementation for http example
* remove unused clone on mutex
* Cargo http-service folder
* WIP: http server
* Revert comments in network service
* Router service and axum implementation structure
* Move bin contents to examples dir
* Add http service and server traits
* HttpMsg definition
* WIP: axum backend
* fix example in Cargo.toml
* Shared axum router and router modification methods
* Http example with axum and metrics service
* make project compile and add Error associated type
* Axum backend shared router fixes
* Dummy service implementation for http example
* remove unused clone on mutex
* Fix typos and remove unused code
* Fix failing tests when feature flags are not set
* Use bytes as a type for payload and response in http service
* Refactored http crate layout into differential services files
* First stab at router service
* Fully piped http bridge system
* Start building bridge helper function
* Refactor bridge builder helper and update example
* impl serialization for metrics data
* Get updated copy of router when processing request
* remove unused code
* fix typo
* [POC]: Http service: support add graphql handler (#47)
* WIP: add graphql endpoint
* support add graphql handler
* remove generic
* fix clippy warnings
* Add post put and patch handlers that expect bytes as body
* Graphql example file
* WIP: Use http post method for graphql related queries
* Parse graphql requests in handler
* Simplify handlers for post and other data methods
* Revert "Simplify handlers for post and other data methods"
This reverts commit 96f2b1821e.
* add tracing and remove comments
* Pass response bytes without any modifications
* Use receive_batch_json for gql request parsing
* Readme for running examples
* fix conflicts
* add a general helper function for graphql
* remove unused function
* cleanup code
* move schema initialization to handle function
* adapt metrics to http service
* fix clippy warnings
* remove unused fn
* fix clippy
* optimize example
Co-authored-by: gusto <bacvinka@gmail.com>
Co-authored-by: Gusto Bacvinka <augustinas.bacvinka@gmail.com>
* Fix cargo build without features
* Simplify handlers for routes with data
Co-authored-by: al8n <scygliu1@gmail.com>
Co-authored-by: Daniel Sanchez Quiros <sanchez.quiros.daniel@gmail.com>
* Add basic encode/decode test
* Use Stream for trait instead of Iterator
* Removed unnecessary pin
* Add custom fountain error
* Add failing path to tests
* Added docs
* Normalized chunks to bytes
* Added settings initialization for fountain codes trait
* Pipe fountain code through consensus
* Implement broadcast block and block reconstruction for Member Overlay
* Fix failing raptor test
* Use seed for raptorq tests
* Use const for topics instead of static
* Clippy happy
* Refactor mempool naming and added settings to backend trait
* Implement mempool networking traits and waku backend
* Transaction networking plumbing in mempool service
* Make TransactionMsg generic
Use bincode to deserialize tx messages
* Make wakuadapter generic over tx and tx-id
* Fix wrong backend type bound
* Adapt to waku beta2
* Thread tx and id together for adapter and pool
* Panic on subscribing error
* Prune unnecessary Id type bounds
* Remove transaction placeholder
* Remove Id bound from waku adapter
* Remove empty transactions module
* ci: add Jenkins and Docker file for PR and doc builds
* Fixing waku-bindings dependacy definitions
* Add post cleanup and musl-dev to Jenkinsfile
* Add go to the Dockerfile for ci
* GOCACHE set to tmp for ci builds
* Use slim-bullseye docker image for ci builds
* Update ci/Jenkinsfile.docs
Co-authored-by: Jakub <i+github@always.fail>
* Add image version and env vars for docs build
* Remove duplicated environment section
* Add missing rust dependencies
* Split jenkins file into linux and macos targets
* Removing github actions for PR checks
* Add explicit versions to the shell.nix dependencies
* Add jenkins libs required for nix
* Use default rust-bin version that includes clippy and fmt
* Add readme to ci folder
Co-authored-by: Jakub <i+github@always.fail>
Youngjoon Lee
Giacomo Pasini
gusto
Daniel Sanchez
Al Liu