Commit Graph

220 Commits

Author SHA1 Message Date
Youngjoon Lee 8c4d1577df
Add mixnet usage docs (#408) 2023-09-15 17:36:28 +09:00
Giacomo Pasini 3121a41b60
Change default network backend to libp2p (#403)
We've been using it for a while and we don't plan to switch back
to Waku anytime soon.
2023-09-15 09:58:26 +02:00
Daniel Sanchez 04d07038dc
Remove unused fountain codes (#407)
* Remove unused fountain codes

* Refactor ProposalChunk to Proposal
2023-09-15 09:39:53 +02:00
Daniel Sanchez e1e2b84921
Block building core (#401)
* Move core block to folder base module

* Added blob select trait

* Added tx select trait

* Implement blob and tx selection filling size

* Added blobs to block constructor
Added block builder module

* Implement block builder

* Remove unnnecesary phantoms

* Add default impl for selections

* Added example

* Fix typos

Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>

* tx to item on generic function

Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>

---------

Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>
2023-09-15 09:39:16 +02:00
gusto 70dc4f3ae6
Simulation branch with one committee fix (#402)
* Tree and branch overlay comparison tests

* Update leader super majority fn in branch overlay

* Cleanup tests
2023-09-15 09:10:46 +03:00
Youngjoon Lee 8449c81d0f
Mixnet PoC base branch (#316)
* Add `mixnode` and `mixnet-client` crate (#302)

* Add `mixnode` binary (#317)

* Integrate mixnet with libp2p network backend (#318)

* Fix #312: proper delays (#321)

* proper delays

* add missing duration param

* tiny fix: compilation error caused by `rand` 0.8 -> 0.7

* use `get_available_port()` for mixnet integration tests (#333)

* add missing comments

* Overwatch mixnet node (#339)

* Add mixnet service and overwatch app

* remove #[tokio::main]

---------

Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>

* fix tests for the overwatch mixnode (#342)

* fix panic when corner case happen in RandomDelayIter (#335)

* Use `log` service for `mixnode` bin (#341)

* Use `wire` for MixnetMessage in libp2p (#347)

* Prevent tmixnet tests from running forever (#363)

* Use random delay when sending msgs to mixnet (#362)

* fix a minor compilation error caused by the latest master

* Fix run output fd (#343)

* add a connection pool

* Exp backoff (#332)

* move mixnet listening into separate task

* add exponential retry for insufficient peers in libp2p

* fix logging

* Fix MutexGuard across await (#373)

* Fix MutexGuard across await

Holding a MutexGuard across an await point is not a good idea.
Removing that solves the issues we had with the mixnet test

* Make mixnode handle bodies coming from the same source concurrently (#372)

---------

Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>

* Move wait at network startup (#338)

We now wait after the call to 'subscribe' to give the network
the time to register peers in the mesh before starting to
publish messages

* Remove unused functions from mixnet connpool (#374)

* Mixnet benchmark (#375)

* merge fixes

* add `connection_pool_size` field to `config.yaml`

* Simplify mixnet topology (#393)

* Simplify bytes and duration range ser/de (#394)

* optimize bytes serde and duration serde

---------

Co-authored-by: Al Liu <scygliu1@gmail.com>
Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com>
Co-authored-by: Giacomo Pasini <Zeegomo@users.noreply.github.com>
2023-09-14 17:38:47 +09:00
Giacomo Pasini 5b7c17e450
Remove failed deserialization error message (#399)
Firstly, a failure in deserialization for a network message is not
an error especially since we're using a public channel.
Secondly, that same channel is shared by different kind of messages
so trying to interpret one as the other will surely lead to a
unsuccessfull attempt.
2023-09-13 12:13:18 +02:00
Giacomo Pasini d72e54f9be
Add full replication implementation for DA (#396)
Add an initial simple but functional implementation for a data
availability protocol.
Full replication simply encodes bytes in a single blob which is
replicated in all nodes.
2023-09-13 11:01:20 +02:00
Giacomo Pasini 1eeeeb1834
Adopt a sans-IO style for DaProtocol (#395)
* Adopt a sans-IO style for DaProtocol

Remove async streams from the DaProtocol trait to avoid creating
dependencies on actual I/O. The protocol it's now transformed into
a state machine with an input and output buffer.

In addition, a method to create an attestation was added to the
trait.

* add forgotten &mut to certify_dispersal

* add forgotten &self param to validate certificate
2023-09-12 16:34:13 +02:00
Daniel Sanchez 7cabddc71f
Fix tracing error import in da service (#392) 2023-09-11 19:21:27 +02:00
Daniel Sanchez 8da13f7012
Da nomos core (#390)
* Refactor da modules
Include da core module in nomos-core

* Include attestation trait

* Added initial approach for Da protocol

* Added empty certificate trait

* Added certificate dispersal method

* Rename validate method to validate attestation

* Clippy happy

* Add validate certificate method
2023-09-11 17:44:26 +02:00
Giacomo Pasini efabf66994
Add default values for libp2p config (#388) 2023-09-11 15:26:01 +02:00
gusto 8e0937beb7
Node envconfig (#382)
* Move config to a new location

* Log config override with env variables

* Network config override with env variables

* Http env vars

* Consensus env vars
2023-09-11 14:56:35 +03:00
gusto ea64b74aa6
Remove codecov token, not required for public repos (#389) 2023-09-11 14:54:21 +03:00
gusto 03b9f69a01
Simulations: make polars dependancy optional (#385)
* Remove explicit target

* Use default profile for windows build

* Use stable-gnu for tests

* Make polars optional feature in simulations

* Remove unnecessary ci debug steps
2023-09-11 14:29:54 +03:00
Daniel Sanchez 3b90cb786c
Da service network (#384)
* Make da backend async

* Added remove blob

* Added send_blob method to network adapter trait

* Added libp2p backend
Implemented blob stream

* Implement attestation stream

* Implement send methods
2023-09-11 12:50:25 +02:00
Daniel Sanchez 96e3c2d499
Da service backend (#381)
* Add basic da module and traits

* Pipe new blobs and internal message handling

* Add and pipe send attestation method

* Add blob trait

* Make da backend async

* Implement mocka backend

* Bound blob in da backend to blob trait

* Added remove blob

* Rename reply to attestation
2023-09-08 11:27:52 +02:00
Giacomo Pasini a79d6c52e6
Add derive feature to serde (#380) 2023-09-07 10:46:13 +02:00
Daniel Sanchez bca27bd27f
Initial DA service sketch (#376)
* Add basic da module and traits

* Pipe new blobs and internal message handling

* Add and pipe send attestation method
2023-09-06 15:11:15 +02:00
Daniel Sanchez 57746f5e76
Block refactor (#368)
* Add sidecars to block

* Use cl and bl prefixes
2023-09-05 13:43:36 +02:00
gusto b9ef37e89e
Add missing strategy and target (#369) 2023-09-04 13:28:07 +03:00
gusto 1cbdbf0bfe
GH Actions: Codecov (#366)
* Add codecov gh action

* Skip integration tests
2023-09-04 10:27:44 +03:00
gusto 83d3d6f56c
GH Actions: Fix Windows build (#367)
* Remove continue-on-error cases

* Skip integration tests for PR and master builds

* Use gnu toolchain on windows

* Use toolchain instead of target
2023-09-04 10:27:13 +03:00
gusto 3be0d1a8a0
Update GH actions for master and prs (#365) 2023-09-01 10:45:53 +03:00
gusto e9b752aa4d
Set node_key from cli args (#361) 2023-08-31 14:48:49 +03:00
gusto 8adb24752a
Lock builds per commit and target (#340) 2023-08-31 14:35:21 +03:00
Daniel Sanchez 26a3910566
Pipe addtx endpoint for libp2p (#345)
* Pipe add tx for both backends

* Cleanup imports
2023-08-31 09:13:35 +02:00
Daniel Sanchez 28596377b7
Plug mempool to libp2p backend (#344)
* Plug mempool to libp2p backend

* Fix except, log error and skip instead.
2023-08-30 13:39:34 +02:00
Daniel Sanchez e5ec2b7739
Update resolver version in workspace (#337)
* Update resolver version in workspace

* Clippy happy
2023-08-29 15:45:48 +02:00
Daniel Sanchez 466cef7623
Fix kzg bench (#336) 2023-08-29 13:27:52 +02:00
Youngjoon Lee 115d45d0f0
Revert/improve `get_available_port()` in tests (#334) 2023-08-29 20:25:05 +09:00
Daniel Sanchez 2bd8ea92b1
DA kzg initial approach (#309)
* Added kzg crate

* Added basic commitment and proof creation methods

* Added blob verification

* DA kzg custom blob sizes (#325)

* Added types module

* Added dynamic size methods

* Added compute and verify test

* Fix using wrong constant on compute_commitment

* Criterion setup and first benchmark

* Move constants to settings struct

* Add tiny docs

* Pad blob is data shorter than it should

---------

Co-authored-by: Gusto <bacvinka@gmail.com>

---------

Co-authored-by: Gusto <bacvinka@gmail.com>
2023-08-28 10:51:46 +02:00
Youngjoon Lee ebbdf14406
Pick ports more randomly for integration tests (#315) 2023-08-21 22:13:52 +09:00
Al Liu 5391382928
Using CSV format by default for simulations (#304) 2023-08-21 12:02:10 +08:00
Al Liu 87f49c0fe9
Optimize serialization/deserialization for `[u8; 32]` (#306)
* optimize simulation app serialization
2023-08-17 23:46:07 +08:00
Daniel Sanchez 61b587a4ce
RS encoding/decoding base methods (#303)
* Add rs lib, implement encoding

* Implement decoding
2023-08-17 09:05:52 +02:00
gusto fe3d39071d
Simulation graceful shutdown fixes (#305)
* Handle finished simulation

* Flush remaining records to the sink

* Write runtime metadata last
2023-08-16 11:08:03 +03:00
Al Liu 2612c77306
Implement research overlay (#291)
* implement research overlay

* Add branch overlay

* Branch overlay config in simulation

* Remove research related extensions

* Use branch_depth and chunks

---------

Co-authored-by: Gusto <bacvinka@gmail.com>
2023-08-10 14:31:15 +03:00
gusto 07ae5a4df9
Add missing build and test flags (#301) 2023-08-10 14:20:19 +03:00
Daniel Sanchez 44367a15a5
Sim update overlay (#298)
* Update sim engine on timeout qc

* Remove rebuild method
2023-08-10 13:18:17 +02:00
Daniel Sanchez ef5935d572
Simulation prune tally (#299)
* Add pruning methods

* Call pruning on node step method

* Fix tally retain closure
2023-08-10 13:12:48 +02:00
gusto da60d8fc95
Parallel node init (#300) 2023-08-10 12:56:02 +03:00
Giacomo Pasini 78c6566d8a
Add libp2p backend to nomos-node (#285)
* Add support for libp2p backend in integration tests

* Add support for libp2p in nomos-node

* change default to waku

* add mutually exclusive features warning

* disable default features to avoid unification

* disable default features

* remove leftover cargo build

* Make sure we are subscribed to libp2p topic at startup

* unify imports

* typo in ci config

* Sequential build and test steps for features

* Add RandomBeaconState to libp2p carnot variant

---------

Co-authored-by: Gusto <bacvinka@gmail.com>
2023-08-09 07:42:08 +02:00
gusto d675585a0f
Test for partial message sending in simulation (#294)
* Test for partial message sending

* Test correctness, typos

* Fix the node capacity flushing

* Not process double timeout qcs in simulations

* Discard older view messages in simulations messages

* Refactor committed_blocks to latest_committed_blocks

* Remove tally default

* Fix condition to root committee parenting

* Bring back pruning

* Clippy happy

---------

Co-authored-by: danielsanchezq <sanchez.quiros.daniel@gmail.com>
2023-08-08 18:00:08 +03:00
Daniel Sanchez 4bdc3ed15a
Add update committees structure for overlay (#295)
* Move roundrobin to leadership module

* Use references in leader selection

* Add membership traits to overlay and create membership module

* Implement committee membership for random beacon state

* Update flat overlay

* Create updateable membership traits and impls

* Update tree overlay

* Update overlay on consensus service

* Update overlay on simulations nomos node

* Update types on tests and modules

* Use chacha for shuffling

* Change to mut slice instead of inner cloning

* Use fisher yates shuffle from scratch

* Stylish and clippy happy
2023-08-08 10:34:02 +02:00
gusto ef72c7a110
Add ci build steps for libp2p node version (#290)
* Add ci build steps for libp2p node version

* Update ci/Jenkinsfile.nightly.integration

Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>

* Fix typos

* Use features in cargo check

* Feature and testcase matrix for integration tests

* Use jenkins matrix to seperate steps for different features

---------

Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>
2023-08-07 18:39:24 +03:00
Giacomo Pasini 4a3d677ea9
Small fixes for libp2p network backend (#280)
* Generate network events for self messages

Waku does that and it's kind of convenient not to handle ourselves
in a different way from the rest.

* Use bigger buffer + fmt

When receiving messages coming from libp2p IWANT requests, it's
common to receive a burst of packest which can cause subscribers
to lag. To account for that, let's increase the buffer in the
broadcast channel.

* Check if topic is being subscribed before self-notification (#292)

* fmt

---------

Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>
2023-08-07 06:00:41 +02:00
Daniel Sanchez 30bf101576
Disable tree rebuild (#293) 2023-08-04 12:17:10 +02:00
Al Liu c16b794517
fix large msg sending logic (#274)
* fix large msg sending logic
2023-08-03 20:05:43 +08:00
Daniel Sanchez fa8e1025f5
Fix: genesis pruning in engine (#289)
* Do not prune genesis

* Fix retain condition
2023-08-03 12:25:33 +02:00