Commit Graph

313 Commits

Author SHA1 Message Date
Giacomo Pasini d72e54f9be
Add full replication implementation for DA (#396)
Add an initial simple but functional implementation for a data
availability protocol.
Full replication simply encodes bytes in a single blob which is
replicated in all nodes.
2023-09-13 11:01:20 +02:00
Giacomo Pasini 1eeeeb1834
Adopt a sans-IO style for DaProtocol (#395)
* Adopt a sans-IO style for DaProtocol

Remove async streams from the DaProtocol trait to avoid creating
dependencies on actual I/O. The protocol it's now transformed into
a state machine with an input and output buffer.

In addition, a method to create an attestation was added to the
trait.

* add forgotten &mut to certify_dispersal

* add forgotten &self param to validate certificate
2023-09-12 16:34:13 +02:00
Daniel Sanchez 7cabddc71f
Fix tracing error import in da service (#392) 2023-09-11 19:21:27 +02:00
Daniel Sanchez 8da13f7012
Da nomos core (#390)
* Refactor da modules
Include da core module in nomos-core

* Include attestation trait

* Added initial approach for Da protocol

* Added empty certificate trait

* Added certificate dispersal method

* Rename validate method to validate attestation

* Clippy happy

* Add validate certificate method
2023-09-11 17:44:26 +02:00
Giacomo Pasini efabf66994
Add default values for libp2p config (#388) 2023-09-11 15:26:01 +02:00
gusto 8e0937beb7
Node envconfig (#382)
* Move config to a new location

* Log config override with env variables

* Network config override with env variables

* Http env vars

* Consensus env vars
2023-09-11 14:56:35 +03:00
gusto ea64b74aa6
Remove codecov token, not required for public repos (#389) 2023-09-11 14:54:21 +03:00
gusto 03b9f69a01
Simulations: make polars dependancy optional (#385)
* Remove explicit target

* Use default profile for windows build

* Use stable-gnu for tests

* Make polars optional feature in simulations

* Remove unnecessary ci debug steps
2023-09-11 14:29:54 +03:00
Daniel Sanchez 3b90cb786c
Da service network (#384)
* Make da backend async

* Added remove blob

* Added send_blob method to network adapter trait

* Added libp2p backend
Implemented blob stream

* Implement attestation stream

* Implement send methods
2023-09-11 12:50:25 +02:00
Daniel Sanchez 96e3c2d499
Da service backend (#381)
* Add basic da module and traits

* Pipe new blobs and internal message handling

* Add and pipe send attestation method

* Add blob trait

* Make da backend async

* Implement mocka backend

* Bound blob in da backend to blob trait

* Added remove blob

* Rename reply to attestation
2023-09-08 11:27:52 +02:00
Giacomo Pasini a79d6c52e6
Add derive feature to serde (#380) 2023-09-07 10:46:13 +02:00
Daniel Sanchez bca27bd27f
Initial DA service sketch (#376)
* Add basic da module and traits

* Pipe new blobs and internal message handling

* Add and pipe send attestation method
2023-09-06 15:11:15 +02:00
Daniel Sanchez 57746f5e76
Block refactor (#368)
* Add sidecars to block

* Use cl and bl prefixes
2023-09-05 13:43:36 +02:00
gusto b9ef37e89e
Add missing strategy and target (#369) 2023-09-04 13:28:07 +03:00
gusto 1cbdbf0bfe
GH Actions: Codecov (#366)
* Add codecov gh action

* Skip integration tests
2023-09-04 10:27:44 +03:00
gusto 83d3d6f56c
GH Actions: Fix Windows build (#367)
* Remove continue-on-error cases

* Skip integration tests for PR and master builds

* Use gnu toolchain on windows

* Use toolchain instead of target
2023-09-04 10:27:13 +03:00
gusto 3be0d1a8a0
Update GH actions for master and prs (#365) 2023-09-01 10:45:53 +03:00
gusto e9b752aa4d
Set node_key from cli args (#361) 2023-08-31 14:48:49 +03:00
gusto 8adb24752a
Lock builds per commit and target (#340) 2023-08-31 14:35:21 +03:00
Daniel Sanchez 26a3910566
Pipe addtx endpoint for libp2p (#345)
* Pipe add tx for both backends

* Cleanup imports
2023-08-31 09:13:35 +02:00
Daniel Sanchez 28596377b7
Plug mempool to libp2p backend (#344)
* Plug mempool to libp2p backend

* Fix except, log error and skip instead.
2023-08-30 13:39:34 +02:00
Daniel Sanchez e5ec2b7739
Update resolver version in workspace (#337)
* Update resolver version in workspace

* Clippy happy
2023-08-29 15:45:48 +02:00
Daniel Sanchez 466cef7623
Fix kzg bench (#336) 2023-08-29 13:27:52 +02:00
Youngjoon Lee 115d45d0f0
Revert/improve `get_available_port()` in tests (#334) 2023-08-29 20:25:05 +09:00
Daniel Sanchez 2bd8ea92b1
DA kzg initial approach (#309)
* Added kzg crate

* Added basic commitment and proof creation methods

* Added blob verification

* DA kzg custom blob sizes (#325)

* Added types module

* Added dynamic size methods

* Added compute and verify test

* Fix using wrong constant on compute_commitment

* Criterion setup and first benchmark

* Move constants to settings struct

* Add tiny docs

* Pad blob is data shorter than it should

---------

Co-authored-by: Gusto <bacvinka@gmail.com>

---------

Co-authored-by: Gusto <bacvinka@gmail.com>
2023-08-28 10:51:46 +02:00
Youngjoon Lee ebbdf14406
Pick ports more randomly for integration tests (#315) 2023-08-21 22:13:52 +09:00
Al Liu 5391382928
Using CSV format by default for simulations (#304) 2023-08-21 12:02:10 +08:00
Al Liu 87f49c0fe9
Optimize serialization/deserialization for `[u8; 32]` (#306)
* optimize simulation app serialization
2023-08-17 23:46:07 +08:00
Daniel Sanchez 61b587a4ce
RS encoding/decoding base methods (#303)
* Add rs lib, implement encoding

* Implement decoding
2023-08-17 09:05:52 +02:00
gusto fe3d39071d
Simulation graceful shutdown fixes (#305)
* Handle finished simulation

* Flush remaining records to the sink

* Write runtime metadata last
2023-08-16 11:08:03 +03:00
Al Liu 2612c77306
Implement research overlay (#291)
* implement research overlay

* Add branch overlay

* Branch overlay config in simulation

* Remove research related extensions

* Use branch_depth and chunks

---------

Co-authored-by: Gusto <bacvinka@gmail.com>
2023-08-10 14:31:15 +03:00
gusto 07ae5a4df9
Add missing build and test flags (#301) 2023-08-10 14:20:19 +03:00
Daniel Sanchez 44367a15a5
Sim update overlay (#298)
* Update sim engine on timeout qc

* Remove rebuild method
2023-08-10 13:18:17 +02:00
Daniel Sanchez ef5935d572
Simulation prune tally (#299)
* Add pruning methods

* Call pruning on node step method

* Fix tally retain closure
2023-08-10 13:12:48 +02:00
gusto da60d8fc95
Parallel node init (#300) 2023-08-10 12:56:02 +03:00
Giacomo Pasini 78c6566d8a
Add libp2p backend to nomos-node (#285)
* Add support for libp2p backend in integration tests

* Add support for libp2p in nomos-node

* change default to waku

* add mutually exclusive features warning

* disable default features to avoid unification

* disable default features

* remove leftover cargo build

* Make sure we are subscribed to libp2p topic at startup

* unify imports

* typo in ci config

* Sequential build and test steps for features

* Add RandomBeaconState to libp2p carnot variant

---------

Co-authored-by: Gusto <bacvinka@gmail.com>
2023-08-09 07:42:08 +02:00
gusto d675585a0f
Test for partial message sending in simulation (#294)
* Test for partial message sending

* Test correctness, typos

* Fix the node capacity flushing

* Not process double timeout qcs in simulations

* Discard older view messages in simulations messages

* Refactor committed_blocks to latest_committed_blocks

* Remove tally default

* Fix condition to root committee parenting

* Bring back pruning

* Clippy happy

---------

Co-authored-by: danielsanchezq <sanchez.quiros.daniel@gmail.com>
2023-08-08 18:00:08 +03:00
Daniel Sanchez 4bdc3ed15a
Add update committees structure for overlay (#295)
* Move roundrobin to leadership module

* Use references in leader selection

* Add membership traits to overlay and create membership module

* Implement committee membership for random beacon state

* Update flat overlay

* Create updateable membership traits and impls

* Update tree overlay

* Update overlay on consensus service

* Update overlay on simulations nomos node

* Update types on tests and modules

* Use chacha for shuffling

* Change to mut slice instead of inner cloning

* Use fisher yates shuffle from scratch

* Stylish and clippy happy
2023-08-08 10:34:02 +02:00
gusto ef72c7a110
Add ci build steps for libp2p node version (#290)
* Add ci build steps for libp2p node version

* Update ci/Jenkinsfile.nightly.integration

Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>

* Fix typos

* Use features in cargo check

* Feature and testcase matrix for integration tests

* Use jenkins matrix to seperate steps for different features

---------

Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>
2023-08-07 18:39:24 +03:00
Giacomo Pasini 4a3d677ea9
Small fixes for libp2p network backend (#280)
* Generate network events for self messages

Waku does that and it's kind of convenient not to handle ourselves
in a different way from the rest.

* Use bigger buffer + fmt

When receiving messages coming from libp2p IWANT requests, it's
common to receive a burst of packest which can cause subscribers
to lag. To account for that, let's increase the buffer in the
broadcast channel.

* Check if topic is being subscribed before self-notification (#292)

* fmt

---------

Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>
2023-08-07 06:00:41 +02:00
Daniel Sanchez 30bf101576
Disable tree rebuild (#293) 2023-08-04 12:17:10 +02:00
Al Liu c16b794517
fix large msg sending logic (#274)
* fix large msg sending logic
2023-08-03 20:05:43 +08:00
Daniel Sanchez fa8e1025f5
Fix: genesis pruning in engine (#289)
* Do not prune genesis

* Fix retain condition
2023-08-03 12:25:33 +02:00
Giacomo Pasini 083d061e46
Add dummy libp2p adapter for mempool (#286) 2023-08-02 17:00:52 +02:00
Giacomo Pasini f8422fc7a8
Add initial implementation of libp2p consensus adapter (#279)
* Add initial implementation of libp2p consensus adapter

Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>

* fix

* Handle all message types received via gossipsub (#283)

* remove todo

Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>

---------

Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>
2023-08-02 14:07:44 +02:00
Giacomo Pasini c2ca46e6a8
Add command to retrieve libp2p network info (#281)
* Add command to retrieve libp2p network info

* fix field name

* fix
2023-08-01 17:04:18 +02:00
Daniel Sanchez fac42cd31d
Prune blocks in consensus engine (#275)
* Add prune older blocks call

* Prune older blocks in simulation app

* Use latest safe view for failure case

* Pick parent view from high qc

* Add safety assertion on pruning method
2023-08-01 16:47:47 +02:00
Daniel Sanchez ae59be5466
Refactor parent committee methods to return optional parent instead of empty one (#282) 2023-08-01 09:19:29 +02:00
Giacomo Pasini f21f1ea10a
Small additions to libp2p (#278)
* Add initial peer config to nomos-libp2p

* Use custom message id to avoid duplicates

* Expose reference to the inner swarm

* move closure into function
2023-08-01 03:26:42 +02:00
gusto 976b1f9577
Add node step time to report (#271)
* Skip step if previous node exeeded step time

* Fix clippy

* Passively keep track of step execution time in node
2023-07-26 10:21:05 +03:00