Commit Graph

68 Commits

Author SHA1 Message Date
GoodDaisy ff8fc577a0
Fix typos (#481)
Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com>
2023-10-31 14:38:42 +01:00
gusto d479721efd
Optional network capacity in simulations (#483)
* Make simulation net capacity optional

* Add cli flag to disable netcap
2023-10-30 06:26:45 +02:00
gusto 54dd96dfff
Simulations overlay topo info (#479)
* Missing json feature for building standalone simapp

* Move dummy sim overlay to tests module

* OverlayInfoExt in simulation app

* Dump overlay info in simapp
2023-10-26 19:43:49 +03:00
gusto 99d77f7e1c
Nomos node tree overlay (#415)
* Use tree overlay in nomos node

* Use tree overlay in tests module

* Handle unexpected consensus vote stream end in tally

* Spawn the next leader first (#425)

* Report unhappy blocks in happy path test (#430)

* report unhappy blocks in the happy path test

* Make threshold configurable for TreeOverlay (#426)

* Modified test, so that all nodes don’t all connect to the first node only (#442)

* merge fix

---------

Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>
Co-authored-by: Al Liu <scygliu1@gmail.com>
2023-10-05 11:56:24 +03:00
Youngjoon Lee 285804e1d5
Remove unnecessary dependencies (#433) 2023-09-25 17:26:21 +09:00
Daniel Sanchez 04d07038dc
Remove unused fountain codes (#407)
* Remove unused fountain codes

* Refactor ProposalChunk to Proposal
2023-09-15 09:39:53 +02:00
Daniel Sanchez e1e2b84921
Block building core (#401)
* Move core block to folder base module

* Added blob select trait

* Added tx select trait

* Implement blob and tx selection filling size

* Added blobs to block constructor
Added block builder module

* Implement block builder

* Remove unnnecesary phantoms

* Add default impl for selections

* Added example

* Fix typos

Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>

* tx to item on generic function

Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>

---------

Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>
2023-09-15 09:39:16 +02:00
gusto 03b9f69a01
Simulations: make polars dependancy optional (#385)
* Remove explicit target

* Use default profile for windows build

* Use stable-gnu for tests

* Make polars optional feature in simulations

* Remove unnecessary ci debug steps
2023-09-11 14:29:54 +03:00
Daniel Sanchez 57746f5e76
Block refactor (#368)
* Add sidecars to block

* Use cl and bl prefixes
2023-09-05 13:43:36 +02:00
Al Liu 5391382928
Using CSV format by default for simulations (#304) 2023-08-21 12:02:10 +08:00
Al Liu 87f49c0fe9
Optimize serialization/deserialization for `[u8; 32]` (#306)
* optimize simulation app serialization
2023-08-17 23:46:07 +08:00
gusto fe3d39071d
Simulation graceful shutdown fixes (#305)
* Handle finished simulation

* Flush remaining records to the sink

* Write runtime metadata last
2023-08-16 11:08:03 +03:00
Al Liu 2612c77306
Implement research overlay (#291)
* implement research overlay

* Add branch overlay

* Branch overlay config in simulation

* Remove research related extensions

* Use branch_depth and chunks

---------

Co-authored-by: Gusto <bacvinka@gmail.com>
2023-08-10 14:31:15 +03:00
Daniel Sanchez 44367a15a5
Sim update overlay (#298)
* Update sim engine on timeout qc

* Remove rebuild method
2023-08-10 13:18:17 +02:00
Daniel Sanchez ef5935d572
Simulation prune tally (#299)
* Add pruning methods

* Call pruning on node step method

* Fix tally retain closure
2023-08-10 13:12:48 +02:00
gusto da60d8fc95
Parallel node init (#300) 2023-08-10 12:56:02 +03:00
gusto d675585a0f
Test for partial message sending in simulation (#294)
* Test for partial message sending

* Test correctness, typos

* Fix the node capacity flushing

* Not process double timeout qcs in simulations

* Discard older view messages in simulations messages

* Refactor committed_blocks to latest_committed_blocks

* Remove tally default

* Fix condition to root committee parenting

* Bring back pruning

* Clippy happy

---------

Co-authored-by: danielsanchezq <sanchez.quiros.daniel@gmail.com>
2023-08-08 18:00:08 +03:00
Daniel Sanchez 4bdc3ed15a
Add update committees structure for overlay (#295)
* Move roundrobin to leadership module

* Use references in leader selection

* Add membership traits to overlay and create membership module

* Implement committee membership for random beacon state

* Update flat overlay

* Create updateable membership traits and impls

* Update tree overlay

* Update overlay on consensus service

* Update overlay on simulations nomos node

* Update types on tests and modules

* Use chacha for shuffling

* Change to mut slice instead of inner cloning

* Use fisher yates shuffle from scratch

* Stylish and clippy happy
2023-08-08 10:34:02 +02:00
Al Liu c16b794517
fix large msg sending logic (#274)
* fix large msg sending logic
2023-08-03 20:05:43 +08:00
Daniel Sanchez fac42cd31d
Prune blocks in consensus engine (#275)
* Add prune older blocks call

* Prune older blocks in simulation app

* Use latest safe view for failure case

* Pick parent view from high qc

* Add safety assertion on pruning method
2023-08-01 16:47:47 +02:00
Daniel Sanchez ae59be5466
Refactor parent committee methods to return optional parent instead of empty one (#282) 2023-08-01 09:19:29 +02:00
gusto 976b1f9577
Add node step time to report (#271)
* Skip step if previous node exeeded step time

* Fix clippy

* Passively keep track of step execution time in node
2023-07-26 10:21:05 +03:00
gusto ef0b0701f3
Simulations nodeid report (#272)
* Add node id to the node state report

* Update config with record settings example
2023-07-25 17:17:26 +03:00
gusto df97ea2543
Simulatios network throughput (#270)
* Use step time from configuration

* PayloadSize trait for inmemory network interface

* Per node network capacity

* Initial carnot message sizes

* Divide provided kbps by step time

* Use std::mem::size_of for msg sizes
2023-07-25 16:03:05 +03:00
gusto 7d64915dd7
Get current view directly from the engine (#269) 2023-07-21 12:30:52 +03:00
gusto 8dd34f81b4
Simulation tally fix (#268)
* Do not remove entry cache when the threshold is reached

* Leader super majority change

* Update leader threshold test
2023-07-21 12:30:32 +03:00
Al Liu 6dee12704d
Change simulation `Node` trait `current_view` method return `View` (#259) 2023-07-20 17:31:55 +08:00
Daniel Sanchez 0866dfc8af
Process single events (#267) 2023-07-20 10:08:09 +02:00
Daniel Sanchez adf935830e
Do not remove entry cache when the threshold is reached (#266)
Co-authored-by: Gusto <bacvinka@gmail.com>
2023-07-19 18:04:08 +02:00
gusto 9f71dbb24c
Simulation network broadcast fix (#262)
* Replace network broadcast msg type to a dedicated channel

* Update tests with broadcast chan

* Replace threadrng with seedable smallrng

* Simplify the broadcast loop
2023-07-18 15:01:29 +03:00
gusto 94e384f609
Tree settings for sim overlay (#257)
* Tree settings for sim overlay

* WIP dyn node simulation

* Allow generic settings and states for dispatched nodes

* Update tests
2023-07-18 14:44:23 +03:00
Al Liu 9467351c10
Finish `View` wrapper (#254)
* finish View wrapper
2023-07-12 21:30:22 +08:00
Al Liu 2135676606
Finish `NodeId` type wrapper (#252)
* add NodeId wrapper
2023-07-11 23:16:49 +08:00
Al Liu a0cb738b9f
Finish `Committee` wrapper (#251)
* Committee type wrapper

---------

Co-authored-by: danielsanchezq <sanchez.quiros.daniel@gmail.com>
2023-07-07 16:08:19 +08:00
Giacomo Pasini 90cf29bf86
Better tests (#232)
* add basic integration tests

* add a way to configure overlay threshold

* Save logs to file in case of failure

* Increase number of test nodes to 10

* fix tests

* use fraction instead of tuple

* fmt
2023-07-04 11:39:28 +02:00
Youngjoon Lee 98aa138b87
feat: enforce TimeoutQc to be constructed only by new() (#229) 2023-06-28 23:37:27 +09:00
Al Liu 224a3a53f5
fix tally_by condition (#234) 2023-06-28 17:56:16 +08:00
Al Liu 4a80690c1f
Add a cli arg to support redirect log (#230)
* add cli arg to support redirect log
2023-06-28 17:10:19 +08:00
gusto 55648e3151
Simulation network broadcast (#225)
* Sim network broadcast method

* Remove list of all nodes in carnot node instance

* Use network broadcast method within nodes

* Add helper method for network payload
2023-06-27 14:20:59 +03:00
Al Liu 18bbf63a3f
update simulation app configuration (#217) 2023-06-22 19:50:34 +08:00
Al Liu 4ccc19f5a3
Support configurable records (#200)
* support configurable records
2023-06-21 16:31:36 +08:00
Daniel Sanchez bed0b9448d
Simulation unhappy path (#193)
* Use elapsed time

* Added timeout

* Extract tally

* Missing elapsed time

* Fix new view leader behaviour

* Fix tests

* Fix timeout double check

* Fix logs

* TimeoutHandler nitpicks

* Clippy happy

* Fix timeout sub

* Modify discard messages comment
2023-06-19 15:27:14 +02:00
gusto faacd10172
Remove dummy node from simapp settings and app itself (#196) 2023-06-17 13:38:59 +03:00
Al Liu 40048fa47b
Fix #183: support structured log for sim app. (#184) 2023-06-16 19:26:48 +08:00
Al Liu 9afd6c007c
Simulation conf polish (#181)
* polish Duration ser/deser and support step_time in conf file
2023-06-15 17:42:42 +08:00
Al Liu b6789b994e
Fix #182: gracefully shutdown (#187)
* finish signal and cleanup
2023-06-15 17:40:29 +08:00
Al Liu e57cb7b3cf
remove RwLock from SimmulationRunnerInner (#189) 2023-06-15 11:16:28 +02:00
Al Liu 27d9f72035
Simulation happy path (#161)
* finish subscriber manager

* optimize subscribe on SimulationRunnerHandle

* fix comment

* replace std locks to parking_lot locks

* move producer initialization out simulate fn

* WIP

* optimize run fn

* update condition

* fix CI

* collect run times

* Add happy-path consensus engine

* tmp

* Fit types from spec (#124)

* Match types to spec

* Remove Output import

* Consensus engine rework (#126)

* rework

* fix test

* clippy happy

---------

Co-authored-by: Giacomo Pasini <Zeegomo@users.noreply.github.com>

* Adapt carnot network adapter interfaces and implementations

* Fix errors

* Update network with engine types

* Fit types yet again

* Remove leadership and old overlay
Create carnot event builder
Added some adjustments

* Add view to vote

* Fix serde derive in consensus-engine

* Add serde feature for engine in core

* Use view in tally

* Move carnot tally to consensus service

* Add new view msg

* Fit engine types in adapter

* Missing serde feature in consensus service

* Implement carnot event builder

* Implement even builder run main tasks

* Fill up view resolver

* Fix errors on network adapter implementations

* Clippy happy

* Extract event handling to independent methods in View

* Fix test

* Refactor carnot event builder (#135)

* refactor

* format

* Discriminate proposal messages (#136)

* Derive block id from wire format (#139)

* Derive block id from wire format

* Derive id on block creation

* Use compile time hash size

* Add leader role (#138)

* add leadership stub

* fix gather_new_views

* fmt

* actually build qc

* remove redundant fields

* add flat overlay (#143)

* add flat overlay

* fix

* sort imports

* fix tests

* Fix waku update

* rewrite data collection add different kind of subscribers

* fix fmt

* Unhappy tally (#137)

* Refactor tally module

* Implement tally for new view messages

* Assess pr comments

* Fix rebase

* simplify tally

---------

Co-authored-by: Giacomo Pasini <g.pasini98@gmail.com>

* fix gather_new_views

* working node

* fix unhappy path

* remove leftover

* Kickstart event building in sim app

* finish event builder

* fix comment

* add Tally

* gather enough new views then construct ProposalBlock event

* Revert "gather enough new views then construct ProposalBlock event"

This reverts commit 87da2bdd0c.

* WIP: CarnotNode

* WIP

* finish event handle

* dump state

* WIP

* finish message sending

* fix some compile errors

* make project compile

* update

* fix fmt and clippy

* optimize json ser/deser and add a config

* update Cargo.toml

* Implement leader proposing (#154)

* Implement leader proposing

* fix fmt

---------

Co-authored-by: al8n <scygliu1@gmail.com>

* fix ser/deser bugs

* fix subscriber bugs

* Fix proposing genesis

* Fix genesis retrieval in consensus-engine

* Bring back general block proposal event

* Fix leaf voting

* fix init node bugs

* add more tracing

* fix empty qc

* fix data race

* fix all panics

* cleanup

* propose new blocks

* fix comment

* do not approve for the same block

* no panics

* fix some comments

* use serde_with

* Bring back genesis on 0

* Fix genesis retrieval
Replace output enum
Vote for genesis proposal

* Genesis methods

* fix StardardQc::genesis()

* fix genesis block bug

* fix PR comment

* fix PR comment

* fix PR comment

* fix PR comment

* fix PR comment

* Fix tally
Fix proposing

* Remove public block building
Added raw method

* Missing fmt

* clippy happy

* fix io stream downcast

* optional stream-type arg, by default we do not run any subscriber

* fmt

* cleanup

* Remove from header block constructor

* Fix duplicated approve (#180)

* fix duplicated approve

* Success tally just on threshold

* Integrate random beacon on happy path

* Fix missing updating beacon

* Replicate consensus output

* Prune older non relevant messages from cache

* Remove view info just again

* Refactor Block deps

* Reverse wrong parent committee call in Consensus engine

* Remove useless event builder settings

* Remove blocks store from event builder

* Remove unnecessary carnot seed

* Remove duplicated proposals check

---------

Co-authored-by: Giacomo Pasini <g.pasini98@gmail.com>
Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com>
Co-authored-by: Giacomo Pasini <Zeegomo@users.noreply.github.com>
2023-06-14 16:52:37 +02:00
Al Liu d864fecd07
chore: replace std locks to parking_lot locks in simulations (#141)
* replace std locks to parking_lot locks
2023-05-31 12:57:42 +08:00
Al Liu 4b880778ab
finish subscriber manager (#140)
* finish subscriber manager
2023-05-11 16:09:53 +08:00