Commit Graph

50 Commits

Author SHA1 Message Date
Al Liu c16b794517
fix large msg sending logic (#274)
* fix large msg sending logic
2023-08-03 20:05:43 +08:00
Daniel Sanchez fac42cd31d
Prune blocks in consensus engine (#275)
* Add prune older blocks call

* Prune older blocks in simulation app

* Use latest safe view for failure case

* Pick parent view from high qc

* Add safety assertion on pruning method
2023-08-01 16:47:47 +02:00
Daniel Sanchez ae59be5466
Refactor parent committee methods to return optional parent instead of empty one (#282) 2023-08-01 09:19:29 +02:00
gusto 976b1f9577
Add node step time to report (#271)
* Skip step if previous node exeeded step time

* Fix clippy

* Passively keep track of step execution time in node
2023-07-26 10:21:05 +03:00
gusto ef0b0701f3
Simulations nodeid report (#272)
* Add node id to the node state report

* Update config with record settings example
2023-07-25 17:17:26 +03:00
gusto df97ea2543
Simulatios network throughput (#270)
* Use step time from configuration

* PayloadSize trait for inmemory network interface

* Per node network capacity

* Initial carnot message sizes

* Divide provided kbps by step time

* Use std::mem::size_of for msg sizes
2023-07-25 16:03:05 +03:00
gusto 7d64915dd7
Get current view directly from the engine (#269) 2023-07-21 12:30:52 +03:00
gusto 8dd34f81b4
Simulation tally fix (#268)
* Do not remove entry cache when the threshold is reached

* Leader super majority change

* Update leader threshold test
2023-07-21 12:30:32 +03:00
Al Liu 6dee12704d
Change simulation `Node` trait `current_view` method return `View` (#259) 2023-07-20 17:31:55 +08:00
Daniel Sanchez 0866dfc8af
Process single events (#267) 2023-07-20 10:08:09 +02:00
Daniel Sanchez adf935830e
Do not remove entry cache when the threshold is reached (#266)
Co-authored-by: Gusto <bacvinka@gmail.com>
2023-07-19 18:04:08 +02:00
gusto 9f71dbb24c
Simulation network broadcast fix (#262)
* Replace network broadcast msg type to a dedicated channel

* Update tests with broadcast chan

* Replace threadrng with seedable smallrng

* Simplify the broadcast loop
2023-07-18 15:01:29 +03:00
gusto 94e384f609
Tree settings for sim overlay (#257)
* Tree settings for sim overlay

* WIP dyn node simulation

* Allow generic settings and states for dispatched nodes

* Update tests
2023-07-18 14:44:23 +03:00
Al Liu 9467351c10
Finish `View` wrapper (#254)
* finish View wrapper
2023-07-12 21:30:22 +08:00
Al Liu 2135676606
Finish `NodeId` type wrapper (#252)
* add NodeId wrapper
2023-07-11 23:16:49 +08:00
Al Liu a0cb738b9f
Finish `Committee` wrapper (#251)
* Committee type wrapper

---------

Co-authored-by: danielsanchezq <sanchez.quiros.daniel@gmail.com>
2023-07-07 16:08:19 +08:00
Giacomo Pasini 90cf29bf86
Better tests (#232)
* add basic integration tests

* add a way to configure overlay threshold

* Save logs to file in case of failure

* Increase number of test nodes to 10

* fix tests

* use fraction instead of tuple

* fmt
2023-07-04 11:39:28 +02:00
Youngjoon Lee 98aa138b87
feat: enforce TimeoutQc to be constructed only by new() (#229) 2023-06-28 23:37:27 +09:00
Al Liu 224a3a53f5
fix tally_by condition (#234) 2023-06-28 17:56:16 +08:00
Al Liu 4a80690c1f
Add a cli arg to support redirect log (#230)
* add cli arg to support redirect log
2023-06-28 17:10:19 +08:00
gusto 55648e3151
Simulation network broadcast (#225)
* Sim network broadcast method

* Remove list of all nodes in carnot node instance

* Use network broadcast method within nodes

* Add helper method for network payload
2023-06-27 14:20:59 +03:00
Al Liu 18bbf63a3f
update simulation app configuration (#217) 2023-06-22 19:50:34 +08:00
Al Liu 4ccc19f5a3
Support configurable records (#200)
* support configurable records
2023-06-21 16:31:36 +08:00
Daniel Sanchez bed0b9448d
Simulation unhappy path (#193)
* Use elapsed time

* Added timeout

* Extract tally

* Missing elapsed time

* Fix new view leader behaviour

* Fix tests

* Fix timeout double check

* Fix logs

* TimeoutHandler nitpicks

* Clippy happy

* Fix timeout sub

* Modify discard messages comment
2023-06-19 15:27:14 +02:00
gusto faacd10172
Remove dummy node from simapp settings and app itself (#196) 2023-06-17 13:38:59 +03:00
Al Liu 40048fa47b
Fix #183: support structured log for sim app. (#184) 2023-06-16 19:26:48 +08:00
Al Liu 9afd6c007c
Simulation conf polish (#181)
* polish Duration ser/deser and support step_time in conf file
2023-06-15 17:42:42 +08:00
Al Liu b6789b994e
Fix #182: gracefully shutdown (#187)
* finish signal and cleanup
2023-06-15 17:40:29 +08:00
Al Liu e57cb7b3cf
remove RwLock from SimmulationRunnerInner (#189) 2023-06-15 11:16:28 +02:00
Al Liu 27d9f72035
Simulation happy path (#161)
* finish subscriber manager

* optimize subscribe on SimulationRunnerHandle

* fix comment

* replace std locks to parking_lot locks

* move producer initialization out simulate fn

* WIP

* optimize run fn

* update condition

* fix CI

* collect run times

* Add happy-path consensus engine

* tmp

* Fit types from spec (#124)

* Match types to spec

* Remove Output import

* Consensus engine rework (#126)

* rework

* fix test

* clippy happy

---------

Co-authored-by: Giacomo Pasini <Zeegomo@users.noreply.github.com>

* Adapt carnot network adapter interfaces and implementations

* Fix errors

* Update network with engine types

* Fit types yet again

* Remove leadership and old overlay
Create carnot event builder
Added some adjustments

* Add view to vote

* Fix serde derive in consensus-engine

* Add serde feature for engine in core

* Use view in tally

* Move carnot tally to consensus service

* Add new view msg

* Fit engine types in adapter

* Missing serde feature in consensus service

* Implement carnot event builder

* Implement even builder run main tasks

* Fill up view resolver

* Fix errors on network adapter implementations

* Clippy happy

* Extract event handling to independent methods in View

* Fix test

* Refactor carnot event builder (#135)

* refactor

* format

* Discriminate proposal messages (#136)

* Derive block id from wire format (#139)

* Derive block id from wire format

* Derive id on block creation

* Use compile time hash size

* Add leader role (#138)

* add leadership stub

* fix gather_new_views

* fmt

* actually build qc

* remove redundant fields

* add flat overlay (#143)

* add flat overlay

* fix

* sort imports

* fix tests

* Fix waku update

* rewrite data collection add different kind of subscribers

* fix fmt

* Unhappy tally (#137)

* Refactor tally module

* Implement tally for new view messages

* Assess pr comments

* Fix rebase

* simplify tally

---------

Co-authored-by: Giacomo Pasini <g.pasini98@gmail.com>

* fix gather_new_views

* working node

* fix unhappy path

* remove leftover

* Kickstart event building in sim app

* finish event builder

* fix comment

* add Tally

* gather enough new views then construct ProposalBlock event

* Revert "gather enough new views then construct ProposalBlock event"

This reverts commit 87da2bdd0c.

* WIP: CarnotNode

* WIP

* finish event handle

* dump state

* WIP

* finish message sending

* fix some compile errors

* make project compile

* update

* fix fmt and clippy

* optimize json ser/deser and add a config

* update Cargo.toml

* Implement leader proposing (#154)

* Implement leader proposing

* fix fmt

---------

Co-authored-by: al8n <scygliu1@gmail.com>

* fix ser/deser bugs

* fix subscriber bugs

* Fix proposing genesis

* Fix genesis retrieval in consensus-engine

* Bring back general block proposal event

* Fix leaf voting

* fix init node bugs

* add more tracing

* fix empty qc

* fix data race

* fix all panics

* cleanup

* propose new blocks

* fix comment

* do not approve for the same block

* no panics

* fix some comments

* use serde_with

* Bring back genesis on 0

* Fix genesis retrieval
Replace output enum
Vote for genesis proposal

* Genesis methods

* fix StardardQc::genesis()

* fix genesis block bug

* fix PR comment

* fix PR comment

* fix PR comment

* fix PR comment

* fix PR comment

* Fix tally
Fix proposing

* Remove public block building
Added raw method

* Missing fmt

* clippy happy

* fix io stream downcast

* optional stream-type arg, by default we do not run any subscriber

* fmt

* cleanup

* Remove from header block constructor

* Fix duplicated approve (#180)

* fix duplicated approve

* Success tally just on threshold

* Integrate random beacon on happy path

* Fix missing updating beacon

* Replicate consensus output

* Prune older non relevant messages from cache

* Remove view info just again

* Refactor Block deps

* Reverse wrong parent committee call in Consensus engine

* Remove useless event builder settings

* Remove blocks store from event builder

* Remove unnecessary carnot seed

* Remove duplicated proposals check

---------

Co-authored-by: Giacomo Pasini <g.pasini98@gmail.com>
Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com>
Co-authored-by: Giacomo Pasini <Zeegomo@users.noreply.github.com>
2023-06-14 16:52:37 +02:00
Al Liu d864fecd07
chore: replace std locks to parking_lot locks in simulations (#141)
* replace std locks to parking_lot locks
2023-05-31 12:57:42 +08:00
Al Liu 4b880778ab
finish subscriber manager (#140)
* finish subscriber manager
2023-05-11 16:09:53 +08:00
gusto 0bf6e6d272
Simulation initialization (#122)
* Use enums for different settings types

* Enum for overlay settings

* Configurable simulation overlay

* Use duration type for network behaviour delays

* Configurable simulation nodes

* Runner for different node types

* Seedable rng

* Convert settings to required objects

* Implement IOStreamSettings deserialization

* Use common run method for different node types

* Configuration for simapp

* Testcase for region distribution

* Use unix time if seed is not provided
2023-05-08 13:06:39 +03:00
Al Liu ea7896f06c
Simulation streaming and gracefully shutdown (#119)
* add stream supports

* add test case

* add polars stream, and force to use stream for the runner

* using arcswap instead RefCell for producers

* finish gracefully shutdown

* - add IOProducer and IOSubscriber
- fix deadlock in sync runner
- fix testcases
2023-04-25 19:14:30 +08:00
Al Liu c2d1ee1291
add rustfmt.toml (#118) 2023-04-19 15:59:51 +08:00
gusto 2206d7a291
Use par_iter in network module (#116)
* Use par_iter in network module

* Pass rng from filter function

* Remove locking for network messages
2023-04-13 17:24:27 +03:00
gusto a87418c908
Test voting steps with various node roles (#114)
* Single step send receive runner tests

* Rename network to overlay state

* Store every overlay that might happen

* Get related nodes for from node id

* Add leader role

* WIP dummy handle events

* Leader and root roles

* Add intent as helper to dummy vote

* Check if has enough votes

* Proposal to vote to new view tests

* Update sync runner tests

* Smallrng in send receive tests

* Send initial vote next view leaders in tests

* Local view struct for node

* Large node number tests

* Fix internal to internal case
2023-04-10 13:18:20 +03:00
Al Liu a19278480d
Pipe output data (#111) 2023-04-04 20:26:57 +08:00
Al Liu d07da7ba5a
add missing ward (#112) 2023-04-03 17:17:14 +08:00
gusto 901ebf4152
Message sending between simulation nodes (#110)
* Dummy node for simulations

* Shared network state for nodes

* Runner one step test

* Beginning of network interface

* Connect dummy node to network

* Network step tests

* Pop messages that are being sent

* Regions send receive tests

* Setup network in sync runner tests

* Dispatch and collect node messages during sim step

* Improve network interface receiver
2023-03-31 12:48:06 +03:00
Al Liu c882a58286
Add StalledViewWard (#109) 2023-03-30 16:35:29 +08:00
Al Liu 61356f1116
Add minmax warding (#108)
* add minmax warding
2023-03-28 00:20:01 +08:00
Al Liu f4b94c8267
Simple warding fix (#107)
* fix negated analyze
2023-03-27 19:16:48 +08:00
Al Liu ae16e8583b
impl ward condition for max view ward (#106)
* impl ward condition for max view ward
2023-03-27 18:45:50 +08:00
Daniel Sanchez 92ef9e5a77
Simulation app runners (#105)
* Integrate new runners, fit types...make it compile

* Added missing runners

* Fix rebased changes

* Make tests pass

* Clippy happy

* More clippy happy
2023-03-24 13:21:10 -07:00
Al Liu 4a90ba6926
chore: types wrapper (#104)
* chore: types wrapper
2023-03-24 19:50:30 +08:00
Al Liu ea589d018a
Simulation App: Make step cost configurable (#102)
* WIP: make step cost configurable

* fix fmt
2023-03-23 16:57:18 +08:00
gusto 2d8c9f1099
Tree overlay for simulation app (#101)
* Add simulations crate

* Firs working runner

* Cleanupt

* Extract overlay from runner

* Fix off id calls

* Fix testing values

* WIP: Tree overlay for simulation app

* Build tree overlay from depth and size

* Tree overlay generation tests

* WIP: Committee role in layout

* Use commtittee role when constructing node

* Pairprogramming role calculation

* WIP: Extract step runner to node

* Aggregate times from different overlay layers

* Runner for overlay with layers

* Use StepRng in runner tests

* Decouple role from node

* Define leader steps and behaviour

* Get possible node ids from the overlay

* Handle unknown state in flat overlay

* Use solver type for deserialization

* Remove refcell from nodes hashmap

* Create layout from the provided node ids

* Break runner loops into seperate functions

* Use for loop to collect node step times

* Revert leader times collection

* Add interegion latency tests

---------

Co-authored-by: danielsanchezq <sanchez.quiros.daniel@gmail.com>
2023-03-23 01:32:54 -07:00
Al Liu 8a1af8c234
Add configuration for simulation app (#94)
* make simulation app compile to wasm

* add configuration

* improve args parsing

* add steps in configuration

---------

Co-authored-by: Daniel Sanchez Quiros <sanchez.quiros.daniel@gmail.com>
2023-03-20 17:13:55 +08:00
Daniel Sanchez 6ce9fdf553
[WIP] Simulation app barebones (#90)
* Add simulations crate

* Kickstart modules

* Add overlay

* Use node steps

* Add initial carnot node

* Implement Carnot node costs simulation

* Firs working runner

* Cleanup

* Extract overlay from runner

* Fix off id calls

* Fix testing values

* Clippy happy

* make simulation app compile to wasm

---------

Co-authored-by: al8n <scygliu1@gmail.com>
2023-03-14 07:15:45 -07:00