Commit Graph

515 Commits

Author SHA1 Message Date
Daniel Sanchez c983eb2260
Add Executor client (#811)
* Refactor paths into common modules

* Kickoff http client

* Add blob to send blob

* Refactor publish method. Will reuse older endpoint in further prs

* Use paths in tests

* Fix more tests
2024-10-08 15:43:58 +00:00
gusto 07c9096924
Integrate dispersal service into executor node (#812) 2024-10-08 17:03:38 +03:00
gusto 22dfb51eba
Executor: Dispersal service mempool adapter (#810)
* Pass metadata when dispersing

* Sketch of executor dispersal mempool adapter
2024-10-08 11:03:34 +03:00
Daniel Sanchez a3cea9a09c
DA: Implement data reconstruction from original columns (#808)
* Implement data reconstruction from original columns

* Make reconstruction public
2024-10-08 07:15:41 +00:00
Alexis c03dec4b3d
ci: build on self-hosted runner (#799)
Signed-off-by: Alexis Pentori <alexis@status.im>
2024-10-07 15:16:02 +03:00
Daniel Sanchez 98e9cde66d
DA: Executor dispersal service (#806)
* Added basic dispersal structure

* Expand impl

* Pull events from network service

* Implement network adapter
Tie types together in service

* Fill up service processing

* Tie up types in kzgrs backend implementation

* Implement disperse

* Pipe encode and dispersal

* Create mempool adapter

* Add mempool adapter

* Tiny comment
2024-10-04 13:06:42 +00:00
gusto d17afcbe4d
Replication behavior membership test (#805) 2024-10-03 20:37:13 +03:00
gusto d7fe88f30b
Remove graylog and related containers (#804) 2024-10-03 14:36:36 +03:00
Caio 93de3afaf9
chose: fk20 benches clippy happy (#800) 2024-10-03 04:43:46 +00:00
Deep Mondal 563bf6f948
DA: kzg main testing parallelize proofs checks (#803) 2024-10-03 04:37:53 +00:00
gusto 66fc046091
Testnet: Kzgrs config (#797)
* Remove out of place wait_online test param

* Cfgsync crate for testnet startup

* Config dispersal to node hosts

* Remove etcd dep and obsolete bash scripts

* Kzgrs test params in testnet

* Nodes listen on all interfaces
2024-10-02 14:26:39 +03:00
gusto b01c4dd8c7
DA: Executor http api (#801)
* Node api handlers and backend modules

* Executor axum api backend

* Expose config functions from node

* Descriptive generics names in http api

* Nomos node metrics feature
2024-10-02 12:18:59 +03:00
Daniel Sanchez 9ecd738d6e
DA: Executor network service (#795)
* Refactor common things in backends

* Further extract to common

* Extract dial_peers

* Pipe out executor events

* Add wrapper settings

* Dial up dispersal peers

* Clippy happy

* dial_peers -> dial_validator_subnetwork_peers

* Add members list to membership trait

* Implement peers selection and dialing up from specification

* Fix tests

* Fix tiny doc
2024-10-01 12:44:13 +00:00
Roman Zajic 2672b7bde4
fix: Coverage and PR workflows - no space left on device (#798)
* test: remove LLVM profiling

* fix: add on push

* test: use release mode

* test: remove Android support from the runner

* test: return back LLVM profiling
- remove dotnet

* fix: remove on push condition

* fix: prune-vm action added

* fix: move actions-checkout

* fix: PR workflow

* fix: remove on push condition from codecov
2024-09-30 23:46:18 +08:00
gusto a30a809ea1
CI: Jenkins risczero setup (#770)
* Remove dep

* Use cargo as jenkins user

* Ensure that toolchain directory is setup before installation

* Debian 12 and gcc deps

* Env variables for risc zero dev mode

* Update testnet and base dockerfiles

* Env vars comments
2024-09-26 18:00:47 +03:00
Daniel Sanchez d83380cf02
Add default template (#769) 2024-09-26 12:41:26 +02:00
Daniel Sanchez 02b6969967
DA: Executor behaviour + swarm (#768)
* Add executor behaviour

* Implement executor behaviour

* Implement executor swarm

* Fix comment

* Clippy happy

* Bubbled up utility methods on swarms
2024-09-26 09:58:49 +00:00
Daniel Sanchez a19a8e8112
Specification related PRs template (#734)
* Added template for specification related PRs

* Added new template
2024-09-26 09:57:53 +00:00
gusto 25270de680
CI: Nighly tests update and docker build gh workflow (#767)
* Github workflow for docker images

* Fix use sections in gh workflow

* Update nightly integration tests
2024-09-25 20:18:56 +03:00
Daniel Sanchez f689372146
Add executor binary crate (#766)
* Create executor binary reusing validator lib

* Use executor in main

* Use storage from config
2024-09-25 09:34:45 +00:00
gusto 915fe3f098
DA: Replication integration (#764)
* Handle replication messages

* Multiple indexes for kzgrs verifier

* Add rocksdb storage to node config

* Configurable delay before sending blob info to mempool

* Dial peers in same subnet for replication

* Update da integration tests

* Update nomos node tests

* Load kzgrs config from file

* SLOW_ENV set true for macos CI

* Assert retrieved blobs in dissemination test

* Kzgrs params deserialize unchecked
2024-09-25 11:27:39 +03:00
Youngjoon Lee ca0eb824aa
Mix: NetworkBehaviour (#765)
* Mix: NetworkBehaviour

* use Waker.wake()

* make clippy happy by defining Config instead of Behaviour::Default
2024-09-24 22:39:59 +09:00
Roman Zajic 8142feaa8c
chore: Da dispersal unit tests update (#720)
* test: prepare to call stream disperse

* fix: formatting

* fix: remove test_stream_disperse_error_cases

* test: utils for cli

* test: test_dispersal_with_swarms
- runtime preparation

* fix: cleanup
- tracing_subscriber

* fix: rewrite without using sampling

* fix: outstanding space

* fix: use executor.run

* fix: formatting

* test: add replies handling

* fix: formatting

* fix: switch to logging system

* fix: executor should wait for correct event
- cleanup

* Update nomos-cli/src/da/network/swarm.rs

Fix ignore value syntax.

Co-authored-by: gusto <bacv@users.noreply.github.com>

---------

Co-authored-by: gusto <bacv@users.noreply.github.com>
2024-09-24 20:37:56 +08:00
gusto 9f4f139771
DA: Integration tests for sampling and dispersal (#733)
* Use common test node in da tests

* Pick ranged subnetids when sampling

* Da network settings for integration tests

* Predefined node keys and deterministic rng

* Disperse kzgrs encoded blob

* Cli swarm fixes

* Increase indexer integration test timeout

* Check dispersal responses in cli da adapter

* DA membership configuration in node tests

* Nomos Cli act as a node in tests

* Increase timeout for dispersal tests

* Different node configurations for da tests

* Collect unique ids for sampling
2024-09-18 16:22:36 +03:00
Roman Zajic fadf6d010b
chore: Improve coverage workflow (#738)
* fix: reduce space requirements
- run tests to get coverage

* fix: schedule on 3am UTC
- use binstall for grcov
2024-09-18 08:43:35 +08:00
Youngjoon Lee 8a1e705792
DA: Fix compliation errors in KZG backend bench (#737) 2024-09-17 18:32:11 +09:00
Youngjoon Lee d2db454765
Add missing error handling: `LedgerError::ParentNotFound` (#630) 2024-09-13 15:07:47 +02:00
Giacomo Pasini b7865b27ec
fix coverage workflow (#736)
* fix coverage workflow

* fix action

* do not run tests

* build binaries

* clean before tests

* only compile tests

* clean dir

* remove intermediate artficats
2024-09-13 15:05:50 +02:00
Giacomo Pasini f2dfd468ac
Integrate PoL into consensus (#721)
* Add PoL crate

* promote cl crate to nomos-node repo

* Integrate PoL into consensus

* remove metal feature

* install risc0 on ci

* more risc0 install

* fix da tests

* pass gh token

* fix tests and run in release mode

* clippy allow

* do not run in release mode

* drop windows in CI

* execute heavy tests nightly or accelerated in macos

* fmt

* configurable slot time

* use xlarge workers

* fix workflows

* remove unstable test

* use xlarge runners

* separate integrations tests

* fix features

* add token to ci

* downgrade runners

* fix integration tests

* revert testing changes and use risc0 dev mode

* fix coverage risc0 install

* increase slot time

* use cargo binstall

* clean before coverage
2024-09-11 01:59:09 +02:00
holisticode 49ac81b950
Implemented get_blob on sampling rocksdb adapter (#732)
* Implemented get_blob on sampling rocksdb adapter

* addressed PR comments
2024-09-06 13:31:51 -05:00
gusto 9835d09b8c
Column index in the sample requests (#731) 2024-09-06 18:39:50 +03:00
gusto f8a73529cf
DA: Incoming sampling request handling (#730)
* WIP: Sample message to behaviour massage translation

* Sampling request handling

* Sampling storage in node

* Sampling rocksdb scaffold
2024-09-06 16:47:38 +03:00
holisticode b31e867f7f
Trigger sampling from DA mempool (#722)
* first iteration, does not compile due to incompatible types

* Trigger sampling in node (#725)

* Sampling service in node

* Sampling in da integration tests

---------

Co-authored-by: gusto <bacv@users.noreply.github.com>
2024-09-04 12:49:29 -05:00
gusto 93509bc09f
DA network in verifier libp2p adapter (#715)
* Use da network in verifier libp2p adapter

* VerifierAdapter in da tests

* Simplify verifier adapter

* Start da_network in da tests
2024-09-03 19:21:30 +03:00
Daniel Sanchez a13f8611ee
DA: Consensus sampling (#708)
* Add sampling relay to consensus and massage all generics

* Pipe in sampling filtering of blob info

* Add mark in block

* Pipe validate block

* Refactor mark_in_block -> mark_complete

* Fix generics on tests

* Fix generics on tests

* Fix rebase

* Cargo fmt after rebase

* Sampling service configuration

* Sampling service config in indexer integration tests

---------

Co-authored-by: Gusto <bacvinka@gmail.com>
2024-09-03 15:02:49 +00:00
Giacomo Pasini 6f6bb61df4
Add PoL (#718)
* Add PoL crate

* promote cl crate to nomos-node repo

* add github action for risc0 proof

* fix actions scrupt

* add metal feature

* fix risc0 install

* remove check test
2024-09-03 14:38:00 +02:00
Roman Zajic c4c5eba642
chore: Da kzgrs-backend unit tests update (#700)
* test: verify column error cases

* test: verify chunks error cases

* test: simplify vec modification

* test: check column_id getter

* fix: remove check column_id getter
- related function will be removed
2024-09-03 18:56:20 +08:00
holisticode efff80de67
Sampling backend implementation (#709)
* implemented backend; correct start of monitoring thread needs to be finished

* Fix kzgrs backend (#710)

* abstract rng

* fix rng abstraction

* replaced subnets vector with HashSet, fixed bugs, added tests

* addressed PR comments

* fix clippy warnings

* Rename TrackingState -> SamplingState

* Short circuit failure on init error

---------

Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com>
Co-authored-by: danielSanchezQ <3danimanimal@gmail.com>
2024-09-03 08:25:12 +00:00
gusto 7dc2111341
DA: Verifier and Indexer in Node (#713)
* Verifier and Indexer attached to node

* Serialization for kzgrs verifier settings
2024-08-30 20:16:21 +03:00
gusto 4ad9ebec46
DA: Nomos Cli dispersal tests (#711)
* Dissemination app in tests module

* Expose remaining executor settings

* Configure peer ids and addresses for cli tests

* Add BlobInfo to mempool via api

* Expose columns number args for nomos cli

* Spawn all nodes in nodes config
2024-08-30 13:02:02 +03:00
holisticode 0cb039d806
DA: Sampling service (#705)
* initial

* first iteration, lots of lifetime and trait issues, does not compile

* Daniel/sampling service (#706)

* Implement sampling service

* Implement libp2p adapter listen_to_sampling_messages

* temporary empty backend mod; implement start_sampling

* addressed PR comments; moved random sampling subnet list generation to backend (todo impl)

* addressed PR nitpicks

* remove empty mod which was failing CI due to cargo fmt

* removed unused code

---------

Co-authored-by: holisticode <holistic.computing@gmail.com>
Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com>
2024-08-27 13:42:52 -05:00
gusto d5ceceff9e
DA Network in node (#704)
* DA Network in node

* Use ed25519 curve for peerIds in libp2p
2024-08-27 16:47:14 +03:00
Daniel Sanchez c611319622
Da: Network addresses (#703)
* Crate address book

* Add addressbook on necesary places
Incorporate to sampling test

* fmt happy
2024-08-27 10:15:45 +00:00
Daniel Sanchez 2f6d265aa6
Implement filler with original side replication (#702) 2024-08-26 10:20:47 +02:00
gusto 37c22d23ce
DA: Network service executor (#701)
* Nomos cli executor swarm

* Executor swarm and backend in nomos-cli

* Libp2p executor adapter in nomos cli

* Update nomos-cli/src/da/network/backend.rs

Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com>

* Use FillFromNodeList with executor backend

* Integrate Executor backend into the cli

* Use subnetwork id where appropriate

* Common clone_deserialize_error

* Use thiserror

* Minimal dialing in cli swarm

* Send dispersal messages async

* Remove unused connection id

---------

Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com>
2024-08-26 11:00:34 +03:00
Daniel Sanchez fcb903c14a
Implement filler v1 (#698) 2024-08-23 08:56:10 +00:00
Daniel Sanchez 5f5b394017
Put swarm to listen at the specified address (#699) 2024-08-23 07:27:39 +00:00
Daniel Sanchez 66de571bfa
Da: network service backend (#695)
* Sketch up types and backend skeleton

* Pipe up sampling events

* Pipe up validation blobs events

* Added docs

* Cargo fmt

* Clippy happy

* Debug log events in validator swarm run method
2024-08-22 10:14:42 +00:00
gusto 3671376691
DA: Store and load multiple columns for the same blob in a single node (#694)
* ColumnIndex type in Blob trait

* Store blob id and column after verification

* Use blob_id+column_idx storage scheme in indexer

* Store and load multiple columns for the same blob

* Add comments
2024-08-21 16:23:36 +03:00
gusto 2ca822e6ec
ColumnIndex in DaBlob (#693)
* ColumnIndex in DaBlob

* Use u16 for column index
2024-08-20 13:15:16 +03:00