1
0
mirror of synced 2025-01-09 15:26:11 +00:00

443 Commits

Author SHA1 Message Date
Roman Zajic
8142feaa8c
chore: Da dispersal unit tests update (#720)
* test: prepare to call stream disperse

* fix: formatting

* fix: remove test_stream_disperse_error_cases

* test: utils for cli

* test: test_dispersal_with_swarms
- runtime preparation

* fix: cleanup
- tracing_subscriber

* fix: rewrite without using sampling

* fix: outstanding space

* fix: use executor.run

* fix: formatting

* test: add replies handling

* fix: formatting

* fix: switch to logging system

* fix: executor should wait for correct event
- cleanup

* Update nomos-cli/src/da/network/swarm.rs

Fix ignore value syntax.

Co-authored-by: gusto <bacv@users.noreply.github.com>

---------

Co-authored-by: gusto <bacv@users.noreply.github.com>
2024-09-24 20:37:56 +08:00
gusto
9f4f139771
DA: Integration tests for sampling and dispersal (#733)
* Use common test node in da tests

* Pick ranged subnetids when sampling

* Da network settings for integration tests

* Predefined node keys and deterministic rng

* Disperse kzgrs encoded blob

* Cli swarm fixes

* Increase indexer integration test timeout

* Check dispersal responses in cli da adapter

* DA membership configuration in node tests

* Nomos Cli act as a node in tests

* Increase timeout for dispersal tests

* Different node configurations for da tests

* Collect unique ids for sampling
2024-09-18 16:22:36 +03:00
Roman Zajic
fadf6d010b
chore: Improve coverage workflow (#738)
* fix: reduce space requirements
- run tests to get coverage

* fix: schedule on 3am UTC
- use binstall for grcov
2024-09-18 08:43:35 +08:00
Youngjoon Lee
8a1e705792
DA: Fix compliation errors in KZG backend bench (#737) 2024-09-17 18:32:11 +09:00
Youngjoon Lee
d2db454765
Add missing error handling: LedgerError::ParentNotFound (#630) 2024-09-13 15:07:47 +02:00
Giacomo Pasini
b7865b27ec
fix coverage workflow (#736)
* fix coverage workflow

* fix action

* do not run tests

* build binaries

* clean before tests

* only compile tests

* clean dir

* remove intermediate artficats
2024-09-13 15:05:50 +02:00
Giacomo Pasini
f2dfd468ac
Integrate PoL into consensus (#721)
* Add PoL crate

* promote cl crate to nomos-node repo

* Integrate PoL into consensus

* remove metal feature

* install risc0 on ci

* more risc0 install

* fix da tests

* pass gh token

* fix tests and run in release mode

* clippy allow

* do not run in release mode

* drop windows in CI

* execute heavy tests nightly or accelerated in macos

* fmt

* configurable slot time

* use xlarge workers

* fix workflows

* remove unstable test

* use xlarge runners

* separate integrations tests

* fix features

* add token to ci

* downgrade runners

* fix integration tests

* revert testing changes and use risc0 dev mode

* fix coverage risc0 install

* increase slot time

* use cargo binstall

* clean before coverage
2024-09-11 01:59:09 +02:00
holisticode
49ac81b950
Implemented get_blob on sampling rocksdb adapter (#732)
* Implemented get_blob on sampling rocksdb adapter

* addressed PR comments
2024-09-06 13:31:51 -05:00
gusto
9835d09b8c
Column index in the sample requests (#731) 2024-09-06 18:39:50 +03:00
gusto
f8a73529cf
DA: Incoming sampling request handling (#730)
* WIP: Sample message to behaviour massage translation

* Sampling request handling

* Sampling storage in node

* Sampling rocksdb scaffold
2024-09-06 16:47:38 +03:00
holisticode
b31e867f7f
Trigger sampling from DA mempool (#722)
* first iteration, does not compile due to incompatible types

* Trigger sampling in node (#725)

* Sampling service in node

* Sampling in da integration tests

---------

Co-authored-by: gusto <bacv@users.noreply.github.com>
2024-09-04 12:49:29 -05:00
gusto
93509bc09f
DA network in verifier libp2p adapter (#715)
* Use da network in verifier libp2p adapter

* VerifierAdapter in da tests

* Simplify verifier adapter

* Start da_network in da tests
2024-09-03 19:21:30 +03:00
Daniel Sanchez
a13f8611ee
DA: Consensus sampling (#708)
* Add sampling relay to consensus and massage all generics

* Pipe in sampling filtering of blob info

* Add mark in block

* Pipe validate block

* Refactor mark_in_block -> mark_complete

* Fix generics on tests

* Fix generics on tests

* Fix rebase

* Cargo fmt after rebase

* Sampling service configuration

* Sampling service config in indexer integration tests

---------

Co-authored-by: Gusto <bacvinka@gmail.com>
2024-09-03 15:02:49 +00:00
Giacomo Pasini
6f6bb61df4
Add PoL (#718)
* Add PoL crate

* promote cl crate to nomos-node repo

* add github action for risc0 proof

* fix actions scrupt

* add metal feature

* fix risc0 install

* remove check test
2024-09-03 14:38:00 +02:00
Roman Zajic
c4c5eba642
chore: Da kzgrs-backend unit tests update (#700)
* test: verify column error cases

* test: verify chunks error cases

* test: simplify vec modification

* test: check column_id getter

* fix: remove check column_id getter
- related function will be removed
2024-09-03 18:56:20 +08:00
holisticode
efff80de67
Sampling backend implementation (#709)
* implemented backend; correct start of monitoring thread needs to be finished

* Fix kzgrs backend (#710)

* abstract rng

* fix rng abstraction

* replaced subnets vector with HashSet, fixed bugs, added tests

* addressed PR comments

* fix clippy warnings

* Rename TrackingState -> SamplingState

* Short circuit failure on init error

---------

Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com>
Co-authored-by: danielSanchezQ <3danimanimal@gmail.com>
2024-09-03 08:25:12 +00:00
gusto
7dc2111341
DA: Verifier and Indexer in Node (#713)
* Verifier and Indexer attached to node

* Serialization for kzgrs verifier settings
2024-08-30 20:16:21 +03:00
gusto
4ad9ebec46
DA: Nomos Cli dispersal tests (#711)
* Dissemination app in tests module

* Expose remaining executor settings

* Configure peer ids and addresses for cli tests

* Add BlobInfo to mempool via api

* Expose columns number args for nomos cli

* Spawn all nodes in nodes config
2024-08-30 13:02:02 +03:00
holisticode
0cb039d806
DA: Sampling service (#705)
* initial

* first iteration, lots of lifetime and trait issues, does not compile

* Daniel/sampling service (#706)

* Implement sampling service

* Implement libp2p adapter listen_to_sampling_messages

* temporary empty backend mod; implement start_sampling

* addressed PR comments; moved random sampling subnet list generation to backend (todo impl)

* addressed PR nitpicks

* remove empty mod which was failing CI due to cargo fmt

* removed unused code

---------

Co-authored-by: holisticode <holistic.computing@gmail.com>
Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com>
2024-08-27 13:42:52 -05:00
gusto
d5ceceff9e
DA Network in node (#704)
* DA Network in node

* Use ed25519 curve for peerIds in libp2p
2024-08-27 16:47:14 +03:00
Daniel Sanchez
c611319622
Da: Network addresses (#703)
* Crate address book

* Add addressbook on necesary places
Incorporate to sampling test

* fmt happy
2024-08-27 10:15:45 +00:00
Daniel Sanchez
2f6d265aa6
Implement filler with original side replication (#702) 2024-08-26 10:20:47 +02:00
gusto
37c22d23ce
DA: Network service executor (#701)
* Nomos cli executor swarm

* Executor swarm and backend in nomos-cli

* Libp2p executor adapter in nomos cli

* Update nomos-cli/src/da/network/backend.rs

Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com>

* Use FillFromNodeList with executor backend

* Integrate Executor backend into the cli

* Use subnetwork id where appropriate

* Common clone_deserialize_error

* Use thiserror

* Minimal dialing in cli swarm

* Send dispersal messages async

* Remove unused connection id

---------

Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com>
2024-08-26 11:00:34 +03:00
Daniel Sanchez
fcb903c14a
Implement filler v1 (#698) 2024-08-23 08:56:10 +00:00
Daniel Sanchez
5f5b394017
Put swarm to listen at the specified address (#699) 2024-08-23 07:27:39 +00:00
Daniel Sanchez
66de571bfa
Da: network service backend (#695)
* Sketch up types and backend skeleton

* Pipe up sampling events

* Pipe up validation blobs events

* Added docs

* Cargo fmt

* Clippy happy

* Debug log events in validator swarm run method
2024-08-22 10:14:42 +00:00
gusto
3671376691
DA: Store and load multiple columns for the same blob in a single node (#694)
* ColumnIndex type in Blob trait

* Store blob id and column after verification

* Use blob_id+column_idx storage scheme in indexer

* Store and load multiple columns for the same blob

* Add comments
2024-08-21 16:23:36 +03:00
gusto
2ca822e6ec
ColumnIndex in DaBlob (#693)
* ColumnIndex in DaBlob

* Use u16 for column index
2024-08-20 13:15:16 +03:00
Daniel Sanchez
2922e9ce74
Da: validator network behaviour & swarm (#691)
* Build up validator behaviour and swarm

* Refactor modules

* More barebones structure on validator swarm

* Refactor protocols into submodule

* Pipe up events stream into swarm impl

* Clippy happy on tests
2024-08-19 16:17:10 +00:00
gusto
c7ba1f56f6
DA: Dispersal in Cli App with mock backend (#690)
* WIP: cli dispersal reenable

* WIP: Nomos da network mock in cli app

* Mock adapter with KzgEncoder in cli

* Send BlobInfo to the mempool

* KzgrsSettings in dispersal cli app

* Dispersal backend mock for executor

* build_attestation_message to build_blob_id refactor

* Pass blob metadata via the cli args
2024-08-19 15:44:55 +03:00
Daniel Sanchez
d2c178f33d
Da: Network sampling (#689)
* Fix reading messages state machine

* Improve waker on behaviour

* Rename test

* First sketch

* Implement replication

* Refactor protocols names

* Clippy happy

* expose behaviour

* Return dummy on outbound for validator behaviour

* Implement behaviour, lacks streams handling

* Cleanup

* Implement stream handling

* Add some documentation and comments

* Clippy fixes after rebase

* Move all neighbours to test_utils module

* Executor only outgoing streams

* First assault at testing behaviours

* Add debug instead of traces

* Added more logs

* Bullshitting tests

* Fix outstream handling, offers hook to send new peer streams

* Fix blob dispersal handling

* Refactor test, use new api

* Imports cleanup

* Working test

* Add docs

* Non overlapping test port

* Added sampling protocol

* Sketch up sampling behaviour types

* Implement behaviour

* Added test

* Mid protocol, not working

* Reimplement using streams (just outgoing, incoming missing)

* Implemented with single streams

* Fix unreachable code reachable

* Fix block test

* Clippy happy

* Better docs
2024-08-16 11:12:34 +00:00
Daniel Sanchez
5f0707c276
Da: Network dispersal (#688)
* Fix reading messages state machine

* Improve waker on behaviour

* Rename test

* First sketch

* Implement replication

* Refactor protocols names

* Clippy happy

* expose behaviour

* Return dummy on outbound for validator behaviour

* Implement behaviour, lacks streams handling

* Cleanup

* Implement stream handling

* Add some documentation and comments

* Clippy fixes after rebase

* Move all neighbours to test_utils module

* Executor only outgoing streams

* First assault at testing behaviours

* Add debug instead of traces

* Added more logs

* Bullshitting tests

* Fix outstream handling, offers hook to send new peer streams

* Fix blob dispersal handling

* Refactor test, use new api

* Imports cleanup

* Working test

* Add docs

* Non overlapping test port
2024-08-12 09:12:08 +02:00
Roman Zajic
4150c6514d
chore: Da kzgrs unit tests update (#676)
* test: blst error conversions

* test: compute roots of unity

* test: variant with Toeplitz1Cache value

* test: verify_column error cases

* test: add DaEncoderParams::new case

* fix: reduce code duplicity for test_verify_column

* test: build_certificate error cases

* test: Certificate related methods

* test: VidCertificate related methods
- Index

* fix: revert changes for kzgrs-backend
2024-08-05 17:57:33 +08:00
gusto
9ad4ee7a3c
DA: DispersedBlobInfo trait in nomos-core and kzgrs-backend (#686)
* DispersedBlobData in nomos-core

* Implement BlobInfo in kzgrs-backend

* Fullreplication BlobInfo

* Use block info in da mempool

* Blob info in consensus

* Blob info in da-verifier

* BlobInfo in da indexer

* BlobInfo in nomos-api

* Verifier and Indexer integration tests with BlobInfo

* Blob info in nomos node

* Import DaEncoder trait
2024-08-02 20:01:18 +03:00
Daniel Sanchez
e380bf4117
Da: network replication fix connection (#685)
* Fix attempt 1 - nonworking

* Added missing open stream request

* Expand outbound lifecycle to openstream state

* Fix reading messages state machine

* Use tracing

* Remove unnecessary loop

* Use stream style for test

* Send 10 messages

* Use quic in test

* Sends and receives

* cargo update fixed the panic

* Refactor test

* reschedule polls

* Clippy cleanup

* Remove leftover features

* Cargo fmt

* Improve waker on behaviour

* Rename test
2024-08-02 16:21:38 +00:00
gusto
2148216bda
CI: Rust 1.80 (#684)
* Allow Nonce to be unused (Clippy)

* Rust 1.80 stable
2024-07-31 15:40:32 +03:00
Daniel Sanchez
91d3558cb2
Create basic structure of mempool network adapter for da sampling (#683)
* Create basic structure of mempool network adapter for da sampling

* Return blob instead from sample method

* fmt
2024-07-26 08:49:24 +00:00
Daniel Sanchez
b9efe1f1ba
Da: network replication core (#680)
* Add core structure
Sketch first version of da network handler

* Added assgination module
Added basic assignation trait

* Added assgination module
Added basic assignation trait

* Refuse connection for non members

* Use proper protocol name

* Split in broadcast/dispersal/sampling modules
Move behaviour and handler to broadcast

* Remove unused broadcast behaviour attributes

* Add basic docs

* Remove unused pending errors on broadcast handler

* Clippy happy

* Refactor Broadcast -> Replication

* Pipe message sending

* Clippy happy

* Clippy fmt

* Refactor outgoing -> outgoing_events

* Pipe out replication

* Include subnetworks id in dispersal and replication messages

* Pipe replication messaging

* fmt

* Fix tests
2024-07-25 17:03:32 +00:00
gusto
9ff90e7d1c
DA: Network message types (#681)
* Split message definitions for broadcast, dispersal and sampling

* Expose dispersal messages via crate

* Create nomos-da-messages crate

* From trait implementations for messages

* Rename broadcast to replication
2024-07-25 12:27:55 +03:00
gusto
5c17221d16
Protoc in testnet and integration tests (#682) 2024-07-25 10:06:54 +03:00
gusto
4cd3d040f3
DA Improve indexer integration test (#679)
* Improve indexer test

* Use workspace gh api token

* Unpin rustls version for libp2p
2024-07-24 18:59:47 +03:00
gusto
fcda9a6da8
DA Prost integration (#678)
* Prost crate integration for DA

* Packing and unpacking for protobuf messages

* Minimal helper macro for dispersal message

* Use protoc in gh actions

* Move dispersal related protocol to nomos-da-dispersal
2024-07-23 18:24:21 +03:00
Daniel Sanchez
f58c296959
Da: verifier bench (#672)
* Use reference for DaBlob on verify

* Add verifier bench

* Fix missing reference refactor

* Clippy happy
2024-07-23 12:43:54 +00:00
Daniel Sanchez
fc815069a2
Just use openssl.dev, no binaries (#677) 2024-07-23 10:32:16 +00:00
Roman Zajic
50f3078e6a
chore: Da full replication unit tests update (#675)
* test: add metadata getter test for VidCertificate

* fix: uncomment assertion for SeqCst

* fix: formatting

* test: get Metadata for Certificate
- generate signature for Certificate

* test: generate hash for Attestation

* fix: add assertion for generate hash

* fix: wrap in test module
- add helper functions

* fix: undo wrap in test module

* test: Hash implementation for Certificate with default Hasher
2024-07-22 11:33:18 +02:00
Daniel Sanchez
8bbbfcfc5f
Core services tracing spans (#673)
* Add cryptarchia tracing span

* Add debug id main span

* Add verifier spawn

* Pin rustls version

---------

Co-authored-by: Gusto <bacvinka@gmail.com>
2024-07-12 10:50:53 +02:00
Daniel Sanchez
77e65c96e4
Added const sizes to encoder bench (#671) 2024-07-03 12:55:05 +02:00
gusto
c9c94c4e55
DA Http Endpoints (#670)
* Api endpoint to add blob and receive attestation

* Nomos DA Certificate serialization

* Kzgrs verification params provider

* Indexer http api using kzgrs

* Plug da http endpoints into nomos api
2024-07-03 12:37:23 +03:00
Daniel Sanchez
126b0b890b
Da: Missing parallel features (#669)
* Added missing ark parallel features

* Remove unnecessary copy

* Toeplitz3 in place

* Fix blob size in encoder bench
2024-07-01 10:54:48 +02:00
05bf69823a
fix missing build settings for libp2p nodes (#668)
Otherwise we get:
```
ERROR: for libp2p-node-1  pull access denied for nomos, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
ERROR: for libp2p-node-2  pull access denied for nomos, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
ERROR: for libp2p-node-3  pull access denied for nomos, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
ERROR: for chatbot  pull access denied for nomos, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
ERROR: pull access denied for nomos, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
```
Required for:
* https://github.com/status-im/infra-misc/pull/294

Signed-off-by: Jakub Sokołowski <jakub@status.im>
2024-06-28 11:13:48 +03:00