* Refactor paths into common modules
* Kickoff http client
* Add blob to send blob
* Refactor publish method. Will reuse older endpoint in further prs
* Use paths in tests
* Fix more tests
* Added basic dispersal structure
* Expand impl
* Pull events from network service
* Implement network adapter
Tie types together in service
* Fill up service processing
* Tie up types in kzgrs backend implementation
* Implement disperse
* Pipe encode and dispersal
* Create mempool adapter
* Add mempool adapter
* Tiny comment
* Remove out of place wait_online test param
* Cfgsync crate for testnet startup
* Config dispersal to node hosts
* Remove etcd dep and obsolete bash scripts
* Kzgrs test params in testnet
* Nodes listen on all interfaces
* Add PoL crate
* promote cl crate to nomos-node repo
* Integrate PoL into consensus
* remove metal feature
* install risc0 on ci
* more risc0 install
* fix da tests
* pass gh token
* fix tests and run in release mode
* clippy allow
* do not run in release mode
* drop windows in CI
* execute heavy tests nightly or accelerated in macos
* fmt
* configurable slot time
* use xlarge workers
* fix workflows
* remove unstable test
* use xlarge runners
* separate integrations tests
* fix features
* add token to ci
* downgrade runners
* fix integration tests
* revert testing changes and use risc0 dev mode
* fix coverage risc0 install
* increase slot time
* use cargo binstall
* clean before coverage
* initial
* first iteration, lots of lifetime and trait issues, does not compile
* Daniel/sampling service (#706)
* Implement sampling service
* Implement libp2p adapter listen_to_sampling_messages
* temporary empty backend mod; implement start_sampling
* addressed PR comments; moved random sampling subnet list generation to backend (todo impl)
* addressed PR nitpicks
* remove empty mod which was failing CI due to cargo fmt
* removed unused code
---------
Co-authored-by: holisticode <holistic.computing@gmail.com>
Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com>
* WIP: cli dispersal reenable
* WIP: Nomos da network mock in cli app
* Mock adapter with KzgEncoder in cli
* Send BlobInfo to the mempool
* KzgrsSettings in dispersal cli app
* Dispersal backend mock for executor
* build_attestation_message to build_blob_id refactor
* Pass blob metadata via the cli args
* Split message definitions for broadcast, dispersal and sampling
* Expose dispersal messages via crate
* Create nomos-da-messages crate
* From trait implementations for messages
* Rename broadcast to replication
* Prost crate integration for DA
* Packing and unpacking for protobuf messages
* Minimal helper macro for dispersal message
* Use protoc in gh actions
* Move dispersal related protocol to nomos-da-dispersal
* Base cleaning of da to new traits/structure
Added new da protocols and types
* DA: KZG+RS core (#632)
* Removed old kzg rs modules
* Added new kzgrs core module
* Implemented bytes_to_polynomial and tests
* Use coefficient form
* Refactor evaluations into method
* Use domain elements instead of roots of unity in tests
* Fix encoding and test
* Clippy happy
* Add comments
* Implement polynomial commitment
* Implement proof generation
* Sketch fn signature for verification
* implement proof verification
* Implemented verification and tests
* Return evaluations from bytes_to_polynomial as well
* Use modular le bytes
* Implement rs encode/decode
* Implement decoding tests
* Implement decode using lagrange
* Cleanup imports
* Da: v1 encoder (#633)
* Added new kzgrs core module
* Implemented bytes_to_polynomial and tests
* Use coefficient form
* Refactor evaluations into method
* Use domain elements instead of roots of unity in tests
* Fix encoding and test
* Clippy happy
* Add comments
* Implement polynomial commitment
* Implement proof generation
* Sketch fn signature for verification
* implement proof verification
* Implemented verification and tests
* Return evaluations from bytes_to_polynomial as well
* Use modular le bytes
* Implement rs encode/decode
* Implement decoding tests
* Implement decode using lagrange
* Added chunksmatrix
* Implement encoder with chunkify
* Added missing files
* Implement commit row commitments
* Implement compute elements (row) proofs
* Fix lib and types exposures
* Implement encoder main methods
* Implement encode method
* Implement chunkify test
Fix related bugs
* Implement compute row kzg commitments
Fix related bugs
* Implement rs encode rows test
Fix related bugs
Refactored API
* Implement row proofs tests
Fix fieldelement encoding/decoding bug
* Implement aggregated commitment test
Implement aggregated column proofs test
* Cleanup
* Fix deps
* Fix tests
* Fix chunk too big test
* Da: v1 verifier (#635)
* Fix encoding and test
* Implement commit row commitments
* Implemented dablob
* Implement verifier new
Implement verifier check column
* Clippy cleanup
* Implement verifier
* Implemented verify column test
* Implemented full verify test
* DA API Payload to Item in mempool (#634)
* Base cleaning of da to new traits/structure
Added new da protocols and types
* DA: KZG+RS core (#632)
* Removed old kzg rs modules
* Added new kzgrs core module
* Implemented bytes_to_polynomial and tests
* Use coefficient form
* Refactor evaluations into method
* Use domain elements instead of roots of unity in tests
* Fix encoding and test
* Clippy happy
* Add comments
* Implement polynomial commitment
* Implement proof generation
* Sketch fn signature for verification
* implement proof verification
* Implemented verification and tests
* Return evaluations from bytes_to_polynomial as well
* Use modular le bytes
* Implement rs encode/decode
* Implement decoding tests
* Implement decode using lagrange
* Cleanup imports
* Reduce abstraction for certificate and vid metadata
* Allow payload to mempool as long as it converts into item
* Da Certificate verifier
* Add mock certificate for core tests
* Mempool payload verification
* Integrate mock verifiers for tx and certs
* Detach verification from cert and tx
* Seperate payload and item in mempools
* Mempools in integration tests
* Remove old cert verifier
* Network payload to item constraints in da mempool
* Update nomos-da/full-replication/src/lib.rs
Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com>
* Sort attestations for cert signature
* Update nomos-da/full-replication/src/lib.rs
Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com>
---------
Co-authored-by: danielsanchezq <sanchez.quiros.daniel@gmail.com>
* DA API Certificate verification (#641)
* Redo certificate verification in mempool
* FullReplication verifier params provider
* Integrate da params provider into the node
* DA API Indexer service (#643)
* Base cleaning of da to new traits/structure
Added new da protocols and types
* Remove da availability crate completely
* Scaffold for da storage service
* Indexer service responsible for storage and blockchain subscription
* Handle index related ops only
* Acquire storage and consensus relays
* Indexer trait
* wip: storage adapter
* Use storage adapter instead of storage
* Add consensus adapter trait for block subscriptions
* Consensus block subscriber adapter
* Use metadata from da core in indexer
* Update nomos-services/data-availability/indexer/src/lib.rs
Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com>
* Update nomos-services/data-availability/indexer/src/lib.rs
Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com>
* Use std::ops::Range for da queries
* Return certificate metadata
* Da storage adapter methods
---------
Co-authored-by: danielsanchezq <sanchez.quiros.daniel@gmail.com>
* Reuse evaluations when computing proofs (#647)
* Reuse precomputed evaluations instead of evaluation polynomial for each proof
* kzgrs benchmarks
* Clippy happy
* DA API indexer implementation (#644)
* Rocksb adapter in da indexer
* Handle da service messages
* Remove indexer trait, use storage directly in the indexer service
* Return unordered indexes range
* Load blob by vid from file
* Use VID in consensus
* Change full replication index type to byte array
* Change payload to cert and item to vid where required
* Service integration tests for indexer
* Feature gate rocksdb backend
* Propagate range response send error
* FRIndex to Index
* VID to VidCertificate
* Pass blobs file dir via settings
* Da v1 multiple proofs bench (#648)
* Parallel proof generation bench
* Added bench comment
* Modify domain to fit exact sizes
* Fix domain in benches
* Force parallelization features in lib
* DA: Implement base structure for verifier service (#627)
* Base cleaning of da to new traits/structure
Added new da protocols and types
* Implement base structure for verifier service
* Added comments and todo!
* Cleanup imports
* Size of VidCert in full replication
* Nomos Da Verifier service crate
* Extension replaced with metadata
* Fix DaIndexer service name
* Storage adapter trait in verifier
* Manage lifecycle and messages in verifier
* Blob trait in core
* Common nomos da storage crate
* Use updated nomos da storage in indexer
* Verifier storage adapter
* Libp2p adaper for verifier
* Kzgrs backend in verifier service
* Fix fmt
* Clippy happy
---------
Co-authored-by: Gusto <bacvinka@gmail.com>
* DA Verifier service integration tests (#650)
* Base cleaning of da to new traits/structure
Added new da protocols and types
* Implement base structure for verifier service
* Added comments and todo!
* Cleanup imports
* Size of VidCert in full replication
* Nomos Da Verifier service crate
* Extension replaced with metadata
* Storage adapter trait in verifier
* Manage lifecycle and messages in verifier
* Common nomos da storage crate
* Use updated nomos da storage in indexer
* Verifier storage adapter
* Libp2p adaper for verifier
* Kzgrs backend in verifier service
* Fix fmt
* Data availability tests module
* Return attestation in service msg response
* Common definitions for da tests
* Serde for kzgrs proofs and commitments
* Da verifier integration test
* WIP nomos-core in kzgrs backend
* Kzgrs blob to common module
* Add client zone to verifier test and check if attestations are created
* Cleanup and comments
* Use libp2p only for verifier and indexer service tests
* Lint in da tests
* Simplify da blob serialization
* Remove attester from nomos-core attestation
* Verifier backend error
---------
Co-authored-by: danielsanchezq <sanchez.quiros.daniel@gmail.com>
* DA Kzgrs Backend Certificate implementation (#651)
* Kzgrs backend certificate definition
* Encoded data to certificate test
* Nomos da domain specific tag
* Handle errors in da certificate creation
* Add nomos core traits to da cert
* Derive ordering traits for da index
* Add failure test cases to kzgrs certificate
* Da v1 benches expand (#658)
* Update benches with more cases
* Expand benches
* Added parallel feature
* Fix test comment
Co-authored-by: Youngjoon Lee <5462944+youngjoon-lee@users.noreply.github.com>
* Remove outdated comment
---------
Co-authored-by: Gusto <bacvinka@gmail.com>
Co-authored-by: gusto <bacv@users.noreply.github.com>
Co-authored-by: Youngjoon Lee <5462944+youngjoon-lee@users.noreply.github.com>
* Switch from Carnot to Cryptarchia
* Do not delete old coins
* Update tests/src/lib.rs
Co-authored-by: davidrusu <davidrusu.me@gmail.com>
* clippy happy
---------
Co-authored-by: davidrusu <davidrusu.me@gmail.com>
* A wrapper crate for prometheus client
* Initial integration of metrics for mempool
* Merge mempool metrics imports
* Add cli flag to enable metrics
* Add nomos metrics service for serving metrics
* Use nomos prometheus metrics in the node
* Rename metrics to registry where applicable
* Expose metrics via http
* Featuregate the metrics service
* Style and fail on encode error
* Add metrics cargo feature for mempool
* humanize array ser/deser
* split fns
* use `const-hex`
* fix fmt
* create `nomos-utils` crate
* Human serde committeeid (#478)
* Human readable serde for CommitteeId
* Deserialize bytes to string if human readable
* Don't allocate if possible in human serde bytes
---------
Co-authored-by: gusto <bacv@users.noreply.github.com>
* Add nomos-cli command to disseminate data across the network
Add nomos-cli to hold various utilities for the node.
To start, this commit adds a command to disseminate some
data through the network and build a certificate of correct
dispersal for inclusion in a block
* fmt
* reorder loop instructions
* Send blobs concurrently
* Add `mixnode` and `mixnet-client` crate (#302)
* Add `mixnode` binary (#317)
* Integrate mixnet with libp2p network backend (#318)
* Fix#312: proper delays (#321)
* proper delays
* add missing duration param
* tiny fix: compilation error caused by `rand` 0.8 -> 0.7
* use `get_available_port()` for mixnet integration tests (#333)
* add missing comments
* Overwatch mixnet node (#339)
* Add mixnet service and overwatch app
* remove #[tokio::main]
---------
Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>
* fix tests for the overwatch mixnode (#342)
* fix panic when corner case happen in RandomDelayIter (#335)
* Use `log` service for `mixnode` bin (#341)
* Use `wire` for MixnetMessage in libp2p (#347)
* Prevent tmixnet tests from running forever (#363)
* Use random delay when sending msgs to mixnet (#362)
* fix a minor compilation error caused by the latest master
* Fix run output fd (#343)
* add a connection pool
* Exp backoff (#332)
* move mixnet listening into separate task
* add exponential retry for insufficient peers in libp2p
* fix logging
* Fix MutexGuard across await (#373)
* Fix MutexGuard across await
Holding a MutexGuard across an await point is not a good idea.
Removing that solves the issues we had with the mixnet test
* Make mixnode handle bodies coming from the same source concurrently (#372)
---------
Co-authored-by: Youngjoon Lee <taxihighway@gmail.com>
* Move wait at network startup (#338)
We now wait after the call to 'subscribe' to give the network
the time to register peers in the mesh before starting to
publish messages
* Remove unused functions from mixnet connpool (#374)
* Mixnet benchmark (#375)
* merge fixes
* add `connection_pool_size` field to `config.yaml`
* Simplify mixnet topology (#393)
* Simplify bytes and duration range ser/de (#394)
* optimize bytes serde and duration serde
---------
Co-authored-by: Al Liu <scygliu1@gmail.com>
Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com>
Co-authored-by: Giacomo Pasini <Zeegomo@users.noreply.github.com>
Add an initial simple but functional implementation for a data
availability protocol.
Full replication simply encodes bytes in a single blob which is
replicated in all nodes.
---------
Co-authored-by: Giacomo Pasini <Zeegomo@users.noreply.github.com>
---------
Co-authored-by: Al Liu <scygliu1@gmail.com>
Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com>
* Cargo http-service folder
* WIP: http server
* Revert comments in network service
* Router service and axum implementation structure
* Move bin contents to examples dir
* Add http service and server traits
* HttpMsg definition
* WIP: axum backend
* fix example in Cargo.toml
* Shared axum router and router modification methods
* Http example with axum and metrics service
* make project compile and add Error associated type
* Axum backend shared router fixes
* Dummy service implementation for http example
* remove unused clone on mutex
* Cargo http-service folder
* WIP: http server
* Revert comments in network service
* Router service and axum implementation structure
* Move bin contents to examples dir
* Add http service and server traits
* HttpMsg definition
* WIP: axum backend
* fix example in Cargo.toml
* Shared axum router and router modification methods
* Http example with axum and metrics service
* make project compile and add Error associated type
* Axum backend shared router fixes
* Dummy service implementation for http example
* remove unused clone on mutex
* Fix typos and remove unused code
* Fix failing tests when feature flags are not set
* Use bytes as a type for payload and response in http service
* Refactored http crate layout into differential services files
* First stab at router service
* Fully piped http bridge system
* Start building bridge helper function
* Refactor bridge builder helper and update example
* impl serialization for metrics data
* Get updated copy of router when processing request
* remove unused code
* fix typo
* [POC]: Http service: support add graphql handler (#47)
* WIP: add graphql endpoint
* support add graphql handler
* remove generic
* fix clippy warnings
* Add post put and patch handlers that expect bytes as body
* Graphql example file
* WIP: Use http post method for graphql related queries
* Parse graphql requests in handler
* Simplify handlers for post and other data methods
* Revert "Simplify handlers for post and other data methods"
This reverts commit 96f2b1821e.
* add tracing and remove comments
* Pass response bytes without any modifications
* Use receive_batch_json for gql request parsing
* Readme for running examples
* fix conflicts
* add a general helper function for graphql
* remove unused function
* cleanup code
* move schema initialization to handle function
* adapt metrics to http service
* fix clippy warnings
* remove unused fn
* fix clippy
* optimize example
Co-authored-by: gusto <bacvinka@gmail.com>
Co-authored-by: Gusto Bacvinka <augustinas.bacvinka@gmail.com>
* Fix cargo build without features
* Simplify handlers for routes with data
Co-authored-by: al8n <scygliu1@gmail.com>
Co-authored-by: Daniel Sanchez Quiros <sanchez.quiros.daniel@gmail.com>
Giacomo Pasini
Youngjoon Lee
Daniel Sanchez
gusto
holisticode
Al Liu
Giacomo Pasini
Youngjoon Lee