Commit Graph

388 Commits

Author SHA1 Message Date
Daniel Sanchez 96a8ef6032
Da: fk20 encoder (#664)
* Use fft for rs

* Specify evaluation domains

* Fix domains in verifier, decoder and dispersal

* Fix domains in verifier, decoder and dispersal

* Fix size in verifier to fit tests
2024-06-25 13:16:25 +02:00
Daniel Sanchez 9adfbdddc6
Da: fk20 implementation (#663)
* Implement fft_g1

* Fix ifft_g1
Added fft/ifft g1 test

* Parallelize i/fft

* Add i/fft benches

* Fix parallel feature tagging

* Fix bench

* Fix parallelization

* Implement fk20

* Add compute roots of unity method

* Fix ranges

* Remove innecesary collect in fft_g1

* Use domaın and ark fft

* Remove custom fft

* Add fk20 benches

* Clippy happy

* Use threadcount for parallelization

---------

Co-authored-by: mgonen <mehmet@status.im>
2024-06-21 12:43:17 +02:00
gusto b4b24e4678
Rust 1.79 stable (#661)
* Rust 1.79 clippy fix

* Rust 1.79 stable in ci and docker

* Allow unused_var in nomos tests

Clippy struggle to recognize the usage of enumeration variable inside one particular testcase.
2024-06-18 10:53:56 +03:00
gusto 572d286f5f
Missing generic param to the mempool metrics (#662) 2024-06-17 16:30:07 +03:00
gusto 1dbc0c236e
Graylog service for the Testnet (#659)
* Graylog service in docker compose

* Retry graylog connection

* Graylog container configuration

* Add tokio sleep instead of std
2024-06-14 14:22:16 +03:00
Daniel Sanchez 7b9643f17e
DA Protocol V1 (#626)
* Base cleaning of da to new traits/structure
Added new da protocols and types

* DA: KZG+RS core (#632)

* Removed old kzg rs modules

* Added new kzgrs core module

* Implemented bytes_to_polynomial and tests

* Use coefficient form

* Refactor evaluations into method

* Use domain elements instead of roots of unity in tests

* Fix encoding and test

* Clippy happy

* Add comments

* Implement polynomial commitment

* Implement proof generation

* Sketch fn signature for verification

* implement proof verification

* Implemented verification and tests

* Return evaluations from bytes_to_polynomial as well

* Use modular le bytes

* Implement rs encode/decode

* Implement decoding tests

* Implement decode using lagrange

* Cleanup imports

* Da: v1 encoder (#633)

* Added new kzgrs core module

* Implemented bytes_to_polynomial and tests

* Use coefficient form

* Refactor evaluations into method

* Use domain elements instead of roots of unity in tests

* Fix encoding and test

* Clippy happy

* Add comments

* Implement polynomial commitment

* Implement proof generation

* Sketch fn signature for verification

* implement proof verification

* Implemented verification and tests

* Return evaluations from bytes_to_polynomial as well

* Use modular le bytes

* Implement rs encode/decode

* Implement decoding tests

* Implement decode using lagrange

* Added chunksmatrix

* Implement encoder with chunkify

* Added missing files

* Implement commit row commitments

* Implement compute elements (row) proofs

* Fix lib and types exposures

* Implement encoder main methods

* Implement encode method

* Implement chunkify test
Fix related bugs

* Implement compute row kzg commitments
Fix related bugs

* Implement rs encode rows test
Fix related bugs
Refactored API

* Implement row proofs tests
Fix fieldelement encoding/decoding bug

* Implement aggregated commitment test
Implement aggregated column proofs test

* Cleanup

* Fix deps

* Fix tests

* Fix chunk too big test

* Da: v1 verifier (#635)

* Fix encoding and test

* Implement commit row commitments

* Implemented dablob

* Implement verifier new
Implement verifier check column

* Clippy cleanup

* Implement verifier

* Implemented verify column test

* Implemented full verify test

* DA API Payload to Item in mempool  (#634)

* Base cleaning of da to new traits/structure
Added new da protocols and types

* DA: KZG+RS core (#632)

* Removed old kzg rs modules

* Added new kzgrs core module

* Implemented bytes_to_polynomial and tests

* Use coefficient form

* Refactor evaluations into method

* Use domain elements instead of roots of unity in tests

* Fix encoding and test

* Clippy happy

* Add comments

* Implement polynomial commitment

* Implement proof generation

* Sketch fn signature for verification

* implement proof verification

* Implemented verification and tests

* Return evaluations from bytes_to_polynomial as well

* Use modular le bytes

* Implement rs encode/decode

* Implement decoding tests

* Implement decode using lagrange

* Cleanup imports

* Reduce abstraction for certificate and vid metadata

* Allow payload to mempool as long as it converts into item

* Da Certificate verifier

* Add mock certificate for core tests

* Mempool payload verification

* Integrate mock verifiers for tx and certs

* Detach verification from cert and tx

* Seperate payload and item in mempools

* Mempools in integration tests

* Remove old cert verifier

* Network payload to item constraints in da mempool

* Update nomos-da/full-replication/src/lib.rs

Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com>

* Sort attestations for cert signature

* Update nomos-da/full-replication/src/lib.rs

Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com>

---------

Co-authored-by: danielsanchezq <sanchez.quiros.daniel@gmail.com>

* DA API Certificate verification  (#641)

* Redo certificate verification in mempool

* FullReplication verifier params provider

* Integrate da params provider into the node

* DA API Indexer service (#643)

* Base cleaning of da to new traits/structure
Added new da protocols and types

* Remove da availability crate completely

* Scaffold for da storage service

* Indexer service responsible for storage and blockchain subscription

* Handle index related ops only

* Acquire storage and consensus relays

* Indexer trait

* wip: storage adapter

* Use storage adapter instead of storage

* Add consensus adapter trait for block subscriptions

* Consensus block subscriber adapter

* Use metadata from da core in indexer

* Update nomos-services/data-availability/indexer/src/lib.rs

Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com>

* Update nomos-services/data-availability/indexer/src/lib.rs

Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com>

* Use std::ops::Range for da queries

* Return certificate metadata

* Da storage adapter methods

---------

Co-authored-by: danielsanchezq <sanchez.quiros.daniel@gmail.com>

* Reuse evaluations when computing proofs (#647)

* Reuse precomputed evaluations instead of evaluation polynomial for each proof

* kzgrs benchmarks

* Clippy happy

* DA API indexer implementation (#644)

* Rocksb adapter in da indexer

* Handle da service messages

* Remove indexer trait, use storage directly in the indexer service

* Return unordered indexes range

* Load blob by vid from file

* Use VID in consensus

* Change full replication index type to byte array

* Change payload to cert and item to vid where required

* Service integration tests for indexer

* Feature gate rocksdb backend

* Propagate range response send error

* FRIndex to Index

* VID to VidCertificate

* Pass blobs file dir via settings

* Da v1 multiple proofs bench (#648)

* Parallel proof generation bench

* Added bench comment

* Modify domain to fit exact sizes

* Fix domain in benches

* Force parallelization features in lib

* DA: Implement base structure for verifier service (#627)

* Base cleaning of da to new traits/structure
Added new da protocols and types

* Implement base structure for verifier service

* Added comments and todo!

* Cleanup imports

* Size of VidCert in full replication

* Nomos Da Verifier service crate

* Extension replaced with metadata

* Fix DaIndexer service name

* Storage adapter trait in verifier

* Manage lifecycle and messages in verifier

* Blob trait in core

* Common nomos da storage crate

* Use updated nomos da storage in indexer

* Verifier storage adapter

* Libp2p adaper for verifier

* Kzgrs backend in verifier service

* Fix fmt

* Clippy happy

---------

Co-authored-by: Gusto <bacvinka@gmail.com>

* DA Verifier service integration tests (#650)

* Base cleaning of da to new traits/structure
Added new da protocols and types

* Implement base structure for verifier service

* Added comments and todo!

* Cleanup imports

* Size of VidCert in full replication

* Nomos Da Verifier service crate

* Extension replaced with metadata

* Storage adapter trait in verifier

* Manage lifecycle and messages in verifier

* Common nomos da storage crate

* Use updated nomos da storage in indexer

* Verifier storage adapter

* Libp2p adaper for verifier

* Kzgrs backend in verifier service

* Fix fmt

* Data availability tests module

* Return attestation in service msg response

* Common definitions for da tests

* Serde for kzgrs proofs and commitments

* Da verifier integration test

* WIP nomos-core in kzgrs backend

* Kzgrs blob to common module

* Add client zone to verifier test and check if attestations are created

* Cleanup and comments

* Use libp2p only for verifier and indexer service tests

* Lint in da tests

* Simplify da blob serialization

* Remove attester from nomos-core attestation

* Verifier backend error

---------

Co-authored-by: danielsanchezq <sanchez.quiros.daniel@gmail.com>

* DA Kzgrs Backend Certificate implementation (#651)

* Kzgrs backend certificate definition

* Encoded data to certificate test

* Nomos da domain specific tag

* Handle errors in da certificate creation

* Add nomos core traits to da cert

* Derive ordering traits for da index

* Add failure test cases to kzgrs certificate

* Da v1 benches expand (#658)

* Update benches with more cases

* Expand benches

* Added parallel feature

* Fix test comment

Co-authored-by: Youngjoon Lee <5462944+youngjoon-lee@users.noreply.github.com>

* Remove outdated comment

---------

Co-authored-by: Gusto <bacvinka@gmail.com>
Co-authored-by: gusto <bacv@users.noreply.github.com>
Co-authored-by: Youngjoon Lee <5462944+youngjoon-lee@users.noreply.github.com>
2024-06-13 11:23:11 +03:00
gusto 7d6ae1b823
Testnet consensus configuration (#656)
* Update metrics for split mempool

* Update required packages for node in docker

* Expose env and cli params for coin

* Set coin params via env in testnet

* Predefined genesis state

* Chain start time from env vars
2024-05-30 12:46:02 +03:00
Youngjoon Lee e904793b7e
Add mdBook skeleton (#640) 2024-04-24 23:18:51 +09:00
Youngjoon Lee cddd215025
update config.yaml for cryptarchia and mixnet (#639) 2024-04-24 22:38:44 +09:00
Youngjoon Lee f98c1c0864
remove duplicate log (#638) 2024-04-24 20:25:55 +09:00
Daniel Sanchez e085b6bef4
Split mempool into tx and da mempool (#636)
* Split mempool into tx and da mempool

* Fix tests

* Differentiate between da and cl mempool msgs in consensus

* Common mempool msg type

---------

Co-authored-by: Gusto <bacvinka@gmail.com>
2024-04-18 15:07:28 +02:00
Giacomo Pasini 7e4d00cc78
Switch from Carnot to Cryptarchia (#624)
* Switch from Carnot to Cryptarchia

* Do not delete old coins

* Update tests/src/lib.rs

Co-authored-by: davidrusu <davidrusu.me@gmail.com>

* clippy happy

---------

Co-authored-by: davidrusu <davidrusu.me@gmail.com>
2024-03-28 12:56:04 +01:00
Al Liu 16444cf32a
Fix race for nomos log service (#577) 2024-03-23 22:02:01 +08:00
gusto b341a11eb2
Metrics to nomos services (#623)
* Remove unused metrics crate from services

* Move prometheus metrics to nomos-services
2024-03-22 14:03:35 +02:00
gusto de6138fc27
Remove old http api implementation (#622) 2024-03-22 12:55:25 +02:00
Daniel Sanchez 2677199ed6
Cryptarchia block subscribe (#617)
* Pipe broadcast channel

* Add subscription to cryptarchia services message

* Clippy happy
2024-03-22 10:45:54 +01:00
Daniel Sanchez 36d441ec21
Use windows msvc (as rocksdb fails with gnu toolchain) (#621) 2024-03-22 10:06:37 +01:00
Al Liu 47487f0be7
Fix clippy (#620) 2024-03-21 16:46:16 +01:00
Youngjoon Lee 40cc0b15b4
Remove unnecessary attributes and add doc comments for Mixnet (#618) 2024-03-21 23:06:32 +09:00
Al Liu fd8ea50dd1
Use rocksdb as storage backend (#585)
* finish rocksdb storage service

* use rocksdb

* create missing cf

* Update Cargo.toml
2024-03-21 12:23:26 +01:00
Youngjoon Lee 502c791318
Implement Stream for MixClient (#616) 2024-03-20 20:13:00 +09:00
Youngjoon Lee d449114044
Replace mixnet backend with mixnet addon (#615) 2024-03-19 19:15:09 +09:00
Youngjoon Lee ed81577ab5
Refactor libp2p StreamSend command more generic (#614) 2024-03-19 10:03:00 +09:00
Giacomo Pasini a933c73245
add cryptarchia leadership (#613) 2024-03-18 16:04:16 +01:00
Giacomo Pasini 1b925d9a3a
Add cryptarchia consensus service (#612)
* add cryptarchia consensus service

* fmt

* clippy happy

* review comments
2024-03-15 15:53:51 +01:00
Youngjoon Lee dbda061f04
Added `mixnet` compilation feature (#610) 2024-03-14 09:53:41 +09:00
Giacomo Pasini 50cff241fe
Refactor block (#609)
* Refactor Block/Header definition

Refactor block/header definition so that it's now responsibility
of the nomos-core crate. This removes definitions in ledger/consensus
crates since there's no need at that level to have an understanding
of the block format.

The new header format supports both carnot and cryptarchia.
2024-03-13 18:46:10 +01:00
Youngjoon Lee e7d591b7bc
Mixnet v1 (#569)
* base

* Remove mixnet client from libp2p network backend (#572)

* Mixnet v1: Remove all mixnet legacies: mixnet crate, mixnode binary, tests, and docker (#573)

* Mixnet v1: Skeleton (#570)

* Use QUIC for libp2p (#580)

* Add Poisson interval function for Mixnet (#575)

* Mixnet network backend skeleton (#586)

* Libp2p stream read/write (#587)

* Emitting packets from mixclient using libp2p stream (#588)

* Handle outputs from mixnode using libp2p stream/gossipsub (#589)

* Refactor poisson (#590)

* Mix client Poisson emission (#591)

* Mix node packet handling (#592)

* Mix Packet / Fragment logic (#593)

* Move FisherYates to `nomos-utils` (#594)

* Mixnet topology (#595)

* Mix client/node unit tests (#596)

* change multiaddr from tcp to udp with quic-v1 (#607)

---------

Co-authored-by: Al Liu <scygliu1@gmail.com>
2024-03-12 15:47:35 +09:00
Youngjoon Lee 98bc66a67c
Expose 18080 for HTTP API in Testnet (#608) 2024-03-12 15:29:51 +09:00
Giacomo Pasini e51865fe33
move ledger into separate crate (#606)
* move ledger into separate crate

* clippy happy
2024-03-11 15:15:17 +01:00
Giacomo Pasini 06d225db20
Make block id generic in mempool (#605) 2024-03-11 11:07:58 +01:00
Giacomo Pasini 31c5f69121
Add orphan leader proofs import (#600)
* Add orphan leader proofs import

* add additional test case covering double import
2024-03-04 16:02:28 +01:00
gusto a036b8adc3
Bump rust (#602) 2024-03-01 13:28:05 +02:00
Giacomo Pasini 2730c2f579
Cryptarchia engine (#583)
* add cryptarchia engine

* address comments

* split into separate files

* clarify comment
2024-02-29 10:51:25 +01:00
Al Liu fde0d29860
RocksDB Storage service (#584)
* finish rocksdb storage service
2024-02-29 12:32:45 +08:00
Jakub Sokołowski bafc6dfb1f
chore(ci): reuse discord send function from library
Provides more info and requires less boilerplate.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
2024-02-14 19:21:39 +01:00
gusto 3f2905817a
Add testnet cli_config and volume (#574)
* Add cli_config and volume

* Grafana config file mode required on linux hosts

* Pin minideb version for openssl compatibility with the build image
2024-01-30 10:04:11 +02:00
gusto 4722b54493
Use chat app as testnet bot (#568) 2024-01-29 11:26:54 +02:00
gusto f33c2613fb
Nomos chat app non interactive (#567)
* Add option to send chat message non iteractively via nomos cli

* Use clap to check if data or file is set

* Require author if message flag set
2024-01-29 09:43:18 +02:00
Youngjoon Lee 42d6816b1b
Write mixnode logs to files in integration tests (#510) 2024-01-24 16:15:11 +09:00
gusto f741590315
Disseminate app file input (#565)
* Accept file option in dissemination app

* File dissemination tests
2024-01-23 18:17:05 +02:00
gusto aa06080baa
Docker compose metrics containers (#564)
* Prometheus container and configuration for testnet

* Add graphana related configuration and container

* Use metrics feature in testnet nodes

* Grafana configuration files and datasources
2024-01-23 18:16:33 +02:00
Giacomo Pasini 16c97c07ec
Add test for big blob dissemination (#563)
* Add tests for dissemination of big blob

* Warn if a requested blob was not returned
2024-01-19 19:08:13 +01:00
Giacomo Pasini aeaf13fc88
Limit in-flight requests in chat app (#562) 2024-01-19 14:17:10 +01:00
Corey d527050633
fix: added openssl to shell.nix (#560) 2024-01-17 09:51:43 -05:00
Al Liu 90e2210bce
bump rayon version to make ra work (#557) 2024-01-15 12:21:33 +08:00
gusto 46d53479a2
Prometheus metrics service (#522)
* A wrapper crate for prometheus client

* Initial integration of metrics for mempool

* Merge mempool metrics imports

* Add cli flag to enable metrics

* Add nomos metrics service for serving metrics

* Use nomos prometheus metrics in the node

* Rename metrics to registry where applicable

* Expose metrics via http

* Featuregate the metrics service

* Style and fail on encode error

* Add metrics cargo feature for mempool
2024-01-12 16:15:12 +02:00
Daniel Sanchez c3b5dc98e4
Rename consensus service to carnot consensus (#556) 2024-01-09 11:08:24 +01:00
Giacomo Pasini b7d1fd9256
Nomos chat fixes (#555)
* Make chat messages unique

Since DA will rightfully ignore duplicated messages, we need to add a nonce to make sure
every message is unique, so that a user can send the same message twice.
This nonce is randomly generated for simplicity.

* Clear in_flight status upon error on chat app
2024-01-08 15:40:19 +01:00
gusto 285300f365
Receive blocks blobs in parallel (#554)
* Receive blocks blobs in parallel

* Update nomos-cli config
2024-01-08 12:56:11 +02:00