Commit Graph

583 Commits

Author SHA1 Message Date
benbierens 983da3c24f
functional implementation 2024-03-12 19:13:06 +01:00
benbierens 7934d415cd
Wires up downloading and unzipping 2024-03-12 19:13:06 +01:00
benbierens c67c2e7cd7
Implements loading backend from cli files or previously downloaded local files 2024-03-12 19:13:06 +01:00
benbierens 83e1347038
Extracts backend initialization to backend-factory 2024-03-12 19:13:06 +01:00
benbierens 63ab4c5064
sets up three paths for backend initialization 2024-03-12 19:13:06 +01:00
benbierens 828484a646
Moves backend creation into start method 2024-03-12 19:13:05 +01:00
benbierens cfca6b4111
Introduces a start method to prover 2024-03-12 19:12:52 +01:00
markspanbroek 4d069599c9
Use real verifier (#737)
* use a real verifying contract address

* contracts: cleanup

* marketplacesuite: set correct circuit files, interval mining

* Proofs tests updates

Contains changes to get the proving tests working reliably.

* integration: use correct circom artifacts for creating proofs

* integration: cleanup

---------

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
2024-03-12 14:20:42 +00:00
markspanbroek b3e57a37e2
Wire up prover (#736)
* wire prover into node

* stricter case object checks

* return correct proof

* misc renames

* adding usefull traces

* fix nodes and tolerance to match expected params

* format challenges in logs

* add circom compat to solidity groth16 convertion

* update

* bump time to give nodes time to load with all circom artifacts

* misc

* misc

* use correct dataset geometry in erasure

* make errors more searchable

* use parens around `=? (await...)` calls

* styling

* styling

* use push raises

* fix to match constructor arguments

* merge master

* merge master

* integration: fix proof parameters for a test

Increased times due to ZK proof generation.
Increased storage requirement because we're now hosting
5 slots instead of 1.

* sales: calculate initial proof at start of period

reason: this ensures that the period (and therefore
the challenge) doesn't change while we're calculating
the proof

* integration: fix proof parameters for tests

Increased times due to waiting on next period.
Fixed data to be of right size.
Updated expected payout due to hosting 5 slots.

* sales: wait for stable proof challenge

When the block pointer is nearing the
wrap-around point, we wait another period
before calculating a proof.

* fix merge conflict

---------

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
2024-03-12 12:10:14 +00:00
markspanbroek 293c676f22
Prover CLI updates (#735)
* rework cli to accept circuit params

* check circom files extension

* adding new required cli changes

* don't use ufcs

* persistence is a command now

* use `nimOldCaseObjects` switch for nim confutils compat

* misc

* Update cli integration tests

* Fix: simulateProofFailures option is not for validator

* moving circom params under `prover` command

* update tests

* Use circuit assets from codex-contract-eth in tests

* Add "prover" cli command to tests

* use correct stores

* make `verifier` a cmd option

* update circuit artifacts path

* fix cli tests

* Update integration tests to use cli commands

Integration tests have been updated to use the new cli commands. The api for usage in the integration tests has also changed a bit.

Proofs tests have been updated to use 5 nodes and 8 blocks of data. The remaining integration tests also need to be updated.

* remove parsedCli from CodexConfig

Instead, parse the cli args on the fly when needed

* remove unneeded gcsafes

* graceful shutdowns

Where possible, do not raise assert, as other nodes in the test may already be running. Instead, raise exceptions, catch them in multinodes.nim, and attempt to do a teardown before failing the test.

`abortOnError` is set to true so that `fail()` will quit immediately, after teardown has been run.

* update testmarketplace to new api, with valid EC params

---------

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
2024-03-12 09:57:13 +00:00
markspanbroek 8589e63d34
Update contract (#734)
* Update codex-contracts-eth

* contracts: update G2Point definition

* integration: disable automatic advancing of time

reason: makes reasoning about timing in tests harder,
because the period is set to 60 seconds in the
marketplace configuration, but this code switches to
a new period every 500 milliseconds

* integration: fix parameters of marketplace payouts test

* integration: update test settings

* integration: fix typo

* integration: workaround for hardhat issue

Subscriptions expire after 5 minutes when using
websockets. Use http and polling instead.

* integration: remove origDatasetSizeInBlocks

* integration: fix proof parameters for test

* integration: do not log output by default

* integration: fix failure rate in test

* integration: fix warning

* integration: include clock in logs

* integration: allow for more periods

5 periods was cutting it close, if we get too much
pointer downtime, then the test would fail
2024-03-12 08:18:25 +00:00
markspanbroek e654e93c71
Fix sales agent (#733)
* clock: add 1 second leeway before acting on timeouts

* sales: do not raise in proving loop when slot is cancelled

Allow the onCancelled callback to handle cancellation, and
the onFailed callback to handle failed requests.

* sales: cleanup proving tests

* sales: fix sales agent tests

* sales: stop cancellation loop when request started, finished or failed

* sales: fix flaky test

* sales: fix another flaky test

* clock: add comment explaining the + 1 second

Co-Authored-By: benbierens <thatbenbierens@gmail.com>

---------

Co-authored-by: benbierens <thatbenbierens@gmail.com>
2024-03-12 06:41:03 +00:00
markspanbroek d1658d7b77
Fix on-chain clock (#732)
* market: use `pending` blocktag when querying onchain state

* clock: use wall clock in integration tests

reason: we'll need to wait for the next period in
integration tests, and we can't do that if the
time doesn't advance

* clock: remove unused field

* integration: use pending block time to get current time

* clock: fix on-chain clock for hardhat

Only use 'latest' block for updates
Only update the first time you see a block

* integration: do not start tests with a very outdated block

* integration: allow for longer expiry period
2024-03-11 16:57:20 +00:00
markspanbroek 1fc0510f2b
build: update nim-ethers to latest chronosv3 branch (#731)
Fixes validator, which uses transaction simulation to
check whether a proof can be marked as missing.

Fixes gas estimations, so they are done on the pending
block, instead of the latest block.
2024-03-11 15:25:41 +00:00
Giuliano Mega 74679e9e2e
abort Codex when startup sequence fails (#738) 2024-03-11 12:31:17 +00:00
Slava 6ee1162989
Update Docker files and building Codex docs (#722)
* Make Build Codex copy/paste friendly

* Add auto-export for env var from files

* Use latest Ubuntu LTS image and install rust via package manager
2024-03-01 12:23:05 +00:00
Ben Bierens 53fdc88def
Feature/eth syncing (#721)
* Delays starting of codex until eth provider is synced

* bumps nim-ethers to 0.7.2

* bumps nim-ethers to 0.7.3

* Adds increasing delay to waitForSync
2024-03-01 07:50:29 +00:00
Ben Bierens 5e7ce52fbe
Fix block retransmit (#651)
* Applies peer-scoped lock to peer task handler.

* Replace async lock with delete-first approach.

* Cleanup some logging

* Adds inFlight flag to WantListEntry

* Clears inflight flag when local retrieval fails.

* Adds test for setting of in-flight

* Adds test for clearing in-flight when lookup fails

* Review comments by Tomasz

---------

Co-authored-by: gmega <giuliano.mega@gmail.com>
2024-02-29 07:37:12 +00:00
Ben Bierens 1c66cb1e83
Adds instruction for rust and how-to for MSYS2 (#715)
* Adds instruction for rust and how-to for MSYS2

* comments by Dmitriy
2024-02-23 08:27:10 +00:00
Giuliano Mega 457567531f
Fixes active cancellation for pending want requests (#714)
* add block cancellation support + tests

* tie issueCancellations into resolveBlocks for proper exception tracking, address comments

* pull cancellation as separate primitive in BlockExcNetwork

* use allFutures, rename issueBlockCancellations -> cancelBlocks

* use trc instead of wrn to register send error

* do not log peer IDs
2024-02-22 14:54:45 +00:00
Slava 1eebaa4ce3
ci: use latest amd64 macos runner (#718)
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2024-02-21 22:10:37 +00:00
Slava c99085c396
Set environment variables from files in Docker (#717)
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2024-02-21 20:50:57 +00:00
Dmitriy Ryajov 9e13d2251a
Prover works with zkey (#711)
* rework backend to instantiate key at initialization

* add groth16 convertes for solidity

* prover taks num samples on construction

* add zkey file

* rework helpers

* rename types

* update tests

* reworked test helpers

* rename types

* rework test

* test all slots artifacts

* bump to latest version
2024-02-19 18:58:39 +00:00
Slava 6b86601597
ci: update actions to the latest major versions (#712)
* ci: update actions to the latest major versions

* ci: add OS to the integration tests logs name

* ci: add CPU and Nim version to the integration tests logs name

* ci: upload integrations tests logs only when they run
2024-02-19 18:55:49 +00:00
Dmitriy Ryajov 3e884430c1
Node use correct stores (#710)
* don't pass erasure

* use correct stores and construct erasure inside the node

* fix tests to match new constructor

* remove prover argument

* review commets

* revert failing on no-prover for now

* small cleanup

* comment out invalid proofs broken test
2024-02-19 10:12:10 -08:00
Eric d70ab59004
refactor: multinode integration test refactor (#662)
* refactor multi node test suite

Refactor the multinode test suite into the marketplace test suite.

- Arbitrary number of nodes can be started with each test: clients, providers, validators
- Hardhat can also be started locally with each test, usually for the purpose of saving and inspecting its log file.
- Log files for all nodes can be persisted on disk, with configuration at the test-level
- Log files, if persisted (as specified in the test), will be persisted to a CI artifact
- Node config is specified at the test-level instead of the suite-level
- Node/Hardhat process starting/stopping is now async, and runs much faster
- Per-node config includes:
  - simulating proof failures
  - logging to file
  - log level
  - log topics
  - storage quota
  - debug (print logs to stdout)
- Tests find next available ports when starting nodes, as closing ports on Windows can lag
- Hardhat is no longer required to be running prior to starting the integration tests (as long as Hardhat is configured to run in the tests).
  - If Hardhat is already running, a snapshot will be taken and reverted before and after each test, respectively.
  - If Hardhat is not already running and configured to run at the test-level, a Hardhat process will be spawned and torn down before and after each test, respectively.

* additional logging for debug purposes

* address PR feedback

- fix spelling
- revert change from catching ProviderError to SignerError -- this should be handled more consistently in the Market abstraction, and will be handled in another PR.
- remove method label from raiseAssert
- remove unused import

* Use API instead of command exec to test for free port

Use chronos `createStreamServer` API to test for free port by binding localhost address and port. Use `ServerFlags.ReuseAddr` to enable reuse of same IP/Port on multiple test runs.

* clean up

* remove upraises annotations from tests

* Update tests to work with updated erasure coding slot sizes

* update dataset size, nodes, tolerance to match valid ec params

Integration tests now have valid dataset sizes (blocks), tolerances, and number of nodes, to work with valid ec params. These values are validated when requested storage.

Print the rest api failure message (via doAssert) when a rest api call fails (eg the rest api may validate some ec params).

All integration tests pass when the async `clock.now` changes are reverted.

* dont use async clock for now

* fix workflow

* move integration logs uplod to reusable

---------

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2024-02-19 04:55:39 +00:00
Dmitriy Ryajov 0497114e44
Use correct blocks count in builder (#709) 2024-02-19 02:19:59 +00:00
Dmitriy Ryajov 3fce267352
Block and manifest as objects (#708)
* make block and manifest objects

* use unsafeAddr since block is an object type now

* make object var
2024-02-19 02:10:16 +00:00
Eric 61ec4275c8
Revert "remove call to `waitFor` from clock.now and make it async (#691)" (#707)
This reverts commit 591be9446a.
2024-02-16 22:12:16 +00:00
Slava 79af5c0e64
Fix Docker entrypoint NAT helper variables (#706)
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2024-02-15 13:57:50 +00:00
Dmitriy Ryajov 6fab252bf1
safer backend release (#701) 2024-02-10 23:17:11 +00:00
Slava 64e6a602fd
Install Rust during Docker builds (#703)
* Install Rust during Docker builds

Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>

* Stop CI matrix on error

---------

Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2024-02-10 17:48:40 +00:00
Dmitriy Ryajov e23159b065
wire in circom backend (#698)
* wire in circom backend

* should contain leafs

* adding circom compad and circuits deps

* update windows build

* fix windows build

* improve test names

* move proving defaults to codextypes

* remove unnedded inmports and move defaults to codextypes

* capture error code on backend failure
2024-02-09 21:40:30 +00:00
Dmitriy Ryajov 825766eea0
Rework builder & sampler (#697)
* add proof initializer for poseidon2 and `$`

* fix padding and block selection

* fix sample selection and input construction

* fix sample selection & more descriptive names

* add concrete types for sampler & builder

* add missing digest calls

* use concrete types

* add sample test fixtures

* use concrete types and don't fetch dummy blocks
2024-02-08 02:27:11 +00:00
Dmitriy Ryajov afec86b3cf
cleanup manifest and wire in strategy (#696) 2024-02-07 20:54:57 +00:00
Slava 1e957da109
CI update (#695)
* fix: update codecov action

* fix: remove nim v2 from Nim-matrix
2024-02-07 19:47:40 +00:00
Dmitriy Ryajov 2fc538337b
rework indexing strategy to use an enum (#694) 2024-02-07 11:13:27 -08:00
markspanbroek 2cf892c467
Smart contracts update: Groth16Proof instead of bytes (#683)
* Smart contracts update: Groth16Proof instead of bytes

* Use dummy verifier for now, until we can create ZK proofs

* Fix tests: submit proof only when slot is filled

* Submit dummy proofs for now

* More detailed log when proof submission failed

* Use dummy verifier for integration tests

For now at least

* Fix mistake in blanket renaming to ethProvider

* Update to latest codex-contracts-eth

* feat: zkey-hash from chain

* Fix zkeyHash

---------

Co-authored-by: Adam Uhlíř <adam@uhlir.dev>
2024-02-07 07:50:35 +01:00
Slava 403b9baf9f
Add Nim-matrix workflow to run on merge queue (#693)
* Add Nim-matrix workflow to run on merge queue

* Use reusable workflows for CI and Nim-matrix
2024-02-06 12:56:27 +02:00
Giuliano Mega 591be9446a
remove call to `waitFor` from clock.now and make it async (#691) 2024-02-05 08:36:28 -03:00
Slava ca0660b07e
Use default short commit length for Codex revision (#690) 2024-01-30 19:58:55 +02:00
markspanbroek fd3c566b25
update asynctest to 0.5.1 (#671)
* update asynctest to 0.5.0

Co-Authored-By: gmega <giuliano.mega@gmail.com>

* update remaining package imports

* add asynctest package wrapper

---------

Co-authored-by: gmega <giuliano.mega@gmail.com>
2024-01-29 17:03:51 -03:00
Slava 95a4484de0
fix: update codex help output (#687) 2024-01-29 19:31:29 +00:00
Slava ae61c29b3f
ci: fix docker workflow after actions update (#689)
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2024-01-29 20:12:07 +02:00
Slava 34095461ae
ci: update actions to lates major versions (#688) 2024-01-29 17:21:52 +02:00
Ben Bierens 0b8811c98d
Debug/sampling (#681)
* Extra logging in sampler

* wip: Fixing sampling issue in padded slot cells

* Cleanup

* Restores tests
2024-01-25 15:34:38 -08:00
Eric de88fd2c53
feat: create logging proxy (#663)
* implement a logging proxy

The logging proxy:
- prevents the need to import chronicles (as well as export except toJson),
- prevents the need to override `writeValue` or use or import nim-json-seralization elsewhere in the codebase, allowing for sole use of utils/json for de/serialization,
- and handles json formatting correctly in chronicles json sinks

* Rename logging -> logutils to avoid ambiguity with common names

* clean up

* add setProperty for JsonRecord, remove nim-json-serialization conflict

* Allow specifying textlines and json format separately

Not specifying a LogFormat will apply the formatting to both textlines and json sinks.

Specifying a LogFormat will apply the formatting to only that sink.

* remove unneeded usages of std/json

We only need to import utils/json instead of std/json

* move serialization from rest/json to utils/json so it can be shared

* fix NoColors ambiguity

Was causing unit tests to fail on Windows.

* Remove nre usage to fix Windows error

Windows was erroring with `could not load: pcre64.dll`. Instead of fixing that error, remove the pcre usage :)

* Add logutils module doc

* Shorten logutils.formatIt for `NBytes`

Both json and textlines formatIt were not needed, and could be combined into one formatIt

* remove debug integration test config

debug output and logformat of json for integration test logs

* Use ## module doc to support docgen

* bump nim-poseidon2 to export fromBytes

Before the changes in this branch, fromBytes was likely being resolved by nim-stew, or other dependency. With the changes in this branch, that dependency was removed and fromBytes could no longer be resolved. By exporting fromBytes from nim-poseidon, the correct resolution is now happening.

* fixes to get compiling after rebasing master

* Add support for Result types being logged using formatIt
2024-01-22 23:35:03 -08:00
Dmitriy Ryajov 72da534856
Wire sampler (#676)
* Setting up testfixture for proof datasampler

* Sets up calculating number of cells in a slot

* Sets up tests for bitwise modulo

* Implements cell index collection

* setting up slot blocks module

* Implements getting treeCID from slot

* implements getting slot blocks by index

* Implements out-of-range check for slot index

* cleanup

* Sets up getting sample from block

* Implements selecting a cell sample from a block

* Implements building a minitree for block cells

* Adds method to get dataset block index from slot block index

* It's running

* splits up indexing

* almost there

* Fixes test. Implementation is now functional

* Refactoring to object-oriented

* Cleanup

* Lining up output type with updated reference code.

* setting up

* Updates expected samples

* Updates proof checking test to match new format

* move builder to own dir

* move sampler to own dir

* fix paths

* various changes to add support for the sampler

* wip sampler implementation

* don't use upraises

* wip sampler integration

* misc

* move tests around

* Various fixes to select correct slot and block index

* removing old tests

* cleanup

* misc

fix tests that work with correct cell indices

* remove unused file

* fixup logging

* add logscope

* truncate entropy to 31 bytes, otherwise it might be > than mod

* forwar getCidAndProof to local store

* misc

* Adds missing test for initial-proving state

* reverting back to correct slot/block indexing

* fix tests for revert

* misc

* misc

---------

Co-authored-by: benbierens <thatbenbierens@gmail.com>
2024-01-17 11:24:34 -08:00
Dmitriy Ryajov 2fc7c75fd2
Contracts handler (#672)
* get rid of unneeded files

* don't reuse batch callback

* move out storage contract handlers

* wip

* add storage handler, to handle storage contracts interactions

* split out node tests

* a bit more cleanup

* remove storage handler, move back into node

* add missing raises:

* wip: add support for rebuilding slot

* split out tests

* wip

* rework indexing strategy to return an iterator

* convert to seq

* minor api change (meh, might revert)

* steping strategy should not die

* allow fetching batches for manifests and indicies

* restored expiry update

* restored expiry update functionality

* avoid closing datastores twice

* correct wrong rename

* fixes sigsegv
2024-01-15 08:45:04 -08:00
Afri 9b9cf9c450
docs: update api for two client test (#675)
Signed-off-by: Afri <58883403+q9f@users.noreply.github.com>
2024-01-15 12:35:55 +01:00