Commit Graph

615 Commits

Author SHA1 Message Date
Jaremy Creechley 96862372e5
wire in prove 2024-05-17 01:03:19 +03:00
Jaremy Creechley 1e86b5c051
wire in prove 2024-05-17 00:59:00 +03:00
Jaremy Creechley 0dfa2cc488
wire in prove 2024-05-17 00:51:57 +03:00
Jaremy Creechley 38ac70809e
add asynthreads utils 2024-05-17 00:39:07 +03:00
Jaremy Creechley 542ba13222
add asynccircom backend 2024-05-17 00:30:11 +03:00
Jaremy Creechley c09c27ad2c
add asynccircom backend 2024-05-17 00:14:37 +03:00
Jaremy Creechley 2af5ed1490
add asynccircom backend 2024-05-17 00:11:52 +03:00
Jaremy Creechley 41167c6a74
add asynccircom backend 2024-05-16 21:49:31 +03:00
Jaremy Creechley 2e26a77e90
add asynccircom backend 2024-05-16 21:46:34 +03:00
Jaremy Creechley 5e39fe8380
add asynccircom backend 2024-05-16 21:43:35 +03:00
Jaremy Creechley 0ac9750308
add asynccircom backend 2024-05-16 21:41:24 +03:00
Jaremy Creechley aa9f2d2a1f
modifying circomcompat for use with asyncthreading 2024-05-16 21:36:02 +03:00
Jaremy Creechley a4010d1844
modifying circomcompat for use with asyncthreading 2024-05-16 21:35:24 +03:00
Jaremy Creechley eb75887631
modifying circomcompat for use with asyncthreading 2024-05-16 21:27:28 +03:00
Jaremy Creechley 7949a253ce
modifying circomcompat for use with asyncthreading 2024-05-16 21:26:18 +03:00
Jaremy Creechley 8f8d51faa4
modifying circomcompat for use with asyncthreading 2024-05-16 21:12:38 +03:00
Adam Uhlíř e208d0b13a
feat: one confirmation for all transactions (#802)
* fix: make sure requestStorage is mined

* fix: correct place to plug confirm

* test: fixing contracts tests

* feat: one confirmation for all transactions

* fix: don't wait for confirmations only mined block
2024-05-14 15:40:29 +00:00
Ben Bierens 9379c7c662
Advertise manifests and trees (#797)
* wip update advertise loop to announce only manifests and trees

* fixes e2e tests

* fixes tests for discovery engine

---------

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2024-05-09 08:03:35 +00:00
Adam Uhlíř 1a0d2d424e
feat: expiry specified with number of seconds (#793) 2024-05-06 17:35:46 +02:00
Ben Bierens 4312e5ca36
Use a copy of slots hashset for iteration. (#801) 2024-05-06 08:12:49 +00:00
Ben Bierens c58d4d7dbe
Bug: Large manifests encoding (#800)
* Adds test for encoding/decoding manifests with large datasets

* switches datasetSize to uint64
2024-04-30 15:27:17 +00:00
Ben Bierens c7bc28d723
Reduce logging during file upload (#792)
* Removes warning

* Updates logging for file upload

* Restores trace for placing block and proof in repo store
2024-04-30 09:31:06 +00:00
Ben Bierens 9365aa4d51
bumps nim-serde (#794) 2024-04-26 09:42:31 +00:00
Ben Bierens 3041f5ff5f
Announce to DHT only tree and manifest CIDs (#788)
* announce only tree and manifest cids

* wip

* Adds tests for selecting which CIDs are announced

* newline

* Review comments by Tomasz
2024-04-24 07:30:02 +00:00
Giuliano Mega 53809129a5
Disable Discv5 bucket and routing table IP limits (#764)
* disable Discv5 bucket and routing table IP limits

* actually configure the discovery protocol
2024-04-02 12:31:52 +00:00
Jaremy Creechley 0a34c37c87
Add Node SPR endpoint (#760)
* Create a node api section, add spr and peerid endpoints, and update twoclient

* tweak docs
2024-04-02 11:13:49 +00:00
Ben Bierens 91bb972e6f
use integer instead of number (#762) 2024-03-29 18:18:41 +00:00
Eric dca9f2ee76
Use circuit artifacts in codex circuit fixtures for tests (#759) 2024-03-29 17:15:21 +00:00
Ben Bierens c97f973410
Adds two traces and bumps nim-ethers (#761) 2024-03-29 11:51:17 +00:00
Ben Bierens 8378ac4dac
Openapi mismatches (#754)
* Aligns response with openapi spec

* Fixes json serialization for local content response

* Review comments by Mark

* Fixes missed rename

* Removes array type for dataList
2024-03-26 15:12:15 +00:00
Ben Bierens 4038eed3c9
Copies openapi yaml into image (#751) 2024-03-25 15:23:30 +00:00
Tomasz Bekas 59d9439ae9
Scheduling erasure coding on another thread (#716)
* Scheduling erasure coding on another thread

* Code review fixes

* Fix for review comments

* Fix missing import

---------

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2024-03-23 09:56:35 +00:00
Adam Uhlíř de1714ed06
feat(api): availabilities patch endpoint (#692)
* feat(api): availabilities patch and reservations endpoints

* test: fixing tests and writing

* test: test reservations endpoint

* chore: feedback implementation

* chore: feedback implementation

* test: fix integration tests
2024-03-21 10:53:45 +00:00
Eric 43f0bf1c1c
refactor: add MarketError (#670)
* Add MarketError

Add MarketError and convert all EthersErrors (ProviderError, SignerError) to MarketError

* Include token contract call in conversion of ethers error
2024-03-21 00:30:04 +00:00
Eric 916e4834d0
Remove codex_use_hardhat compilation flag (#741)
This flag was originally used to change OnChainClock behavior when using hardhat as an ethereum chain source due to a very strange bug which would mark the timestamp of new blocks as one second off the timestamp they should have been.

The issue has since been worked around in another manner, and thus this flag is no longer needed.
2024-03-19 11:08:54 +00:00
Eric f567f4ec15
refactor: use serde for json de/serialization instead of utils/json (#704)
* json > nim-serde bump

Should wait until serde is integrated into nim-ethers before making these changes as there will be less import exceptions required.

* bump nim-serde

* change func to proc due to chronicles side effects

* import serde into utils/json, use as proxy

import nim-serde into utils/json and use utils/json as a proxy for serde functions, including overloading `%` and `fromJson` for application types.

* update tests to use serde

* bump serde to latest

* remove testjson -- no longer needed

* bump serde in nimble

* updates to reconcile rebase with master
2024-03-19 03:25:13 +00:00
Giuliano Mega b4de53f436
update path for libp2p (#743) 2024-03-18 22:41:22 +00:00
Ben Bierens 8f65f578db
Pulls in latest contracts. (#742) 2024-03-18 13:41:25 +00:00
Giuliano Mega d33804f700
fixes double lookups when block does not exist (#739)
* fixes double lookups when block does not exist

* handle timeouts on requestBlock

* fix indentation which was causing an integration test to fail
2024-03-15 21:50:56 +00:00
markspanbroek 4d069599c9
Use real verifier (#737)
* use a real verifying contract address

* contracts: cleanup

* marketplacesuite: set correct circuit files, interval mining

* Proofs tests updates

Contains changes to get the proving tests working reliably.

* integration: use correct circom artifacts for creating proofs

* integration: cleanup

---------

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
2024-03-12 14:20:42 +00:00
markspanbroek b3e57a37e2
Wire up prover (#736)
* wire prover into node

* stricter case object checks

* return correct proof

* misc renames

* adding usefull traces

* fix nodes and tolerance to match expected params

* format challenges in logs

* add circom compat to solidity groth16 convertion

* update

* bump time to give nodes time to load with all circom artifacts

* misc

* misc

* use correct dataset geometry in erasure

* make errors more searchable

* use parens around `=? (await...)` calls

* styling

* styling

* use push raises

* fix to match constructor arguments

* merge master

* merge master

* integration: fix proof parameters for a test

Increased times due to ZK proof generation.
Increased storage requirement because we're now hosting
5 slots instead of 1.

* sales: calculate initial proof at start of period

reason: this ensures that the period (and therefore
the challenge) doesn't change while we're calculating
the proof

* integration: fix proof parameters for tests

Increased times due to waiting on next period.
Fixed data to be of right size.
Updated expected payout due to hosting 5 slots.

* sales: wait for stable proof challenge

When the block pointer is nearing the
wrap-around point, we wait another period
before calculating a proof.

* fix merge conflict

---------

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
2024-03-12 12:10:14 +00:00
markspanbroek 293c676f22
Prover CLI updates (#735)
* rework cli to accept circuit params

* check circom files extension

* adding new required cli changes

* don't use ufcs

* persistence is a command now

* use `nimOldCaseObjects` switch for nim confutils compat

* misc

* Update cli integration tests

* Fix: simulateProofFailures option is not for validator

* moving circom params under `prover` command

* update tests

* Use circuit assets from codex-contract-eth in tests

* Add "prover" cli command to tests

* use correct stores

* make `verifier` a cmd option

* update circuit artifacts path

* fix cli tests

* Update integration tests to use cli commands

Integration tests have been updated to use the new cli commands. The api for usage in the integration tests has also changed a bit.

Proofs tests have been updated to use 5 nodes and 8 blocks of data. The remaining integration tests also need to be updated.

* remove parsedCli from CodexConfig

Instead, parse the cli args on the fly when needed

* remove unneeded gcsafes

* graceful shutdowns

Where possible, do not raise assert, as other nodes in the test may already be running. Instead, raise exceptions, catch them in multinodes.nim, and attempt to do a teardown before failing the test.

`abortOnError` is set to true so that `fail()` will quit immediately, after teardown has been run.

* update testmarketplace to new api, with valid EC params

---------

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
2024-03-12 09:57:13 +00:00
markspanbroek 8589e63d34
Update contract (#734)
* Update codex-contracts-eth

* contracts: update G2Point definition

* integration: disable automatic advancing of time

reason: makes reasoning about timing in tests harder,
because the period is set to 60 seconds in the
marketplace configuration, but this code switches to
a new period every 500 milliseconds

* integration: fix parameters of marketplace payouts test

* integration: update test settings

* integration: fix typo

* integration: workaround for hardhat issue

Subscriptions expire after 5 minutes when using
websockets. Use http and polling instead.

* integration: remove origDatasetSizeInBlocks

* integration: fix proof parameters for test

* integration: do not log output by default

* integration: fix failure rate in test

* integration: fix warning

* integration: include clock in logs

* integration: allow for more periods

5 periods was cutting it close, if we get too much
pointer downtime, then the test would fail
2024-03-12 08:18:25 +00:00
markspanbroek e654e93c71
Fix sales agent (#733)
* clock: add 1 second leeway before acting on timeouts

* sales: do not raise in proving loop when slot is cancelled

Allow the onCancelled callback to handle cancellation, and
the onFailed callback to handle failed requests.

* sales: cleanup proving tests

* sales: fix sales agent tests

* sales: stop cancellation loop when request started, finished or failed

* sales: fix flaky test

* sales: fix another flaky test

* clock: add comment explaining the + 1 second

Co-Authored-By: benbierens <thatbenbierens@gmail.com>

---------

Co-authored-by: benbierens <thatbenbierens@gmail.com>
2024-03-12 06:41:03 +00:00
markspanbroek d1658d7b77
Fix on-chain clock (#732)
* market: use `pending` blocktag when querying onchain state

* clock: use wall clock in integration tests

reason: we'll need to wait for the next period in
integration tests, and we can't do that if the
time doesn't advance

* clock: remove unused field

* integration: use pending block time to get current time

* clock: fix on-chain clock for hardhat

Only use 'latest' block for updates
Only update the first time you see a block

* integration: do not start tests with a very outdated block

* integration: allow for longer expiry period
2024-03-11 16:57:20 +00:00
markspanbroek 1fc0510f2b
build: update nim-ethers to latest chronosv3 branch (#731)
Fixes validator, which uses transaction simulation to
check whether a proof can be marked as missing.

Fixes gas estimations, so they are done on the pending
block, instead of the latest block.
2024-03-11 15:25:41 +00:00
Giuliano Mega 74679e9e2e
abort Codex when startup sequence fails (#738) 2024-03-11 12:31:17 +00:00
Slava 6ee1162989
Update Docker files and building Codex docs (#722)
* Make Build Codex copy/paste friendly

* Add auto-export for env var from files

* Use latest Ubuntu LTS image and install rust via package manager
2024-03-01 12:23:05 +00:00
Ben Bierens 53fdc88def
Feature/eth syncing (#721)
* Delays starting of codex until eth provider is synced

* bumps nim-ethers to 0.7.2

* bumps nim-ethers to 0.7.3

* Adds increasing delay to waitForSync
2024-03-01 07:50:29 +00:00
Ben Bierens 5e7ce52fbe
Fix block retransmit (#651)
* Applies peer-scoped lock to peer task handler.

* Replace async lock with delete-first approach.

* Cleanup some logging

* Adds inFlight flag to WantListEntry

* Clears inflight flag when local retrieval fails.

* Adds test for setting of in-flight

* Adds test for clearing in-flight when lookup fails

* Review comments by Tomasz

---------

Co-authored-by: gmega <giuliano.mega@gmail.com>
2024-02-29 07:37:12 +00:00