Commit Graph

302 Commits

Author SHA1 Message Date
Ben Bierens 9379c7c662
Advertise manifests and trees (#797)
* wip update advertise loop to announce only manifests and trees

* fixes e2e tests

* fixes tests for discovery engine

---------

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2024-05-09 08:03:35 +00:00
Adam Uhlíř 1a0d2d424e
feat: expiry specified with number of seconds (#793) 2024-05-06 17:35:46 +02:00
Ben Bierens 4312e5ca36
Use a copy of slots hashset for iteration. (#801) 2024-05-06 08:12:49 +00:00
Ben Bierens c58d4d7dbe
Bug: Large manifests encoding (#800)
* Adds test for encoding/decoding manifests with large datasets

* switches datasetSize to uint64
2024-04-30 15:27:17 +00:00
Ben Bierens c7bc28d723
Reduce logging during file upload (#792)
* Removes warning

* Updates logging for file upload

* Restores trace for placing block and proof in repo store
2024-04-30 09:31:06 +00:00
Ben Bierens 3041f5ff5f
Announce to DHT only tree and manifest CIDs (#788)
* announce only tree and manifest cids

* wip

* Adds tests for selecting which CIDs are announced

* newline

* Review comments by Tomasz
2024-04-24 07:30:02 +00:00
Giuliano Mega 53809129a5
Disable Discv5 bucket and routing table IP limits (#764)
* disable Discv5 bucket and routing table IP limits

* actually configure the discovery protocol
2024-04-02 12:31:52 +00:00
Jaremy Creechley 0a34c37c87
Add Node SPR endpoint (#760)
* Create a node api section, add spr and peerid endpoints, and update twoclient

* tweak docs
2024-04-02 11:13:49 +00:00
Ben Bierens c97f973410
Adds two traces and bumps nim-ethers (#761) 2024-03-29 11:51:17 +00:00
Ben Bierens 8378ac4dac
Openapi mismatches (#754)
* Aligns response with openapi spec

* Fixes json serialization for local content response

* Review comments by Mark

* Fixes missed rename

* Removes array type for dataList
2024-03-26 15:12:15 +00:00
Tomasz Bekas 59d9439ae9
Scheduling erasure coding on another thread (#716)
* Scheduling erasure coding on another thread

* Code review fixes

* Fix for review comments

* Fix missing import

---------

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2024-03-23 09:56:35 +00:00
Adam Uhlíř de1714ed06
feat(api): availabilities patch endpoint (#692)
* feat(api): availabilities patch and reservations endpoints

* test: fixing tests and writing

* test: test reservations endpoint

* chore: feedback implementation

* chore: feedback implementation

* test: fix integration tests
2024-03-21 10:53:45 +00:00
Eric 43f0bf1c1c
refactor: add MarketError (#670)
* Add MarketError

Add MarketError and convert all EthersErrors (ProviderError, SignerError) to MarketError

* Include token contract call in conversion of ethers error
2024-03-21 00:30:04 +00:00
Eric 916e4834d0
Remove codex_use_hardhat compilation flag (#741)
This flag was originally used to change OnChainClock behavior when using hardhat as an ethereum chain source due to a very strange bug which would mark the timestamp of new blocks as one second off the timestamp they should have been.

The issue has since been worked around in another manner, and thus this flag is no longer needed.
2024-03-19 11:08:54 +00:00
Eric f567f4ec15
refactor: use serde for json de/serialization instead of utils/json (#704)
* json > nim-serde bump

Should wait until serde is integrated into nim-ethers before making these changes as there will be less import exceptions required.

* bump nim-serde

* change func to proc due to chronicles side effects

* import serde into utils/json, use as proxy

import nim-serde into utils/json and use utils/json as a proxy for serde functions, including overloading `%` and `fromJson` for application types.

* update tests to use serde

* bump serde to latest

* remove testjson -- no longer needed

* bump serde in nimble

* updates to reconcile rebase with master
2024-03-19 03:25:13 +00:00
Giuliano Mega d33804f700
fixes double lookups when block does not exist (#739)
* fixes double lookups when block does not exist

* handle timeouts on requestBlock

* fix indentation which was causing an integration test to fail
2024-03-15 21:50:56 +00:00
markspanbroek 4d069599c9
Use real verifier (#737)
* use a real verifying contract address

* contracts: cleanup

* marketplacesuite: set correct circuit files, interval mining

* Proofs tests updates

Contains changes to get the proving tests working reliably.

* integration: use correct circom artifacts for creating proofs

* integration: cleanup

---------

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
2024-03-12 14:20:42 +00:00
markspanbroek b3e57a37e2
Wire up prover (#736)
* wire prover into node

* stricter case object checks

* return correct proof

* misc renames

* adding usefull traces

* fix nodes and tolerance to match expected params

* format challenges in logs

* add circom compat to solidity groth16 convertion

* update

* bump time to give nodes time to load with all circom artifacts

* misc

* misc

* use correct dataset geometry in erasure

* make errors more searchable

* use parens around `=? (await...)` calls

* styling

* styling

* use push raises

* fix to match constructor arguments

* merge master

* merge master

* integration: fix proof parameters for a test

Increased times due to ZK proof generation.
Increased storage requirement because we're now hosting
5 slots instead of 1.

* sales: calculate initial proof at start of period

reason: this ensures that the period (and therefore
the challenge) doesn't change while we're calculating
the proof

* integration: fix proof parameters for tests

Increased times due to waiting on next period.
Fixed data to be of right size.
Updated expected payout due to hosting 5 slots.

* sales: wait for stable proof challenge

When the block pointer is nearing the
wrap-around point, we wait another period
before calculating a proof.

* fix merge conflict

---------

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
2024-03-12 12:10:14 +00:00
markspanbroek 293c676f22
Prover CLI updates (#735)
* rework cli to accept circuit params

* check circom files extension

* adding new required cli changes

* don't use ufcs

* persistence is a command now

* use `nimOldCaseObjects` switch for nim confutils compat

* misc

* Update cli integration tests

* Fix: simulateProofFailures option is not for validator

* moving circom params under `prover` command

* update tests

* Use circuit assets from codex-contract-eth in tests

* Add "prover" cli command to tests

* use correct stores

* make `verifier` a cmd option

* update circuit artifacts path

* fix cli tests

* Update integration tests to use cli commands

Integration tests have been updated to use the new cli commands. The api for usage in the integration tests has also changed a bit.

Proofs tests have been updated to use 5 nodes and 8 blocks of data. The remaining integration tests also need to be updated.

* remove parsedCli from CodexConfig

Instead, parse the cli args on the fly when needed

* remove unneeded gcsafes

* graceful shutdowns

Where possible, do not raise assert, as other nodes in the test may already be running. Instead, raise exceptions, catch them in multinodes.nim, and attempt to do a teardown before failing the test.

`abortOnError` is set to true so that `fail()` will quit immediately, after teardown has been run.

* update testmarketplace to new api, with valid EC params

---------

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
2024-03-12 09:57:13 +00:00
markspanbroek 8589e63d34
Update contract (#734)
* Update codex-contracts-eth

* contracts: update G2Point definition

* integration: disable automatic advancing of time

reason: makes reasoning about timing in tests harder,
because the period is set to 60 seconds in the
marketplace configuration, but this code switches to
a new period every 500 milliseconds

* integration: fix parameters of marketplace payouts test

* integration: update test settings

* integration: fix typo

* integration: workaround for hardhat issue

Subscriptions expire after 5 minutes when using
websockets. Use http and polling instead.

* integration: remove origDatasetSizeInBlocks

* integration: fix proof parameters for test

* integration: do not log output by default

* integration: fix failure rate in test

* integration: fix warning

* integration: include clock in logs

* integration: allow for more periods

5 periods was cutting it close, if we get too much
pointer downtime, then the test would fail
2024-03-12 08:18:25 +00:00
markspanbroek e654e93c71
Fix sales agent (#733)
* clock: add 1 second leeway before acting on timeouts

* sales: do not raise in proving loop when slot is cancelled

Allow the onCancelled callback to handle cancellation, and
the onFailed callback to handle failed requests.

* sales: cleanup proving tests

* sales: fix sales agent tests

* sales: stop cancellation loop when request started, finished or failed

* sales: fix flaky test

* sales: fix another flaky test

* clock: add comment explaining the + 1 second

Co-Authored-By: benbierens <thatbenbierens@gmail.com>

---------

Co-authored-by: benbierens <thatbenbierens@gmail.com>
2024-03-12 06:41:03 +00:00
markspanbroek d1658d7b77
Fix on-chain clock (#732)
* market: use `pending` blocktag when querying onchain state

* clock: use wall clock in integration tests

reason: we'll need to wait for the next period in
integration tests, and we can't do that if the
time doesn't advance

* clock: remove unused field

* integration: use pending block time to get current time

* clock: fix on-chain clock for hardhat

Only use 'latest' block for updates
Only update the first time you see a block

* integration: do not start tests with a very outdated block

* integration: allow for longer expiry period
2024-03-11 16:57:20 +00:00
Giuliano Mega 74679e9e2e
abort Codex when startup sequence fails (#738) 2024-03-11 12:31:17 +00:00
Ben Bierens 53fdc88def
Feature/eth syncing (#721)
* Delays starting of codex until eth provider is synced

* bumps nim-ethers to 0.7.2

* bumps nim-ethers to 0.7.3

* Adds increasing delay to waitForSync
2024-03-01 07:50:29 +00:00
Ben Bierens 5e7ce52fbe
Fix block retransmit (#651)
* Applies peer-scoped lock to peer task handler.

* Replace async lock with delete-first approach.

* Cleanup some logging

* Adds inFlight flag to WantListEntry

* Clears inflight flag when local retrieval fails.

* Adds test for setting of in-flight

* Adds test for clearing in-flight when lookup fails

* Review comments by Tomasz

---------

Co-authored-by: gmega <giuliano.mega@gmail.com>
2024-02-29 07:37:12 +00:00
Giuliano Mega 457567531f
Fixes active cancellation for pending want requests (#714)
* add block cancellation support + tests

* tie issueCancellations into resolveBlocks for proper exception tracking, address comments

* pull cancellation as separate primitive in BlockExcNetwork

* use allFutures, rename issueBlockCancellations -> cancelBlocks

* use trc instead of wrn to register send error

* do not log peer IDs
2024-02-22 14:54:45 +00:00
Dmitriy Ryajov 9e13d2251a
Prover works with zkey (#711)
* rework backend to instantiate key at initialization

* add groth16 convertes for solidity

* prover taks num samples on construction

* add zkey file

* rework helpers

* rename types

* update tests

* reworked test helpers

* rename types

* rework test

* test all slots artifacts

* bump to latest version
2024-02-19 18:58:39 +00:00
Dmitriy Ryajov 3e884430c1
Node use correct stores (#710)
* don't pass erasure

* use correct stores and construct erasure inside the node

* fix tests to match new constructor

* remove prover argument

* review commets

* revert failing on no-prover for now

* small cleanup

* comment out invalid proofs broken test
2024-02-19 10:12:10 -08:00
Eric d70ab59004
refactor: multinode integration test refactor (#662)
* refactor multi node test suite

Refactor the multinode test suite into the marketplace test suite.

- Arbitrary number of nodes can be started with each test: clients, providers, validators
- Hardhat can also be started locally with each test, usually for the purpose of saving and inspecting its log file.
- Log files for all nodes can be persisted on disk, with configuration at the test-level
- Log files, if persisted (as specified in the test), will be persisted to a CI artifact
- Node config is specified at the test-level instead of the suite-level
- Node/Hardhat process starting/stopping is now async, and runs much faster
- Per-node config includes:
  - simulating proof failures
  - logging to file
  - log level
  - log topics
  - storage quota
  - debug (print logs to stdout)
- Tests find next available ports when starting nodes, as closing ports on Windows can lag
- Hardhat is no longer required to be running prior to starting the integration tests (as long as Hardhat is configured to run in the tests).
  - If Hardhat is already running, a snapshot will be taken and reverted before and after each test, respectively.
  - If Hardhat is not already running and configured to run at the test-level, a Hardhat process will be spawned and torn down before and after each test, respectively.

* additional logging for debug purposes

* address PR feedback

- fix spelling
- revert change from catching ProviderError to SignerError -- this should be handled more consistently in the Market abstraction, and will be handled in another PR.
- remove method label from raiseAssert
- remove unused import

* Use API instead of command exec to test for free port

Use chronos `createStreamServer` API to test for free port by binding localhost address and port. Use `ServerFlags.ReuseAddr` to enable reuse of same IP/Port on multiple test runs.

* clean up

* remove upraises annotations from tests

* Update tests to work with updated erasure coding slot sizes

* update dataset size, nodes, tolerance to match valid ec params

Integration tests now have valid dataset sizes (blocks), tolerances, and number of nodes, to work with valid ec params. These values are validated when requested storage.

Print the rest api failure message (via doAssert) when a rest api call fails (eg the rest api may validate some ec params).

All integration tests pass when the async `clock.now` changes are reverted.

* dont use async clock for now

* fix workflow

* move integration logs uplod to reusable

---------

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2024-02-19 04:55:39 +00:00
Dmitriy Ryajov 0497114e44
Use correct blocks count in builder (#709) 2024-02-19 02:19:59 +00:00
Dmitriy Ryajov 3fce267352
Block and manifest as objects (#708)
* make block and manifest objects

* use unsafeAddr since block is an object type now

* make object var
2024-02-19 02:10:16 +00:00
Eric 61ec4275c8
Revert "remove call to `waitFor` from clock.now and make it async (#691)" (#707)
This reverts commit 591be9446a.
2024-02-16 22:12:16 +00:00
Dmitriy Ryajov 6fab252bf1
safer backend release (#701) 2024-02-10 23:17:11 +00:00
Dmitriy Ryajov e23159b065
wire in circom backend (#698)
* wire in circom backend

* should contain leafs

* adding circom compad and circuits deps

* update windows build

* fix windows build

* improve test names

* move proving defaults to codextypes

* remove unnedded inmports and move defaults to codextypes

* capture error code on backend failure
2024-02-09 21:40:30 +00:00
Dmitriy Ryajov 825766eea0
Rework builder & sampler (#697)
* add proof initializer for poseidon2 and `$`

* fix padding and block selection

* fix sample selection and input construction

* fix sample selection & more descriptive names

* add concrete types for sampler & builder

* add missing digest calls

* use concrete types

* add sample test fixtures

* use concrete types and don't fetch dummy blocks
2024-02-08 02:27:11 +00:00
Dmitriy Ryajov afec86b3cf
cleanup manifest and wire in strategy (#696) 2024-02-07 20:54:57 +00:00
Dmitriy Ryajov 2fc538337b
rework indexing strategy to use an enum (#694) 2024-02-07 11:13:27 -08:00
markspanbroek 2cf892c467
Smart contracts update: Groth16Proof instead of bytes (#683)
* Smart contracts update: Groth16Proof instead of bytes

* Use dummy verifier for now, until we can create ZK proofs

* Fix tests: submit proof only when slot is filled

* Submit dummy proofs for now

* More detailed log when proof submission failed

* Use dummy verifier for integration tests

For now at least

* Fix mistake in blanket renaming to ethProvider

* Update to latest codex-contracts-eth

* feat: zkey-hash from chain

* Fix zkeyHash

---------

Co-authored-by: Adam Uhlíř <adam@uhlir.dev>
2024-02-07 07:50:35 +01:00
Giuliano Mega 591be9446a
remove call to `waitFor` from clock.now and make it async (#691) 2024-02-05 08:36:28 -03:00
Slava ca0660b07e
Use default short commit length for Codex revision (#690) 2024-01-30 19:58:55 +02:00
Slava 95a4484de0
fix: update codex help output (#687) 2024-01-29 19:31:29 +00:00
Ben Bierens 0b8811c98d
Debug/sampling (#681)
* Extra logging in sampler

* wip: Fixing sampling issue in padded slot cells

* Cleanup

* Restores tests
2024-01-25 15:34:38 -08:00
Eric de88fd2c53
feat: create logging proxy (#663)
* implement a logging proxy

The logging proxy:
- prevents the need to import chronicles (as well as export except toJson),
- prevents the need to override `writeValue` or use or import nim-json-seralization elsewhere in the codebase, allowing for sole use of utils/json for de/serialization,
- and handles json formatting correctly in chronicles json sinks

* Rename logging -> logutils to avoid ambiguity with common names

* clean up

* add setProperty for JsonRecord, remove nim-json-serialization conflict

* Allow specifying textlines and json format separately

Not specifying a LogFormat will apply the formatting to both textlines and json sinks.

Specifying a LogFormat will apply the formatting to only that sink.

* remove unneeded usages of std/json

We only need to import utils/json instead of std/json

* move serialization from rest/json to utils/json so it can be shared

* fix NoColors ambiguity

Was causing unit tests to fail on Windows.

* Remove nre usage to fix Windows error

Windows was erroring with `could not load: pcre64.dll`. Instead of fixing that error, remove the pcre usage :)

* Add logutils module doc

* Shorten logutils.formatIt for `NBytes`

Both json and textlines formatIt were not needed, and could be combined into one formatIt

* remove debug integration test config

debug output and logformat of json for integration test logs

* Use ## module doc to support docgen

* bump nim-poseidon2 to export fromBytes

Before the changes in this branch, fromBytes was likely being resolved by nim-stew, or other dependency. With the changes in this branch, that dependency was removed and fromBytes could no longer be resolved. By exporting fromBytes from nim-poseidon, the correct resolution is now happening.

* fixes to get compiling after rebasing master

* Add support for Result types being logged using formatIt
2024-01-22 23:35:03 -08:00
Dmitriy Ryajov 72da534856
Wire sampler (#676)
* Setting up testfixture for proof datasampler

* Sets up calculating number of cells in a slot

* Sets up tests for bitwise modulo

* Implements cell index collection

* setting up slot blocks module

* Implements getting treeCID from slot

* implements getting slot blocks by index

* Implements out-of-range check for slot index

* cleanup

* Sets up getting sample from block

* Implements selecting a cell sample from a block

* Implements building a minitree for block cells

* Adds method to get dataset block index from slot block index

* It's running

* splits up indexing

* almost there

* Fixes test. Implementation is now functional

* Refactoring to object-oriented

* Cleanup

* Lining up output type with updated reference code.

* setting up

* Updates expected samples

* Updates proof checking test to match new format

* move builder to own dir

* move sampler to own dir

* fix paths

* various changes to add support for the sampler

* wip sampler implementation

* don't use upraises

* wip sampler integration

* misc

* move tests around

* Various fixes to select correct slot and block index

* removing old tests

* cleanup

* misc

fix tests that work with correct cell indices

* remove unused file

* fixup logging

* add logscope

* truncate entropy to 31 bytes, otherwise it might be > than mod

* forwar getCidAndProof to local store

* misc

* Adds missing test for initial-proving state

* reverting back to correct slot/block indexing

* fix tests for revert

* misc

* misc

---------

Co-authored-by: benbierens <thatbenbierens@gmail.com>
2024-01-17 11:24:34 -08:00
Dmitriy Ryajov 2fc7c75fd2
Contracts handler (#672)
* get rid of unneeded files

* don't reuse batch callback

* move out storage contract handlers

* wip

* add storage handler, to handle storage contracts interactions

* split out node tests

* a bit more cleanup

* remove storage handler, move back into node

* add missing raises:

* wip: add support for rebuilding slot

* split out tests

* wip

* rework indexing strategy to return an iterator

* convert to seq

* minor api change (meh, might revert)

* steping strategy should not die

* allow fetching batches for manifests and indicies

* restored expiry update

* restored expiry update functionality

* avoid closing datastores twice

* correct wrong rename

* fixes sigsegv
2024-01-15 08:45:04 -08:00
Dmitriy Ryajov 8b12934fe2
Build slots (#668)
Wiring in slots builder functionality into `requestStorage`
2024-01-11 08:45:23 -08:00
Eric b186afbc4f
bump ethers, remove upraises from callbacks (#664) 2024-01-09 15:51:34 +11:00
Dmitriy Ryajov fffb674bba
Integrate slot builder (#666)
* rework merkle tree support

* rename merkletree -> codexmerkletree

* treed and proof encoding/decoding

* style

* adding codex merkle and coders tests

* use default hash codec

* proof size changed

* add from nodes test

* shorte file names

* wip poseidon tree

* shorten file names

* root returns a result

* import poseidon tests

* fix merge issues and cleanup a few warnings

* setting up slot builder

* Getting cids in slot

* ensures blocks are devisable by number of slots

* wip

* Implements indexing strategies

* Swaps in indexing strategy into erasure.

* wires slot and indexing tests up

* Fixes issue where indexing strategy stepped gives wrong values for smallest of ranges

* debugs indexing strategies

* Can select slot blocks

* finding number of pad cells

* Implements building slot tree

* finishes implementing slot builder

* Adds check that block size is a multiple of cell size

* Cleanup slotbuilder

* Review comments by Tomasz

* Fixes issue where ecK was used as numberOfSlots.

* rework merkle tree support

* deps

* rename merkletree -> codexmerkletree

* treed and proof encoding/decoding

* style

* adding codex merkle and coders tests

* remove new codecs for now

* proof size changed

* add from nodes test

* shorte file names

* wip poseidon tree

* shorten file names

* fix bad `elements` iter

* bump

* bump

* wip

* reworking slotbuilder

* move out of manifest

* expose getCidAndProof

* import index strat...

* remove getMHash

* remove unused artifacts

* alias zero

* add digest for multihash

* merge issues

* remove unused hashes

* add option to result converter

* misc

* fix tests

* add helper to derive EC block count

* rename method

* misc

* bump

* extract slot root building into own proc

* revert to manifest to accessor

---------

Co-authored-by: benbierens <thatbenbierens@gmail.com>
2024-01-08 14:52:46 -08:00
Dmitriy Ryajov b8ee2ac71e
Update multicodecs (#665)
* rework merkle tree support

* rename merkletree -> codexmerkletree

* treed and proof encoding/decoding

* style

* adding codex merkle and coders tests

* use default hash codec

* proof size changed

* add from nodes test

* shorte file names

* wip poseidon tree

* shorten file names

* root returns a result

* import poseidon tests

* update multicodecs

* consolidating codex types and adding new codecs

* update codec

* remove temp codecs constants

* move codecs related stuff out

* updating codecs

* misc

* updating sizes since block size was adjusted to 64kb

* fix merge issues and cleanup a few warnings
2023-12-22 13:04:01 +01:00
Dmitriy Ryajov 52c5578c46
Rework merkle tree (#654)
* rework merkle tree support

* deps

* rename merkletree -> codexmerkletree

* treed and proof encoding/decoding

* small change to invoke proof verification

* rename merkletree to codexmerkletree

* style

* adding codex merkle and coders tests

* fixup imports

* remove new codecs for now

* bump deps

* adding trace statement

* properly serde of manifest block codecs

* use default hash codec

* add more trace logging to aid debugging

* misc

* remove double import

* revert un-needded change

* proof size changed

* bump poseidon2

* add from nodes test

* shorte file names

* remove upraises

* wip poseidon tree

* adjust file names

* misc

* shorten file names

* fix bad `elements` iter

* don't do asserts

* add fromNodes and converters

* root and getProof now return result

* add poseidon2 tree tests

* root now returns result

* misc

* had to make merkletree a ref, because nim blows up otherwise

* root returns a result

* root returns a result

* import poseidon tests

* bump

* merkle poseidon2 digest

* misc

* add merkle digest tests

* bump

* don't use checksuite

* Update tests/codex/merkletree/generictreetests.nim

Co-authored-by: markspanbroek <mark@spanbroek.net>
Signed-off-by: Dmitriy Ryajov <dryajov@gmail.com>

* Update codex/merkletree/merkletree.nim

Co-authored-by: markspanbroek <mark@spanbroek.net>
Signed-off-by: Dmitriy Ryajov <dryajov@gmail.com>

* Update codex/merkletree/merkletree.nim

Co-authored-by: markspanbroek <mark@spanbroek.net>
Signed-off-by: Dmitriy Ryajov <dryajov@gmail.com>

* Update tests/codex/merkletree/generictreetests.nim

Co-authored-by: markspanbroek <mark@spanbroek.net>
Signed-off-by: Dmitriy Ryajov <dryajov@gmail.com>

* missing return

* make toBool private (it's still needed otherwise comparison won't work)

* added `digestTree` that returns a tree and `digest` for root

* test against both poseidon trees - codex and poseidon2

* shorten merkle tree names

* don't compare trees - it's going to be too slow

* move comparison to mekrle helper

* remove merkle utils

---------

Signed-off-by: Dmitriy Ryajov <dryajov@gmail.com>
Co-authored-by: markspanbroek <mark@spanbroek.net>
2023-12-21 06:41:43 +00:00
Eric f293082ae9
Reverts logging-proxy, commit 27f585eb6f (#660) 2023-12-20 13:24:40 +11:00
Eric 27f585eb6f
feat: create logging proxy (#653)
* implement a logging proxy

The logging proxy:
- prevents the need to import chronicles (as well as export except toJson),
- prevents the need to override `writeValue` or use or import nim-json-seralization elsewhere in the codebase, allowing for sole use of utils/json for de/serialization,
- and handles json formatting correctly in chronicles json sinks

* Rename logging -> logutils to avoid ambiguity with common names

* clean up

* add setProperty for JsonRecord, remove nim-json-serialization conflict

* Allow specifying textlines and json format separately

Not specifying a LogFormat will apply the formatting to both textlines and json sinks.

Specifying a LogFormat will apply the formatting to only that sink.

* remove unneeded usages of std/json

We only need to import utils/json instead of std/json

* move serialization from rest/json to utils/json so it can be shared

* fix NoColors ambiguity

Was causing unit tests to fail on Windows.

* Remove nre usage to fix Windows error

Windows was erroring with `could not load: pcre64.dll`. Instead of fixing that error, remove the pcre usage :)

* Add logutils module doc

* Shorten logutils.formatIt for `NBytes`

Both json and textlines formatIt were not needed, and could be combined into one formatIt

* remove debug integration test config

debug output and logformat of json for integration test logs

* Use ## module doc to support docgen
2023-12-19 22:12:47 +00:00
Eric df6b9c6760
Workaround for Hardhat last block timestamp bug (#644)
* Workaround for Hardhat timestamp bug

Likely due to a Hardhat bug in which the callbacks for subscription events are called and awaited before updating its local understanding of the last block time, Hardhat will report a block time in the `newHeads` event that is generally 1 second before the time reported from `getLatestBlock.timestamp`. This was causing issues with the OnChainClock's offset and therefore the `now()` used by the `OnChainClock` would sometimes be off by a second (or more), causing tests to fail.

This commit introduce a `codex_use_hardhat` compilation flag, that when set, will always get the latest block timestamp from Hardhat via the `getLatestBlock.timestamp` RPC call for `OnChainClock.now` calls. Otherwise, the last block timestamp reported in the `newHeads` event will be used.

Update the docker dist tests compilation flag for simulated proof failures (it was not correct), and explicitly add the `codex_use_hardhat=false` for clarity.

* enable simulated proof failures for coverage

* comment out failing test on linux -- will be replaced

* bump codex contracts eth

* add back clock offset for non-hardhat cases

* bump codex-contracts-eth

increases pointer by 67 blocks each period increase

* Add `codex_use_hardhat` flag to coverage tests
2023-12-20 08:06:24 +11:00
Eric 2c621e0421
chore: improve log information (#656)
* Add proof downtime to validator logs

* Add previous state to state machine transition log
2023-12-19 04:29:18 +00:00
Eric 39554c8ea9
sales was never marked as running and caused a hang (#655) 2023-12-19 01:28:17 +00:00
Ben Bierens f1b1fe9152
Adds endpoint to expose quota information (#652) 2023-12-14 10:57:16 +00:00
Adam Uhlíř e62cb96bde
feat: cleanup returning availability (#650)
Co-authored-by: markspanbroek <mark@spanbroek.net>
2023-12-13 20:58:17 +01:00
Ben Bierens 0c3d1dd563
Verifiable manifests (#642)
* Adds test for encoding/decoding protected manifest

* Setting up verifiable manifest

* mysterious mysteries

* Successful encoding test for verifiable manifests

* extracts toF out of users of manifest code

* Update codex/manifest/coders.nim

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Signed-off-by: Ben Bierens <39762930+benbierens@users.noreply.github.com>

* Review comments by Dmitriy

* Adds missing verifiable print to $ method.

* Replace poseidon2 F type with int as temporary stand-in for verification hashes

* Replaces verification hash placeholder with CID

---------

Signed-off-by: Ben Bierens <39762930+benbierens@users.noreply.github.com>
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2023-12-12 08:11:54 +00:00
Ben Bierens a9f8090bb4
Add challenge to prove callback (#649)
* Sets up passing proof challenge to onProve callback

* Implements tests for proof challenge

* Removes unused import
2023-12-11 11:29:15 +01:00
Eric 3907ca4095
Add get active slot /slots/{slotId} to REST api, use utils/json (#645)
* Add get active slot /slots/{slotId} to REST api, use utils/json

- Add endpoint /slots/{slotId} to get an active SalesAgent from the Sales module. Used in integration tests to test when a sale has reached a certain state. Those integration test changes will be included in a larger PR, coming later.
- Add OpenAPI changes for new endpoint and associated components
- Use utils/json instead of nim-json-serialization. Required exemption of imports from several packages that export nim-json-serialization by default.

* Only except `toJson` from import/export of chronicles
2023-12-07 12:16:36 +11:00
Eric 86a6ef9215
remove waitFor cancelAndWait in proving state (#643)
In proving.onCancelled, the `waitFor state.loop.cancelAndWait()` was never completing. Turns out this was not needed, because when changing states, the current state's run is cancelled, which automatically cancels the state prove loop, because it is a child of proving.run. Therefore, the logic to cancelAndWait the prove loop was removed as it was not needed.
2023-12-06 14:20:14 +11:00
Adam Uhlíř b38146d3f7
fix: check expiration is before request end (#641) 2023-12-05 14:25:28 +01:00
Adam Uhlíř 0e4387d1b3
refactor: move expiry update from fetchBatched (#634) 2023-11-28 22:04:11 +01:00
Dmitriy Ryajov 22c31046a7
Dont dial self (#633)
* don't dial self

* revert style changes

* filter out self on dial

* add helpers to `peerId` and `isSelf`

* don't fire up discovery eaguerly

* allow excluding multiple peers in sendWantHave

* revert style changes

* move self check to discovery.find

* readd eaguer dht lookup is required in some cases

* revert style changes

* misc

* drop peer first, before queueing a dht lookup

* moar style changes

* use isSelf
2023-11-27 10:25:53 -08:00
markspanbroek b62ccf5a8a
Update Request: remove PoR, add merkle root (#590)
* [build] update codex-contracts-eth

* [contracts] update request parameters

- Remove PoR parameters and totalChunks
- Add merkleRoot
2023-11-27 12:25:01 +00:00
Adam Uhlíř 79fce39dbf
fix: require expiry from api (#629)
Co-authored-by: markspanbroek <mark@spanbroek.net>
2023-11-22 11:35:26 +00:00
Adam Uhlíř 8681a40ee7
feat: update expiry when data downloaded and slot filled (#619)
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
Co-authored-by: markspanbroek <mark@spanbroek.net>
2023-11-22 10:09:12 +00:00
Dmitriy Ryajov ec8d0c98b2
Fix REST endpoints semantics (#612)
* Fix REST endpoints semantics

* update endpoint description

* update, operation id

* Adding enum support

* make enum descerializer public

* add support for listing manifests

* test `/data` endpoint to list local manifests

* debug leftovers

* remove commented out line
2023-11-20 16:14:06 -08:00
Tomasz Bekas 70efd13c73
Fix block delete by treeCid and index (#623) 2023-11-20 14:48:56 +01:00
Ben Bierens bece1b88a1
Feat/bump questionable (#627)
* Bumps questionable version to 0.10.12

* removes unnecessary questionnable bindings.

* Fixes tests

* unnecessary whitespaces
2023-11-17 13:49:45 +01:00
Ben Bierens 40d7714a67
Fixes json formatting for NodeId (#614) 2023-11-15 13:29:17 +00:00
Dmitriy Ryajov 4d50ecf504
handle exceptions gracefuly in rest api (#617) 2023-11-14 11:05:43 -08:00
Dmitriy Ryajov 8d61391073
Light cleanup (#616)
mostly whitespaces cleanup
2023-11-14 09:52:27 -08:00
Dmitriy Ryajov a1725594a8
Fix json serialization for cid and enums (#615)
* use str on JString types, `$` will preserve `"`

* Adding enum support

* deserialize cid test

* make enum descerializer public

* unify fromJson for objects and refs

* add enum descerialization testing
2023-11-14 07:23:50 -08:00
Tomasz Bekas 2396c4d76d
Blockexchange uses merkle root and index to fetch blocks (#566)
* Blockexchange uses merkle root and index to fetch blocks

* Links the network store getTree to the local store.

* Update codex/stores/repostore.nim

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Signed-off-by: Tomasz Bekas <tomasz.bekas@gmail.com>

* Rework erasure.nim to include recent cleanup

* Revert accidential changes to lib versions

* Addressing review comments

* Storing proofs instead of trees

* Fix a comment

* Fix broken tests

* Fix for broken testerasure.nim

* Addressing PR comments

---------

Signed-off-by: Tomasz Bekas <tomasz.bekas@gmail.com>
Co-authored-by: benbierens <thatbenbierens@gmail.com>
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2023-11-14 13:02:17 +01:00
Ben Bierens 34a9c5e88d
Storage/purchases endpoint (#609)
* adds list of purchase ids to storage API

* adds list of purchase ids to storage API

* Updates openAPI

* fixes openAPI type
2023-11-13 11:30:27 +00:00
Ben Bierens 80e2ef4eb5
Debug/info shows announce addresses (#610)
* Shows announce addresses in debug/info endpoint.

* Shows announce addresses in debug/info endpoint.
2023-11-12 09:58:40 +01:00
Ben Bierens cb02962231
Adds endpoint for listing files (manifests) in node. Useful for demo UI. (#599)
* Adds endpoint for listing files (manifests) in node. Useful for demo UI.

* Moves upload/download/files into content API calls.

* Cleans up json serialization for manifest

* Cleans up some more json serialization

* Moves block iteration and decoding to node.nim

* Moves api methods into their own init procs.

* Applies RestContent api object.

* Replaces format methods with Rest objects in json.nim

* Unused import

* Review comments by Adam

* Fixes issue where content/local endpoint clashes with content/cid.

* faulty merge resolution

* Renames content API to data.

* Fixes faulty rebase

* Adds test for data/local API

* Renames local and download api.
2023-11-09 08:47:09 +00:00
Eric 7d4ea878d2
Support logging to file (#558)
* Support logging to file

* Log the entire config and fix build error

* Downgrade log level for "starting codex node" config output

* bump ethers to prevent nonce gaps

* fix tests
2023-11-09 16:35:55 +11:00
Adam Uhlíř c0bec2f899
feat: ensure block expiry (#597)
* feat: update block expiry

* chore: feedback implementation

* chore: feedback implementation

* chore: feedback implementation
2023-11-06 08:10:30 +00:00
Slava 11cd2c46ad
Rename metrics (#478) (#603) 2023-11-03 17:21:54 +02:00
Adam Uhlíř 2fc71cf81b
feat: partial payouts for cancelled requests (#561) 2023-10-24 10:12:54 +00:00
Dmitriy Ryajov 8a7d74e6b2
removing old por proofs implementation (#593) 2023-10-23 07:58:07 -07:00
Adam Uhlíř c28627d16f
chore(logging): purchases logging (#567) 2023-10-19 10:12:49 +02:00
Adam Uhlíř 5f4218ba1c
fix: use increaseAllowance for token approving (#565) 2023-10-16 13:09:54 +02:00
Ben Bierens 040434aa19
renames log properties named 'count' (#560) 2023-10-03 13:33:01 +02:00
Ben Bierens fede106118
Numbered log entries (#559)
* add numbered log entries (hack)

* Changed from CLI argument to compiler define

* Use uint64 for counting

---------

Co-authored-by: gmega <giuliano.mega@gmail.com>
2023-10-03 11:53:58 +02:00
Eric 570a1f7b67
[marketplace] Availability improvements (#535)
## Problem
When Availabilities are created, the amount of bytes in the Availability are reserved in the repo, so those bytes on disk cannot be written to otherwise. When a request for storage is received by a node, if a previously created Availability is matched, an attempt will be made to fill a slot in the request (more accurately, the request's slots are added to the SlotQueue, and eventually those slots will be processed). During download, bytes that were reserved for the Availability were released (as they were written to disk). To prevent more bytes from being released than were reserved in the Availability, the Availability was marked as used during the download, so that no other requests would match the Availability, and therefore no new downloads (and byte releases) would begin. The unfortunate downside to this, is that the number of Availabilities a node has determines the download concurrency capacity. If, for example, a node creates a single Availability that covers all available disk space the operator is willing to use, that single Availability would mean that only one download could occur at a time, meaning the node could potentially miss out on storage opportunities.

## Solution
To alleviate the concurrency issue, each time a slot is processed, a Reservation is created, which takes size (aka reserved bytes) away from the Availability and stores them in the Reservation object. This can be done as many times as needed as long as there are enough bytes remaining in the Availability. Therefore, concurrent downloads are no longer limited by the number of Availabilities. Instead, they would more likely be limited to the SlotQueue's `maxWorkers`.

From a database design perspective, an Availability has zero or more Reservations.

Reservations are persisted in the RepoStore's metadata, along with Availabilities. The metadata store key path for Reservations is ` meta / sales / reservations / <availabilityId> / <reservationId>`, while Availabilities are stored one level up, eg `meta / sales / reservations / <availabilityId> `, allowing all Reservations for an Availability to be queried (this is not currently needed, but may be useful when work to restore Availability size is implemented, more on this later).

### Lifecycle
When a reservation is created, its size is deducted from the Availability, and when a reservation is deleted, any remaining size (bytes not written to disk) is returned to the Availability. If the request finishes, is cancelled (expired), or an error occurs, the Reservation is deleted (and any undownloaded bytes returned to the Availability). In addition, when the Sales module starts, any Reservations that are not actively being used in a filled slot, are deleted.

Having a Reservation persisted until after a storage request is completed, will allow for the originally set Availability size to be reclaimed once a request contract has been completed. This is a feature that is yet to be implemented, however the work in this PR is a step in the direction towards enabling this.

### Unknowns
Reservation size is determined by the `StorageAsk.slotSize`. If during download, more bytes than `slotSize` are attempted to be downloaded than this, then the Reservation update will fail, and the state machine will move to a `SaleErrored` state, deleting the Reservation. This will likely prevent the slot from being filled.

### Notes
Based on #514
2023-09-29 14:33:08 +10:00
Adam Uhlíř 2f1c778d02
fix: unknown state goes to payout when slot state is finished (#555) 2023-09-27 15:57:41 +02:00
Dmitriy Ryajov 25ea7fd0b2
Erasure cleanup (#545)
* cleanup erasure coding

* moar cleanup

* fix off by 1 issues in tests

* style

* consolidate decoding data code

* simplify tuple unpacking

* fix retrieve purchase

We don't support single blocks for now

* Apply suggestions from code review

Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
Signed-off-by: Dmitriy Ryajov <dryajov@gmail.com>

---------

Signed-off-by: Dmitriy Ryajov <dryajov@gmail.com>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
2023-09-25 07:31:10 -07:00
markspanbroek 71cd35112b
Taiko L2 (#483)
* [docs] fix two client scenario: add missing collateral

* [integration] separate step to wait for node to be started

* [cli] add option to specify ethereum private key

* Remove unused imports

* Fix warnings

* [integration] move type definitions to correct place

* [integration] wait a bit longer for a node to start in debug mode

When e.g. running against Taiko test net rpc, the node start
takes longer

* [integration] simplify handling of codex node and client

* [integration] add Taiko integration test

* [contracts] await token approval confirmation before next tx

* [contracts] deployment address of marketplace on Taiko

* [cli] --eth-private-key now takes a file name

Instead of supplying the private key on the command line,
expect the private key to be in a file with the correct
permissions.

* [utils] Fixes undeclared `activeChroniclesStream` on Windows

* [build] update nim-ethers to include PR #52

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>

* [cli] Better error messages when reading eth private key

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>

* [integration] simplify reading of cmd line arguments

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>

* [build] update to latest version of nim-ethers

* [contracts] updated contract address for Taiko L2

* [build] update codex contracts to latest version

---------

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
2023-09-13 16:17:56 +02:00
Adam Uhlíř ae89db1eea
fix: sales concurrency bug (#537) 2023-09-05 16:47:29 +02:00
markspanbroek d3a22a7b7b
Fix slot queue push (#542)
* [sales] remove availability check before adding to slot queue

* [sales] add missing return statement

* [tests] remove 'eventuallyCheck' helper

* [sales] remove reservations from slot queue

* [tests] rename module `eventually` -> `always`

* [sales] increase slot queue size

Because it will now also hold items for which we haven't
checked availability yet.
2023-09-04 16:42:09 +02:00
Ben Bierens d279eebd69
Start discovery after announce address is updated (#536)
* Start discovery after announce address is updated

* Moves switch start-stop to codex.nim. Adjusts timing to solve issue
2023-09-04 11:12:20 +02:00
Ben Bierens 545e0d47e1
Slows down block maintenance iteration (#543)
* Slows down block maintenance iteration

* Fixes unstable integration test.
2023-09-04 11:12:14 +02:00
Eric 37b3d99c3d
Improve integration testing client (CodexClient) and json serialization (#514)
* Improve integration testing client (CodexClient) and json serialization

The current client used for integration testing against the REST endpoints for Codex accepts and passes primitive types. This caused a hard to diagnose bug where a `uint` was not being deserialized correctly.

In addition, the json de/serializing done between the CodexClient and REST client was not easy to read and was not tested.

These changes bring non-primitive types to most of the CodexClient functions, allowing us to lean on the compiler to ensure we're providing correct typings. More importantly, a json de/serialization util was created as a drop-in replacement for the std/json lib, with the main two differences being that field serialization is opt-in (instead of opt-out as in the case of json_serialization) and serialization errors are captured and logged, making debugging serialization issues much easier.

* Update integration test to use nodes=2 and tolerance=1

* clean up
2023-09-01 15:44:41 +10:00
Adam Uhlíř 098baf47d1
chore: improve logging (#526) 2023-08-28 11:33:44 +02:00
Ben Bierens f10c3f3934
Debug/extra logging (#509)
* logs

* poll log

* more logs

* time log for upload steps

* Adds metric and log for block retrieval time

* adds some logging

* move logging for storestream block indices

* Log at start and end of block iteration

* applies branch blockstore-bugfix

* Cleanup

* Cleanup
2023-08-22 08:35:16 +02:00
Ben Bierens 238771eb73
Repostore crash debug (#521)
* repostore listBlocks reads too much data

* Add assert to make sure data is not queried
2023-08-22 08:26:58 +02:00
Adam Uhlíř f459a2c6f6
refactor: merging proving module into sales (#469)
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
2023-08-21 12:26:43 +02:00