Commit Graph

628 Commits

Author SHA1 Message Date
Tomasz Bekas 3ae73197c2
Bandaid for failing erasure coding (#855) 2024-07-03 14:44:00 +00:00
Giuliano Mega fbf1b51f57
Prover workshop band-aid (#853)
* add prover bandaid

* Improve error message text

Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
Signed-off-by: Giuliano Mega <giuliano.mega@gmail.com>

---------

Signed-off-by: Giuliano Mega <giuliano.mega@gmail.com>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
2024-07-03 11:27:56 +00:00
Eric 67facb4b2a
feat(rest): adds erasure coding constraints when requesting storage (#848)
* Rest API: add erasure coding constraints when requesting storage

* clean up

* Make error message for "dataset too small" more informative.

* fix API integration test

---------

Co-authored-by: gmega <giuliano.mega@gmail.com>
2024-06-27 21:26:19 +00:00
Giuliano Mega b004ca75f6
Bump Nim to 1.6.21 (#851)
* bump Nim to 1.6.21 (range type reset fixes)

* remove incompatible versions from compiler matrix
2024-06-27 19:58:27 +00:00
Tomasz Bekas 11eaebfd77
Fix verifiable manifest constructor (#844)
* Fix verifiable manifest constructor

* Add integration test for verifiable manifest download

Add integration test for testing download of verifiable dataset after creating request for storage

* add missing import

* add testecbug to integration suite

* Remove hardhat instance from integration test

* change description, drop echo

---------

Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
Co-authored-by: gmega <giuliano.mega@gmail.com>
2024-06-26 20:02:39 +00:00
Eric 5b131611ac
bump nim-leopard to handle no parity data (#845) 2024-06-26 15:41:06 +00:00
Eric a79bbd459a
prevent node crashing with `not val.isNil` (#843) 2024-06-26 14:22:44 +00:00
Giuliano Mega a55b676a42
provisional fix so EC errors do not crash the node on download (#841) 2024-06-25 21:38:53 +00:00
Slava ad53066131
ci: add verify_circuit=true to the releases (#840) 2024-06-25 18:31:50 +00:00
Giuliano Mega 8138ef5afd
Fix verifiable manifest initialization (#839)
* fix verifiable manifest initialization

* fix linearstrategy, use verifiableStrategy to select blocks for slots

* check for both strategies in attribute inheritance test
2024-06-21 22:50:56 +00:00
Giuliano Mega 4619260dc1
Fix StoreStream so it doesn't return parity bytes (#838)
* fix storestream so it doesn\'t return parity bits for protected/verifiable manifests

* use Cid.example instead of creating a mock manually
2024-06-21 11:09:59 +00:00
Tomasz Bekas ec7faa21b5
Block deletion with ref count & repostore refactor (#631) 2024-06-20 22:46:06 +00:00
Adam Uhlíř 1a57341b7d
fix: createReservation lock (#825)
* fix: createReservation lock

* fix: additional locking places

* fix: acquire lock

* chore: feedback

Co-authored-by: markspanbroek <mark@spanbroek.net>
Signed-off-by: Adam Uhlíř <adam@uhlir.dev>

* feat: withLock template and fixed tests

* fix: use proc for MockReservations constructor

* chore: feedback

Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
Signed-off-by: Adam Uhlíř <adam@uhlir.dev>

* chore: feedback implementation

---------

Signed-off-by: Adam Uhlíř <adam@uhlir.dev>
Co-authored-by: markspanbroek <mark@spanbroek.net>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
2024-06-20 10:28:01 +00:00
Giuliano Mega 6e9bdf1d7e
Allow compiler version to be pinned consistently (#832)
* allow compiler version to be pinned consistently

* set pin to v1.6.14 to make it equivalent to current master
2024-06-17 18:59:25 +00:00
Giuliano Mega e124035772
remove stale and misleading dependency version metadata (#833) 2024-06-17 14:31:10 +00:00
Eric e422c9065f
config flag to enable CORS response headers for data downloads (#834) 2024-06-17 11:33:21 +00:00
Giuliano Mega d524252f1f
make CI build against branch head instead of merge (#831) 2024-06-13 01:54:50 +00:00
Tomasz Bekas f51ef528b1
Rework AsyncIter (#811)
* Rework AsyncIter

* Add tests for finishing iter on error

* Improved error handling for  and additional tests

* Use new style of constructors

* Handle future cancellation

* Docs for constructors
2024-06-10 22:47:29 +00:00
Ben Bierens fe9d9705f1
Fix/spec intsizes (#829)
* Sets int64 formats for byte-counting integers in openapi spec

* uints instead of ints

* uint64 not supported by every openAPI client
2024-06-06 09:44:58 +00:00
Ben Bierens e8f4658ea5
Sets int64 formats for byte-counting integers in openapi spec (#828)
* Sets int64 formats for byte-counting integers in openapi spec

* uints instead of ints
2024-06-06 06:20:46 +00:00
Slava a4a50fd714
Add Release workflow (#827)
* Install specific Rust version and add Linux arm64 support (#749)

* Add release workflow (#749)

* Update Rust requirements in the Building guide (#749)

* Fix rust_version input usage (#749)

* Update comments (#749)

* Use ARCH_OVERRIDE only for i386 (#749)

* Add an alternative method to install Rust (#749)

* Do not upload builds to the Codex network (#749)
2024-06-05 12:56:41 +00:00
Ben Bierens 3246c43174
Updates rust in docker. Pins circom-compat to pinned FFI branch (#818)
* Updates rust in docker. Pins circom-compat to pinned FFI branch

* Extracts builder image from codex.Dockerfile

* Updates readme

* Use single Dockerfile

* Revert readme back

---------

Co-authored-by: Veaceslav Doina <20563034+veaceslavdoina@users.noreply.github.com>
2024-05-31 08:23:11 +00:00
Ben Bierens bd8fedaf28
Metadata in LevelDB (#806)
* pulls in datastore-leveldb update

* bump

* Applies LevelDb as metadata store. Adds option for repostore.

* Sets submodule to main branch

* I can do syntax, me

* Removes wildcard from metadata query key

* Applies leveldb instead of sqlite-in-memory for tests

* Restores query key wildcard.

* Pins nim-datastore to latest master

* bumps leveldb to 0.1.4

---------

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2024-05-30 06:57:10 +00:00
Eric e6a387e8e8
feat[marketplace]: add slot queue pausing (#752)
* add seen flag

* Add MockSlotQueueItem and better prioritisation tests

* Update seen priority, and include in SlotQueueItem.init

* Re-add processed slots to queue

Re-add processed slots to queue if the sale was ignored or errored

* add pausing of queue

- when processing slots in queue, pause queue if item was marked seen
- if availability size is increased, trigger onAvailabilityAdded callback
- in sales, on availability added, clear 'seen' flags, then unpause the queue
- when items pushed to the queue, unpause the queue

* remove unused NoMatchingAvailabilityError from slotqueue

The slot queue should also have nothing to do with availabilities

* when all availabilities are empty, pause the queue

An empty availability is defined as size < DefaultBlockSize as this means even the smallest possible request could not be served. However, this is up for discussion.

* remove availability from onAvailabilitiesEmptied callback

* refactor onAvailabilityAdded and onAvailabilitiesEmptied

onAvailabilityAdded and onAvailabilitiesEmptied are now only called from reservations.update (and eventually reservations.delete once implemented).

- Add empty routine for Availability and Reservation
- Add allEmpty routine for Availability and Reservation, which returns true when all all Availability or Reservation objects in the datastore are empty.

* SlotQueue test support updates

* Sales module test support updates

* Reservations module tests for queue pausing

* Sales module tests for queue pausing

Includes tests for sales states cancelled, errored, ignored to ensure onCleanUp is called with correct parameters

* SlotQueue module tests for queue pausing

* fix existing sales test

* PR feedback

- indent `self.unpause`
- update comment for `clearSeenFlags`

* reprocessSlot in SaleErrored only when coming from downloading

* remove pausing of queue when availabilities are "emptied"

Queue pausing when all availiabilies are "emptied" is not necessary, given that the node would not be able to service slots once all its availabilities' freeSize are too small for the slots in the queue, and would then be paused anyway.

Add test that asserts the queue is paused once the freeSpace of availabilities drops too low to fill slots in the queue.

* Update clearing of seen flags

The asyncheapqueue update overload would need to check index bounds and ultimately a different solution was found using the mitems iterator.

* fix test

request.id was different before updating request.ask.slots, and that id was used to set the state in mockmarket.

* Change filled/cleanup future to nil, so no await is needed

* add wait to allow items to be added to queue

* do not unpause queue when seen items are pushed

* re-add seen item back to queue once paused

Previously, when a seen item was processed, it was first popped off the queue, then the queue was paused waiting to process that item once the queue was unpaused. Now, when a seen item is processed, it is popped off the queue, the queue is paused, then the item is re-added to the queue and the queue will wait until unpaused before it will continue popping items off the queue. If the item was not re-added to the queue, it would have been processed immediately once unpaused, however there may have been other items with higher priority pushed to the queue in the meantime. The queue would not be unpaused if those added items were already seen. In particular, this may happen when ignored items due to lack of availability are re-added to a paused queue. Those ignored items will likely have a higher priority than the item that was just seen (due to it having been processed first), causing the queue to the be paused.

* address PR comments
2024-05-26 00:38:38 +00:00
Jaremy Creechley efd46148b0
Pr add prover benchmark tool (#790)
* initial setup

* reorg

* figuring out basic shell commands

* benchmarks

* benchmarks

* Sets up environment for running benchmarks

* updates

* integrate setup and proving

* updates

* adding outputs

* cleanup

* check failure

* benchmarks

* benchmarks

* benchmarks

* benchmarks

* benchmarks

* benchmarks

* formatting

* fix running larger sizes

* use larger ceremony file size

* use larger ceremony file size

* use larger ceremony file size

* restore benchmarks

* cleanup

* cleanup

* cleanup

* cleanup

* cleanup

* cleanup

* cleanup

* cleanup

* cleanup

* cleanup

* refactor env

* refactor env

* refactor env

* refactor env

* refactor env

* rename

* cleanup

* cleanup

* cleanup

* cleanup

* cleanup

* cleanup

* readme

* readme

* merge

* initial splitout of codex ark prover cli

* opts

* copying nimcli opts

* copying nimcli opts

* copying nimcli opts

* updating ark cli

* updating ark cli

* updating ark cli

* updating ark cli

* updating ark cli

* updating ark cli

* updating ark cli

* updating ark cli

* docs

* remove file

* add param

* add benchmarkLoops param

* update benchmark formatting

* update benchmark formatting

* update benchmark formatting

* update benchmark formatting

* fix naming

* fix serde version

* Apply suggestions from code review

cleanup wording

Signed-off-by: Dmitriy Ryajov <dryajov@gmail.com>

---------

Signed-off-by: Dmitriy Ryajov <dryajov@gmail.com>
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2024-05-23 09:28:17 -07:00
markspanbroek 3046b7636c
Cleanup integration tests (#757)
* integration: move REST API tests into their own module

* integration: move upload and download tests into their own module

* integration: move purchasing tests into their own module

* integration: move marketplace tests to the right module

* integration: mine a block *after* starting nodes

To ensure that tests involving multiple nodes do
not start with out-of-sync clocks

* Fix: do not swallow CancellationErrors

* integration: avoid underflow in UInt256

* network: remove unnecessary error handling

No Exceptions can occur, only Defects, because everything
is asyncSpawned.

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>

* network: do not raise in asyncSpawned proc

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>

---------

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2024-05-23 15:29:30 +00:00
Giuliano Mega 97b538961c
Updates tutorial with entry on how to run Codex with its Markeplace (#765)
* update help info

* Marketplace guide WiP

* add API requests

* reword titles

* reword Guides section

* simplify geth startup instruction

* make instructions more clear

* clarify expiry trial-and-error

* minor rewording

* Update docs/Marketplace.md

Signed-off-by: Jaremy Creechley <creechley@gmail.com>

* Update docs/Marketplace.md

Signed-off-by: Jaremy Creechley <creechley@gmail.com>

* Update docs/Marketplace.md

Signed-off-by: Jaremy Creechley <creechley@gmail.com>

* Update docs/Marketplace.md

Signed-off-by: Jaremy Creechley <creechley@gmail.com>

* Feat/updated guide jc tweaks (#769)

* more tweaks
* fix network json
* fix commands
* fix datetime
* fix purchase tolerance

* Apply suggestions from code review

Signed-off-by: Jaremy Creechley <creechley@gmail.com>

* Update docs/Marketplace.md

It's data not data1 and need the "@"

Signed-off-by: Jaremy Creechley <creechley@gmail.com>

* use debug/info by default for guide

* add slashing high level at footnote

* rephrase meaning of --data

---------

Signed-off-by: Jaremy Creechley <creechley@gmail.com>
Co-authored-by: Jaremy Creechley <creechley@gmail.com>
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2024-05-22 16:43:33 +00:00
Ben Bierens 8bf44e24ee
inlines copying of hashset (#809) 2024-05-17 00:57:30 +00:00
Ben Bierens a0b12e85bf
Update logging for download (#799)
* Updates logging for file upload

* Restores trace for placing block and proof in repo store

* Reduces logging while transmitting blocks

* unnecessary formatter

* Clean up some more download related traces

* much better

* Review comment by dryajov
2024-05-16 10:06:12 -07:00
Adam Uhlíř e208d0b13a
feat: one confirmation for all transactions (#802)
* fix: make sure requestStorage is mined

* fix: correct place to plug confirm

* test: fixing contracts tests

* feat: one confirmation for all transactions

* fix: don't wait for confirmations only mined block
2024-05-14 15:40:29 +00:00
Ben Bierens 9379c7c662
Advertise manifests and trees (#797)
* wip update advertise loop to announce only manifests and trees

* fixes e2e tests

* fixes tests for discovery engine

---------

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2024-05-09 08:03:35 +00:00
Adam Uhlíř 1a0d2d424e
feat: expiry specified with number of seconds (#793) 2024-05-06 17:35:46 +02:00
Ben Bierens 4312e5ca36
Use a copy of slots hashset for iteration. (#801) 2024-05-06 08:12:49 +00:00
Ben Bierens c58d4d7dbe
Bug: Large manifests encoding (#800)
* Adds test for encoding/decoding manifests with large datasets

* switches datasetSize to uint64
2024-04-30 15:27:17 +00:00
Ben Bierens c7bc28d723
Reduce logging during file upload (#792)
* Removes warning

* Updates logging for file upload

* Restores trace for placing block and proof in repo store
2024-04-30 09:31:06 +00:00
Ben Bierens 9365aa4d51
bumps nim-serde (#794) 2024-04-26 09:42:31 +00:00
Ben Bierens 3041f5ff5f
Announce to DHT only tree and manifest CIDs (#788)
* announce only tree and manifest cids

* wip

* Adds tests for selecting which CIDs are announced

* newline

* Review comments by Tomasz
2024-04-24 07:30:02 +00:00
Giuliano Mega 53809129a5
Disable Discv5 bucket and routing table IP limits (#764)
* disable Discv5 bucket and routing table IP limits

* actually configure the discovery protocol
2024-04-02 12:31:52 +00:00
Jaremy Creechley 0a34c37c87
Add Node SPR endpoint (#760)
* Create a node api section, add spr and peerid endpoints, and update twoclient

* tweak docs
2024-04-02 11:13:49 +00:00
Ben Bierens 91bb972e6f
use integer instead of number (#762) 2024-03-29 18:18:41 +00:00
Eric dca9f2ee76
Use circuit artifacts in codex circuit fixtures for tests (#759) 2024-03-29 17:15:21 +00:00
Ben Bierens c97f973410
Adds two traces and bumps nim-ethers (#761) 2024-03-29 11:51:17 +00:00
Ben Bierens 8378ac4dac
Openapi mismatches (#754)
* Aligns response with openapi spec

* Fixes json serialization for local content response

* Review comments by Mark

* Fixes missed rename

* Removes array type for dataList
2024-03-26 15:12:15 +00:00
Ben Bierens 4038eed3c9
Copies openapi yaml into image (#751) 2024-03-25 15:23:30 +00:00
Tomasz Bekas 59d9439ae9
Scheduling erasure coding on another thread (#716)
* Scheduling erasure coding on another thread

* Code review fixes

* Fix for review comments

* Fix missing import

---------

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2024-03-23 09:56:35 +00:00
Adam Uhlíř de1714ed06
feat(api): availabilities patch endpoint (#692)
* feat(api): availabilities patch and reservations endpoints

* test: fixing tests and writing

* test: test reservations endpoint

* chore: feedback implementation

* chore: feedback implementation

* test: fix integration tests
2024-03-21 10:53:45 +00:00
Eric 43f0bf1c1c
refactor: add MarketError (#670)
* Add MarketError

Add MarketError and convert all EthersErrors (ProviderError, SignerError) to MarketError

* Include token contract call in conversion of ethers error
2024-03-21 00:30:04 +00:00
Eric 916e4834d0
Remove codex_use_hardhat compilation flag (#741)
This flag was originally used to change OnChainClock behavior when using hardhat as an ethereum chain source due to a very strange bug which would mark the timestamp of new blocks as one second off the timestamp they should have been.

The issue has since been worked around in another manner, and thus this flag is no longer needed.
2024-03-19 11:08:54 +00:00
Eric f567f4ec15
refactor: use serde for json de/serialization instead of utils/json (#704)
* json > nim-serde bump

Should wait until serde is integrated into nim-ethers before making these changes as there will be less import exceptions required.

* bump nim-serde

* change func to proc due to chronicles side effects

* import serde into utils/json, use as proxy

import nim-serde into utils/json and use utils/json as a proxy for serde functions, including overloading `%` and `fromJson` for application types.

* update tests to use serde

* bump serde to latest

* remove testjson -- no longer needed

* bump serde in nimble

* updates to reconcile rebase with master
2024-03-19 03:25:13 +00:00
Giuliano Mega b4de53f436
update path for libp2p (#743) 2024-03-18 22:41:22 +00:00