* add changes to use chronos v4 in compat mode
* switch chronos to compat fix branch
* use nimbus-build-system with configurable Nim repo
* add missing imports
* add missing await
* bump compat
* pin nim version in Makefile
* add await instead of asyncSpawn to advertisement queue loop
* bump DHT to v0.5.0
* allow error state of `onBatch` to propagate upwards in test code
* pin Nim compiler commit to avoid fetching stale branch
* make CI build against branch head instead of merge
* fix handling of return values in testslotqueue
* add logs and test
* add Merkle proof checks
* factor out Circom input normalization, fix proof input serialization
* add test and update existing ones
* update circuit assets
* add back trace message
* switch contracts to fix branch
* update codex-contracts-eth to latest
* do not expose prove with prenormalized inputs
* add prover bandaid
* Improve error message text
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
Signed-off-by: Giuliano Mega <giuliano.mega@gmail.com>
---------
Signed-off-by: Giuliano Mega <giuliano.mega@gmail.com>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
* Rest API: add erasure coding constraints when requesting storage
* clean up
* Make error message for "dataset too small" more informative.
* fix API integration test
---------
Co-authored-by: gmega <giuliano.mega@gmail.com>
* Fix verifiable manifest constructor
* Add integration test for verifiable manifest download
Add integration test for testing download of verifiable dataset after creating request for storage
* add missing import
* add testecbug to integration suite
* Remove hardhat instance from integration test
* change description, drop echo
---------
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
Co-authored-by: gmega <giuliano.mega@gmail.com>
* fix verifiable manifest initialization
* fix linearstrategy, use verifiableStrategy to select blocks for slots
* check for both strategies in attribute inheritance test
* Rework AsyncIter
* Add tests for finishing iter on error
* Improved error handling for and additional tests
* Use new style of constructors
* Handle future cancellation
* Docs for constructors
* Install specific Rust version and add Linux arm64 support (#749)
* Add release workflow (#749)
* Update Rust requirements in the Building guide (#749)
* Fix rust_version input usage (#749)
* Update comments (#749)
* Use ARCH_OVERRIDE only for i386 (#749)
* Add an alternative method to install Rust (#749)
* Do not upload builds to the Codex network (#749)
* pulls in datastore-leveldb update
* bump
* Applies LevelDb as metadata store. Adds option for repostore.
* Sets submodule to main branch
* I can do syntax, me
* Removes wildcard from metadata query key
* Applies leveldb instead of sqlite-in-memory for tests
* Restores query key wildcard.
* Pins nim-datastore to latest master
* bumps leveldb to 0.1.4
---------
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
* add seen flag
* Add MockSlotQueueItem and better prioritisation tests
* Update seen priority, and include in SlotQueueItem.init
* Re-add processed slots to queue
Re-add processed slots to queue if the sale was ignored or errored
* add pausing of queue
- when processing slots in queue, pause queue if item was marked seen
- if availability size is increased, trigger onAvailabilityAdded callback
- in sales, on availability added, clear 'seen' flags, then unpause the queue
- when items pushed to the queue, unpause the queue
* remove unused NoMatchingAvailabilityError from slotqueue
The slot queue should also have nothing to do with availabilities
* when all availabilities are empty, pause the queue
An empty availability is defined as size < DefaultBlockSize as this means even the smallest possible request could not be served. However, this is up for discussion.
* remove availability from onAvailabilitiesEmptied callback
* refactor onAvailabilityAdded and onAvailabilitiesEmptied
onAvailabilityAdded and onAvailabilitiesEmptied are now only called from reservations.update (and eventually reservations.delete once implemented).
- Add empty routine for Availability and Reservation
- Add allEmpty routine for Availability and Reservation, which returns true when all all Availability or Reservation objects in the datastore are empty.
* SlotQueue test support updates
* Sales module test support updates
* Reservations module tests for queue pausing
* Sales module tests for queue pausing
Includes tests for sales states cancelled, errored, ignored to ensure onCleanUp is called with correct parameters
* SlotQueue module tests for queue pausing
* fix existing sales test
* PR feedback
- indent `self.unpause`
- update comment for `clearSeenFlags`
* reprocessSlot in SaleErrored only when coming from downloading
* remove pausing of queue when availabilities are "emptied"
Queue pausing when all availiabilies are "emptied" is not necessary, given that the node would not be able to service slots once all its availabilities' freeSize are too small for the slots in the queue, and would then be paused anyway.
Add test that asserts the queue is paused once the freeSpace of availabilities drops too low to fill slots in the queue.
* Update clearing of seen flags
The asyncheapqueue update overload would need to check index bounds and ultimately a different solution was found using the mitems iterator.
* fix test
request.id was different before updating request.ask.slots, and that id was used to set the state in mockmarket.
* Change filled/cleanup future to nil, so no await is needed
* add wait to allow items to be added to queue
* do not unpause queue when seen items are pushed
* re-add seen item back to queue once paused
Previously, when a seen item was processed, it was first popped off the queue, then the queue was paused waiting to process that item once the queue was unpaused. Now, when a seen item is processed, it is popped off the queue, the queue is paused, then the item is re-added to the queue and the queue will wait until unpaused before it will continue popping items off the queue. If the item was not re-added to the queue, it would have been processed immediately once unpaused, however there may have been other items with higher priority pushed to the queue in the meantime. The queue would not be unpaused if those added items were already seen. In particular, this may happen when ignored items due to lack of availability are re-added to a paused queue. Those ignored items will likely have a higher priority than the item that was just seen (due to it having been processed first), causing the queue to the be paused.
* address PR comments
* integration: move REST API tests into their own module
* integration: move upload and download tests into their own module
* integration: move purchasing tests into their own module
* integration: move marketplace tests to the right module
* integration: mine a block *after* starting nodes
To ensure that tests involving multiple nodes do
not start with out-of-sync clocks
* Fix: do not swallow CancellationErrors
* integration: avoid underflow in UInt256
* network: remove unnecessary error handling
No Exceptions can occur, only Defects, because everything
is asyncSpawned.
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
* network: do not raise in asyncSpawned proc
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
---------
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
* Updates logging for file upload
* Restores trace for placing block and proof in repo store
* Reduces logging while transmitting blocks
* unnecessary formatter
* Clean up some more download related traces
* much better
* Review comment by dryajov
* fix: make sure requestStorage is mined
* fix: correct place to plug confirm
* test: fixing contracts tests
* feat: one confirmation for all transactions
* fix: don't wait for confirmations only mined block
* Aligns response with openapi spec
* Fixes json serialization for local content response
* Review comments by Mark
* Fixes missed rename
* Removes array type for dataList
Eric
Arnaud
Giuliano Mega
Slava
Tomasz Bekas
Adam Uhlíř
Ben Bierens
Jaremy Creechley
markspanbroek