Due to a bug in hardhat, the latest block timestamp was out of sync from the newHeads block timestamp which was causing some timing issues in the tests. To fix this, a compile flag, `codex_testing` (which was changed from `codex_enable_proof_failures`) is used to determine the bevhaviour of clock.now. If `codex_testing` is true, clock.now uses the latest block timestamp, else it uses the cached block.timestamp that was populated in the `newHeads` event.
In proving.onCancelled, the `waitFor state.loop.cancelAndWait()` was never completing. Turns out this was not needed, because when changing states, the current state's run is cancelled, which automatically cancels the state prove loop, because it is a child of proving.run. Therefore, the logic to cancelAndWait the prove loop was removed as it was not needed.
- advance to next period and mine every 500ms
- update eventually checks in tests to check for number of periods, not time
- change log file format to include test name
- works with hardhat already running
- works with ethersuite to revert snapshot (ethProvider instead of provider)
- OnChainClock.now() gets latest block timestamp
Support logging to file
Log the entire config and fix build error
Process slot queue on reservation callback onMarkUnused
Add Reservation object, rename reserve > create
refactor Reservations api to include Reservation CRUD
All tests that use the Reservation module updated
- add requestId and slotIndex to Reservation (hopefully these will prove to be useful when we persist Reservations until request are completed, to add back bytes to Availability)
- add querying of all reservations, with accompanying tests
- change from find to findAvailabilities
- move onCleanUp from SalesContext to SalesAgent as it was getting overwritten for each slot processed
- remove sales agent AFTER deleting reservation, as this was causing some SIGSEGVs
- retrofit testsales and testslotqueue to match updated Reservations module API
Add deletion of inactive reservations on sales load
clean up
add exception message detail util
Apply to onStore errors as we are seeing undetailed errors in the dist tests logs
add missing file
change slotsize to reflect current implemenation
Fix slotSize to reduce by one block
Revert change to slotSize that reduces it by one block
Add additional test check for querying reservations/availabilities
filter past requests based on availability
Because availability filtering on push was removed, when availability is added and past storage request events are queried, those requests need to be filtered by availability before being added to the queue.
Revert "filter past requests based on availability"
This reverts commit 0c2362658b523e0de425794b1fa30ebd53bd30ae.
Add debugging for dist tests
Add cancel on error during filling state
When calling fillSlot, any transaction errors that occur (possibly during estimate gas) will cause that tx to be replaced with a cancellation transaction (a 0-valued tx to ourselves).
more debug logging
fix build
wait for fillSlot to be mined (so that a revert error can be extracted)
fix confirmation of fillSlot
add more debugging
moar debugging
debugging: change echo to trace
bump ethers to add chronicles
fix contracts tests
switch to earlier nim-ethers commit
bump json-rpc and nim-ethers
bump nim-ethers to prevent parsing newHeads log from crashing sales state machine
moar debugging
moar debugging
moar debugging
bump ethers to fix "key not found: data" error
ethers debug logging
bump ethers - better Transaction object deserialization
bump ethers to replay tx with past tx format
bump ethers to add serialization for PastTransaction
ethers bump: better logging, separate logic for revert reason string
fix build
ethers: update revert reason retreival
remove unneeded confirm
ethers: try replay without decrementing blockNumber
ethers: include type and chainId in replayed txs
ethers: add gas into and remove type from replayed txs
ensure gas is being serialized in Transaction
ethers: fix build error
bump ethers: rebased on top of cancel tx due to estimateGas error PR
Fix proving with latest ethers bump (tested on master)
Update multinode suite for better simulateFailedProofs enabling, add proofs test
Improve multinode suite for better debug options, including logging to file
There is a 503 "sales unavailable" error
improve multinode test suite, add logging to file
In proving.onCancelled, the `waitFor state.loop.cancelAndWait()` was never completing. Turns out this was not needed, because when changing states, the current state's run is cancelled, which automatically cancels the state prove loop, because it is a child of proving.run. Therefore, the logic to cancelAndWait the prove loop was removed as it was not needed.
* Fix REST endpoints semantics
* update endpoint description
* update, operation id
* Adding enum support
* make enum descerializer public
* add support for listing manifests
* test `/data` endpoint to list local manifests
* debug leftovers
* remove commented out line
* use str on JString types, `$` will preserve `"`
* Adding enum support
* deserialize cid test
* make enum descerializer public
* unify fromJson for objects and refs
* add enum descerialization testing
* Blockexchange uses merkle root and index to fetch blocks
* Links the network store getTree to the local store.
* Update codex/stores/repostore.nim
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Signed-off-by: Tomasz Bekas <tomasz.bekas@gmail.com>
* Rework erasure.nim to include recent cleanup
* Revert accidential changes to lib versions
* Addressing review comments
* Storing proofs instead of trees
* Fix a comment
* Fix broken tests
* Fix for broken testerasure.nim
* Addressing PR comments
---------
Signed-off-by: Tomasz Bekas <tomasz.bekas@gmail.com>
Co-authored-by: benbierens <thatbenbierens@gmail.com>
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
* Adds endpoint for listing files (manifests) in node. Useful for demo UI.
* Moves upload/download/files into content API calls.
* Cleans up json serialization for manifest
* Cleans up some more json serialization
* Moves block iteration and decoding to node.nim
* Moves api methods into their own init procs.
* Applies RestContent api object.
* Replaces format methods with Rest objects in json.nim
* Unused import
* Review comments by Adam
* Fixes issue where content/local endpoint clashes with content/cid.
* faulty merge resolution
* Renames content API to data.
* Fixes faulty rebase
* Adds test for data/local API
* Renames local and download api.
## Problem
When Availabilities are created, the amount of bytes in the Availability are reserved in the repo, so those bytes on disk cannot be written to otherwise. When a request for storage is received by a node, if a previously created Availability is matched, an attempt will be made to fill a slot in the request (more accurately, the request's slots are added to the SlotQueue, and eventually those slots will be processed). During download, bytes that were reserved for the Availability were released (as they were written to disk). To prevent more bytes from being released than were reserved in the Availability, the Availability was marked as used during the download, so that no other requests would match the Availability, and therefore no new downloads (and byte releases) would begin. The unfortunate downside to this, is that the number of Availabilities a node has determines the download concurrency capacity. If, for example, a node creates a single Availability that covers all available disk space the operator is willing to use, that single Availability would mean that only one download could occur at a time, meaning the node could potentially miss out on storage opportunities.
## Solution
To alleviate the concurrency issue, each time a slot is processed, a Reservation is created, which takes size (aka reserved bytes) away from the Availability and stores them in the Reservation object. This can be done as many times as needed as long as there are enough bytes remaining in the Availability. Therefore, concurrent downloads are no longer limited by the number of Availabilities. Instead, they would more likely be limited to the SlotQueue's `maxWorkers`.
From a database design perspective, an Availability has zero or more Reservations.
Reservations are persisted in the RepoStore's metadata, along with Availabilities. The metadata store key path for Reservations is ` meta / sales / reservations / <availabilityId> / <reservationId>`, while Availabilities are stored one level up, eg `meta / sales / reservations / <availabilityId> `, allowing all Reservations for an Availability to be queried (this is not currently needed, but may be useful when work to restore Availability size is implemented, more on this later).
### Lifecycle
When a reservation is created, its size is deducted from the Availability, and when a reservation is deleted, any remaining size (bytes not written to disk) is returned to the Availability. If the request finishes, is cancelled (expired), or an error occurs, the Reservation is deleted (and any undownloaded bytes returned to the Availability). In addition, when the Sales module starts, any Reservations that are not actively being used in a filled slot, are deleted.
Having a Reservation persisted until after a storage request is completed, will allow for the originally set Availability size to be reclaimed once a request contract has been completed. This is a feature that is yet to be implemented, however the work in this PR is a step in the direction towards enabling this.
### Unknowns
Reservation size is determined by the `StorageAsk.slotSize`. If during download, more bytes than `slotSize` are attempted to be downloaded than this, then the Reservation update will fail, and the state machine will move to a `SaleErrored` state, deleting the Reservation. This will likely prevent the slot from being filled.
### Notes
Based on #514
* cleanup erasure coding
* moar cleanup
* fix off by 1 issues in tests
* style
* consolidate decoding data code
* simplify tuple unpacking
* fix retrieve purchase
We don't support single blocks for now
* Apply suggestions from code review
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
Signed-off-by: Dmitriy Ryajov <dryajov@gmail.com>
---------
Signed-off-by: Dmitriy Ryajov <dryajov@gmail.com>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
