* Adds endpoint for listing files (manifests) in node. Useful for demo UI.
* Moves upload/download/files into content API calls.
* Cleans up json serialization for manifest
* Cleans up some more json serialization
* Moves block iteration and decoding to node.nim
* Moves api methods into their own init procs.
* Applies RestContent api object.
* Replaces format methods with Rest objects in json.nim
* Unused import
* Review comments by Adam
* Fixes issue where content/local endpoint clashes with content/cid.
* faulty merge resolution
* Renames content API to data.
* Fixes faulty rebase
* Adds test for data/local API
* Renames local and download api.
## Problem
When Availabilities are created, the amount of bytes in the Availability are reserved in the repo, so those bytes on disk cannot be written to otherwise. When a request for storage is received by a node, if a previously created Availability is matched, an attempt will be made to fill a slot in the request (more accurately, the request's slots are added to the SlotQueue, and eventually those slots will be processed). During download, bytes that were reserved for the Availability were released (as they were written to disk). To prevent more bytes from being released than were reserved in the Availability, the Availability was marked as used during the download, so that no other requests would match the Availability, and therefore no new downloads (and byte releases) would begin. The unfortunate downside to this, is that the number of Availabilities a node has determines the download concurrency capacity. If, for example, a node creates a single Availability that covers all available disk space the operator is willing to use, that single Availability would mean that only one download could occur at a time, meaning the node could potentially miss out on storage opportunities.
## Solution
To alleviate the concurrency issue, each time a slot is processed, a Reservation is created, which takes size (aka reserved bytes) away from the Availability and stores them in the Reservation object. This can be done as many times as needed as long as there are enough bytes remaining in the Availability. Therefore, concurrent downloads are no longer limited by the number of Availabilities. Instead, they would more likely be limited to the SlotQueue's `maxWorkers`.
From a database design perspective, an Availability has zero or more Reservations.
Reservations are persisted in the RepoStore's metadata, along with Availabilities. The metadata store key path for Reservations is ` meta / sales / reservations / <availabilityId> / <reservationId>`, while Availabilities are stored one level up, eg `meta / sales / reservations / <availabilityId> `, allowing all Reservations for an Availability to be queried (this is not currently needed, but may be useful when work to restore Availability size is implemented, more on this later).
### Lifecycle
When a reservation is created, its size is deducted from the Availability, and when a reservation is deleted, any remaining size (bytes not written to disk) is returned to the Availability. If the request finishes, is cancelled (expired), or an error occurs, the Reservation is deleted (and any undownloaded bytes returned to the Availability). In addition, when the Sales module starts, any Reservations that are not actively being used in a filled slot, are deleted.
Having a Reservation persisted until after a storage request is completed, will allow for the originally set Availability size to be reclaimed once a request contract has been completed. This is a feature that is yet to be implemented, however the work in this PR is a step in the direction towards enabling this.
### Unknowns
Reservation size is determined by the `StorageAsk.slotSize`. If during download, more bytes than `slotSize` are attempted to be downloaded than this, then the Reservation update will fail, and the state machine will move to a `SaleErrored` state, deleting the Reservation. This will likely prevent the slot from being filled.
### Notes
Based on #514
* cleanup erasure coding
* moar cleanup
* fix off by 1 issues in tests
* style
* consolidate decoding data code
* simplify tuple unpacking
* fix retrieve purchase
We don't support single blocks for now
* Apply suggestions from code review
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
Signed-off-by: Dmitriy Ryajov <dryajov@gmail.com>
---------
Signed-off-by: Dmitriy Ryajov <dryajov@gmail.com>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
* [docs] fix two client scenario: add missing collateral
* [integration] separate step to wait for node to be started
* [cli] add option to specify ethereum private key
* Remove unused imports
* Fix warnings
* [integration] move type definitions to correct place
* [integration] wait a bit longer for a node to start in debug mode
When e.g. running against Taiko test net rpc, the node start
takes longer
* [integration] simplify handling of codex node and client
* [integration] add Taiko integration test
* [contracts] await token approval confirmation before next tx
* [contracts] deployment address of marketplace on Taiko
* [cli] --eth-private-key now takes a file name
Instead of supplying the private key on the command line,
expect the private key to be in a file with the correct
permissions.
* [utils] Fixes undeclared `activeChroniclesStream` on Windows
* [build] update nim-ethers to include PR #52
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
* [cli] Better error messages when reading eth private key
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
* [integration] simplify reading of cmd line arguments
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
* [build] update to latest version of nim-ethers
* [contracts] updated contract address for Taiko L2
* [build] update codex contracts to latest version
---------
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
* [sales] remove availability check before adding to slot queue
* [sales] add missing return statement
* [tests] remove 'eventuallyCheck' helper
* [sales] remove reservations from slot queue
* [tests] rename module `eventually` -> `always`
* [sales] increase slot queue size
Because it will now also hold items for which we haven't
checked availability yet.
* Improve integration testing client (CodexClient) and json serialization
The current client used for integration testing against the REST endpoints for Codex accepts and passes primitive types. This caused a hard to diagnose bug where a `uint` was not being deserialized correctly.
In addition, the json de/serializing done between the CodexClient and REST client was not easy to read and was not tested.
These changes bring non-primitive types to most of the CodexClient functions, allowing us to lean on the compiler to ensure we're providing correct typings. More importantly, a json de/serialization util was created as a drop-in replacement for the std/json lib, with the main two differences being that field serialization is opt-in (instead of opt-out as in the case of json_serialization) and serialization errors are captured and logged, making debugging serialization issues much easier.
* Update integration test to use nodes=2 and tolerance=1
* clean up
* logs
* poll log
* more logs
* time log for upload steps
* Adds metric and log for block retrieval time
* adds some logging
* move logging for storestream block indices
* Log at start and end of block iteration
* applies branch blockstore-bugfix
* Cleanup
* Cleanup
Add handling of empty blocks in the RepoStore.
* Add empty block handling to repostore for put, del, has
Also added tests for all empty block handling blockstore operations. This showed there was an ambiguous identifier present for `hasBlock`, so one of the two `hasBlock` definitions was removed in `repostore`.
* Change CacheStore to RepoStore in testerasure
As CacheStore is not used in the node, update the Datastore used in the erasure coding tests to be a RepoStore. This ensures that the K > 1 cases are being tested, where they will produce empty padding blocks in the erasure-coded manifests.
* extra utilities and tweaks
* add atlas lock
* update ignores
* break build into it's own script
* update url rules
* base off codexdht's
* compile fixes for Nim 1.6.14
* update submodules
* convert mapFailure to procs to work around type resolution issues
* add toml parser for multiaddress
* change error type on keyutils
* bump nimbus build to use 1.6.14
* update gitignore
* adding new deps submodules
* bump nim ci version
* even more fixes
* more libp2p changes
* update keys
* fix eventually function
* adding coverage test file
* move coverage to build.nims
* use nimcache/coverage
* move libp2p import for tests into helper.nim
* remove named bin
* bug fixes for networkpeers (from Dmitriy)
---------
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
