* initial implementation of repo store
* allow isManifest on multicodec
* rework with new blockstore
* add raw codec
* rework listBlocks
* remove fsstore
* reworking with repostore
* bump datastore
* fix listBlocks iterator
* adding store's common tests
* run common store tests
* remove fsstore backend tests
* bump datastore
* add `listBlocks` tests
* listBlocks filter based on block type
* disabling tests in need of rewriting
* allow passing block type
* move BlockNotFoundError definition
* fix tests
* increase default advertise loop sleep to 10 mins
* use `self`
* add cache quota functionality
* pass meta store and start repo
* add `CacheQuotaNamespace`
* pass meta store
* bump datastore to latest master
* don't use os `/` as key separator
* Added quota limits support
* tests for quota limits
* add block expiration key
* remove unnesesary space
* use idleAsync in listBlocks
* proper test name
* re-add contrlC try/except
* add storage quota and block ttl config options
* clarify comments
* change expires key format
* check for block presence before storing
* bump datastore
* use dht with fixed datastore `has`
* bump datastore to latest master
* bump dht to latest master
* attach `wait` to handle fut
* fix crash when can't find a CID and timeout hits
the exception was not raised, thus a Nil was returned that was
than wrapped in an option, leading to crash.
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
* add inFlight flag
* adding pending blocks tests
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
Co-authored-by: Csaba Kiraly <csaba.kiraly@gmail.com>
* [purchasing] Simplify test
* [utils] Move StorageRequest.example up one level
* [purchasing] Load purchases from market
* [purchasing] load purchase states
* Implement myRequest() and getState() methods for OnChainMarket
* [proofs] Fix intermittently failing tests
Ensures that examples of proofs in tests are never of length 0;
these are considered invalid proofs by the smart contract logic.
* [contracts] Fix failing test
With the new solidity contracts update, a contract can only
be paid out after it started.
* [market] Add method to get request end time
* [purchasing] wait until purchase is finished
Purchase.wait() would previously wait until purchase
was started, now we wait until it is finished.
* [purchasing] Handle 'finished' and 'failed' states
* [marketplace] move to failed state once request fails
- Add support for subscribing to request failure events.
- Add supporting contract tests for subscribing to request failure events.
- Allow the PurchaseStarted state to move to PurchaseFailure once a request failure event is emitted
- Add supporting tests for moving from PurchaseStarted to PurchaseFailure
- Add state transition tests for PurchaseUnknown.
* [marketplace] Fix test with longer sleepAsync
* [integration] Add function to restart a codex node
* [purchasing] Set client address before requesting storage
To prevent the purchase id (which equals the request id)
from changing once it's been submitted.
* [contracts] Fix: OnChainMarket.getState()
Had the wrong method signature before
* [purchasing] Load purchases on node start
* [purchasing] Rename state 'PurchaseError' to 'PurchaseErrored'
Allows for an exception type called 'PurchaseError'
* [purchasing] Load purchases in background
No longer calls market.getRequest() for every purchase
on node start.
* [contracts] Add `$` for RequestId, SlotId and Nonce
To aid with debugging
* [purchasing] Add Purchasing.stop()
To ensure that all contract interactions have both a
start() and a stop() for
* [tests] Remove sleepAsync where possible
Use `eventually` loop instead, to make sure that we're
not waiting unnecessarily.
* [integration] Fix: handle non-json response in test
* [purchasing] Add purchase state to json
* [integration] Ensure that purchase is submitted before restart
Fixes test failure on slower CI
* [purchasing] re-implement `description` as method
Allows description to be set in the same module where the
state type is defined.
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
* [contracts] fix typo
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
* [market] Use more generic error type
Should we decide to change the provider type later
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
* setup and persist private key
* return dht record spr
* helper to remap multiaddr ip and port
* set/update discovery and announce addrs
* add nat and discovery IPs
* allow for announce and DHT addresses separatelly
* update tests
* check for nat or discoveryIp
* fix integration tests
* misc align
* don't share data dirs and and set bootstrap node
* add log scope
* remap announceAddrs after node start
* simplify discovery initialization
* make nat and disc-ip required
* add log scope don't init dht spr in constructor
* bump dht
* dissallow `0.0.0.0` for `--nat`
* add format for cid
* cid formatIt change
* track nim-libp2p-unstable
* rework probuf serialization for por
* add missing include
* removing nim protobuf serialization
* rollback to dht to main
* remove protobuf serialization import
1. use `ProviderError` to catch contract errors instaed of `JsonRpcProviderError`.
2. Add tests for `isProofRequired`, `willProofBeRequired`, and `getProofEnd` when slot is empty (and is call is reverted)
3. bump nim-ethers to branch that has new `reverts` API and `ProviderError` support.
4. Update to new `reverts` API.
Support storage contract parameters to free slots after too many proofs missed, as well as fail a contract after too many slots have been freed.
# Conflicts:
# vendor/dagger-contracts
Remove `isSlotCancelled` and `isCancelled` from proving, as it did not fit in the module.
Update the proving module to not rely on checking the contract to understand if a request was cancelled. Instead, `proofEnd` was modified in `dagger-contracts` such that it returns a past timestamp when the contract is cancelled. This successfully removes
- remove RequestState as no longer being used
- remove RequestState check on purchase timeout
- add tests for `withdrawFunds` and `subscribeRequestCancelled`
- update `dagger-contracts` so that `RequestCancelled.requestId` property is indexed in the event
- rename `ContractId` to `SlotId`
- rename `Proving.contracts` to `Proving.slots`
- change signatures of `isSlotCancelled` and `isCancelled` to use `SlotId` and `RequestId` types, respectively.
- change all references to `RequestId`, `SlotId`
Add or remove proof requirements when a request contract’s state changes. When a request sale has completed (for a slot), the host who purchased that slot now must provide regular proofs for the data they are contracted to hold. This is now enforced by adding the slotId to the HashSet of Ids for which to require proofs. When a request has been cancelled (not all slots were filled before the request expired), proofs no longer need to be provided and the slotId is removed from teh HashSet.
Add `isCancelled` and `isSlotCancelled` checks to query the contract state without relying the on the state context variable in the contract. Because contract state can only be updated in a transaction, and the client withdrawing funds is responsible for changing the contract state to “Cancelled”, the `isCancelled` and `isSlotCancelled` functions were introduced to check the state regardless of whether or not the client had already withdrawn their funds.
When a request for storage times out (not enough slots filled), the client will initiate a withdraw request to retrieve its funds out of the contract, setting the state of the request to RequestState.Cancelled. The client will also emit a RequestCancelled event for others to listen to (ie hosts will need to listen for this event to withdraw its collateral).
Add unit test that checks for emission of RequestCancelled after request is purchased request expires.
Update dagger-contracts dependency to commit that holds the changes supporting withdrawing of funds.
Increase blocksize from ~8KiB to ~64KiB and remove useless unbounded prefetching on download
* increased upload/download speed on my Win10 Zen3 box to 50/150 MB/s (with FSStore)
* made manifest files 8x smaller
* two more changes in the tests to make them pass
Closes#263
* upgrading nimbus build system to latest
* use new `updateRecord`
* bump nim-libp2p-dht
* bump nimbus build system
* bumping deps
* update to latest dht
Make RequestId, SlotId, Nonce, PurcahseId distinct types.
Add/modify conversions to support the distinct type (ABI encoding/decoding, JSON encoding, REST decoding).
Update tests
- rename `ContractId` to `SlotId`
- add `RequestId`, `PurchaseId`, `Nonce` types as aliases of `array[32, byte]`
- rename `Proving.contracts` to `Proving.slots`
- change signatures of `isSlotCancelled` and `isCancelled` to use `SlotId` and `RequestId` types, respectively.
- change all references to `RequestId`, `SlotId`, and `PurchaseId`
The initial goal of this patch was to allow to download of a file via REST API in exactly the same size as it was uploaded, which required adding fields Chunker.offset and Manifest.originalBytes to keep that size. On top of that, we added more integrity checks to operations on Manifest, and reorganized TestNode.nim to test the actual interaction between node.store and node.retrieve operations.
Note that the wire format of Manifest was changed, so we need to recreate all BlockStores.
* Download without padding
* Fixed chunker tests
* Chunker: get rid of RabinChunker
* Verify offset in the chunker tests
* Use manifest.originalBytesPadded in StoreStream.size
* StoreStream: replace emptyBlock with zeroMem
* Manifest.bytes: compute how many bytes corresponding StoreStream(Manifest, pad) will return
* Manifest: verify originalBytes and originalLen on new/encode/decode
Also set originalBytes in each Manifest creation/update scenario
* Manifest: comments, split code into sections
* Reordered parameters to deal with int64 size in 32-bit builds
* TestNode.nim: combine Store and Retrieve tests
1. Instead of copy-pasting code from node.nim, new test calls node.store() and node.retrieve() in order to check that they can correctly store and then retrieve data
2. New test compares only file contents, manifest contents considered an implementation detail
3. New test chunks at odd chunkSize=BlockSize/1.618 in order to ensure that data retrieved correctly even when buffer sizes mismatch
* TestNode.nim: code refactoring
* Manifest.add: one more test
* Manifest.verify: return Result instead of raising Defect
* Node.store: added blockSize parameter
change return type for `method getBlock` from `Future[?!(?Block)]` to
`Future[?!Block]`
use `type BlockNotFoundError = object of CodexError` to differentiate between
"block not found in the store" and other errors
also make some logic and error handling/messages more consistent across
BlockStore implementations
closes#177closes#182closes#210
alternative to #205, #209
ECC decoding can be done with any encoded.K blocks, and now we start it once any K blocks arrived.
Previous code essentially waited for all blocks to arrive (or fail to arrive), that may mean unnecessary delays.
New code will sometimes ECC-decode when parity blocks were retrieved faster that data blocks (instead of waiting for data blocks from network), but we expect that it's OK due to extremely fast Leopard decoding (~~500 MB/s)
* [contracts] ContractInteractions.new() now requires account parameter
* [cli] Only perform Ethereum interactions when --eth-account is specified
* [cli] Add --persistence option that is disabled by default
* [cli] Use Option for ethDeployment parameter
* [node] Better error reporting when Ethereum node cannot be reached
also fix a discrepancy where cli option `--cache-size` is documented as
`0 disables the cache` in `codex/conf.nim`, but previously the value `0` would
result in a cache being constructed with default parameter values for
`CacheStore.new()`
Closes#180
BlockStore API got new return types (rationale in https://github.com/status-im/nim-codex/issues/123#issuecomment-1163797753):
- getBlock: Future[?! (?Block)]
- putBlock/delBlock/listBlocks: Future[?!void]
- hasBlock: Future[?!bool]
Plus refactored readOnce(StoreStream) and check received data in its tests.
And replaced local use of AsyncHeapQueue with seq.sort.
As discussed in team:
- Retrieving of blocks that are mentioned in the
manifest should be moved into a BlockStore.
- Sales should be given an instance of BlockStore,
and an instance of StorageProofs, instead of the
onStore, onClear, and onProve callbacks
Reason: with the new design there are no offers and
selection, making this less like a negotation, and
more like an agent trying to fulfill a storage request
* initial implementation of storage proofs upload
* make sure proof verifies with after deserializing
* add por store
* rename por store to stp store
* rename porstore to stpstore
* add support for host discovery to discovery mock
* add tags upload network tests
Dmitriy Ryajov
markspanbroek
Michael Bradley
Eric Mastro
Bulat-Ziganshin