Commit Graph

282 Commits

Author SHA1 Message Date
Eric Mastro 8dd110765f [marketplace] Catch slot empty, return default values 2022-10-25 15:10:35 +11:00
Eric Mastro 9d218c88a5 [marketplace] Remove `isSlotCancelled` and `isCancelled`
Remove `isSlotCancelled` and `isCancelled` from proving, as it did not fit in the module.

Update the proving module to not rely on checking the contract to understand if a request was cancelled. Instead, `proofEnd` was modified in `dagger-contracts` such that it returns a past timestamp when the contract is cancelled. This successfully removes
2022-10-25 15:10:35 +11:00
Eric Mastro b4a14e00f7 [marketplace] address more PR comments
- remove RequestState as no longer being used
- remove RequestState check on purchase timeout
- add tests for `withdrawFunds` and `subscribeRequestCancelled`
- update `dagger-contracts` so that `RequestCancelled.requestId` property is indexed in the event
2022-10-25 15:10:35 +11:00
Eric Mastro 5cb74b6638 [chore] additional clean up of `array[32, byte]` types
- rename `ContractId` to `SlotId`
- rename `Proving.contracts` to `Proving.slots`
- change signatures of `isSlotCancelled` and `isCancelled` to use `SlotId` and `RequestId` types, respectively.
- change all references to `RequestId`, `SlotId`
2022-10-25 15:10:35 +11:00
Eric Mastro 372f827982 [marketplace] add/remove proofs for contract state
Add or remove proof requirements when a request contract’s state changes. When a request sale has completed (for a slot), the host who purchased that slot now must provide regular proofs for the data they are contracted to hold. This is now enforced by adding the slotId to the HashSet of Ids for which to require proofs. When a request has been cancelled (not all slots were filled before the request expired), proofs no longer need to be provided and the slotId is removed from teh HashSet.

Add `isCancelled` and `isSlotCancelled` checks to query the contract state without relying the on the state context variable in the contract. Because contract state can only be updated in a transaction, and the client withdrawing funds is responsible for changing the contract state to “Cancelled”, the `isCancelled` and `isSlotCancelled` functions were introduced to check the state regardless of whether or not the client had already withdrawn their funds.
2022-10-25 15:10:35 +11:00
Eric Mastro 0c3fbad470 [purchasing] Withdraw funds when request times out
When a request for storage times out (not enough slots filled), the client will initiate a withdraw request to retrieve its funds out of the contract, setting the state of the request to RequestState.Cancelled. The client will also emit a RequestCancelled event for others to listen to (ie hosts will need to listen for this event to withdraw its collateral).

Add unit test that checks for emission of RequestCancelled after request is purchased request expires.

Update dagger-contracts dependency to commit that holds the changes supporting withdrawing of funds.
2022-10-25 15:10:35 +11:00
Bulat-Ziganshin 9939d85b74
Replace protobuf serialization for Block exchange with minprotobuf (#271)
* minprotobuf serialization for Block exchange

* Handle decoding errors by stopping peer connection
2022-10-13 18:58:57 -05:00
Bulat-Ziganshin 6e6f40016c
Changes required for reasonable upload/download speed (#265)
Increase blocksize from ~8KiB to ~64KiB and remove useless unbounded prefetching on download
* increased upload/download speed on my Win10 Zen3 box to 50/150 MB/s (with FSStore)
* made manifest files 8x smaller
* two more changes in the tests to make them pass
Closes #263
2022-10-06 23:01:25 +03:00
Dmitriy Ryajov 106b04340a
Remove local providers (#268)
* add remove provider method

* remove failed providers

* bumping dht to latest master
2022-10-05 10:01:21 -06:00
Mark Spanbroek 58f0439f37 [contracts] Use Duration instead of float for clock offset
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
2022-10-03 15:17:03 +02:00
Mark Spanbroek 4e6d84850a [contracts] Fix intermittently failing clock test
Use a sub-second offset to avoid off-by-one errors in the
test at the second boundary.

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
2022-10-03 15:17:03 +02:00
Dmitriy Ryajov 8d315b38b8
Dht maintenance (#267)
* protocol start is now async

* bumping to maintenance branch
2022-10-01 11:08:44 -06:00
Dmitriy Ryajov 69bd359287
Persist Dht providers (#257)
* providers store integration
2022-09-29 20:16:59 -06:00
Dmitriy Ryajov 7e2f917c84
Dht providers lru (#247)
* upgrading nimbus build system to latest

* use new `updateRecord`

* bump nim-libp2p-dht

* bump nimbus build system

* bumping deps

* update to latest dht
2022-09-13 13:37:49 -06:00
Bulat-Ziganshin df1522c9f8
Fixed message.proto to match IPFS specs (#245)
We incorrectly assigned code 2 to the payload field instead of code 3:

Our code: 8c59392526/codex/blockexchange/protobuf/message.proto (L50-L52)

While original code: 0fa397581c/message/pb/message.proto (L43-L45)

And [documentation](https://github.com/ipfs/specs/blob/main/BITSWAP.md#bitswap-120-wire-format).
2022-09-12 21:32:12 -06:00
Eric Mastro 58f3335cc1 [fix] Fix C-compilation error failing tests 2022-08-26 13:29:09 +10:00
Eric Mastro 8f116cd01e [chore] make RequestId, SlotId, Nonce, PurcahseId distinct
Make RequestId, SlotId, Nonce, PurcahseId distinct types.

Add/modify conversions to support the distinct type (ABI encoding/decoding, JSON encoding, REST decoding).

Update tests
2022-08-26 13:29:09 +10:00
Eric Mastro 6df5a7cf54 [chore] clean up `array[32, byte]` types
- rename `ContractId` to `SlotId`
- add `RequestId`, `PurchaseId`, `Nonce` types as aliases of `array[32, byte]`
- rename `Proving.contracts` to `Proving.slots`
- change signatures of `isSlotCancelled` and `isCancelled` to use `SlotId` and `RequestId` types, respectively.
- change all references to `RequestId`, `SlotId`, and `PurchaseId`
2022-08-26 13:29:09 +10:00
Bulat-Ziganshin f24ded0f76
Download files without padding (#218)
The initial goal of this patch was to allow to download of a file via REST API in exactly the same size as it was uploaded, which required adding fields Chunker.offset and Manifest.originalBytes to keep that size. On top of that, we added more integrity checks to operations on Manifest, and reorganized TestNode.nim to test the actual interaction between node.store and node.retrieve operations.

Note that the wire format of Manifest was changed, so we need to recreate all BlockStores.

* Download without padding
* Fixed chunker tests
* Chunker: get rid of RabinChunker
* Verify offset in the chunker tests
* Use manifest.originalBytesPadded in StoreStream.size
* StoreStream: replace emptyBlock with zeroMem
* Manifest.bytes: compute how many bytes corresponding StoreStream(Manifest, pad) will return
* Manifest: verify originalBytes and originalLen on new/encode/decode
Also set originalBytes in each Manifest creation/update scenario
* Manifest: comments, split code into sections
* Reordered parameters to deal with int64 size in 32-bit builds
* TestNode.nim: combine Store and Retrieve tests
1. Instead of copy-pasting code from node.nim, new test calls node.store() and node.retrieve() in order to check that they can correctly store and then retrieve data
2. New test compares only file contents, manifest contents considered an implementation detail
3. New test chunks at odd chunkSize=BlockSize/1.618 in order to ensure that data retrieved correctly even when buffer sizes mismatch
* TestNode.nim: code refactoring
* Manifest.add: one more test
* Manifest.verify: return Result instead of raising Defect
* Node.store: added blockSize parameter
2022-08-24 15:15:59 +03:00
Dmitriy Ryajov 4bc701652f
Adding metrics (#203)
* adding inflight discovery metric

* adding metrics instructions and default dashboard

* spelling fixes
2022-08-23 10:11:21 -06:00
Michael Bradley, Jr 3d823dcbc6 [stores] update getBlock return type
change return type for `method getBlock` from `Future[?!(?Block)]` to
`Future[?!Block]`

use `type BlockNotFoundError = object of CodexError` to differentiate between
"block not found in the store" and other errors

also make some logic and error handling/messages more consistent across
BlockStore implementations

closes #177
closes #182
closes #210

alternative to #205, #209
2022-08-22 20:52:51 -05:00
Bulat-Ziganshin 7ec039f822
Quickfix: start erasure decoding once any K blocks arrived (#199)
ECC decoding can be done with any encoded.K blocks, and now we start it once any K blocks arrived.

Previous code essentially waited for all blocks to arrive (or fail to arrive), that may mean unnecessary delays.

New code will sometimes ECC-decode when parity blocks were retrieved faster that data blocks (instead of waiting for data blocks from network), but we expect that it's OK due to extremely fast Leopard decoding (~~500 MB/s)
2022-08-19 21:21:55 -06:00
Michael Bradley, Jr 5aa42541b3 [build] update nim-datastore
also make changes to `codex/codex.nim` and tests for SQLiteStore re: changes in
nim-datastore

closes #178
2022-08-10 13:13:07 -05:00
markspanbroek 097a4cfd67
Better command line options for Ethereum (#181)
* [contracts] ContractInteractions.new() now requires account parameter

* [cli] Only perform Ethereum interactions when --eth-account is specified

* [cli] Add --persistence option that is disabled by default

* [cli] Use Option for ethDeployment parameter

* [node] Better error reporting when Ethereum node cannot be reached
2022-08-09 06:29:06 +02:00
Michael Bradley, Jr eebcf7e7b2 [node] custom iterator/query for SQLiteStore's `method listBlocks` 2022-08-08 17:48:37 -05:00
Michael Bradley, Jr 2ecf750959 [node] support self.cache=nil in SQLiteStore
also fix a discrepancy where cli option `--cache-size` is documented as
`0 disables the cache` in `codex/conf.nim`, but previously the value `0` would
result in a cache being constructed with default parameter values for
`CacheStore.new()`

Closes #180
2022-08-08 16:24:45 -05:00
Mark Spanbroek 7f5310d640 [rest api] Allow user to specify amount of hosts 2022-08-08 14:00:55 +02:00
Mark Spanbroek fd6f8a1386 [sales] Pass along request and slot index in onProve callback 2022-08-08 14:00:55 +02:00
Mark Spanbroek cb5427ca95 [sales] Pass along request and slot index in onStore callback 2022-08-08 14:00:55 +02:00
Mark Spanbroek 3a3d083345 [contracts] Replace size by slotSize 2022-08-08 14:00:55 +02:00
Mark Spanbroek bf11ef5b55 [sales] Use correct slot price 2022-08-08 14:00:55 +02:00
Mark Spanbroek 057de4e928 [sales] Choose a slot index at random 2022-08-08 14:00:55 +02:00
Mark Spanbroek b4b6fdb7c6 [contracts] subscribe to SlotFilled event 2022-08-08 14:00:55 +02:00
Mark Spanbroek 7995fd71d5 [market] remove fulfillRequest()
It is superseded by fillSlot()
2022-08-08 14:00:55 +02:00
Mark Spanbroek 04e0c54e95 [market] Remove old getHost() overload
Is superseded by the new overload that includes
a slot id parameter.
2022-08-08 14:00:55 +02:00
Mark Spanbroek 698baeef35 [purchasing] Remove Purchase.selected
There no longer exists a single host that
is selected for a contract.
2022-08-08 14:00:55 +02:00
Mark Spanbroek dc95c863d2 [sales] replace fulfillRequest() by fillSlot() 2022-08-08 14:00:55 +02:00
Mark Spanbroek 156cd5ba73 [market] Replace fulfillRequest() by fillSlot() 2022-08-08 14:00:55 +02:00
Mark Spanbroek 67e4a28ed0 [contracts] Replace fulfillRequest() by fillSlot() 2022-08-08 14:00:55 +02:00
Mark Spanbroek 0494b3617b [contracts] replace totalNodes and nodeId by slots 2022-08-08 14:00:55 +02:00
Mark Spanbroek 360c03e7c4 [contracts] Rename: maxPrice -> reward 2022-08-08 14:00:55 +02:00
Michael Bradley, Jr 75666d01bf [node] add SQLiteStore backend
Closes #138
2022-08-08 02:12:43 -05:00
Dmitriy Ryajov 625facff4a
Better batching (#170)
* add cleaner batching

* pass block instead of cid

* cleanup manifest fetching

* bug: reference correct cid version

* add batch tests

* make sure batch is always <= batchSize
2022-07-29 14:04:12 -06:00
Dmitriy Ryajov 48368893c9
Add limits for inflight requests (#169)
* convert network to async

* use async network api

* test with async network

* test concurrent send limits
2022-07-29 10:19:34 -06:00
Dmitriy Ryajov 7ccde112f8
Contract store (#161)
* split out manifest and block retrieval

* adding test for `fetchManifest`

* raise exceptions in onStore
2022-07-28 11:44:59 -06:00
Bulat-Ziganshin 0bfe26440e
WIP: Harmonize BlockStore API (remaining functions) (#123) (#130)
BlockStore API got new return types (rationale in https://github.com/status-im/nim-codex/issues/123#issuecomment-1163797753):
- getBlock: Future[?! (?Block)]
- putBlock/delBlock/listBlocks: Future[?!void]
- hasBlock: Future[?!bool]

Plus refactored readOnce(StoreStream) and check received data in its tests.

And replaced local use of AsyncHeapQueue with seq.sort.
2022-07-28 03:39:17 +03:00
Mark Spanbroek afdb5be2d4 Add TODOs to indicate wanted architecture
As discussed in team:
- Retrieving of blocks that are mentioned in the
  manifest should be moved into a BlockStore.
- Sales should be given an instance of BlockStore,
  and an instance of StorageProofs, instead of the
  onStore, onClear, and onProve callbacks
2022-07-20 13:43:20 +02:00
Mark Spanbroek 97a5237d44 [sales] Add documentation 2022-07-20 13:43:20 +02:00
Mark Spanbroek 571aa6f694 [sales] Consistent names for callbacks 2022-07-20 13:43:20 +02:00
Mark Spanbroek 592a3dce4d [sales] Add callback for clearing storage when no longer needed 2022-07-20 13:43:20 +02:00
Mark Spanbroek adcb91a5d5 [node] Store datasets locally when sales asks for it 2022-07-20 13:43:20 +02:00
Mark Spanbroek 7bc5280596 [node] rename PrefetchBatch -> Batch
Reason: in a next commit we'll use the same
constant to fetch data in batches
2022-07-20 13:43:20 +02:00
Mark Spanbroek d4025f4a5f [manifest] Extract logic for decoding a manifest given a cid 2022-07-20 13:43:20 +02:00
Mark Spanbroek e06211f827 [sales] rename: negotiation -> sales agent
Reason: with the new design there are no offers and
selection, making this less like a negotation, and
more like an agent trying to fulfill a storage request
2022-07-20 13:43:20 +02:00
Mark Spanbroek 96ca1de768 Cleanup: remove StorageOffer everywhere 2022-07-20 13:43:20 +02:00
Mark Spanbroek bd2fba50c7 [purchasing] Fix integration test 2022-07-20 13:43:20 +02:00
Mark Spanbroek 4a4588e2a0 [node] Set callbacks for Sales to retrieve and prove posession of data 2022-07-20 13:43:20 +02:00
Mark Spanbroek 94f6838d19 [node] Fix: ensure that retrieve works for small files 2022-07-20 13:43:20 +02:00
Mark Spanbroek f14eeb688b [WIP sales] Handle request expiry 2022-07-20 13:43:20 +02:00
Mark Spanbroek 9438aba5d2 [WIP sales] Handle other host fulfilling request 2022-07-20 13:43:20 +02:00
Mark Spanbroek 7f864570bd [WIP sales] call onSale() 2022-07-20 13:43:20 +02:00
Mark Spanbroek a2570f9380 [WIP sales] fulfill request 2022-07-20 13:43:20 +02:00
Mark Spanbroek 5abd5dd674 [WIP sales] make storage available again upon failure 2022-07-20 13:43:20 +02:00
Mark Spanbroek dcb8f8a7f7 [WIP sales] Generate proof of storage 2022-07-20 13:43:20 +02:00
Mark Spanbroek 07dbb6ae18 [WIP sales] Retrieve data 2022-07-20 13:43:20 +02:00
Mark Spanbroek 3e6d51754d [WIP sales] Make tests pass by commenting out code 2022-07-20 13:43:20 +02:00
Mark Spanbroek 0636c94b74 [market] Add market.getSigner() 2022-07-20 13:43:20 +02:00
Mark Spanbroek 03b92a2067 [market] Add market.getHost() 2022-07-20 13:43:20 +02:00
Mark Spanbroek b414ecd67e [market] Add market.getRequest() 2022-07-20 13:43:20 +02:00
Mark Spanbroek 4b5cfe8e41 [purchasing] Update to latest dagger-contracts 2022-07-20 13:43:20 +02:00
Mark Spanbroek b7ab9481d9 [market] Update to latest dagger-contracts 2022-07-20 13:43:20 +02:00
Mark Spanbroek e352a181e4 [contracts] Update to latest dagger-contracts 2022-07-20 13:43:20 +02:00
Bulat-Ziganshin 15ae157e61
BlockStore: delBlock got new type (#123)
Now every store traces only its own operations and failures (currently implemented only for delBlock)
2022-06-28 19:10:05 +03:00
Dmitriy Ryajov cf4a0a393e
Json info (#115)
* make info return a json with an spr

* add an `exec` target to the makefile
2022-06-14 17:34:56 -06:00
Dmitriy Ryajov 28ebb16767
adding AsyncStreamWrapper (#110)
* Adding AsyncStreamWrapper type

* use asyncstreamwrapper to avoid copies

* make size an operator

* export asyncstreamwrapper
2022-06-14 09:19:35 -06:00
Eric Mastro 3b0e93eab4
[chore] bump nim-ethers to 0.1.7 (#107)
* [chore] bump nim-ethers to 0.1.7

* [chore] make block handler callback async

nim-ethers 0.1.7 makes the subscription callbacks async.
2022-06-02 16:09:53 -06:00
Dmitriy Ryajov 6ce7e23767
Upload authenticators (#108)
* initial implementation of storage proofs upload

* make sure proof verifies with after deserializing

* add por store

* rename por store to stp store

* rename porstore to stpstore

* add support for host discovery to discovery mock

* add tags upload network tests
2022-05-25 20:29:31 -06:00
Dmitriy Ryajov 738738c3c6
Adding storage proofs store (#109)
* add por store

* rename por store to stp store

* add stp tests
2022-05-24 18:42:49 -06:00
Dmitriy Ryajov 56b80d6f6d
Por serialize (#106)
* move por into storage proofs

* use SeekableStream

* adding serialization for por

* remove leftovers

* add empty block support

* add basic por test

* rename block exchange for consistency

* add storageproofstests

* moving timing to storageproofs

* fix imports

* fix imports

* fix imports

* add top level exports

* move delete blocks helper to helpers

* more import/export fixes

* cleanup

* more import fixes

* fix unused warnings

* detect corrupt blocks tests

* add serialization tests

* move init method around

* bump asynctest

* fix CID version

* get rid of warning

* wip: fix CI

* increase CI timeout
2022-05-23 23:24:15 -06:00
Dmitriy Ryajov 6ad7a6bb96
Batched blocks (#105)
* remove sleep

* throttle prefetch

* break loop if conn closed

* logging

* prefetch blocks in batches

* add blocks batch constant
2022-05-20 10:53:34 -06:00
Dmitriy Ryajov d4f3ebc867
reorg engine directory structure (#104) 2022-05-19 16:28:53 -06:00
Michael Bradley 209343087c
Change every dagger to codex (#102) 2022-05-19 13:56:03 -06:00