4bc701652f
Adding metrics ( #203 )
...
* adding inflight discovery metric
* adding metrics instructions and default dashboard
* spelling fixes
2022-08-23 10:11:21 -06:00
3d823dcbc6
[stores] update getBlock return type
...
change return type for `method getBlock` from `Future[?!(?Block)]` to
`Future[?!Block]`
use `type BlockNotFoundError = object of CodexError` to differentiate between
"block not found in the store" and other errors
also make some logic and error handling/messages more consistent across
BlockStore implementations
closes #177
closes #182
closes #210
alternative to #205 , #209
2022-08-22 20:52:51 -05:00
7ec039f822
Quickfix: start erasure decoding once any K blocks arrived ( #199 )
...
ECC decoding can be done with any encoded.K blocks, and now we start it once any K blocks arrived.
Previous code essentially waited for all blocks to arrive (or fail to arrive), that may mean unnecessary delays.
New code will sometimes ECC-decode when parity blocks were retrieved faster that data blocks (instead of waiting for data blocks from network), but we expect that it's OK due to extremely fast Leopard decoding (~~500 MB/s)
2022-08-19 21:21:55 -06:00
5aa42541b3
[build] update nim-datastore
...
also make changes to `codex/codex.nim` and tests for SQLiteStore re: changes in
nim-datastore
closes #178
2022-08-10 13:13:07 -05:00
097a4cfd67
Better command line options for Ethereum ( #181 )
...
* [contracts] ContractInteractions.new() now requires account parameter
* [cli] Only perform Ethereum interactions when --eth-account is specified
* [cli] Add --persistence option that is disabled by default
* [cli] Use Option for ethDeployment parameter
* [node] Better error reporting when Ethereum node cannot be reached
2022-08-09 06:29:06 +02:00
eebcf7e7b2
[node] custom iterator/query for SQLiteStore's `method listBlocks`
2022-08-08 17:48:37 -05:00
2ecf750959
[node] support self.cache=nil in SQLiteStore
...
also fix a discrepancy where cli option `--cache-size` is documented as
`0 disables the cache` in `codex/conf.nim`, but previously the value `0` would
result in a cache being constructed with default parameter values for
`CacheStore.new()`
Closes #180
2022-08-08 16:24:45 -05:00
7f5310d640
[rest api] Allow user to specify amount of hosts
2022-08-08 14:00:55 +02:00
fd6f8a1386
[sales] Pass along request and slot index in onProve callback
2022-08-08 14:00:55 +02:00
cb5427ca95
[sales] Pass along request and slot index in onStore callback
2022-08-08 14:00:55 +02:00
3a3d083345
[contracts] Replace size by slotSize
2022-08-08 14:00:55 +02:00
bf11ef5b55
[sales] Use correct slot price
2022-08-08 14:00:55 +02:00
057de4e928
[sales] Choose a slot index at random
2022-08-08 14:00:55 +02:00
b4b6fdb7c6
[contracts] subscribe to SlotFilled event
2022-08-08 14:00:55 +02:00
7995fd71d5
[market] remove fulfillRequest()
...
It is superseded by fillSlot()
2022-08-08 14:00:55 +02:00
04e0c54e95
[market] Remove old getHost() overload
...
Is superseded by the new overload that includes
a slot id parameter.
2022-08-08 14:00:55 +02:00
698baeef35
[purchasing] Remove Purchase.selected
...
There no longer exists a single host that
is selected for a contract.
2022-08-08 14:00:55 +02:00
dc95c863d2
[sales] replace fulfillRequest() by fillSlot()
2022-08-08 14:00:55 +02:00
156cd5ba73
[market] Replace fulfillRequest() by fillSlot()
2022-08-08 14:00:55 +02:00
67e4a28ed0
[contracts] Replace fulfillRequest() by fillSlot()
2022-08-08 14:00:55 +02:00
0494b3617b
[contracts] replace totalNodes and nodeId by slots
2022-08-08 14:00:55 +02:00
360c03e7c4
[contracts] Rename: maxPrice -> reward
2022-08-08 14:00:55 +02:00
75666d01bf
[node] add SQLiteStore backend
...
Closes #138
2022-08-08 02:12:43 -05:00
625facff4a
Better batching ( #170 )
...
* add cleaner batching
* pass block instead of cid
* cleanup manifest fetching
* bug: reference correct cid version
* add batch tests
* make sure batch is always <= batchSize
2022-07-29 14:04:12 -06:00
48368893c9
Add limits for inflight requests ( #169 )
...
* convert network to async
* use async network api
* test with async network
* test concurrent send limits
2022-07-29 10:19:34 -06:00
7ccde112f8
Contract store ( #161 )
...
* split out manifest and block retrieval
* adding test for `fetchManifest`
* raise exceptions in onStore
2022-07-28 11:44:59 -06:00
0bfe26440e
WIP: Harmonize BlockStore API (remaining functions) ( #123 ) ( #130 )
...
BlockStore API got new return types (rationale in https://github.com/status-im/nim-codex/issues/123#issuecomment-1163797753 ):
- getBlock: Future[?! (?Block)]
- putBlock/delBlock/listBlocks: Future[?!void]
- hasBlock: Future[?!bool]
Plus refactored readOnce(StoreStream) and check received data in its tests.
And replaced local use of AsyncHeapQueue with seq.sort.
2022-07-28 03:39:17 +03:00
afdb5be2d4
Add TODOs to indicate wanted architecture
...
As discussed in team:
- Retrieving of blocks that are mentioned in the
manifest should be moved into a BlockStore.
- Sales should be given an instance of BlockStore,
and an instance of StorageProofs, instead of the
onStore, onClear, and onProve callbacks
2022-07-20 13:43:20 +02:00
97a5237d44
[sales] Add documentation
2022-07-20 13:43:20 +02:00
571aa6f694
[sales] Consistent names for callbacks
2022-07-20 13:43:20 +02:00
592a3dce4d
[sales] Add callback for clearing storage when no longer needed
2022-07-20 13:43:20 +02:00
adcb91a5d5
[node] Store datasets locally when sales asks for it
2022-07-20 13:43:20 +02:00
7bc5280596
[node] rename PrefetchBatch -> Batch
...
Reason: in a next commit we'll use the same
constant to fetch data in batches
2022-07-20 13:43:20 +02:00
d4025f4a5f
[manifest] Extract logic for decoding a manifest given a cid
2022-07-20 13:43:20 +02:00
e06211f827
[sales] rename: negotiation -> sales agent
...
Reason: with the new design there are no offers and
selection, making this less like a negotation, and
more like an agent trying to fulfill a storage request
2022-07-20 13:43:20 +02:00
96ca1de768
Cleanup: remove StorageOffer everywhere
2022-07-20 13:43:20 +02:00
bd2fba50c7
[purchasing] Fix integration test
2022-07-20 13:43:20 +02:00
4a4588e2a0
[node] Set callbacks for Sales to retrieve and prove posession of data
2022-07-20 13:43:20 +02:00
94f6838d19
[node] Fix: ensure that retrieve works for small files
2022-07-20 13:43:20 +02:00
f14eeb688b
[WIP sales] Handle request expiry
2022-07-20 13:43:20 +02:00
9438aba5d2
[WIP sales] Handle other host fulfilling request
2022-07-20 13:43:20 +02:00
7f864570bd
[WIP sales] call onSale()
2022-07-20 13:43:20 +02:00
a2570f9380
[WIP sales] fulfill request
2022-07-20 13:43:20 +02:00
5abd5dd674
[WIP sales] make storage available again upon failure
2022-07-20 13:43:20 +02:00
dcb8f8a7f7
[WIP sales] Generate proof of storage
2022-07-20 13:43:20 +02:00
07dbb6ae18
[WIP sales] Retrieve data
2022-07-20 13:43:20 +02:00
3e6d51754d
[WIP sales] Make tests pass by commenting out code
2022-07-20 13:43:20 +02:00
0636c94b74
[market] Add market.getSigner()
2022-07-20 13:43:20 +02:00
03b92a2067
[market] Add market.getHost()
2022-07-20 13:43:20 +02:00
b414ecd67e
[market] Add market.getRequest()
2022-07-20 13:43:20 +02:00
4b5cfe8e41
[purchasing] Update to latest dagger-contracts
2022-07-20 13:43:20 +02:00
b7ab9481d9
[market] Update to latest dagger-contracts
2022-07-20 13:43:20 +02:00
e352a181e4
[contracts] Update to latest dagger-contracts
2022-07-20 13:43:20 +02:00
15ae157e61
BlockStore: delBlock got new type ( #123 )
...
Now every store traces only its own operations and failures (currently implemented only for delBlock)
2022-06-28 19:10:05 +03:00
cf4a0a393e
Json info ( #115 )
...
* make info return a json with an spr
* add an `exec` target to the makefile
2022-06-14 17:34:56 -06:00
28ebb16767
adding AsyncStreamWrapper ( #110 )
...
* Adding AsyncStreamWrapper type
* use asyncstreamwrapper to avoid copies
* make size an operator
* export asyncstreamwrapper
2022-06-14 09:19:35 -06:00
3b0e93eab4
[chore] bump nim-ethers to 0.1.7 ( #107 )
...
* [chore] bump nim-ethers to 0.1.7
* [chore] make block handler callback async
nim-ethers 0.1.7 makes the subscription callbacks async.
2022-06-02 16:09:53 -06:00
6ce7e23767
Upload authenticators ( #108 )
...
* initial implementation of storage proofs upload
* make sure proof verifies with after deserializing
* add por store
* rename por store to stp store
* rename porstore to stpstore
* add support for host discovery to discovery mock
* add tags upload network tests
2022-05-25 20:29:31 -06:00
738738c3c6
Adding storage proofs store ( #109 )
...
* add por store
* rename por store to stp store
* add stp tests
2022-05-24 18:42:49 -06:00
56b80d6f6d
Por serialize ( #106 )
...
* move por into storage proofs
* use SeekableStream
* adding serialization for por
* remove leftovers
* add empty block support
* add basic por test
* rename block exchange for consistency
* add storageproofstests
* moving timing to storageproofs
* fix imports
* fix imports
* fix imports
* add top level exports
* move delete blocks helper to helpers
* more import/export fixes
* cleanup
* more import fixes
* fix unused warnings
* detect corrupt blocks tests
* add serialization tests
* move init method around
* bump asynctest
* fix CID version
* get rid of warning
* wip: fix CI
* increase CI timeout
2022-05-23 23:24:15 -06:00
6ad7a6bb96
Batched blocks ( #105 )
...
* remove sleep
* throttle prefetch
* break loop if conn closed
* logging
* prefetch blocks in batches
* add blocks batch constant
2022-05-20 10:53:34 -06:00
d4f3ebc867
reorg engine directory structure ( #104 )
2022-05-19 16:28:53 -06:00
209343087c
Change every dagger to codex ( #102 )
2022-05-19 13:56:03 -06:00
Dmitriy Ryajov
Michael Bradley, Jr
Bulat-Ziganshin
markspanbroek
Eric Mastro