Commit Graph

312 Commits

Author SHA1 Message Date
Dmitriy Ryajov 4bc701652f
Adding metrics (#203)
* adding inflight discovery metric

* adding metrics instructions and default dashboard

* spelling fixes
2022-08-23 10:11:21 -06:00
Michael Bradley, Jr 3d823dcbc6 [stores] update getBlock return type
change return type for `method getBlock` from `Future[?!(?Block)]` to
`Future[?!Block]`

use `type BlockNotFoundError = object of CodexError` to differentiate between
"block not found in the store" and other errors

also make some logic and error handling/messages more consistent across
BlockStore implementations

closes #177
closes #182
closes #210

alternative to #205, #209
2022-08-22 20:52:51 -05:00
Michael Bradley, Jr e0e2d7b583 [build] bump questionable 2022-08-22 20:52:51 -05:00
Bulat-Ziganshin 7ec039f822
Quickfix: start erasure decoding once any K blocks arrived (#199)
ECC decoding can be done with any encoded.K blocks, and now we start it once any K blocks arrived.

Previous code essentially waited for all blocks to arrive (or fail to arrive), that may mean unnecessary delays.

New code will sometimes ECC-decode when parity blocks were retrieved faster that data blocks (instead of waiting for data blocks from network), but we expect that it's OK due to extremely fast Leopard decoding (~~500 MB/s)
2022-08-19 21:21:55 -06:00
Dmitriy Ryajov e48450d671
Updating download endpoint curl (#212) 2022-08-19 21:20:25 -06:00
Michael Bradley, Jr 5aa42541b3 [build] update nim-datastore
also make changes to `codex/codex.nim` and tests for SQLiteStore re: changes in
nim-datastore

closes #178
2022-08-10 13:13:07 -05:00
markspanbroek 097a4cfd67
Better command line options for Ethereum (#181)
* [contracts] ContractInteractions.new() now requires account parameter

* [cli] Only perform Ethereum interactions when --eth-account is specified

* [cli] Add --persistence option that is disabled by default

* [cli] Use Option for ethDeployment parameter

* [node] Better error reporting when Ethereum node cannot be reached
2022-08-09 06:29:06 +02:00
Michael Bradley, Jr eebcf7e7b2 [node] custom iterator/query for SQLiteStore's `method listBlocks` 2022-08-08 17:48:37 -05:00
Michael Bradley, Jr 2ecf750959 [node] support self.cache=nil in SQLiteStore
also fix a discrepancy where cli option `--cache-size` is documented as
`0 disables the cache` in `codex/conf.nim`, but previously the value `0` would
result in a cache being constructed with default parameter values for
`CacheStore.new()`

Closes #180
2022-08-08 16:24:45 -05:00
Mark Spanbroek b7df2d151c [build] Update dagger-contracts 2022-08-08 14:00:55 +02:00
Mark Spanbroek 7f5310d640 [rest api] Allow user to specify amount of hosts 2022-08-08 14:00:55 +02:00
Mark Spanbroek fd6f8a1386 [sales] Pass along request and slot index in onProve callback 2022-08-08 14:00:55 +02:00
Mark Spanbroek cb5427ca95 [sales] Pass along request and slot index in onStore callback 2022-08-08 14:00:55 +02:00
Mark Spanbroek 3a3d083345 [contracts] Replace size by slotSize 2022-08-08 14:00:55 +02:00
Mark Spanbroek 6ec2d3bf8f Update dagger-contracts 2022-08-08 14:00:55 +02:00
Mark Spanbroek bf11ef5b55 [sales] Use correct slot price 2022-08-08 14:00:55 +02:00
Mark Spanbroek 057de4e928 [sales] Choose a slot index at random 2022-08-08 14:00:55 +02:00
Mark Spanbroek b4b6fdb7c6 [contracts] subscribe to SlotFilled event 2022-08-08 14:00:55 +02:00
Mark Spanbroek 7995fd71d5 [market] remove fulfillRequest()
It is superseded by fillSlot()
2022-08-08 14:00:55 +02:00
Mark Spanbroek 04e0c54e95 [market] Remove old getHost() overload
Is superseded by the new overload that includes
a slot id parameter.
2022-08-08 14:00:55 +02:00
Mark Spanbroek 698baeef35 [purchasing] Remove Purchase.selected
There no longer exists a single host that
is selected for a contract.
2022-08-08 14:00:55 +02:00
Mark Spanbroek ac6d17e895 [purchasing] Remove fulfillRequest() from test 2022-08-08 14:00:55 +02:00
Mark Spanbroek dc95c863d2 [sales] replace fulfillRequest() by fillSlot() 2022-08-08 14:00:55 +02:00
Mark Spanbroek 156cd5ba73 [market] Replace fulfillRequest() by fillSlot() 2022-08-08 14:00:55 +02:00
Mark Spanbroek 67e4a28ed0 [contracts] Replace fulfillRequest() by fillSlot() 2022-08-08 14:00:55 +02:00
Mark Spanbroek 0494b3617b [contracts] replace totalNodes and nodeId by slots 2022-08-08 14:00:55 +02:00
Mark Spanbroek 360c03e7c4 [contracts] Rename: maxPrice -> reward 2022-08-08 14:00:55 +02:00
Mark Spanbroek 5a00eddfba Update dagger-contracts 2022-08-08 14:00:55 +02:00
Michael Bradley, Jr 75666d01bf [node] add SQLiteStore backend
Closes #138
2022-08-08 02:12:43 -05:00
Michael Bradley, Jr 92d83bb1d2 [build] add github.com/arnetheduck/nim-sqlite3-abi to vendor 2022-08-08 02:12:43 -05:00
Michael Bradley, Jr 679df23b6f [build] add github.com/status-im/nim-datastore to vendor 2022-08-08 02:12:43 -05:00
Dmitriy Ryajov 625facff4a
Better batching (#170)
* add cleaner batching

* pass block instead of cid

* cleanup manifest fetching

* bug: reference correct cid version

* add batch tests

* make sure batch is always <= batchSize
2022-07-29 14:04:12 -06:00
Dmitriy Ryajov 48368893c9
Add limits for inflight requests (#169)
* convert network to async

* use async network api

* test with async network

* test concurrent send limits
2022-07-29 10:19:34 -06:00
Dmitriy Ryajov 0b786b383f
adding missing `/` to upload example 2022-07-28 12:03:26 -06:00
Dmitriy Ryajov 7ccde112f8
Contract store (#161)
* split out manifest and block retrieval

* adding test for `fetchManifest`

* raise exceptions in onStore
2022-07-28 11:44:59 -06:00
Bulat-Ziganshin 0bfe26440e
WIP: Harmonize BlockStore API (remaining functions) (#123) (#130)
BlockStore API got new return types (rationale in https://github.com/status-im/nim-codex/issues/123#issuecomment-1163797753):
- getBlock: Future[?! (?Block)]
- putBlock/delBlock/listBlocks: Future[?!void]
- hasBlock: Future[?!bool]

Plus refactored readOnce(StoreStream) and check received data in its tests.

And replaced local use of AsyncHeapQueue with seq.sort.
2022-07-28 03:39:17 +03:00
Mark Spanbroek afdb5be2d4 Add TODOs to indicate wanted architecture
As discussed in team:
- Retrieving of blocks that are mentioned in the
  manifest should be moved into a BlockStore.
- Sales should be given an instance of BlockStore,
  and an instance of StorageProofs, instead of the
  onStore, onClear, and onProve callbacks
2022-07-20 13:43:20 +02:00
Mark Spanbroek 13bbf2b052 Fix test that was failing intermittently
Give test more time to complete but do not increase
sleep time more than necessary, by introducing
waiting loop for test purposes.
2022-07-20 13:43:20 +02:00
Mark Spanbroek 6df0c9ab40 [integration] set expiry always to high value
Reasoning: a purchase is now completed when
the first proof comes in, so the test will no
longer wait until the request has expired.
2022-07-20 13:43:20 +02:00
Mark Spanbroek 97a5237d44 [sales] Add documentation 2022-07-20 13:43:20 +02:00
Mark Spanbroek 571aa6f694 [sales] Consistent names for callbacks 2022-07-20 13:43:20 +02:00
Mark Spanbroek 592a3dce4d [sales] Add callback for clearing storage when no longer needed 2022-07-20 13:43:20 +02:00
Mark Spanbroek adcb91a5d5 [node] Store datasets locally when sales asks for it 2022-07-20 13:43:20 +02:00
Mark Spanbroek 7bc5280596 [node] rename PrefetchBatch -> Batch
Reason: in a next commit we'll use the same
constant to fetch data in batches
2022-07-20 13:43:20 +02:00
Mark Spanbroek d4025f4a5f [manifest] Extract logic for decoding a manifest given a cid 2022-07-20 13:43:20 +02:00
Mark Spanbroek e06211f827 [sales] rename: negotiation -> sales agent
Reason: with the new design there are no offers and
selection, making this less like a negotation, and
more like an agent trying to fulfill a storage request
2022-07-20 13:43:20 +02:00
Mark Spanbroek 96ca1de768 Cleanup: remove StorageOffer everywhere 2022-07-20 13:43:20 +02:00
Mark Spanbroek bd2fba50c7 [purchasing] Fix integration test 2022-07-20 13:43:20 +02:00
Mark Spanbroek 4a4588e2a0 [node] Set callbacks for Sales to retrieve and prove posession of data 2022-07-20 13:43:20 +02:00
Mark Spanbroek 94f6838d19 [node] Fix: ensure that retrieve works for small files 2022-07-20 13:43:20 +02:00