Implement blockstore manager which executes block storage operations on its block stores, in the order to which they were added to the manager, typically in the order of most local (fastest, eg cache) to least local (slowest, eg filesystem or perhaps a network filesystem). As an example, given a `BlockStoreManager` instantiated with a `@[MemoryStore, FSStore]`, retrieving a block would first attempt to get from the `MemoryStore`, and if not found, attempt to get from the `FSStore`.
Remove all dependencies on `BlockStores` (typically in the shape of `localstore`) and instead depend on `BlockStoreManager` via the `BlockExcEngine`.
Modify the role of the `BlockExcEngine` to make a “local vs remote” decision on block access/storage. For all operations other than retrieving blocks, this means simply going to the `BlockStoreManager`. For retrieving blocks, however, this means going first to the `BlockStoreManager`, and then if not found, going to the Dagger network (via pending block and want/have lists).
Remove `NetworkStore` as its two purposes were to defer block retrieval from a local store first, then go to the block exchange to requeest a block from the Dagger network. `BlockStoreManager` takes care of going to local storage first, and the block exchange engine handles going to Dagger network if retrieval from the store manager fails.
### Notes
1. Future work may want to consider breaking up `BlockExcEngine` further in to three modules:
- `BlockExcEngine` (depends on `WantHave`, `DHT`)
- `WantHave`
- `DHT` (work is in progress)
Co-authored-by: Michael Bradley <michaelsbradleyjr@gmail.com>
Downloading a CID that is unknown to the network would wait for the blockexchange to try to get the data. However, a subsequent API call, eg upload for that CID, or a blockexchange timeout would cause the dagger node to crash, due to an attempt to complete a future that had already been completed.
We were able to narrow this down to an attempt to call `resp.finish()` when the download block had already returned an `RestApiResponse` (and never sent any chunks to the response from the libp2p stream).
Change the HTTP response code for an unknown cid from 400 to 404.
Fix spelling mistake “cunk”.
Co-authored-by: Michael Bradley <michaelsbradleyjr@gmail.com>
Co-authored-by: Michael Bradley <michaelsbradleyjr@gmail.com>
* adding basic comments to rest api
* Update dagger/rest/api.nim
Co-authored-by: Michael Bradley <michaelsbradleyjr@gmail.com>
Co-authored-by: Michael Bradley <michaelsbradleyjr@gmail.com>
Rationale: In Nim 1.2.x this warning was triggered
too easily, this was fixed in Nim 1.4.x. Because
1.2.x is still used, we disable this warning to ensure
that the long list of ObservableStores warnings does not
obfuscates other warnings.
* fix compilation issues with latest libp2p
* change package description
* fix CI compiler
* try with nim 1.4.2
* try with 1.4.2
* use more standardized ci config
* set reasonable log level
* remove nim version dependency
* reverting to 1.4.6 as otherwise it fails on amd64
* use PeerInfo in event handlers
* use CidV1 and raw multicodec as default
* add block stream abstraction
* raises defect
* adding dataset abstraction
* move blockstream into own dir
* reorg files and fix tests
* rename dataset to blockset
* wip
* wip
* adding basic test for treehash algo
* run blockset tests along with with the rest
* remove obsolete contents
* fix chunker tests
* rename bitswap and move to stores
* rename bitwsap to blockexc and move to stores
* moare project structure reorg
