Increase blocksize from ~8KiB to ~64KiB and remove useless unbounded prefetching on download
* increased upload/download speed on my Win10 Zen3 box to 50/150 MB/s (with FSStore)
* made manifest files 8x smaller
* two more changes in the tests to make them pass
Closes#263
Make RequestId, SlotId, Nonce, PurcahseId distinct types.
Add/modify conversions to support the distinct type (ABI encoding/decoding, JSON encoding, REST decoding).
Update tests
- rename `ContractId` to `SlotId`
- add `RequestId`, `PurchaseId`, `Nonce` types as aliases of `array[32, byte]`
- rename `Proving.contracts` to `Proving.slots`
- change signatures of `isSlotCancelled` and `isCancelled` to use `SlotId` and `RequestId` types, respectively.
- change all references to `RequestId`, `SlotId`, and `PurchaseId`
The initial goal of this patch was to allow to download of a file via REST API in exactly the same size as it was uploaded, which required adding fields Chunker.offset and Manifest.originalBytes to keep that size. On top of that, we added more integrity checks to operations on Manifest, and reorganized TestNode.nim to test the actual interaction between node.store and node.retrieve operations.
Note that the wire format of Manifest was changed, so we need to recreate all BlockStores.
* Download without padding
* Fixed chunker tests
* Chunker: get rid of RabinChunker
* Verify offset in the chunker tests
* Use manifest.originalBytesPadded in StoreStream.size
* StoreStream: replace emptyBlock with zeroMem
* Manifest.bytes: compute how many bytes corresponding StoreStream(Manifest, pad) will return
* Manifest: verify originalBytes and originalLen on new/encode/decode
Also set originalBytes in each Manifest creation/update scenario
* Manifest: comments, split code into sections
* Reordered parameters to deal with int64 size in 32-bit builds
* TestNode.nim: combine Store and Retrieve tests
1. Instead of copy-pasting code from node.nim, new test calls node.store() and node.retrieve() in order to check that they can correctly store and then retrieve data
2. New test compares only file contents, manifest contents considered an implementation detail
3. New test chunks at odd chunkSize=BlockSize/1.618 in order to ensure that data retrieved correctly even when buffer sizes mismatch
* TestNode.nim: code refactoring
* Manifest.add: one more test
* Manifest.verify: return Result instead of raising Defect
* Node.store: added blockSize parameter
change return type for `method getBlock` from `Future[?!(?Block)]` to
`Future[?!Block]`
use `type BlockNotFoundError = object of CodexError` to differentiate between
"block not found in the store" and other errors
also make some logic and error handling/messages more consistent across
BlockStore implementations
closes#177closes#182closes#210
alternative to #205, #209
* [contracts] ContractInteractions.new() now requires account parameter
* [cli] Only perform Ethereum interactions when --eth-account is specified
* [cli] Add --persistence option that is disabled by default
* [cli] Use Option for ethDeployment parameter
* [node] Better error reporting when Ethereum node cannot be reached
also fix a discrepancy where cli option `--cache-size` is documented as
`0 disables the cache` in `codex/conf.nim`, but previously the value `0` would
result in a cache being constructed with default parameter values for
`CacheStore.new()`
Closes#180
BlockStore API got new return types (rationale in https://github.com/status-im/nim-codex/issues/123#issuecomment-1163797753):
- getBlock: Future[?! (?Block)]
- putBlock/delBlock/listBlocks: Future[?!void]
- hasBlock: Future[?!bool]
Plus refactored readOnce(StoreStream) and check received data in its tests.
And replaced local use of AsyncHeapQueue with seq.sort.
`waitFor` has been moved into the main body of integration test
`nodes negotiate ...` to reduce blockage on the main thread. That change, along
with widening the expiry in `when defined(windows)` (and bumping up
`timeout-minutes` in our CI workflow) allows Windows CI builds to succeed more
consistently, i.e. without timing out.
* initial implementation of storage proofs upload
* make sure proof verifies with after deserializing
* add por store
* rename por store to stp store
* rename porstore to stpstore
* add support for host discovery to discovery mock
* add tags upload network tests
* don't force logging syncs
* Add failing test
* wip discovery engine
* re-add chronicles sinks
* wip
* move network related stuff to own folder
* move peer related stuff to own folder
* extract discovery into it's own engine
* update imports
* move pending blocks into engine module
* add top level exports
* update imports
* update import paths
* update imports
* support for inflight request filtering and tests
* use `remove` instead of `del`
* fix sorting in `selectCheapest`
* re-org test file structure
* fix to use discovery engine
* file re-org
* fix compilation
* fixup discovery to use async handlers
* more re-org
* rework with support for discovery engine
* add logging
* use defaults
* wip: reworking with discoveryengine
* wip: more test fixes
* more logging
* use ordered table
* use `bt` for blocktype Block
* fix tests
* make tests work with discovery engine
* expose all node components
* fix to work with discovery engine
* wip
* propagate cancellation in listBlocks
* start/stop disc engine in blockexc engine
* remove disc engine start/stop
* wire up discovery engine
* misc comments and imports
* pass discovery to dagger node
* set sleep timers
* unused imports
* misc
* don't spawn a task, await it
* don't await handlers
* trace logging
* reduce default sleep time
Co-authored-by: Tanguy <tanguy@status.im>
* rework discovery with async queues
* increase max message size for large manifests
* increase sleep time to 100 millis
* pass config
* check for nil on start/stop
* fix tests and split out discovery tests
* don't auto mount network
* add discovery tests
* rework moc discovery
* move discovery moc to disc dir
* don't force logging syncs
* don't force moc discovery on all tests
* rework discovery with methods
* add top level utils file
* don't use asyncCheck
* don't pass entire blocks to list blocks calback
* spelling
* - don't send want reqs to peers reporting the cid
- Don't request blocks directly on presense update, use `requestBlock`
* bug, nodes should not have blocks in local store
* Add failing test
* prefetch blocks so that download isn't serial
* if request already pending, return the handle
* fire discovery if no peers report block as have
* only query discovery if not enough nodes for cid
* wrap async req in template
* use non awaiting version of queue routines
* rework E2E tests as unittest
* re-add chronicles sinks
Co-authored-by: Tanguy <tanguy@status.im>
* rework discovery with async queues
* misc style changes
* increase max message size for large manifests
* use upraises and avoid exceptions on key access
* increase sleep time to 100 millis
* pass config
* make list blocks trigger a callback on each block
* check for nil on start/stop
* fix tests and split out discovery tests
* don't auto mount network
* add list block tests
* add discovery tests
* rework moc discovery
* move discovery moc to disc dir
* don't force logging syncs
* don't force moc discovery on all tests
* rework discovery with methods
* add top level utils file
* don't use asyncCheck
* don't pass entire blocks to list blocks calback
* spelling
* Add Proving object, which maintains contract id's to watch
* [proving] invoke callback when proof is required
# Conflicts:
# dagger/por/timing/periods.nim
# dagger/por/timing/prooftiming.nim
* [proving] check proof requirements for all our contracts
# Conflicts:
# tests/dagger/helpers/mockprooftiming.nim
* Update vendor/dagger-contracts
* [proving] call onProofRequired() when proof is required soon
* [proving] stop checking contracts that have ended
* [proving] Remove duplicated funcs
* [proving] Implement ProofTiming on top of smart contract
* [proving] Fix race condition in waitUntilNextPeriod()
Sometimes waitUntilNextPeriod would take a while to
determine the current period, leading to unexpected results.
Splits waitUntilNextPeriod() into getCurrentPeriod()
and waitUntilPeriod(), to ensure that we're really waiting
for the period that we think we're waiting for.
* wip: adding request for storage endpoint
* wire in erasure coding
* fix tests for erasure coding
* put type definitions into separate file
* integrate erasure coding
* change run/shutdown to start/stop
* temporary sleep, otherwise the fsstore blocks
* use storestream for retrieval
* use `len` prop on manifest
* remove read pattern
* make blocksize and chunksize consistent
* fix tests
* fix chunker tests - now padded by default
Introduce new nimble 'test' task that runs *all* tests.
Renames testAll to testDagger, because testAll didn't
run all tests, it forgot to run the contract tests.
* feat: introduce LRU cache
Replace `MemoryStore` with LRU caching mechanism.
`lrucache` library was forked to https://github.com/status-im/lrucache.nim.
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
# Conflicts:
# dagger/dagger.nim
# dagger/stores.nim
# dagger/stores/manager.nim
# tests/dagger/blockexc/testengine.nim
# tests/dagger/helpers/nodeutils.nim
# tests/dagger/testnode.nim
# tests/dagger/teststores.nim
* feat: introduce --cache-size CLI option
Allow for a value of `0` to disable the cache.
# Conflicts:
# dagger/dagger.nim
* allow dynamic block size in cache
allow block size to be variable/dynamic
update lrucache to use updated lrucache dep
Using removeLru proc, added tests
* Refactor CacheStore init block
Co-authored-by: Michael Bradley, Jr <michaelsbradleyjr@gmail.com>
Integrate dagger contracts from `nim-dagger-contracts` repo.
Add `dagger-contracts`, `nim-web3`, and all of `nim-web3`’s transitive deps as submodule deps to `nim-dagger`. Note: `nim-web3` and its transitive deps may no longer be needed when we switch to `nim-ethers`.
Add a `testContracts` nimble task to test all of the contracts functionality. Namely, this spins up an ethereum simulator, deploys the contracts (in `dagger-contracts`), runs the contract tests, and finally, regardless of success/error, kills the ethereum sim processes. The nimble task can be run with `./env.sh nimble testContracts`.
We also tested `nim-dagger-contracts` as a submodule dep of `nim-dagger`, and while the tests run as expected, the preference is to merge `nim-dagger-contracts` inside of `nim-dagger` for ease of parallel development. There’s also a high probability that `nim-dagger-contracts` is not being used as a dep by other projects. Are there any strong objections to this?
Co-authored-by: Michael Bradley <michaelsbradleyjr@gmail.com>
Bulat-Ziganshin
Dmitriy Ryajov
Eric Mastro
Michael Bradley, Jr
markspanbroek
Tanguy
Zahary Karadjov