BlockStore API got new return types (rationale in https://github.com/status-im/nim-codex/issues/123#issuecomment-1163797753):
- getBlock: Future[?! (?Block)]
- putBlock/delBlock/listBlocks: Future[?!void]
- hasBlock: Future[?!bool]
Plus refactored readOnce(StoreStream) and check received data in its tests.
And replaced local use of AsyncHeapQueue with seq.sort.
`waitFor` has been moved into the main body of integration test
`nodes negotiate ...` to reduce blockage on the main thread. That change, along
with widening the expiry in `when defined(windows)` (and bumping up
`timeout-minutes` in our CI workflow) allows Windows CI builds to succeed more
consistently, i.e. without timing out.
* initial implementation of storage proofs upload
* make sure proof verifies with after deserializing
* add por store
* rename por store to stp store
* rename porstore to stpstore
* add support for host discovery to discovery mock
* add tags upload network tests
* don't force logging syncs
* Add failing test
* wip discovery engine
* re-add chronicles sinks
* wip
* move network related stuff to own folder
* move peer related stuff to own folder
* extract discovery into it's own engine
* update imports
* move pending blocks into engine module
* add top level exports
* update imports
* update import paths
* update imports
* support for inflight request filtering and tests
* use `remove` instead of `del`
* fix sorting in `selectCheapest`
* re-org test file structure
* fix to use discovery engine
* file re-org
* fix compilation
* fixup discovery to use async handlers
* more re-org
* rework with support for discovery engine
* add logging
* use defaults
* wip: reworking with discoveryengine
* wip: more test fixes
* more logging
* use ordered table
* use `bt` for blocktype Block
* fix tests
* make tests work with discovery engine
* expose all node components
* fix to work with discovery engine
* wip
* propagate cancellation in listBlocks
* start/stop disc engine in blockexc engine
* remove disc engine start/stop
* wire up discovery engine
* misc comments and imports
* pass discovery to dagger node
* set sleep timers
* unused imports
* misc
* don't spawn a task, await it
* don't await handlers
* trace logging
* reduce default sleep time
Co-authored-by: Tanguy <tanguy@status.im>
* rework discovery with async queues
* increase max message size for large manifests
* increase sleep time to 100 millis
* pass config
* check for nil on start/stop
* fix tests and split out discovery tests
* don't auto mount network
* add discovery tests
* rework moc discovery
* move discovery moc to disc dir
* don't force logging syncs
* don't force moc discovery on all tests
* rework discovery with methods
* add top level utils file
* don't use asyncCheck
* don't pass entire blocks to list blocks calback
* spelling
* - don't send want reqs to peers reporting the cid
- Don't request blocks directly on presense update, use `requestBlock`
* bug, nodes should not have blocks in local store
* Add failing test
* prefetch blocks so that download isn't serial
* if request already pending, return the handle
* fire discovery if no peers report block as have
* only query discovery if not enough nodes for cid
* wrap async req in template
* use non awaiting version of queue routines
* rework E2E tests as unittest
* re-add chronicles sinks
Co-authored-by: Tanguy <tanguy@status.im>
* rework discovery with async queues
* misc style changes
* increase max message size for large manifests
* use upraises and avoid exceptions on key access
* increase sleep time to 100 millis
* pass config
* make list blocks trigger a callback on each block
* check for nil on start/stop
* fix tests and split out discovery tests
* don't auto mount network
* add list block tests
* add discovery tests
* rework moc discovery
* move discovery moc to disc dir
* don't force logging syncs
* don't force moc discovery on all tests
* rework discovery with methods
* add top level utils file
* don't use asyncCheck
* don't pass entire blocks to list blocks calback
* spelling
* Add Proving object, which maintains contract id's to watch
* [proving] invoke callback when proof is required
# Conflicts:
# dagger/por/timing/periods.nim
# dagger/por/timing/prooftiming.nim
* [proving] check proof requirements for all our contracts
# Conflicts:
# tests/dagger/helpers/mockprooftiming.nim
* Update vendor/dagger-contracts
* [proving] call onProofRequired() when proof is required soon
* [proving] stop checking contracts that have ended
* [proving] Remove duplicated funcs
* [proving] Implement ProofTiming on top of smart contract
* [proving] Fix race condition in waitUntilNextPeriod()
Sometimes waitUntilNextPeriod would take a while to
determine the current period, leading to unexpected results.
Splits waitUntilNextPeriod() into getCurrentPeriod()
and waitUntilPeriod(), to ensure that we're really waiting
for the period that we think we're waiting for.
* wip: adding request for storage endpoint
* wire in erasure coding
* fix tests for erasure coding
* put type definitions into separate file
* integrate erasure coding
* change run/shutdown to start/stop
* temporary sleep, otherwise the fsstore blocks
* use storestream for retrieval
* use `len` prop on manifest
* remove read pattern
* make blocksize and chunksize consistent
* fix tests
* fix chunker tests - now padded by default
Introduce new nimble 'test' task that runs *all* tests.
Renames testAll to testDagger, because testAll didn't
run all tests, it forgot to run the contract tests.
* feat: introduce LRU cache
Replace `MemoryStore` with LRU caching mechanism.
`lrucache` library was forked to https://github.com/status-im/lrucache.nim.
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
# Conflicts:
# dagger/dagger.nim
# dagger/stores.nim
# dagger/stores/manager.nim
# tests/dagger/blockexc/testengine.nim
# tests/dagger/helpers/nodeutils.nim
# tests/dagger/testnode.nim
# tests/dagger/teststores.nim
* feat: introduce --cache-size CLI option
Allow for a value of `0` to disable the cache.
# Conflicts:
# dagger/dagger.nim
* allow dynamic block size in cache
allow block size to be variable/dynamic
update lrucache to use updated lrucache dep
Using removeLru proc, added tests
* Refactor CacheStore init block
Co-authored-by: Michael Bradley, Jr <michaelsbradleyjr@gmail.com>
Integrate dagger contracts from `nim-dagger-contracts` repo.
Add `dagger-contracts`, `nim-web3`, and all of `nim-web3`’s transitive deps as submodule deps to `nim-dagger`. Note: `nim-web3` and its transitive deps may no longer be needed when we switch to `nim-ethers`.
Add a `testContracts` nimble task to test all of the contracts functionality. Namely, this spins up an ethereum simulator, deploys the contracts (in `dagger-contracts`), runs the contract tests, and finally, regardless of success/error, kills the ethereum sim processes. The nimble task can be run with `./env.sh nimble testContracts`.
We also tested `nim-dagger-contracts` as a submodule dep of `nim-dagger`, and while the tests run as expected, the preference is to merge `nim-dagger-contracts` inside of `nim-dagger` for ease of parallel development. There’s also a high probability that `nim-dagger-contracts` is not being used as a dep by other projects. Are there any strong objections to this?
Co-authored-by: Michael Bradley <michaelsbradleyjr@gmail.com>
* fix compilation issues with latest libp2p
* change package description
* fix CI compiler
* try with nim 1.4.2
* try with 1.4.2
* use more standardized ci config
* set reasonable log level
* remove nim version dependency
* reverting to 1.4.6 as otherwise it fails on amd64
* use PeerInfo in event handlers
* use CidV1 and raw multicodec as default
* add block stream abstraction
* raises defect
* adding dataset abstraction
* move blockstream into own dir
* reorg files and fix tests
* rename dataset to blockset
* wip
* wip
* adding basic test for treehash algo
* run blockset tests along with with the rest
* remove obsolete contents
* fix chunker tests
* rename bitswap and move to stores
* rename bitwsap to blockexc and move to stores
* moare project structure reorg
* moving protobuf into bitswap
* adding block type
* reworking bitswap
* adding chunker
* adding license header
* use 1.2.6
* adding fixed size chunker
* add blockstore
* add iterator to chunker
* more bitswap changes
* rename ipfs to dagger
* rename to dagger
* blockstore inherits from BlockProvider
* wip - add core block handling logic
* smal changes
* use proper block store methods
* adding asynq heapqueue
* wip prepare for bitswap task runner
* adding `$`
* adding memory store and tests
* fixed chunking
* extracted desicion engine from bitswap
* added helper random funcs
* adding testing helpers
* only handle seqs
* add peer events
* cleanup pending blocks on blockstore event
* allow nil handlers
* move protobuf type helpers
* allow initializing block from Cid
* testing and fixes
* small fixes
* expose `<`
* spelling
* default value
* spelling
* pending blocks manager
* adding stores manager
* more tests a wip around bitswap
* small changes
* merge bitswap and engine for now
* for now run only the new poc's tests
* add a more complete ci setup
* use template in map
* remove p2pd
* remove go
* dont use asyncCheck
* few small changes
* adding ability to update items
* adding multiple task runners
* handle cancelation properly
* use Result instead of throwing
* wip bitswap tests
* moving things around
* split out engine again
* add request and handlers interface
* fix tests
* wip - engine tests
* remove unused imports
* fix tests
* cleanup block requesting logic
* add block request tests
* more block requests
* add support for max heap
* don't use result
* use max heap & send block presence in task handler
* add task handler tests
* rename store to localStore
* cleanup & logging
* cancel task on stop
* don't depend on local store for events
* dont use heap queue for wants
* add chronicles
* fix issue with peer wants
* add test for delayed block sends
* remove obsolete tests
* wip chunker
* run all tests
* add todo
* misc
* remove irrelevant files
* removing more files
* adding helpers for bitswap tests
* moved bitswap file
* misc
* make blocks timeout longer
* adjust block timeout
* speedup test
* compile with threads
* import missing crypto
* misc
* disable threads for now
* fix 32 bit platforms
* re-enable threads support in tests
Mark Spanbroek
Michael Bradley, Jr
Dmitriy Ryajov
Bulat-Ziganshin
Tanguy
Zahary Karadjov
Eric Mastro