nim-codex/tests/dagger/blockexc/testblockexc.nim

257 lines
7.4 KiB
Nim
Raw Normal View History

Poc 2 (#7) * moving protobuf into bitswap * adding block type * reworking bitswap * adding chunker * adding license header * use 1.2.6 * adding fixed size chunker * add blockstore * add iterator to chunker * more bitswap changes * rename ipfs to dagger * rename to dagger * blockstore inherits from BlockProvider * wip - add core block handling logic * smal changes * use proper block store methods * adding asynq heapqueue * wip prepare for bitswap task runner * adding `$` * adding memory store and tests * fixed chunking * extracted desicion engine from bitswap * added helper random funcs * adding testing helpers * only handle seqs * add peer events * cleanup pending blocks on blockstore event * allow nil handlers * move protobuf type helpers * allow initializing block from Cid * testing and fixes * small fixes * expose `<` * spelling * default value * spelling * pending blocks manager * adding stores manager * more tests a wip around bitswap * small changes * merge bitswap and engine for now * for now run only the new poc's tests * add a more complete ci setup * use template in map * remove p2pd * remove go * dont use asyncCheck * few small changes * adding ability to update items * adding multiple task runners * handle cancelation properly * use Result instead of throwing * wip bitswap tests * moving things around * split out engine again * add request and handlers interface * fix tests * wip - engine tests * remove unused imports * fix tests * cleanup block requesting logic * add block request tests * more block requests * add support for max heap * don't use result * use max heap & send block presence in task handler * add task handler tests * rename store to localStore * cleanup & logging * cancel task on stop * don't depend on local store for events * dont use heap queue for wants * add chronicles * fix issue with peer wants * add test for delayed block sends * remove obsolete tests * wip chunker * run all tests * add todo * misc * remove irrelevant files * removing more files * adding helpers for bitswap tests * moved bitswap file * misc * make blocks timeout longer * adjust block timeout * speedup test * compile with threads * import missing crypto * misc * disable threads for now * fix 32 bit platforms * re-enable threads support in tests
2021-02-26 00:23:22 +00:00
import std/sequtils
import std/algorithm
import pkg/asynctest
import pkg/chronos
import pkg/stew/byteutils
Poc 2 (#7) * moving protobuf into bitswap * adding block type * reworking bitswap * adding chunker * adding license header * use 1.2.6 * adding fixed size chunker * add blockstore * add iterator to chunker * more bitswap changes * rename ipfs to dagger * rename to dagger * blockstore inherits from BlockProvider * wip - add core block handling logic * smal changes * use proper block store methods * adding asynq heapqueue * wip prepare for bitswap task runner * adding `$` * adding memory store and tests * fixed chunking * extracted desicion engine from bitswap * added helper random funcs * adding testing helpers * only handle seqs * add peer events * cleanup pending blocks on blockstore event * allow nil handlers * move protobuf type helpers * allow initializing block from Cid * testing and fixes * small fixes * expose `<` * spelling * default value * spelling * pending blocks manager * adding stores manager * more tests a wip around bitswap * small changes * merge bitswap and engine for now * for now run only the new poc's tests * add a more complete ci setup * use template in map * remove p2pd * remove go * dont use asyncCheck * few small changes * adding ability to update items * adding multiple task runners * handle cancelation properly * use Result instead of throwing * wip bitswap tests * moving things around * split out engine again * add request and handlers interface * fix tests * wip - engine tests * remove unused imports * fix tests * cleanup block requesting logic * add block request tests * more block requests * add support for max heap * don't use result * use max heap & send block presence in task handler * add task handler tests * rename store to localStore * cleanup & logging * cancel task on stop * don't depend on local store for events * dont use heap queue for wants * add chronicles * fix issue with peer wants * add test for delayed block sends * remove obsolete tests * wip chunker * run all tests * add todo * misc * remove irrelevant files * removing more files * adding helpers for bitswap tests * moved bitswap file * misc * make blocks timeout longer * adjust block timeout * speedup test * compile with threads * import missing crypto * misc * disable threads for now * fix 32 bit platforms * re-enable threads support in tests
2021-02-26 00:23:22 +00:00
import pkg/libp2p
import pkg/libp2p/errors
import pkg/dagger/rng
import pkg/dagger/stores
import pkg/dagger/blockexchange
Poc 2 (#7) * moving protobuf into bitswap * adding block type * reworking bitswap * adding chunker * adding license header * use 1.2.6 * adding fixed size chunker * add blockstore * add iterator to chunker * more bitswap changes * rename ipfs to dagger * rename to dagger * blockstore inherits from BlockProvider * wip - add core block handling logic * smal changes * use proper block store methods * adding asynq heapqueue * wip prepare for bitswap task runner * adding `$` * adding memory store and tests * fixed chunking * extracted desicion engine from bitswap * added helper random funcs * adding testing helpers * only handle seqs * add peer events * cleanup pending blocks on blockstore event * allow nil handlers * move protobuf type helpers * allow initializing block from Cid * testing and fixes * small fixes * expose `<` * spelling * default value * spelling * pending blocks manager * adding stores manager * more tests a wip around bitswap * small changes * merge bitswap and engine for now * for now run only the new poc's tests * add a more complete ci setup * use template in map * remove p2pd * remove go * dont use asyncCheck * few small changes * adding ability to update items * adding multiple task runners * handle cancelation properly * use Result instead of throwing * wip bitswap tests * moving things around * split out engine again * add request and handlers interface * fix tests * wip - engine tests * remove unused imports * fix tests * cleanup block requesting logic * add block request tests * more block requests * add support for max heap * don't use result * use max heap & send block presence in task handler * add task handler tests * rename store to localStore * cleanup & logging * cancel task on stop * don't depend on local store for events * dont use heap queue for wants * add chronicles * fix issue with peer wants * add test for delayed block sends * remove obsolete tests * wip chunker * run all tests * add todo * misc * remove irrelevant files * removing more files * adding helpers for bitswap tests * moved bitswap file * misc * make blocks timeout longer * adjust block timeout * speedup test * compile with threads * import missing crypto * misc * disable threads for now * fix 32 bit platforms * re-enable threads support in tests
2021-02-26 00:23:22 +00:00
import pkg/dagger/chunker
import pkg/dagger/blocktype as bt
import ../helpers
import ../examples
Poc 2 (#7) * moving protobuf into bitswap * adding block type * reworking bitswap * adding chunker * adding license header * use 1.2.6 * adding fixed size chunker * add blockstore * add iterator to chunker * more bitswap changes * rename ipfs to dagger * rename to dagger * blockstore inherits from BlockProvider * wip - add core block handling logic * smal changes * use proper block store methods * adding asynq heapqueue * wip prepare for bitswap task runner * adding `$` * adding memory store and tests * fixed chunking * extracted desicion engine from bitswap * added helper random funcs * adding testing helpers * only handle seqs * add peer events * cleanup pending blocks on blockstore event * allow nil handlers * move protobuf type helpers * allow initializing block from Cid * testing and fixes * small fixes * expose `<` * spelling * default value * spelling * pending blocks manager * adding stores manager * more tests a wip around bitswap * small changes * merge bitswap and engine for now * for now run only the new poc's tests * add a more complete ci setup * use template in map * remove p2pd * remove go * dont use asyncCheck * few small changes * adding ability to update items * adding multiple task runners * handle cancelation properly * use Result instead of throwing * wip bitswap tests * moving things around * split out engine again * add request and handlers interface * fix tests * wip - engine tests * remove unused imports * fix tests * cleanup block requesting logic * add block request tests * more block requests * add support for max heap * don't use result * use max heap & send block presence in task handler * add task handler tests * rename store to localStore * cleanup & logging * cancel task on stop * don't depend on local store for events * dont use heap queue for wants * add chronicles * fix issue with peer wants * add test for delayed block sends * remove obsolete tests * wip chunker * run all tests * add todo * misc * remove irrelevant files * removing more files * adding helpers for bitswap tests * moved bitswap file * misc * make blocks timeout longer * adjust block timeout * speedup test * compile with threads * import missing crypto * misc * disable threads for now * fix 32 bit platforms * re-enable threads support in tests
2021-02-26 00:23:22 +00:00
Node setup (#32) * settup basic nim node * adding http utils * adding confutils * rough rest api proto * adding missing deps * turn tls emulation off * adding toml serialization * wip * adding missing deps * make sure to clean old state in teardown * adding file upload rest endpoint * renaming blockexchange to networkstore * updating nim-presto * updating libp2p * wip adding streaming upload * reworked chunking * bump to latest unstable * adding asyncfutures stream * make streamable * deleting unused files * reworking stores api * use new stores api * rework blockset and remove blockstream * don't return option from constructor * rework chunker * wip implement upload * fix tests * move unrelated logic to engine * don't print entire message * logging * basic encode/decode to/from dag-pb * add basic upload/download support * fix tests * renaming blockset to manifest * don't pass config to node * remove config and use new manifest * wip: make endpoints more reliable * wip: adding node tests * include correct manifest test * removing asyncfutures * proper chunking of files * simplify stream reading * test with encoding/decoding with many blocks * add block storing tests * adding retrieval test * add logging * tidy up chunker * tidy up manifest and node * use default chunk size * fix tests * fix tests * make sure Eof is set properly * wip * minor cleanup * add file utils * cleanup config * splitout DaggerServer and "main" * remove events since they are not used * add broadcast method to network peer * add and wire localstore * use localstore in the node * wip * logging * move file utils * use the constant * updating deps * fix memstore * use latest libp2p unstable * fix tests * rework block streaming * don't fail storing if the block already exists * add helper info endpoint * correct comment * rename localstore to fsstore * fix tests * remove unused tests * add test to retrieve one block * move some test files around * consolidate setup * Update dagger/blockexchange/engine.nim Co-authored-by: Tanguy <tanguy@status.im> * typo * better block path handling * don't inherit rootobj * remove useless template * Update tests/dagger/blockexc/testblockexc.nim Co-authored-by: markspanbroek <mark@spanbroek.net> * use isMainModule * use proper flag for starter/stoped * cleanup optional use * wrap in isMainModule * use `cancelAndAwait` * remove unused imports * wip * don't use optional * use functional error api * rework store tests and add fs tests * Block.new() to Block.init() * don't use optional for engine blocks * use result instead of optional for getBlock * remove unused imports * move stopping servers to `shutdown` * use result instead of optional * rework with results * fix tests * use waitFor in signal handlers * error helper * use `?` and mapFailure where possible * remove unnecesary `=?` * improve empty cid digest initialization Co-authored-by: Tanguy <tanguy@status.im> Co-authored-by: markspanbroek <mark@spanbroek.net>
2022-01-10 15:32:56 +00:00
suite "NetworkStore engine - 2 nodes":
Poc 2 (#7) * moving protobuf into bitswap * adding block type * reworking bitswap * adding chunker * adding license header * use 1.2.6 * adding fixed size chunker * add blockstore * add iterator to chunker * more bitswap changes * rename ipfs to dagger * rename to dagger * blockstore inherits from BlockProvider * wip - add core block handling logic * smal changes * use proper block store methods * adding asynq heapqueue * wip prepare for bitswap task runner * adding `$` * adding memory store and tests * fixed chunking * extracted desicion engine from bitswap * added helper random funcs * adding testing helpers * only handle seqs * add peer events * cleanup pending blocks on blockstore event * allow nil handlers * move protobuf type helpers * allow initializing block from Cid * testing and fixes * small fixes * expose `<` * spelling * default value * spelling * pending blocks manager * adding stores manager * more tests a wip around bitswap * small changes * merge bitswap and engine for now * for now run only the new poc's tests * add a more complete ci setup * use template in map * remove p2pd * remove go * dont use asyncCheck * few small changes * adding ability to update items * adding multiple task runners * handle cancelation properly * use Result instead of throwing * wip bitswap tests * moving things around * split out engine again * add request and handlers interface * fix tests * wip - engine tests * remove unused imports * fix tests * cleanup block requesting logic * add block request tests * more block requests * add support for max heap * don't use result * use max heap & send block presence in task handler * add task handler tests * rename store to localStore * cleanup & logging * cancel task on stop * don't depend on local store for events * dont use heap queue for wants * add chronicles * fix issue with peer wants * add test for delayed block sends * remove obsolete tests * wip chunker * run all tests * add todo * misc * remove irrelevant files * removing more files * adding helpers for bitswap tests * moved bitswap file * misc * make blocks timeout longer * adjust block timeout * speedup test * compile with threads * import missing crypto * misc * disable threads for now * fix 32 bit platforms * re-enable threads support in tests
2021-02-26 00:23:22 +00:00
let
Node setup (#32) * settup basic nim node * adding http utils * adding confutils * rough rest api proto * adding missing deps * turn tls emulation off * adding toml serialization * wip * adding missing deps * make sure to clean old state in teardown * adding file upload rest endpoint * renaming blockexchange to networkstore * updating nim-presto * updating libp2p * wip adding streaming upload * reworked chunking * bump to latest unstable * adding asyncfutures stream * make streamable * deleting unused files * reworking stores api * use new stores api * rework blockset and remove blockstream * don't return option from constructor * rework chunker * wip implement upload * fix tests * move unrelated logic to engine * don't print entire message * logging * basic encode/decode to/from dag-pb * add basic upload/download support * fix tests * renaming blockset to manifest * don't pass config to node * remove config and use new manifest * wip: make endpoints more reliable * wip: adding node tests * include correct manifest test * removing asyncfutures * proper chunking of files * simplify stream reading * test with encoding/decoding with many blocks * add block storing tests * adding retrieval test * add logging * tidy up chunker * tidy up manifest and node * use default chunk size * fix tests * fix tests * make sure Eof is set properly * wip * minor cleanup * add file utils * cleanup config * splitout DaggerServer and "main" * remove events since they are not used * add broadcast method to network peer * add and wire localstore * use localstore in the node * wip * logging * move file utils * use the constant * updating deps * fix memstore * use latest libp2p unstable * fix tests * rework block streaming * don't fail storing if the block already exists * add helper info endpoint * correct comment * rename localstore to fsstore * fix tests * remove unused tests * add test to retrieve one block * move some test files around * consolidate setup * Update dagger/blockexchange/engine.nim Co-authored-by: Tanguy <tanguy@status.im> * typo * better block path handling * don't inherit rootobj * remove useless template * Update tests/dagger/blockexc/testblockexc.nim Co-authored-by: markspanbroek <mark@spanbroek.net> * use isMainModule * use proper flag for starter/stoped * cleanup optional use * wrap in isMainModule * use `cancelAndAwait` * remove unused imports * wip * don't use optional * use functional error api * rework store tests and add fs tests * Block.new() to Block.init() * don't use optional for engine blocks * use result instead of optional for getBlock * remove unused imports * move stopping servers to `shutdown` * use result instead of optional * rework with results * fix tests * use waitFor in signal handlers * error helper * use `?` and mapFailure where possible * remove unnecesary `=?` * improve empty cid digest initialization Co-authored-by: Tanguy <tanguy@status.im> Co-authored-by: markspanbroek <mark@spanbroek.net>
2022-01-10 15:32:56 +00:00
chunker1 = RandomChunker.new(Rng.instance(), size = 1024, chunkSize = 256)
chunker2 = RandomChunker.new(Rng.instance(), size = 1024, chunkSize = 256)
Poc 2 (#7) * moving protobuf into bitswap * adding block type * reworking bitswap * adding chunker * adding license header * use 1.2.6 * adding fixed size chunker * add blockstore * add iterator to chunker * more bitswap changes * rename ipfs to dagger * rename to dagger * blockstore inherits from BlockProvider * wip - add core block handling logic * smal changes * use proper block store methods * adding asynq heapqueue * wip prepare for bitswap task runner * adding `$` * adding memory store and tests * fixed chunking * extracted desicion engine from bitswap * added helper random funcs * adding testing helpers * only handle seqs * add peer events * cleanup pending blocks on blockstore event * allow nil handlers * move protobuf type helpers * allow initializing block from Cid * testing and fixes * small fixes * expose `<` * spelling * default value * spelling * pending blocks manager * adding stores manager * more tests a wip around bitswap * small changes * merge bitswap and engine for now * for now run only the new poc's tests * add a more complete ci setup * use template in map * remove p2pd * remove go * dont use asyncCheck * few small changes * adding ability to update items * adding multiple task runners * handle cancelation properly * use Result instead of throwing * wip bitswap tests * moving things around * split out engine again * add request and handlers interface * fix tests * wip - engine tests * remove unused imports * fix tests * cleanup block requesting logic * add block request tests * more block requests * add support for max heap * don't use result * use max heap & send block presence in task handler * add task handler tests * rename store to localStore * cleanup & logging * cancel task on stop * don't depend on local store for events * dont use heap queue for wants * add chronicles * fix issue with peer wants * add test for delayed block sends * remove obsolete tests * wip chunker * run all tests * add todo * misc * remove irrelevant files * removing more files * adding helpers for bitswap tests * moved bitswap file * misc * make blocks timeout longer * adjust block timeout * speedup test * compile with threads * import missing crypto * misc * disable threads for now * fix 32 bit platforms * re-enable threads support in tests
2021-02-26 00:23:22 +00:00
var
switch1, switch2: Switch
2021-04-19 14:37:38 +00:00
wallet1, wallet2: WalletRef
pricing1, pricing2: Pricing
network1, network2: BlockExcNetwork
Node setup (#32) * settup basic nim node * adding http utils * adding confutils * rough rest api proto * adding missing deps * turn tls emulation off * adding toml serialization * wip * adding missing deps * make sure to clean old state in teardown * adding file upload rest endpoint * renaming blockexchange to networkstore * updating nim-presto * updating libp2p * wip adding streaming upload * reworked chunking * bump to latest unstable * adding asyncfutures stream * make streamable * deleting unused files * reworking stores api * use new stores api * rework blockset and remove blockstream * don't return option from constructor * rework chunker * wip implement upload * fix tests * move unrelated logic to engine * don't print entire message * logging * basic encode/decode to/from dag-pb * add basic upload/download support * fix tests * renaming blockset to manifest * don't pass config to node * remove config and use new manifest * wip: make endpoints more reliable * wip: adding node tests * include correct manifest test * removing asyncfutures * proper chunking of files * simplify stream reading * test with encoding/decoding with many blocks * add block storing tests * adding retrieval test * add logging * tidy up chunker * tidy up manifest and node * use default chunk size * fix tests * fix tests * make sure Eof is set properly * wip * minor cleanup * add file utils * cleanup config * splitout DaggerServer and "main" * remove events since they are not used * add broadcast method to network peer * add and wire localstore * use localstore in the node * wip * logging * move file utils * use the constant * updating deps * fix memstore * use latest libp2p unstable * fix tests * rework block streaming * don't fail storing if the block already exists * add helper info endpoint * correct comment * rename localstore to fsstore * fix tests * remove unused tests * add test to retrieve one block * move some test files around * consolidate setup * Update dagger/blockexchange/engine.nim Co-authored-by: Tanguy <tanguy@status.im> * typo * better block path handling * don't inherit rootobj * remove useless template * Update tests/dagger/blockexc/testblockexc.nim Co-authored-by: markspanbroek <mark@spanbroek.net> * use isMainModule * use proper flag for starter/stoped * cleanup optional use * wrap in isMainModule * use `cancelAndAwait` * remove unused imports * wip * don't use optional * use functional error api * rework store tests and add fs tests * Block.new() to Block.init() * don't use optional for engine blocks * use result instead of optional for getBlock * remove unused imports * move stopping servers to `shutdown` * use result instead of optional * rework with results * fix tests * use waitFor in signal handlers * error helper * use `?` and mapFailure where possible * remove unnecesary `=?` * improve empty cid digest initialization Co-authored-by: Tanguy <tanguy@status.im> Co-authored-by: markspanbroek <mark@spanbroek.net>
2022-01-10 15:32:56 +00:00
blockexc1, blockexc2: NetworkStore
Poc 2 (#7) * moving protobuf into bitswap * adding block type * reworking bitswap * adding chunker * adding license header * use 1.2.6 * adding fixed size chunker * add blockstore * add iterator to chunker * more bitswap changes * rename ipfs to dagger * rename to dagger * blockstore inherits from BlockProvider * wip - add core block handling logic * smal changes * use proper block store methods * adding asynq heapqueue * wip prepare for bitswap task runner * adding `$` * adding memory store and tests * fixed chunking * extracted desicion engine from bitswap * added helper random funcs * adding testing helpers * only handle seqs * add peer events * cleanup pending blocks on blockstore event * allow nil handlers * move protobuf type helpers * allow initializing block from Cid * testing and fixes * small fixes * expose `<` * spelling * default value * spelling * pending blocks manager * adding stores manager * more tests a wip around bitswap * small changes * merge bitswap and engine for now * for now run only the new poc's tests * add a more complete ci setup * use template in map * remove p2pd * remove go * dont use asyncCheck * few small changes * adding ability to update items * adding multiple task runners * handle cancelation properly * use Result instead of throwing * wip bitswap tests * moving things around * split out engine again * add request and handlers interface * fix tests * wip - engine tests * remove unused imports * fix tests * cleanup block requesting logic * add block request tests * more block requests * add support for max heap * don't use result * use max heap & send block presence in task handler * add task handler tests * rename store to localStore * cleanup & logging * cancel task on stop * don't depend on local store for events * dont use heap queue for wants * add chronicles * fix issue with peer wants * add test for delayed block sends * remove obsolete tests * wip chunker * run all tests * add todo * misc * remove irrelevant files * removing more files * adding helpers for bitswap tests * moved bitswap file * misc * make blocks timeout longer * adjust block timeout * speedup test * compile with threads * import missing crypto * misc * disable threads for now * fix 32 bit platforms * re-enable threads support in tests
2021-02-26 00:23:22 +00:00
peerId1, peerId2: PeerID
peerCtx1, peerCtx2: BlockExcPeerCtx
Node setup (#32) * settup basic nim node * adding http utils * adding confutils * rough rest api proto * adding missing deps * turn tls emulation off * adding toml serialization * wip * adding missing deps * make sure to clean old state in teardown * adding file upload rest endpoint * renaming blockexchange to networkstore * updating nim-presto * updating libp2p * wip adding streaming upload * reworked chunking * bump to latest unstable * adding asyncfutures stream * make streamable * deleting unused files * reworking stores api * use new stores api * rework blockset and remove blockstream * don't return option from constructor * rework chunker * wip implement upload * fix tests * move unrelated logic to engine * don't print entire message * logging * basic encode/decode to/from dag-pb * add basic upload/download support * fix tests * renaming blockset to manifest * don't pass config to node * remove config and use new manifest * wip: make endpoints more reliable * wip: adding node tests * include correct manifest test * removing asyncfutures * proper chunking of files * simplify stream reading * test with encoding/decoding with many blocks * add block storing tests * adding retrieval test * add logging * tidy up chunker * tidy up manifest and node * use default chunk size * fix tests * fix tests * make sure Eof is set properly * wip * minor cleanup * add file utils * cleanup config * splitout DaggerServer and "main" * remove events since they are not used * add broadcast method to network peer * add and wire localstore * use localstore in the node * wip * logging * move file utils * use the constant * updating deps * fix memstore * use latest libp2p unstable * fix tests * rework block streaming * don't fail storing if the block already exists * add helper info endpoint * correct comment * rename localstore to fsstore * fix tests * remove unused tests * add test to retrieve one block * move some test files around * consolidate setup * Update dagger/blockexchange/engine.nim Co-authored-by: Tanguy <tanguy@status.im> * typo * better block path handling * don't inherit rootobj * remove useless template * Update tests/dagger/blockexc/testblockexc.nim Co-authored-by: markspanbroek <mark@spanbroek.net> * use isMainModule * use proper flag for starter/stoped * cleanup optional use * wrap in isMainModule * use `cancelAndAwait` * remove unused imports * wip * don't use optional * use functional error api * rework store tests and add fs tests * Block.new() to Block.init() * don't use optional for engine blocks * use result instead of optional for getBlock * remove unused imports * move stopping servers to `shutdown` * use result instead of optional * rework with results * fix tests * use waitFor in signal handlers * error helper * use `?` and mapFailure where possible * remove unnecesary `=?` * improve empty cid digest initialization Co-authored-by: Tanguy <tanguy@status.im> Co-authored-by: markspanbroek <mark@spanbroek.net>
2022-01-10 15:32:56 +00:00
blocks1, blocks2: seq[bt.Block]
engine1, engine2: BlockExcEngine
localStore1, localStore2: BlockStore
Poc 2 (#7) * moving protobuf into bitswap * adding block type * reworking bitswap * adding chunker * adding license header * use 1.2.6 * adding fixed size chunker * add blockstore * add iterator to chunker * more bitswap changes * rename ipfs to dagger * rename to dagger * blockstore inherits from BlockProvider * wip - add core block handling logic * smal changes * use proper block store methods * adding asynq heapqueue * wip prepare for bitswap task runner * adding `$` * adding memory store and tests * fixed chunking * extracted desicion engine from bitswap * added helper random funcs * adding testing helpers * only handle seqs * add peer events * cleanup pending blocks on blockstore event * allow nil handlers * move protobuf type helpers * allow initializing block from Cid * testing and fixes * small fixes * expose `<` * spelling * default value * spelling * pending blocks manager * adding stores manager * more tests a wip around bitswap * small changes * merge bitswap and engine for now * for now run only the new poc's tests * add a more complete ci setup * use template in map * remove p2pd * remove go * dont use asyncCheck * few small changes * adding ability to update items * adding multiple task runners * handle cancelation properly * use Result instead of throwing * wip bitswap tests * moving things around * split out engine again * add request and handlers interface * fix tests * wip - engine tests * remove unused imports * fix tests * cleanup block requesting logic * add block request tests * more block requests * add support for max heap * don't use result * use max heap & send block presence in task handler * add task handler tests * rename store to localStore * cleanup & logging * cancel task on stop * don't depend on local store for events * dont use heap queue for wants * add chronicles * fix issue with peer wants * add test for delayed block sends * remove obsolete tests * wip chunker * run all tests * add todo * misc * remove irrelevant files * removing more files * adding helpers for bitswap tests * moved bitswap file * misc * make blocks timeout longer * adjust block timeout * speedup test * compile with threads * import missing crypto * misc * disable threads for now * fix 32 bit platforms * re-enable threads support in tests
2021-02-26 00:23:22 +00:00
setup:
Node setup (#32) * settup basic nim node * adding http utils * adding confutils * rough rest api proto * adding missing deps * turn tls emulation off * adding toml serialization * wip * adding missing deps * make sure to clean old state in teardown * adding file upload rest endpoint * renaming blockexchange to networkstore * updating nim-presto * updating libp2p * wip adding streaming upload * reworked chunking * bump to latest unstable * adding asyncfutures stream * make streamable * deleting unused files * reworking stores api * use new stores api * rework blockset and remove blockstream * don't return option from constructor * rework chunker * wip implement upload * fix tests * move unrelated logic to engine * don't print entire message * logging * basic encode/decode to/from dag-pb * add basic upload/download support * fix tests * renaming blockset to manifest * don't pass config to node * remove config and use new manifest * wip: make endpoints more reliable * wip: adding node tests * include correct manifest test * removing asyncfutures * proper chunking of files * simplify stream reading * test with encoding/decoding with many blocks * add block storing tests * adding retrieval test * add logging * tidy up chunker * tidy up manifest and node * use default chunk size * fix tests * fix tests * make sure Eof is set properly * wip * minor cleanup * add file utils * cleanup config * splitout DaggerServer and "main" * remove events since they are not used * add broadcast method to network peer * add and wire localstore * use localstore in the node * wip * logging * move file utils * use the constant * updating deps * fix memstore * use latest libp2p unstable * fix tests * rework block streaming * don't fail storing if the block already exists * add helper info endpoint * correct comment * rename localstore to fsstore * fix tests * remove unused tests * add test to retrieve one block * move some test files around * consolidate setup * Update dagger/blockexchange/engine.nim Co-authored-by: Tanguy <tanguy@status.im> * typo * better block path handling * don't inherit rootobj * remove useless template * Update tests/dagger/blockexc/testblockexc.nim Co-authored-by: markspanbroek <mark@spanbroek.net> * use isMainModule * use proper flag for starter/stoped * cleanup optional use * wrap in isMainModule * use `cancelAndAwait` * remove unused imports * wip * don't use optional * use functional error api * rework store tests and add fs tests * Block.new() to Block.init() * don't use optional for engine blocks * use result instead of optional for getBlock * remove unused imports * move stopping servers to `shutdown` * use result instead of optional * rework with results * fix tests * use waitFor in signal handlers * error helper * use `?` and mapFailure where possible * remove unnecesary `=?` * improve empty cid digest initialization Co-authored-by: Tanguy <tanguy@status.im> Co-authored-by: markspanbroek <mark@spanbroek.net>
2022-01-10 15:32:56 +00:00
while true:
let chunk = await chunker1.getBytes()
if chunk.len <= 0:
break
blocks1.add(bt.Block.new(chunk).tryGet())
Node setup (#32) * settup basic nim node * adding http utils * adding confutils * rough rest api proto * adding missing deps * turn tls emulation off * adding toml serialization * wip * adding missing deps * make sure to clean old state in teardown * adding file upload rest endpoint * renaming blockexchange to networkstore * updating nim-presto * updating libp2p * wip adding streaming upload * reworked chunking * bump to latest unstable * adding asyncfutures stream * make streamable * deleting unused files * reworking stores api * use new stores api * rework blockset and remove blockstream * don't return option from constructor * rework chunker * wip implement upload * fix tests * move unrelated logic to engine * don't print entire message * logging * basic encode/decode to/from dag-pb * add basic upload/download support * fix tests * renaming blockset to manifest * don't pass config to node * remove config and use new manifest * wip: make endpoints more reliable * wip: adding node tests * include correct manifest test * removing asyncfutures * proper chunking of files * simplify stream reading * test with encoding/decoding with many blocks * add block storing tests * adding retrieval test * add logging * tidy up chunker * tidy up manifest and node * use default chunk size * fix tests * fix tests * make sure Eof is set properly * wip * minor cleanup * add file utils * cleanup config * splitout DaggerServer and "main" * remove events since they are not used * add broadcast method to network peer * add and wire localstore * use localstore in the node * wip * logging * move file utils * use the constant * updating deps * fix memstore * use latest libp2p unstable * fix tests * rework block streaming * don't fail storing if the block already exists * add helper info endpoint * correct comment * rename localstore to fsstore * fix tests * remove unused tests * add test to retrieve one block * move some test files around * consolidate setup * Update dagger/blockexchange/engine.nim Co-authored-by: Tanguy <tanguy@status.im> * typo * better block path handling * don't inherit rootobj * remove useless template * Update tests/dagger/blockexc/testblockexc.nim Co-authored-by: markspanbroek <mark@spanbroek.net> * use isMainModule * use proper flag for starter/stoped * cleanup optional use * wrap in isMainModule * use `cancelAndAwait` * remove unused imports * wip * don't use optional * use functional error api * rework store tests and add fs tests * Block.new() to Block.init() * don't use optional for engine blocks * use result instead of optional for getBlock * remove unused imports * move stopping servers to `shutdown` * use result instead of optional * rework with results * fix tests * use waitFor in signal handlers * error helper * use `?` and mapFailure where possible * remove unnecesary `=?` * improve empty cid digest initialization Co-authored-by: Tanguy <tanguy@status.im> Co-authored-by: markspanbroek <mark@spanbroek.net>
2022-01-10 15:32:56 +00:00
while true:
let chunk = await chunker2.getBytes()
if chunk.len <= 0:
break
blocks2.add(bt.Block.new(chunk).tryGet())
Poc 2 (#7) * moving protobuf into bitswap * adding block type * reworking bitswap * adding chunker * adding license header * use 1.2.6 * adding fixed size chunker * add blockstore * add iterator to chunker * more bitswap changes * rename ipfs to dagger * rename to dagger * blockstore inherits from BlockProvider * wip - add core block handling logic * smal changes * use proper block store methods * adding asynq heapqueue * wip prepare for bitswap task runner * adding `$` * adding memory store and tests * fixed chunking * extracted desicion engine from bitswap * added helper random funcs * adding testing helpers * only handle seqs * add peer events * cleanup pending blocks on blockstore event * allow nil handlers * move protobuf type helpers * allow initializing block from Cid * testing and fixes * small fixes * expose `<` * spelling * default value * spelling * pending blocks manager * adding stores manager * more tests a wip around bitswap * small changes * merge bitswap and engine for now * for now run only the new poc's tests * add a more complete ci setup * use template in map * remove p2pd * remove go * dont use asyncCheck * few small changes * adding ability to update items * adding multiple task runners * handle cancelation properly * use Result instead of throwing * wip bitswap tests * moving things around * split out engine again * add request and handlers interface * fix tests * wip - engine tests * remove unused imports * fix tests * cleanup block requesting logic * add block request tests * more block requests * add support for max heap * don't use result * use max heap & send block presence in task handler * add task handler tests * rename store to localStore * cleanup & logging * cancel task on stop * don't depend on local store for events * dont use heap queue for wants * add chronicles * fix issue with peer wants * add test for delayed block sends * remove obsolete tests * wip chunker * run all tests * add todo * misc * remove irrelevant files * removing more files * adding helpers for bitswap tests * moved bitswap file * misc * make blocks timeout longer * adjust block timeout * speedup test * compile with threads * import missing crypto * misc * disable threads for now * fix 32 bit platforms * re-enable threads support in tests
2021-02-26 00:23:22 +00:00
switch1 = newStandardSwitch()
switch2 = newStandardSwitch()
2021-04-19 14:37:38 +00:00
wallet1 = WalletRef.example
wallet2 = WalletRef.example
pricing1 = Pricing.example
pricing2 = Pricing.example
Node setup (#32) * settup basic nim node * adding http utils * adding confutils * rough rest api proto * adding missing deps * turn tls emulation off * adding toml serialization * wip * adding missing deps * make sure to clean old state in teardown * adding file upload rest endpoint * renaming blockexchange to networkstore * updating nim-presto * updating libp2p * wip adding streaming upload * reworked chunking * bump to latest unstable * adding asyncfutures stream * make streamable * deleting unused files * reworking stores api * use new stores api * rework blockset and remove blockstream * don't return option from constructor * rework chunker * wip implement upload * fix tests * move unrelated logic to engine * don't print entire message * logging * basic encode/decode to/from dag-pb * add basic upload/download support * fix tests * renaming blockset to manifest * don't pass config to node * remove config and use new manifest * wip: make endpoints more reliable * wip: adding node tests * include correct manifest test * removing asyncfutures * proper chunking of files * simplify stream reading * test with encoding/decoding with many blocks * add block storing tests * adding retrieval test * add logging * tidy up chunker * tidy up manifest and node * use default chunk size * fix tests * fix tests * make sure Eof is set properly * wip * minor cleanup * add file utils * cleanup config * splitout DaggerServer and "main" * remove events since they are not used * add broadcast method to network peer * add and wire localstore * use localstore in the node * wip * logging * move file utils * use the constant * updating deps * fix memstore * use latest libp2p unstable * fix tests * rework block streaming * don't fail storing if the block already exists * add helper info endpoint * correct comment * rename localstore to fsstore * fix tests * remove unused tests * add test to retrieve one block * move some test files around * consolidate setup * Update dagger/blockexchange/engine.nim Co-authored-by: Tanguy <tanguy@status.im> * typo * better block path handling * don't inherit rootobj * remove useless template * Update tests/dagger/blockexc/testblockexc.nim Co-authored-by: markspanbroek <mark@spanbroek.net> * use isMainModule * use proper flag for starter/stoped * cleanup optional use * wrap in isMainModule * use `cancelAndAwait` * remove unused imports * wip * don't use optional * use functional error api * rework store tests and add fs tests * Block.new() to Block.init() * don't use optional for engine blocks * use result instead of optional for getBlock * remove unused imports * move stopping servers to `shutdown` * use result instead of optional * rework with results * fix tests * use waitFor in signal handlers * error helper * use `?` and mapFailure where possible * remove unnecesary `=?` * improve empty cid digest initialization Co-authored-by: Tanguy <tanguy@status.im> Co-authored-by: markspanbroek <mark@spanbroek.net>
2022-01-10 15:32:56 +00:00
await switch1.start()
await switch2.start()
Poc 2 (#7) * moving protobuf into bitswap * adding block type * reworking bitswap * adding chunker * adding license header * use 1.2.6 * adding fixed size chunker * add blockstore * add iterator to chunker * more bitswap changes * rename ipfs to dagger * rename to dagger * blockstore inherits from BlockProvider * wip - add core block handling logic * smal changes * use proper block store methods * adding asynq heapqueue * wip prepare for bitswap task runner * adding `$` * adding memory store and tests * fixed chunking * extracted desicion engine from bitswap * added helper random funcs * adding testing helpers * only handle seqs * add peer events * cleanup pending blocks on blockstore event * allow nil handlers * move protobuf type helpers * allow initializing block from Cid * testing and fixes * small fixes * expose `<` * spelling * default value * spelling * pending blocks manager * adding stores manager * more tests a wip around bitswap * small changes * merge bitswap and engine for now * for now run only the new poc's tests * add a more complete ci setup * use template in map * remove p2pd * remove go * dont use asyncCheck * few small changes * adding ability to update items * adding multiple task runners * handle cancelation properly * use Result instead of throwing * wip bitswap tests * moving things around * split out engine again * add request and handlers interface * fix tests * wip - engine tests * remove unused imports * fix tests * cleanup block requesting logic * add block request tests * more block requests * add support for max heap * don't use result * use max heap & send block presence in task handler * add task handler tests * rename store to localStore * cleanup & logging * cancel task on stop * don't depend on local store for events * dont use heap queue for wants * add chronicles * fix issue with peer wants * add test for delayed block sends * remove obsolete tests * wip chunker * run all tests * add todo * misc * remove irrelevant files * removing more files * adding helpers for bitswap tests * moved bitswap file * misc * make blocks timeout longer * adjust block timeout * speedup test * compile with threads * import missing crypto * misc * disable threads for now * fix 32 bit platforms * re-enable threads support in tests
2021-02-26 00:23:22 +00:00
peerId1 = switch1.peerInfo.peerId
peerId2 = switch2.peerInfo.peerId
localStore1 = CacheStore.new(blocks1.mapIt( it ))
network1 = BlockExcNetwork.new(switch = switch1)
Node setup (#32) * settup basic nim node * adding http utils * adding confutils * rough rest api proto * adding missing deps * turn tls emulation off * adding toml serialization * wip * adding missing deps * make sure to clean old state in teardown * adding file upload rest endpoint * renaming blockexchange to networkstore * updating nim-presto * updating libp2p * wip adding streaming upload * reworked chunking * bump to latest unstable * adding asyncfutures stream * make streamable * deleting unused files * reworking stores api * use new stores api * rework blockset and remove blockstream * don't return option from constructor * rework chunker * wip implement upload * fix tests * move unrelated logic to engine * don't print entire message * logging * basic encode/decode to/from dag-pb * add basic upload/download support * fix tests * renaming blockset to manifest * don't pass config to node * remove config and use new manifest * wip: make endpoints more reliable * wip: adding node tests * include correct manifest test * removing asyncfutures * proper chunking of files * simplify stream reading * test with encoding/decoding with many blocks * add block storing tests * adding retrieval test * add logging * tidy up chunker * tidy up manifest and node * use default chunk size * fix tests * fix tests * make sure Eof is set properly * wip * minor cleanup * add file utils * cleanup config * splitout DaggerServer and "main" * remove events since they are not used * add broadcast method to network peer * add and wire localstore * use localstore in the node * wip * logging * move file utils * use the constant * updating deps * fix memstore * use latest libp2p unstable * fix tests * rework block streaming * don't fail storing if the block already exists * add helper info endpoint * correct comment * rename localstore to fsstore * fix tests * remove unused tests * add test to retrieve one block * move some test files around * consolidate setup * Update dagger/blockexchange/engine.nim Co-authored-by: Tanguy <tanguy@status.im> * typo * better block path handling * don't inherit rootobj * remove useless template * Update tests/dagger/blockexc/testblockexc.nim Co-authored-by: markspanbroek <mark@spanbroek.net> * use isMainModule * use proper flag for starter/stoped * cleanup optional use * wrap in isMainModule * use `cancelAndAwait` * remove unused imports * wip * don't use optional * use functional error api * rework store tests and add fs tests * Block.new() to Block.init() * don't use optional for engine blocks * use result instead of optional for getBlock * remove unused imports * move stopping servers to `shutdown` * use result instead of optional * rework with results * fix tests * use waitFor in signal handlers * error helper * use `?` and mapFailure where possible * remove unnecesary `=?` * improve empty cid digest initialization Co-authored-by: Tanguy <tanguy@status.im> Co-authored-by: markspanbroek <mark@spanbroek.net>
2022-01-10 15:32:56 +00:00
engine1 = BlockExcEngine.new(localStore1, wallet1, network1)
blockexc1 = NetworkStore.new(engine1, localStore1)
Poc 2 (#7) * moving protobuf into bitswap * adding block type * reworking bitswap * adding chunker * adding license header * use 1.2.6 * adding fixed size chunker * add blockstore * add iterator to chunker * more bitswap changes * rename ipfs to dagger * rename to dagger * blockstore inherits from BlockProvider * wip - add core block handling logic * smal changes * use proper block store methods * adding asynq heapqueue * wip prepare for bitswap task runner * adding `$` * adding memory store and tests * fixed chunking * extracted desicion engine from bitswap * added helper random funcs * adding testing helpers * only handle seqs * add peer events * cleanup pending blocks on blockstore event * allow nil handlers * move protobuf type helpers * allow initializing block from Cid * testing and fixes * small fixes * expose `<` * spelling * default value * spelling * pending blocks manager * adding stores manager * more tests a wip around bitswap * small changes * merge bitswap and engine for now * for now run only the new poc's tests * add a more complete ci setup * use template in map * remove p2pd * remove go * dont use asyncCheck * few small changes * adding ability to update items * adding multiple task runners * handle cancelation properly * use Result instead of throwing * wip bitswap tests * moving things around * split out engine again * add request and handlers interface * fix tests * wip - engine tests * remove unused imports * fix tests * cleanup block requesting logic * add block request tests * more block requests * add support for max heap * don't use result * use max heap & send block presence in task handler * add task handler tests * rename store to localStore * cleanup & logging * cancel task on stop * don't depend on local store for events * dont use heap queue for wants * add chronicles * fix issue with peer wants * add test for delayed block sends * remove obsolete tests * wip chunker * run all tests * add todo * misc * remove irrelevant files * removing more files * adding helpers for bitswap tests * moved bitswap file * misc * make blocks timeout longer * adjust block timeout * speedup test * compile with threads * import missing crypto * misc * disable threads for now * fix 32 bit platforms * re-enable threads support in tests
2021-02-26 00:23:22 +00:00
switch1.mount(network1)
localStore2 = CacheStore.new(blocks2.mapIt( it ))
network2 = BlockExcNetwork.new(switch = switch2)
Node setup (#32) * settup basic nim node * adding http utils * adding confutils * rough rest api proto * adding missing deps * turn tls emulation off * adding toml serialization * wip * adding missing deps * make sure to clean old state in teardown * adding file upload rest endpoint * renaming blockexchange to networkstore * updating nim-presto * updating libp2p * wip adding streaming upload * reworked chunking * bump to latest unstable * adding asyncfutures stream * make streamable * deleting unused files * reworking stores api * use new stores api * rework blockset and remove blockstream * don't return option from constructor * rework chunker * wip implement upload * fix tests * move unrelated logic to engine * don't print entire message * logging * basic encode/decode to/from dag-pb * add basic upload/download support * fix tests * renaming blockset to manifest * don't pass config to node * remove config and use new manifest * wip: make endpoints more reliable * wip: adding node tests * include correct manifest test * removing asyncfutures * proper chunking of files * simplify stream reading * test with encoding/decoding with many blocks * add block storing tests * adding retrieval test * add logging * tidy up chunker * tidy up manifest and node * use default chunk size * fix tests * fix tests * make sure Eof is set properly * wip * minor cleanup * add file utils * cleanup config * splitout DaggerServer and "main" * remove events since they are not used * add broadcast method to network peer * add and wire localstore * use localstore in the node * wip * logging * move file utils * use the constant * updating deps * fix memstore * use latest libp2p unstable * fix tests * rework block streaming * don't fail storing if the block already exists * add helper info endpoint * correct comment * rename localstore to fsstore * fix tests * remove unused tests * add test to retrieve one block * move some test files around * consolidate setup * Update dagger/blockexchange/engine.nim Co-authored-by: Tanguy <tanguy@status.im> * typo * better block path handling * don't inherit rootobj * remove useless template * Update tests/dagger/blockexc/testblockexc.nim Co-authored-by: markspanbroek <mark@spanbroek.net> * use isMainModule * use proper flag for starter/stoped * cleanup optional use * wrap in isMainModule * use `cancelAndAwait` * remove unused imports * wip * don't use optional * use functional error api * rework store tests and add fs tests * Block.new() to Block.init() * don't use optional for engine blocks * use result instead of optional for getBlock * remove unused imports * move stopping servers to `shutdown` * use result instead of optional * rework with results * fix tests * use waitFor in signal handlers * error helper * use `?` and mapFailure where possible * remove unnecesary `=?` * improve empty cid digest initialization Co-authored-by: Tanguy <tanguy@status.im> Co-authored-by: markspanbroek <mark@spanbroek.net>
2022-01-10 15:32:56 +00:00
engine2 = BlockExcEngine.new(localStore2, wallet2, network2)
blockexc2 = NetworkStore.new(engine2, localStore2)
Poc 2 (#7) * moving protobuf into bitswap * adding block type * reworking bitswap * adding chunker * adding license header * use 1.2.6 * adding fixed size chunker * add blockstore * add iterator to chunker * more bitswap changes * rename ipfs to dagger * rename to dagger * blockstore inherits from BlockProvider * wip - add core block handling logic * smal changes * use proper block store methods * adding asynq heapqueue * wip prepare for bitswap task runner * adding `$` * adding memory store and tests * fixed chunking * extracted desicion engine from bitswap * added helper random funcs * adding testing helpers * only handle seqs * add peer events * cleanup pending blocks on blockstore event * allow nil handlers * move protobuf type helpers * allow initializing block from Cid * testing and fixes * small fixes * expose `<` * spelling * default value * spelling * pending blocks manager * adding stores manager * more tests a wip around bitswap * small changes * merge bitswap and engine for now * for now run only the new poc's tests * add a more complete ci setup * use template in map * remove p2pd * remove go * dont use asyncCheck * few small changes * adding ability to update items * adding multiple task runners * handle cancelation properly * use Result instead of throwing * wip bitswap tests * moving things around * split out engine again * add request and handlers interface * fix tests * wip - engine tests * remove unused imports * fix tests * cleanup block requesting logic * add block request tests * more block requests * add support for max heap * don't use result * use max heap & send block presence in task handler * add task handler tests * rename store to localStore * cleanup & logging * cancel task on stop * don't depend on local store for events * dont use heap queue for wants * add chronicles * fix issue with peer wants * add test for delayed block sends * remove obsolete tests * wip chunker * run all tests * add todo * misc * remove irrelevant files * removing more files * adding helpers for bitswap tests * moved bitswap file * misc * make blocks timeout longer * adjust block timeout * speedup test * compile with threads * import missing crypto * misc * disable threads for now * fix 32 bit platforms * re-enable threads support in tests
2021-02-26 00:23:22 +00:00
switch2.mount(network2)
await allFuturesThrowing(
Node setup (#32) * settup basic nim node * adding http utils * adding confutils * rough rest api proto * adding missing deps * turn tls emulation off * adding toml serialization * wip * adding missing deps * make sure to clean old state in teardown * adding file upload rest endpoint * renaming blockexchange to networkstore * updating nim-presto * updating libp2p * wip adding streaming upload * reworked chunking * bump to latest unstable * adding asyncfutures stream * make streamable * deleting unused files * reworking stores api * use new stores api * rework blockset and remove blockstream * don't return option from constructor * rework chunker * wip implement upload * fix tests * move unrelated logic to engine * don't print entire message * logging * basic encode/decode to/from dag-pb * add basic upload/download support * fix tests * renaming blockset to manifest * don't pass config to node * remove config and use new manifest * wip: make endpoints more reliable * wip: adding node tests * include correct manifest test * removing asyncfutures * proper chunking of files * simplify stream reading * test with encoding/decoding with many blocks * add block storing tests * adding retrieval test * add logging * tidy up chunker * tidy up manifest and node * use default chunk size * fix tests * fix tests * make sure Eof is set properly * wip * minor cleanup * add file utils * cleanup config * splitout DaggerServer and "main" * remove events since they are not used * add broadcast method to network peer * add and wire localstore * use localstore in the node * wip * logging * move file utils * use the constant * updating deps * fix memstore * use latest libp2p unstable * fix tests * rework block streaming * don't fail storing if the block already exists * add helper info endpoint * correct comment * rename localstore to fsstore * fix tests * remove unused tests * add test to retrieve one block * move some test files around * consolidate setup * Update dagger/blockexchange/engine.nim Co-authored-by: Tanguy <tanguy@status.im> * typo * better block path handling * don't inherit rootobj * remove useless template * Update tests/dagger/blockexc/testblockexc.nim Co-authored-by: markspanbroek <mark@spanbroek.net> * use isMainModule * use proper flag for starter/stoped * cleanup optional use * wrap in isMainModule * use `cancelAndAwait` * remove unused imports * wip * don't use optional * use functional error api * rework store tests and add fs tests * Block.new() to Block.init() * don't use optional for engine blocks * use result instead of optional for getBlock * remove unused imports * move stopping servers to `shutdown` * use result instead of optional * rework with results * fix tests * use waitFor in signal handlers * error helper * use `?` and mapFailure where possible * remove unnecesary `=?` * improve empty cid digest initialization Co-authored-by: Tanguy <tanguy@status.im> Co-authored-by: markspanbroek <mark@spanbroek.net>
2022-01-10 15:32:56 +00:00
engine1.start(),
engine2.start(),
Poc 2 (#7) * moving protobuf into bitswap * adding block type * reworking bitswap * adding chunker * adding license header * use 1.2.6 * adding fixed size chunker * add blockstore * add iterator to chunker * more bitswap changes * rename ipfs to dagger * rename to dagger * blockstore inherits from BlockProvider * wip - add core block handling logic * smal changes * use proper block store methods * adding asynq heapqueue * wip prepare for bitswap task runner * adding `$` * adding memory store and tests * fixed chunking * extracted desicion engine from bitswap * added helper random funcs * adding testing helpers * only handle seqs * add peer events * cleanup pending blocks on blockstore event * allow nil handlers * move protobuf type helpers * allow initializing block from Cid * testing and fixes * small fixes * expose `<` * spelling * default value * spelling * pending blocks manager * adding stores manager * more tests a wip around bitswap * small changes * merge bitswap and engine for now * for now run only the new poc's tests * add a more complete ci setup * use template in map * remove p2pd * remove go * dont use asyncCheck * few small changes * adding ability to update items * adding multiple task runners * handle cancelation properly * use Result instead of throwing * wip bitswap tests * moving things around * split out engine again * add request and handlers interface * fix tests * wip - engine tests * remove unused imports * fix tests * cleanup block requesting logic * add block request tests * more block requests * add support for max heap * don't use result * use max heap & send block presence in task handler * add task handler tests * rename store to localStore * cleanup & logging * cancel task on stop * don't depend on local store for events * dont use heap queue for wants * add chronicles * fix issue with peer wants * add test for delayed block sends * remove obsolete tests * wip chunker * run all tests * add todo * misc * remove irrelevant files * removing more files * adding helpers for bitswap tests * moved bitswap file * misc * make blocks timeout longer * adjust block timeout * speedup test * compile with threads * import missing crypto * misc * disable threads for now * fix 32 bit platforms * re-enable threads support in tests
2021-02-26 00:23:22 +00:00
)
# initialize our want lists
blockexc1.engine.wantList = blocks2.mapIt( it.cid )
blockexc2.engine.wantList = blocks1.mapIt( it.cid )
Poc 2 (#7) * moving protobuf into bitswap * adding block type * reworking bitswap * adding chunker * adding license header * use 1.2.6 * adding fixed size chunker * add blockstore * add iterator to chunker * more bitswap changes * rename ipfs to dagger * rename to dagger * blockstore inherits from BlockProvider * wip - add core block handling logic * smal changes * use proper block store methods * adding asynq heapqueue * wip prepare for bitswap task runner * adding `$` * adding memory store and tests * fixed chunking * extracted desicion engine from bitswap * added helper random funcs * adding testing helpers * only handle seqs * add peer events * cleanup pending blocks on blockstore event * allow nil handlers * move protobuf type helpers * allow initializing block from Cid * testing and fixes * small fixes * expose `<` * spelling * default value * spelling * pending blocks manager * adding stores manager * more tests a wip around bitswap * small changes * merge bitswap and engine for now * for now run only the new poc's tests * add a more complete ci setup * use template in map * remove p2pd * remove go * dont use asyncCheck * few small changes * adding ability to update items * adding multiple task runners * handle cancelation properly * use Result instead of throwing * wip bitswap tests * moving things around * split out engine again * add request and handlers interface * fix tests * wip - engine tests * remove unused imports * fix tests * cleanup block requesting logic * add block request tests * more block requests * add support for max heap * don't use result * use max heap & send block presence in task handler * add task handler tests * rename store to localStore * cleanup & logging * cancel task on stop * don't depend on local store for events * dont use heap queue for wants * add chronicles * fix issue with peer wants * add test for delayed block sends * remove obsolete tests * wip chunker * run all tests * add todo * misc * remove irrelevant files * removing more files * adding helpers for bitswap tests * moved bitswap file * misc * make blocks timeout longer * adjust block timeout * speedup test * compile with threads * import missing crypto * misc * disable threads for now * fix 32 bit platforms * re-enable threads support in tests
2021-02-26 00:23:22 +00:00
pricing1.address = wallet1.address
pricing2.address = wallet2.address
blockexc1.engine.pricing = pricing1.some
blockexc2.engine.pricing = pricing2.some
Poc 2 (#7) * moving protobuf into bitswap * adding block type * reworking bitswap * adding chunker * adding license header * use 1.2.6 * adding fixed size chunker * add blockstore * add iterator to chunker * more bitswap changes * rename ipfs to dagger * rename to dagger * blockstore inherits from BlockProvider * wip - add core block handling logic * smal changes * use proper block store methods * adding asynq heapqueue * wip prepare for bitswap task runner * adding `$` * adding memory store and tests * fixed chunking * extracted desicion engine from bitswap * added helper random funcs * adding testing helpers * only handle seqs * add peer events * cleanup pending blocks on blockstore event * allow nil handlers * move protobuf type helpers * allow initializing block from Cid * testing and fixes * small fixes * expose `<` * spelling * default value * spelling * pending blocks manager * adding stores manager * more tests a wip around bitswap * small changes * merge bitswap and engine for now * for now run only the new poc's tests * add a more complete ci setup * use template in map * remove p2pd * remove go * dont use asyncCheck * few small changes * adding ability to update items * adding multiple task runners * handle cancelation properly * use Result instead of throwing * wip bitswap tests * moving things around * split out engine again * add request and handlers interface * fix tests * wip - engine tests * remove unused imports * fix tests * cleanup block requesting logic * add block request tests * more block requests * add support for max heap * don't use result * use max heap & send block presence in task handler * add task handler tests * rename store to localStore * cleanup & logging * cancel task on stop * don't depend on local store for events * dont use heap queue for wants * add chronicles * fix issue with peer wants * add test for delayed block sends * remove obsolete tests * wip chunker * run all tests * add todo * misc * remove irrelevant files * removing more files * adding helpers for bitswap tests * moved bitswap file * misc * make blocks timeout longer * adjust block timeout * speedup test * compile with threads * import missing crypto * misc * disable threads for now * fix 32 bit platforms * re-enable threads support in tests
2021-02-26 00:23:22 +00:00
await switch1.connect(
switch2.peerInfo.peerId,
switch2.peerInfo.addrs)
await sleepAsync(1.seconds) # give some time to exchange lists
peerCtx2 = blockexc1.engine.getPeerCtx(peerId2)
peerCtx1 = blockexc2.engine.getPeerCtx(peerId1)
Poc 2 (#7) * moving protobuf into bitswap * adding block type * reworking bitswap * adding chunker * adding license header * use 1.2.6 * adding fixed size chunker * add blockstore * add iterator to chunker * more bitswap changes * rename ipfs to dagger * rename to dagger * blockstore inherits from BlockProvider * wip - add core block handling logic * smal changes * use proper block store methods * adding asynq heapqueue * wip prepare for bitswap task runner * adding `$` * adding memory store and tests * fixed chunking * extracted desicion engine from bitswap * added helper random funcs * adding testing helpers * only handle seqs * add peer events * cleanup pending blocks on blockstore event * allow nil handlers * move protobuf type helpers * allow initializing block from Cid * testing and fixes * small fixes * expose `<` * spelling * default value * spelling * pending blocks manager * adding stores manager * more tests a wip around bitswap * small changes * merge bitswap and engine for now * for now run only the new poc's tests * add a more complete ci setup * use template in map * remove p2pd * remove go * dont use asyncCheck * few small changes * adding ability to update items * adding multiple task runners * handle cancelation properly * use Result instead of throwing * wip bitswap tests * moving things around * split out engine again * add request and handlers interface * fix tests * wip - engine tests * remove unused imports * fix tests * cleanup block requesting logic * add block request tests * more block requests * add support for max heap * don't use result * use max heap & send block presence in task handler * add task handler tests * rename store to localStore * cleanup & logging * cancel task on stop * don't depend on local store for events * dont use heap queue for wants * add chronicles * fix issue with peer wants * add test for delayed block sends * remove obsolete tests * wip chunker * run all tests * add todo * misc * remove irrelevant files * removing more files * adding helpers for bitswap tests * moved bitswap file * misc * make blocks timeout longer * adjust block timeout * speedup test * compile with threads * import missing crypto * misc * disable threads for now * fix 32 bit platforms * re-enable threads support in tests
2021-02-26 00:23:22 +00:00
teardown:
await allFuturesThrowing(
Node setup (#32) * settup basic nim node * adding http utils * adding confutils * rough rest api proto * adding missing deps * turn tls emulation off * adding toml serialization * wip * adding missing deps * make sure to clean old state in teardown * adding file upload rest endpoint * renaming blockexchange to networkstore * updating nim-presto * updating libp2p * wip adding streaming upload * reworked chunking * bump to latest unstable * adding asyncfutures stream * make streamable * deleting unused files * reworking stores api * use new stores api * rework blockset and remove blockstream * don't return option from constructor * rework chunker * wip implement upload * fix tests * move unrelated logic to engine * don't print entire message * logging * basic encode/decode to/from dag-pb * add basic upload/download support * fix tests * renaming blockset to manifest * don't pass config to node * remove config and use new manifest * wip: make endpoints more reliable * wip: adding node tests * include correct manifest test * removing asyncfutures * proper chunking of files * simplify stream reading * test with encoding/decoding with many blocks * add block storing tests * adding retrieval test * add logging * tidy up chunker * tidy up manifest and node * use default chunk size * fix tests * fix tests * make sure Eof is set properly * wip * minor cleanup * add file utils * cleanup config * splitout DaggerServer and "main" * remove events since they are not used * add broadcast method to network peer * add and wire localstore * use localstore in the node * wip * logging * move file utils * use the constant * updating deps * fix memstore * use latest libp2p unstable * fix tests * rework block streaming * don't fail storing if the block already exists * add helper info endpoint * correct comment * rename localstore to fsstore * fix tests * remove unused tests * add test to retrieve one block * move some test files around * consolidate setup * Update dagger/blockexchange/engine.nim Co-authored-by: Tanguy <tanguy@status.im> * typo * better block path handling * don't inherit rootobj * remove useless template * Update tests/dagger/blockexc/testblockexc.nim Co-authored-by: markspanbroek <mark@spanbroek.net> * use isMainModule * use proper flag for starter/stoped * cleanup optional use * wrap in isMainModule * use `cancelAndAwait` * remove unused imports * wip * don't use optional * use functional error api * rework store tests and add fs tests * Block.new() to Block.init() * don't use optional for engine blocks * use result instead of optional for getBlock * remove unused imports * move stopping servers to `shutdown` * use result instead of optional * rework with results * fix tests * use waitFor in signal handlers * error helper * use `?` and mapFailure where possible * remove unnecesary `=?` * improve empty cid digest initialization Co-authored-by: Tanguy <tanguy@status.im> Co-authored-by: markspanbroek <mark@spanbroek.net>
2022-01-10 15:32:56 +00:00
engine1.stop(),
engine2.stop(),
Poc 2 (#7) * moving protobuf into bitswap * adding block type * reworking bitswap * adding chunker * adding license header * use 1.2.6 * adding fixed size chunker * add blockstore * add iterator to chunker * more bitswap changes * rename ipfs to dagger * rename to dagger * blockstore inherits from BlockProvider * wip - add core block handling logic * smal changes * use proper block store methods * adding asynq heapqueue * wip prepare for bitswap task runner * adding `$` * adding memory store and tests * fixed chunking * extracted desicion engine from bitswap * added helper random funcs * adding testing helpers * only handle seqs * add peer events * cleanup pending blocks on blockstore event * allow nil handlers * move protobuf type helpers * allow initializing block from Cid * testing and fixes * small fixes * expose `<` * spelling * default value * spelling * pending blocks manager * adding stores manager * more tests a wip around bitswap * small changes * merge bitswap and engine for now * for now run only the new poc's tests * add a more complete ci setup * use template in map * remove p2pd * remove go * dont use asyncCheck * few small changes * adding ability to update items * adding multiple task runners * handle cancelation properly * use Result instead of throwing * wip bitswap tests * moving things around * split out engine again * add request and handlers interface * fix tests * wip - engine tests * remove unused imports * fix tests * cleanup block requesting logic * add block request tests * more block requests * add support for max heap * don't use result * use max heap & send block presence in task handler * add task handler tests * rename store to localStore * cleanup & logging * cancel task on stop * don't depend on local store for events * dont use heap queue for wants * add chronicles * fix issue with peer wants * add test for delayed block sends * remove obsolete tests * wip chunker * run all tests * add todo * misc * remove irrelevant files * removing more files * adding helpers for bitswap tests * moved bitswap file * misc * make blocks timeout longer * adjust block timeout * speedup test * compile with threads * import missing crypto * misc * disable threads for now * fix 32 bit platforms * re-enable threads support in tests
2021-02-26 00:23:22 +00:00
switch1.stop(),
switch2.stop())
test "should exchange want lists on connect":
check not isNil(peerCtx1)
check not isNil(peerCtx2)
check:
peerCtx1.peerHave.mapIt( $it ).sorted(cmp[string]) ==
blockexc2.engine.wantList.mapIt( $it ).sorted(cmp[string])
Poc 2 (#7) * moving protobuf into bitswap * adding block type * reworking bitswap * adding chunker * adding license header * use 1.2.6 * adding fixed size chunker * add blockstore * add iterator to chunker * more bitswap changes * rename ipfs to dagger * rename to dagger * blockstore inherits from BlockProvider * wip - add core block handling logic * smal changes * use proper block store methods * adding asynq heapqueue * wip prepare for bitswap task runner * adding `$` * adding memory store and tests * fixed chunking * extracted desicion engine from bitswap * added helper random funcs * adding testing helpers * only handle seqs * add peer events * cleanup pending blocks on blockstore event * allow nil handlers * move protobuf type helpers * allow initializing block from Cid * testing and fixes * small fixes * expose `<` * spelling * default value * spelling * pending blocks manager * adding stores manager * more tests a wip around bitswap * small changes * merge bitswap and engine for now * for now run only the new poc's tests * add a more complete ci setup * use template in map * remove p2pd * remove go * dont use asyncCheck * few small changes * adding ability to update items * adding multiple task runners * handle cancelation properly * use Result instead of throwing * wip bitswap tests * moving things around * split out engine again * add request and handlers interface * fix tests * wip - engine tests * remove unused imports * fix tests * cleanup block requesting logic * add block request tests * more block requests * add support for max heap * don't use result * use max heap & send block presence in task handler * add task handler tests * rename store to localStore * cleanup & logging * cancel task on stop * don't depend on local store for events * dont use heap queue for wants * add chronicles * fix issue with peer wants * add test for delayed block sends * remove obsolete tests * wip chunker * run all tests * add todo * misc * remove irrelevant files * removing more files * adding helpers for bitswap tests * moved bitswap file * misc * make blocks timeout longer * adjust block timeout * speedup test * compile with threads * import missing crypto * misc * disable threads for now * fix 32 bit platforms * re-enable threads support in tests
2021-02-26 00:23:22 +00:00
peerCtx2.peerHave.mapIt( $it ).sorted(cmp[string]) ==
blockexc1.engine.wantList.mapIt( $it ).sorted(cmp[string])
Poc 2 (#7) * moving protobuf into bitswap * adding block type * reworking bitswap * adding chunker * adding license header * use 1.2.6 * adding fixed size chunker * add blockstore * add iterator to chunker * more bitswap changes * rename ipfs to dagger * rename to dagger * blockstore inherits from BlockProvider * wip - add core block handling logic * smal changes * use proper block store methods * adding asynq heapqueue * wip prepare for bitswap task runner * adding `$` * adding memory store and tests * fixed chunking * extracted desicion engine from bitswap * added helper random funcs * adding testing helpers * only handle seqs * add peer events * cleanup pending blocks on blockstore event * allow nil handlers * move protobuf type helpers * allow initializing block from Cid * testing and fixes * small fixes * expose `<` * spelling * default value * spelling * pending blocks manager * adding stores manager * more tests a wip around bitswap * small changes * merge bitswap and engine for now * for now run only the new poc's tests * add a more complete ci setup * use template in map * remove p2pd * remove go * dont use asyncCheck * few small changes * adding ability to update items * adding multiple task runners * handle cancelation properly * use Result instead of throwing * wip bitswap tests * moving things around * split out engine again * add request and handlers interface * fix tests * wip - engine tests * remove unused imports * fix tests * cleanup block requesting logic * add block request tests * more block requests * add support for max heap * don't use result * use max heap & send block presence in task handler * add task handler tests * rename store to localStore * cleanup & logging * cancel task on stop * don't depend on local store for events * dont use heap queue for wants * add chronicles * fix issue with peer wants * add test for delayed block sends * remove obsolete tests * wip chunker * run all tests * add todo * misc * remove irrelevant files * removing more files * adding helpers for bitswap tests * moved bitswap file * misc * make blocks timeout longer * adjust block timeout * speedup test * compile with threads * import missing crypto * misc * disable threads for now * fix 32 bit platforms * re-enable threads support in tests
2021-02-26 00:23:22 +00:00
test "exchanges accounts on connect":
check peerCtx1.account.?address == pricing1.address.some
check peerCtx2.account.?address == pricing2.address.some
Poc 2 (#7) * moving protobuf into bitswap * adding block type * reworking bitswap * adding chunker * adding license header * use 1.2.6 * adding fixed size chunker * add blockstore * add iterator to chunker * more bitswap changes * rename ipfs to dagger * rename to dagger * blockstore inherits from BlockProvider * wip - add core block handling logic * smal changes * use proper block store methods * adding asynq heapqueue * wip prepare for bitswap task runner * adding `$` * adding memory store and tests * fixed chunking * extracted desicion engine from bitswap * added helper random funcs * adding testing helpers * only handle seqs * add peer events * cleanup pending blocks on blockstore event * allow nil handlers * move protobuf type helpers * allow initializing block from Cid * testing and fixes * small fixes * expose `<` * spelling * default value * spelling * pending blocks manager * adding stores manager * more tests a wip around bitswap * small changes * merge bitswap and engine for now * for now run only the new poc's tests * add a more complete ci setup * use template in map * remove p2pd * remove go * dont use asyncCheck * few small changes * adding ability to update items * adding multiple task runners * handle cancelation properly * use Result instead of throwing * wip bitswap tests * moving things around * split out engine again * add request and handlers interface * fix tests * wip - engine tests * remove unused imports * fix tests * cleanup block requesting logic * add block request tests * more block requests * add support for max heap * don't use result * use max heap & send block presence in task handler * add task handler tests * rename store to localStore * cleanup & logging * cancel task on stop * don't depend on local store for events * dont use heap queue for wants * add chronicles * fix issue with peer wants * add test for delayed block sends * remove obsolete tests * wip chunker * run all tests * add todo * misc * remove irrelevant files * removing more files * adding helpers for bitswap tests * moved bitswap file * misc * make blocks timeout longer * adjust block timeout * speedup test * compile with threads * import missing crypto * misc * disable threads for now * fix 32 bit platforms * re-enable threads support in tests
2021-02-26 00:23:22 +00:00
test "should send want-have for block":
let blk = bt.Block.new("Block 1".toBytes).tryGet()
Node setup (#32) * settup basic nim node * adding http utils * adding confutils * rough rest api proto * adding missing deps * turn tls emulation off * adding toml serialization * wip * adding missing deps * make sure to clean old state in teardown * adding file upload rest endpoint * renaming blockexchange to networkstore * updating nim-presto * updating libp2p * wip adding streaming upload * reworked chunking * bump to latest unstable * adding asyncfutures stream * make streamable * deleting unused files * reworking stores api * use new stores api * rework blockset and remove blockstream * don't return option from constructor * rework chunker * wip implement upload * fix tests * move unrelated logic to engine * don't print entire message * logging * basic encode/decode to/from dag-pb * add basic upload/download support * fix tests * renaming blockset to manifest * don't pass config to node * remove config and use new manifest * wip: make endpoints more reliable * wip: adding node tests * include correct manifest test * removing asyncfutures * proper chunking of files * simplify stream reading * test with encoding/decoding with many blocks * add block storing tests * adding retrieval test * add logging * tidy up chunker * tidy up manifest and node * use default chunk size * fix tests * fix tests * make sure Eof is set properly * wip * minor cleanup * add file utils * cleanup config * splitout DaggerServer and "main" * remove events since they are not used * add broadcast method to network peer * add and wire localstore * use localstore in the node * wip * logging * move file utils * use the constant * updating deps * fix memstore * use latest libp2p unstable * fix tests * rework block streaming * don't fail storing if the block already exists * add helper info endpoint * correct comment * rename localstore to fsstore * fix tests * remove unused tests * add test to retrieve one block * move some test files around * consolidate setup * Update dagger/blockexchange/engine.nim Co-authored-by: Tanguy <tanguy@status.im> * typo * better block path handling * don't inherit rootobj * remove useless template * Update tests/dagger/blockexc/testblockexc.nim Co-authored-by: markspanbroek <mark@spanbroek.net> * use isMainModule * use proper flag for starter/stoped * cleanup optional use * wrap in isMainModule * use `cancelAndAwait` * remove unused imports * wip * don't use optional * use functional error api * rework store tests and add fs tests * Block.new() to Block.init() * don't use optional for engine blocks * use result instead of optional for getBlock * remove unused imports * move stopping servers to `shutdown` * use result instead of optional * rework with results * fix tests * use waitFor in signal handlers * error helper * use `?` and mapFailure where possible * remove unnecesary `=?` * improve empty cid digest initialization Co-authored-by: Tanguy <tanguy@status.im> Co-authored-by: markspanbroek <mark@spanbroek.net>
2022-01-10 15:32:56 +00:00
check await blockexc2.engine.localStore.putBlock(blk)
Poc 2 (#7) * moving protobuf into bitswap * adding block type * reworking bitswap * adding chunker * adding license header * use 1.2.6 * adding fixed size chunker * add blockstore * add iterator to chunker * more bitswap changes * rename ipfs to dagger * rename to dagger * blockstore inherits from BlockProvider * wip - add core block handling logic * smal changes * use proper block store methods * adding asynq heapqueue * wip prepare for bitswap task runner * adding `$` * adding memory store and tests * fixed chunking * extracted desicion engine from bitswap * added helper random funcs * adding testing helpers * only handle seqs * add peer events * cleanup pending blocks on blockstore event * allow nil handlers * move protobuf type helpers * allow initializing block from Cid * testing and fixes * small fixes * expose `<` * spelling * default value * spelling * pending blocks manager * adding stores manager * more tests a wip around bitswap * small changes * merge bitswap and engine for now * for now run only the new poc's tests * add a more complete ci setup * use template in map * remove p2pd * remove go * dont use asyncCheck * few small changes * adding ability to update items * adding multiple task runners * handle cancelation properly * use Result instead of throwing * wip bitswap tests * moving things around * split out engine again * add request and handlers interface * fix tests * wip - engine tests * remove unused imports * fix tests * cleanup block requesting logic * add block request tests * more block requests * add support for max heap * don't use result * use max heap & send block presence in task handler * add task handler tests * rename store to localStore * cleanup & logging * cancel task on stop * don't depend on local store for events * dont use heap queue for wants * add chronicles * fix issue with peer wants * add test for delayed block sends * remove obsolete tests * wip chunker * run all tests * add todo * misc * remove irrelevant files * removing more files * adding helpers for bitswap tests * moved bitswap file * misc * make blocks timeout longer * adjust block timeout * speedup test * compile with threads * import missing crypto * misc * disable threads for now * fix 32 bit platforms * re-enable threads support in tests
2021-02-26 00:23:22 +00:00
let entry = Entry(
`block`: blk.cid.data.buffer,
priority: 1,
cancel: false,
wantType: WantType.wantBlock,
sendDontHave: false)
peerCtx1.peerWants.add(entry)
Node setup (#32) * settup basic nim node * adding http utils * adding confutils * rough rest api proto * adding missing deps * turn tls emulation off * adding toml serialization * wip * adding missing deps * make sure to clean old state in teardown * adding file upload rest endpoint * renaming blockexchange to networkstore * updating nim-presto * updating libp2p * wip adding streaming upload * reworked chunking * bump to latest unstable * adding asyncfutures stream * make streamable * deleting unused files * reworking stores api * use new stores api * rework blockset and remove blockstream * don't return option from constructor * rework chunker * wip implement upload * fix tests * move unrelated logic to engine * don't print entire message * logging * basic encode/decode to/from dag-pb * add basic upload/download support * fix tests * renaming blockset to manifest * don't pass config to node * remove config and use new manifest * wip: make endpoints more reliable * wip: adding node tests * include correct manifest test * removing asyncfutures * proper chunking of files * simplify stream reading * test with encoding/decoding with many blocks * add block storing tests * adding retrieval test * add logging * tidy up chunker * tidy up manifest and node * use default chunk size * fix tests * fix tests * make sure Eof is set properly * wip * minor cleanup * add file utils * cleanup config * splitout DaggerServer and "main" * remove events since they are not used * add broadcast method to network peer * add and wire localstore * use localstore in the node * wip * logging * move file utils * use the constant * updating deps * fix memstore * use latest libp2p unstable * fix tests * rework block streaming * don't fail storing if the block already exists * add helper info endpoint * correct comment * rename localstore to fsstore * fix tests * remove unused tests * add test to retrieve one block * move some test files around * consolidate setup * Update dagger/blockexchange/engine.nim Co-authored-by: Tanguy <tanguy@status.im> * typo * better block path handling * don't inherit rootobj * remove useless template * Update tests/dagger/blockexc/testblockexc.nim Co-authored-by: markspanbroek <mark@spanbroek.net> * use isMainModule * use proper flag for starter/stoped * cleanup optional use * wrap in isMainModule * use `cancelAndAwait` * remove unused imports * wip * don't use optional * use functional error api * rework store tests and add fs tests * Block.new() to Block.init() * don't use optional for engine blocks * use result instead of optional for getBlock * remove unused imports * move stopping servers to `shutdown` * use result instead of optional * rework with results * fix tests * use waitFor in signal handlers * error helper * use `?` and mapFailure where possible * remove unnecesary `=?` * improve empty cid digest initialization Co-authored-by: Tanguy <tanguy@status.im> Co-authored-by: markspanbroek <mark@spanbroek.net>
2022-01-10 15:32:56 +00:00
check blockexc2
.engine
.taskQueue
.pushOrUpdateNoWait(peerCtx1).isOk
Poc 2 (#7) * moving protobuf into bitswap * adding block type * reworking bitswap * adding chunker * adding license header * use 1.2.6 * adding fixed size chunker * add blockstore * add iterator to chunker * more bitswap changes * rename ipfs to dagger * rename to dagger * blockstore inherits from BlockProvider * wip - add core block handling logic * smal changes * use proper block store methods * adding asynq heapqueue * wip prepare for bitswap task runner * adding `$` * adding memory store and tests * fixed chunking * extracted desicion engine from bitswap * added helper random funcs * adding testing helpers * only handle seqs * add peer events * cleanup pending blocks on blockstore event * allow nil handlers * move protobuf type helpers * allow initializing block from Cid * testing and fixes * small fixes * expose `<` * spelling * default value * spelling * pending blocks manager * adding stores manager * more tests a wip around bitswap * small changes * merge bitswap and engine for now * for now run only the new poc's tests * add a more complete ci setup * use template in map * remove p2pd * remove go * dont use asyncCheck * few small changes * adding ability to update items * adding multiple task runners * handle cancelation properly * use Result instead of throwing * wip bitswap tests * moving things around * split out engine again * add request and handlers interface * fix tests * wip - engine tests * remove unused imports * fix tests * cleanup block requesting logic * add block request tests * more block requests * add support for max heap * don't use result * use max heap & send block presence in task handler * add task handler tests * rename store to localStore * cleanup & logging * cancel task on stop * don't depend on local store for events * dont use heap queue for wants * add chronicles * fix issue with peer wants * add test for delayed block sends * remove obsolete tests * wip chunker * run all tests * add todo * misc * remove irrelevant files * removing more files * adding helpers for bitswap tests * moved bitswap file * misc * make blocks timeout longer * adjust block timeout * speedup test * compile with threads * import missing crypto * misc * disable threads for now * fix 32 bit platforms * re-enable threads support in tests
2021-02-26 00:23:22 +00:00
await sleepAsync(100.millis)
check blockexc1.engine.localStore.hasBlock(blk.cid)
Poc 2 (#7) * moving protobuf into bitswap * adding block type * reworking bitswap * adding chunker * adding license header * use 1.2.6 * adding fixed size chunker * add blockstore * add iterator to chunker * more bitswap changes * rename ipfs to dagger * rename to dagger * blockstore inherits from BlockProvider * wip - add core block handling logic * smal changes * use proper block store methods * adding asynq heapqueue * wip prepare for bitswap task runner * adding `$` * adding memory store and tests * fixed chunking * extracted desicion engine from bitswap * added helper random funcs * adding testing helpers * only handle seqs * add peer events * cleanup pending blocks on blockstore event * allow nil handlers * move protobuf type helpers * allow initializing block from Cid * testing and fixes * small fixes * expose `<` * spelling * default value * spelling * pending blocks manager * adding stores manager * more tests a wip around bitswap * small changes * merge bitswap and engine for now * for now run only the new poc's tests * add a more complete ci setup * use template in map * remove p2pd * remove go * dont use asyncCheck * few small changes * adding ability to update items * adding multiple task runners * handle cancelation properly * use Result instead of throwing * wip bitswap tests * moving things around * split out engine again * add request and handlers interface * fix tests * wip - engine tests * remove unused imports * fix tests * cleanup block requesting logic * add block request tests * more block requests * add support for max heap * don't use result * use max heap & send block presence in task handler * add task handler tests * rename store to localStore * cleanup & logging * cancel task on stop * don't depend on local store for events * dont use heap queue for wants * add chronicles * fix issue with peer wants * add test for delayed block sends * remove obsolete tests * wip chunker * run all tests * add todo * misc * remove irrelevant files * removing more files * adding helpers for bitswap tests * moved bitswap file * misc * make blocks timeout longer * adjust block timeout * speedup test * compile with threads * import missing crypto * misc * disable threads for now * fix 32 bit platforms * re-enable threads support in tests
2021-02-26 00:23:22 +00:00
test "should get blocks from remote":
Node setup (#32) * settup basic nim node * adding http utils * adding confutils * rough rest api proto * adding missing deps * turn tls emulation off * adding toml serialization * wip * adding missing deps * make sure to clean old state in teardown * adding file upload rest endpoint * renaming blockexchange to networkstore * updating nim-presto * updating libp2p * wip adding streaming upload * reworked chunking * bump to latest unstable * adding asyncfutures stream * make streamable * deleting unused files * reworking stores api * use new stores api * rework blockset and remove blockstream * don't return option from constructor * rework chunker * wip implement upload * fix tests * move unrelated logic to engine * don't print entire message * logging * basic encode/decode to/from dag-pb * add basic upload/download support * fix tests * renaming blockset to manifest * don't pass config to node * remove config and use new manifest * wip: make endpoints more reliable * wip: adding node tests * include correct manifest test * removing asyncfutures * proper chunking of files * simplify stream reading * test with encoding/decoding with many blocks * add block storing tests * adding retrieval test * add logging * tidy up chunker * tidy up manifest and node * use default chunk size * fix tests * fix tests * make sure Eof is set properly * wip * minor cleanup * add file utils * cleanup config * splitout DaggerServer and "main" * remove events since they are not used * add broadcast method to network peer * add and wire localstore * use localstore in the node * wip * logging * move file utils * use the constant * updating deps * fix memstore * use latest libp2p unstable * fix tests * rework block streaming * don't fail storing if the block already exists * add helper info endpoint * correct comment * rename localstore to fsstore * fix tests * remove unused tests * add test to retrieve one block * move some test files around * consolidate setup * Update dagger/blockexchange/engine.nim Co-authored-by: Tanguy <tanguy@status.im> * typo * better block path handling * don't inherit rootobj * remove useless template * Update tests/dagger/blockexc/testblockexc.nim Co-authored-by: markspanbroek <mark@spanbroek.net> * use isMainModule * use proper flag for starter/stoped * cleanup optional use * wrap in isMainModule * use `cancelAndAwait` * remove unused imports * wip * don't use optional * use functional error api * rework store tests and add fs tests * Block.new() to Block.init() * don't use optional for engine blocks * use result instead of optional for getBlock * remove unused imports * move stopping servers to `shutdown` * use result instead of optional * rework with results * fix tests * use waitFor in signal handlers * error helper * use `?` and mapFailure where possible * remove unnecesary `=?` * improve empty cid digest initialization Co-authored-by: Tanguy <tanguy@status.im> Co-authored-by: markspanbroek <mark@spanbroek.net>
2022-01-10 15:32:56 +00:00
let blocks = await allFinished(
blocks2.mapIt( blockexc1.getBlock(it.cid) ))
check blocks.mapIt( !it.read ) == blocks2
Poc 2 (#7) * moving protobuf into bitswap * adding block type * reworking bitswap * adding chunker * adding license header * use 1.2.6 * adding fixed size chunker * add blockstore * add iterator to chunker * more bitswap changes * rename ipfs to dagger * rename to dagger * blockstore inherits from BlockProvider * wip - add core block handling logic * smal changes * use proper block store methods * adding asynq heapqueue * wip prepare for bitswap task runner * adding `$` * adding memory store and tests * fixed chunking * extracted desicion engine from bitswap * added helper random funcs * adding testing helpers * only handle seqs * add peer events * cleanup pending blocks on blockstore event * allow nil handlers * move protobuf type helpers * allow initializing block from Cid * testing and fixes * small fixes * expose `<` * spelling * default value * spelling * pending blocks manager * adding stores manager * more tests a wip around bitswap * small changes * merge bitswap and engine for now * for now run only the new poc's tests * add a more complete ci setup * use template in map * remove p2pd * remove go * dont use asyncCheck * few small changes * adding ability to update items * adding multiple task runners * handle cancelation properly * use Result instead of throwing * wip bitswap tests * moving things around * split out engine again * add request and handlers interface * fix tests * wip - engine tests * remove unused imports * fix tests * cleanup block requesting logic * add block request tests * more block requests * add support for max heap * don't use result * use max heap & send block presence in task handler * add task handler tests * rename store to localStore * cleanup & logging * cancel task on stop * don't depend on local store for events * dont use heap queue for wants * add chronicles * fix issue with peer wants * add test for delayed block sends * remove obsolete tests * wip chunker * run all tests * add todo * misc * remove irrelevant files * removing more files * adding helpers for bitswap tests * moved bitswap file * misc * make blocks timeout longer * adjust block timeout * speedup test * compile with threads * import missing crypto * misc * disable threads for now * fix 32 bit platforms * re-enable threads support in tests
2021-02-26 00:23:22 +00:00
test "remote should send blocks when available":
let blk = bt.Block.new("Block 1".toBytes).tryGet()
Poc 2 (#7) * moving protobuf into bitswap * adding block type * reworking bitswap * adding chunker * adding license header * use 1.2.6 * adding fixed size chunker * add blockstore * add iterator to chunker * more bitswap changes * rename ipfs to dagger * rename to dagger * blockstore inherits from BlockProvider * wip - add core block handling logic * smal changes * use proper block store methods * adding asynq heapqueue * wip prepare for bitswap task runner * adding `$` * adding memory store and tests * fixed chunking * extracted desicion engine from bitswap * added helper random funcs * adding testing helpers * only handle seqs * add peer events * cleanup pending blocks on blockstore event * allow nil handlers * move protobuf type helpers * allow initializing block from Cid * testing and fixes * small fixes * expose `<` * spelling * default value * spelling * pending blocks manager * adding stores manager * more tests a wip around bitswap * small changes * merge bitswap and engine for now * for now run only the new poc's tests * add a more complete ci setup * use template in map * remove p2pd * remove go * dont use asyncCheck * few small changes * adding ability to update items * adding multiple task runners * handle cancelation properly * use Result instead of throwing * wip bitswap tests * moving things around * split out engine again * add request and handlers interface * fix tests * wip - engine tests * remove unused imports * fix tests * cleanup block requesting logic * add block request tests * more block requests * add support for max heap * don't use result * use max heap & send block presence in task handler * add task handler tests * rename store to localStore * cleanup & logging * cancel task on stop * don't depend on local store for events * dont use heap queue for wants * add chronicles * fix issue with peer wants * add test for delayed block sends * remove obsolete tests * wip chunker * run all tests * add todo * misc * remove irrelevant files * removing more files * adding helpers for bitswap tests * moved bitswap file * misc * make blocks timeout longer * adjust block timeout * speedup test * compile with threads * import missing crypto * misc * disable threads for now * fix 32 bit platforms * re-enable threads support in tests
2021-02-26 00:23:22 +00:00
# should fail retrieving block from remote
Node setup (#32) * settup basic nim node * adding http utils * adding confutils * rough rest api proto * adding missing deps * turn tls emulation off * adding toml serialization * wip * adding missing deps * make sure to clean old state in teardown * adding file upload rest endpoint * renaming blockexchange to networkstore * updating nim-presto * updating libp2p * wip adding streaming upload * reworked chunking * bump to latest unstable * adding asyncfutures stream * make streamable * deleting unused files * reworking stores api * use new stores api * rework blockset and remove blockstream * don't return option from constructor * rework chunker * wip implement upload * fix tests * move unrelated logic to engine * don't print entire message * logging * basic encode/decode to/from dag-pb * add basic upload/download support * fix tests * renaming blockset to manifest * don't pass config to node * remove config and use new manifest * wip: make endpoints more reliable * wip: adding node tests * include correct manifest test * removing asyncfutures * proper chunking of files * simplify stream reading * test with encoding/decoding with many blocks * add block storing tests * adding retrieval test * add logging * tidy up chunker * tidy up manifest and node * use default chunk size * fix tests * fix tests * make sure Eof is set properly * wip * minor cleanup * add file utils * cleanup config * splitout DaggerServer and "main" * remove events since they are not used * add broadcast method to network peer * add and wire localstore * use localstore in the node * wip * logging * move file utils * use the constant * updating deps * fix memstore * use latest libp2p unstable * fix tests * rework block streaming * don't fail storing if the block already exists * add helper info endpoint * correct comment * rename localstore to fsstore * fix tests * remove unused tests * add test to retrieve one block * move some test files around * consolidate setup * Update dagger/blockexchange/engine.nim Co-authored-by: Tanguy <tanguy@status.im> * typo * better block path handling * don't inherit rootobj * remove useless template * Update tests/dagger/blockexc/testblockexc.nim Co-authored-by: markspanbroek <mark@spanbroek.net> * use isMainModule * use proper flag for starter/stoped * cleanup optional use * wrap in isMainModule * use `cancelAndAwait` * remove unused imports * wip * don't use optional * use functional error api * rework store tests and add fs tests * Block.new() to Block.init() * don't use optional for engine blocks * use result instead of optional for getBlock * remove unused imports * move stopping servers to `shutdown` * use result instead of optional * rework with results * fix tests * use waitFor in signal handlers * error helper * use `?` and mapFailure where possible * remove unnecesary `=?` * improve empty cid digest initialization Co-authored-by: Tanguy <tanguy@status.im> Co-authored-by: markspanbroek <mark@spanbroek.net>
2022-01-10 15:32:56 +00:00
check not await blockexc1.getBlock(blk.cid)
Poc 2 (#7) * moving protobuf into bitswap * adding block type * reworking bitswap * adding chunker * adding license header * use 1.2.6 * adding fixed size chunker * add blockstore * add iterator to chunker * more bitswap changes * rename ipfs to dagger * rename to dagger * blockstore inherits from BlockProvider * wip - add core block handling logic * smal changes * use proper block store methods * adding asynq heapqueue * wip prepare for bitswap task runner * adding `$` * adding memory store and tests * fixed chunking * extracted desicion engine from bitswap * added helper random funcs * adding testing helpers * only handle seqs * add peer events * cleanup pending blocks on blockstore event * allow nil handlers * move protobuf type helpers * allow initializing block from Cid * testing and fixes * small fixes * expose `<` * spelling * default value * spelling * pending blocks manager * adding stores manager * more tests a wip around bitswap * small changes * merge bitswap and engine for now * for now run only the new poc's tests * add a more complete ci setup * use template in map * remove p2pd * remove go * dont use asyncCheck * few small changes * adding ability to update items * adding multiple task runners * handle cancelation properly * use Result instead of throwing * wip bitswap tests * moving things around * split out engine again * add request and handlers interface * fix tests * wip - engine tests * remove unused imports * fix tests * cleanup block requesting logic * add block request tests * more block requests * add support for max heap * don't use result * use max heap & send block presence in task handler * add task handler tests * rename store to localStore * cleanup & logging * cancel task on stop * don't depend on local store for events * dont use heap queue for wants * add chronicles * fix issue with peer wants * add test for delayed block sends * remove obsolete tests * wip chunker * run all tests * add todo * misc * remove irrelevant files * removing more files * adding helpers for bitswap tests * moved bitswap file * misc * make blocks timeout longer * adjust block timeout * speedup test * compile with threads * import missing crypto * misc * disable threads for now * fix 32 bit platforms * re-enable threads support in tests
2021-02-26 00:23:22 +00:00
.withTimeout(100.millis) # should expire
# first put the required block in the local store
Node setup (#32) * settup basic nim node * adding http utils * adding confutils * rough rest api proto * adding missing deps * turn tls emulation off * adding toml serialization * wip * adding missing deps * make sure to clean old state in teardown * adding file upload rest endpoint * renaming blockexchange to networkstore * updating nim-presto * updating libp2p * wip adding streaming upload * reworked chunking * bump to latest unstable * adding asyncfutures stream * make streamable * deleting unused files * reworking stores api * use new stores api * rework blockset and remove blockstream * don't return option from constructor * rework chunker * wip implement upload * fix tests * move unrelated logic to engine * don't print entire message * logging * basic encode/decode to/from dag-pb * add basic upload/download support * fix tests * renaming blockset to manifest * don't pass config to node * remove config and use new manifest * wip: make endpoints more reliable * wip: adding node tests * include correct manifest test * removing asyncfutures * proper chunking of files * simplify stream reading * test with encoding/decoding with many blocks * add block storing tests * adding retrieval test * add logging * tidy up chunker * tidy up manifest and node * use default chunk size * fix tests * fix tests * make sure Eof is set properly * wip * minor cleanup * add file utils * cleanup config * splitout DaggerServer and "main" * remove events since they are not used * add broadcast method to network peer * add and wire localstore * use localstore in the node * wip * logging * move file utils * use the constant * updating deps * fix memstore * use latest libp2p unstable * fix tests * rework block streaming * don't fail storing if the block already exists * add helper info endpoint * correct comment * rename localstore to fsstore * fix tests * remove unused tests * add test to retrieve one block * move some test files around * consolidate setup * Update dagger/blockexchange/engine.nim Co-authored-by: Tanguy <tanguy@status.im> * typo * better block path handling * don't inherit rootobj * remove useless template * Update tests/dagger/blockexc/testblockexc.nim Co-authored-by: markspanbroek <mark@spanbroek.net> * use isMainModule * use proper flag for starter/stoped * cleanup optional use * wrap in isMainModule * use `cancelAndAwait` * remove unused imports * wip * don't use optional * use functional error api * rework store tests and add fs tests * Block.new() to Block.init() * don't use optional for engine blocks * use result instead of optional for getBlock * remove unused imports * move stopping servers to `shutdown` * use result instead of optional * rework with results * fix tests * use waitFor in signal handlers * error helper * use `?` and mapFailure where possible * remove unnecesary `=?` * improve empty cid digest initialization Co-authored-by: Tanguy <tanguy@status.im> Co-authored-by: markspanbroek <mark@spanbroek.net>
2022-01-10 15:32:56 +00:00
check await blockexc2.engine.localStore.putBlock(blk)
Poc 2 (#7) * moving protobuf into bitswap * adding block type * reworking bitswap * adding chunker * adding license header * use 1.2.6 * adding fixed size chunker * add blockstore * add iterator to chunker * more bitswap changes * rename ipfs to dagger * rename to dagger * blockstore inherits from BlockProvider * wip - add core block handling logic * smal changes * use proper block store methods * adding asynq heapqueue * wip prepare for bitswap task runner * adding `$` * adding memory store and tests * fixed chunking * extracted desicion engine from bitswap * added helper random funcs * adding testing helpers * only handle seqs * add peer events * cleanup pending blocks on blockstore event * allow nil handlers * move protobuf type helpers * allow initializing block from Cid * testing and fixes * small fixes * expose `<` * spelling * default value * spelling * pending blocks manager * adding stores manager * more tests a wip around bitswap * small changes * merge bitswap and engine for now * for now run only the new poc's tests * add a more complete ci setup * use template in map * remove p2pd * remove go * dont use asyncCheck * few small changes * adding ability to update items * adding multiple task runners * handle cancelation properly * use Result instead of throwing * wip bitswap tests * moving things around * split out engine again * add request and handlers interface * fix tests * wip - engine tests * remove unused imports * fix tests * cleanup block requesting logic * add block request tests * more block requests * add support for max heap * don't use result * use max heap & send block presence in task handler * add task handler tests * rename store to localStore * cleanup & logging * cancel task on stop * don't depend on local store for events * dont use heap queue for wants * add chronicles * fix issue with peer wants * add test for delayed block sends * remove obsolete tests * wip chunker * run all tests * add todo * misc * remove irrelevant files * removing more files * adding helpers for bitswap tests * moved bitswap file * misc * make blocks timeout longer * adjust block timeout * speedup test * compile with threads * import missing crypto * misc * disable threads for now * fix 32 bit platforms * re-enable threads support in tests
2021-02-26 00:23:22 +00:00
# second trigger blockexc to resolve any pending requests
Poc 2 (#7) * moving protobuf into bitswap * adding block type * reworking bitswap * adding chunker * adding license header * use 1.2.6 * adding fixed size chunker * add blockstore * add iterator to chunker * more bitswap changes * rename ipfs to dagger * rename to dagger * blockstore inherits from BlockProvider * wip - add core block handling logic * smal changes * use proper block store methods * adding asynq heapqueue * wip prepare for bitswap task runner * adding `$` * adding memory store and tests * fixed chunking * extracted desicion engine from bitswap * added helper random funcs * adding testing helpers * only handle seqs * add peer events * cleanup pending blocks on blockstore event * allow nil handlers * move protobuf type helpers * allow initializing block from Cid * testing and fixes * small fixes * expose `<` * spelling * default value * spelling * pending blocks manager * adding stores manager * more tests a wip around bitswap * small changes * merge bitswap and engine for now * for now run only the new poc's tests * add a more complete ci setup * use template in map * remove p2pd * remove go * dont use asyncCheck * few small changes * adding ability to update items * adding multiple task runners * handle cancelation properly * use Result instead of throwing * wip bitswap tests * moving things around * split out engine again * add request and handlers interface * fix tests * wip - engine tests * remove unused imports * fix tests * cleanup block requesting logic * add block request tests * more block requests * add support for max heap * don't use result * use max heap & send block presence in task handler * add task handler tests * rename store to localStore * cleanup & logging * cancel task on stop * don't depend on local store for events * dont use heap queue for wants * add chronicles * fix issue with peer wants * add test for delayed block sends * remove obsolete tests * wip chunker * run all tests * add todo * misc * remove irrelevant files * removing more files * adding helpers for bitswap tests * moved bitswap file * misc * make blocks timeout longer * adjust block timeout * speedup test * compile with threads * import missing crypto * misc * disable threads for now * fix 32 bit platforms * re-enable threads support in tests
2021-02-26 00:23:22 +00:00
# for the block
Node setup (#32) * settup basic nim node * adding http utils * adding confutils * rough rest api proto * adding missing deps * turn tls emulation off * adding toml serialization * wip * adding missing deps * make sure to clean old state in teardown * adding file upload rest endpoint * renaming blockexchange to networkstore * updating nim-presto * updating libp2p * wip adding streaming upload * reworked chunking * bump to latest unstable * adding asyncfutures stream * make streamable * deleting unused files * reworking stores api * use new stores api * rework blockset and remove blockstream * don't return option from constructor * rework chunker * wip implement upload * fix tests * move unrelated logic to engine * don't print entire message * logging * basic encode/decode to/from dag-pb * add basic upload/download support * fix tests * renaming blockset to manifest * don't pass config to node * remove config and use new manifest * wip: make endpoints more reliable * wip: adding node tests * include correct manifest test * removing asyncfutures * proper chunking of files * simplify stream reading * test with encoding/decoding with many blocks * add block storing tests * adding retrieval test * add logging * tidy up chunker * tidy up manifest and node * use default chunk size * fix tests * fix tests * make sure Eof is set properly * wip * minor cleanup * add file utils * cleanup config * splitout DaggerServer and "main" * remove events since they are not used * add broadcast method to network peer * add and wire localstore * use localstore in the node * wip * logging * move file utils * use the constant * updating deps * fix memstore * use latest libp2p unstable * fix tests * rework block streaming * don't fail storing if the block already exists * add helper info endpoint * correct comment * rename localstore to fsstore * fix tests * remove unused tests * add test to retrieve one block * move some test files around * consolidate setup * Update dagger/blockexchange/engine.nim Co-authored-by: Tanguy <tanguy@status.im> * typo * better block path handling * don't inherit rootobj * remove useless template * Update tests/dagger/blockexc/testblockexc.nim Co-authored-by: markspanbroek <mark@spanbroek.net> * use isMainModule * use proper flag for starter/stoped * cleanup optional use * wrap in isMainModule * use `cancelAndAwait` * remove unused imports * wip * don't use optional * use functional error api * rework store tests and add fs tests * Block.new() to Block.init() * don't use optional for engine blocks * use result instead of optional for getBlock * remove unused imports * move stopping servers to `shutdown` * use result instead of optional * rework with results * fix tests * use waitFor in signal handlers * error helper * use `?` and mapFailure where possible * remove unnecesary `=?` * improve empty cid digest initialization Co-authored-by: Tanguy <tanguy@status.im> Co-authored-by: markspanbroek <mark@spanbroek.net>
2022-01-10 15:32:56 +00:00
check await blockexc2.putBlock(blk)
Poc 2 (#7) * moving protobuf into bitswap * adding block type * reworking bitswap * adding chunker * adding license header * use 1.2.6 * adding fixed size chunker * add blockstore * add iterator to chunker * more bitswap changes * rename ipfs to dagger * rename to dagger * blockstore inherits from BlockProvider * wip - add core block handling logic * smal changes * use proper block store methods * adding asynq heapqueue * wip prepare for bitswap task runner * adding `$` * adding memory store and tests * fixed chunking * extracted desicion engine from bitswap * added helper random funcs * adding testing helpers * only handle seqs * add peer events * cleanup pending blocks on blockstore event * allow nil handlers * move protobuf type helpers * allow initializing block from Cid * testing and fixes * small fixes * expose `<` * spelling * default value * spelling * pending blocks manager * adding stores manager * more tests a wip around bitswap * small changes * merge bitswap and engine for now * for now run only the new poc's tests * add a more complete ci setup * use template in map * remove p2pd * remove go * dont use asyncCheck * few small changes * adding ability to update items * adding multiple task runners * handle cancelation properly * use Result instead of throwing * wip bitswap tests * moving things around * split out engine again * add request and handlers interface * fix tests * wip - engine tests * remove unused imports * fix tests * cleanup block requesting logic * add block request tests * more block requests * add support for max heap * don't use result * use max heap & send block presence in task handler * add task handler tests * rename store to localStore * cleanup & logging * cancel task on stop * don't depend on local store for events * dont use heap queue for wants * add chronicles * fix issue with peer wants * add test for delayed block sends * remove obsolete tests * wip chunker * run all tests * add todo * misc * remove irrelevant files * removing more files * adding helpers for bitswap tests * moved bitswap file * misc * make blocks timeout longer * adjust block timeout * speedup test * compile with threads * import missing crypto * misc * disable threads for now * fix 32 bit platforms * re-enable threads support in tests
2021-02-26 00:23:22 +00:00
Node setup (#32) * settup basic nim node * adding http utils * adding confutils * rough rest api proto * adding missing deps * turn tls emulation off * adding toml serialization * wip * adding missing deps * make sure to clean old state in teardown * adding file upload rest endpoint * renaming blockexchange to networkstore * updating nim-presto * updating libp2p * wip adding streaming upload * reworked chunking * bump to latest unstable * adding asyncfutures stream * make streamable * deleting unused files * reworking stores api * use new stores api * rework blockset and remove blockstream * don't return option from constructor * rework chunker * wip implement upload * fix tests * move unrelated logic to engine * don't print entire message * logging * basic encode/decode to/from dag-pb * add basic upload/download support * fix tests * renaming blockset to manifest * don't pass config to node * remove config and use new manifest * wip: make endpoints more reliable * wip: adding node tests * include correct manifest test * removing asyncfutures * proper chunking of files * simplify stream reading * test with encoding/decoding with many blocks * add block storing tests * adding retrieval test * add logging * tidy up chunker * tidy up manifest and node * use default chunk size * fix tests * fix tests * make sure Eof is set properly * wip * minor cleanup * add file utils * cleanup config * splitout DaggerServer and "main" * remove events since they are not used * add broadcast method to network peer * add and wire localstore * use localstore in the node * wip * logging * move file utils * use the constant * updating deps * fix memstore * use latest libp2p unstable * fix tests * rework block streaming * don't fail storing if the block already exists * add helper info endpoint * correct comment * rename localstore to fsstore * fix tests * remove unused tests * add test to retrieve one block * move some test files around * consolidate setup * Update dagger/blockexchange/engine.nim Co-authored-by: Tanguy <tanguy@status.im> * typo * better block path handling * don't inherit rootobj * remove useless template * Update tests/dagger/blockexc/testblockexc.nim Co-authored-by: markspanbroek <mark@spanbroek.net> * use isMainModule * use proper flag for starter/stoped * cleanup optional use * wrap in isMainModule * use `cancelAndAwait` * remove unused imports * wip * don't use optional * use functional error api * rework store tests and add fs tests * Block.new() to Block.init() * don't use optional for engine blocks * use result instead of optional for getBlock * remove unused imports * move stopping servers to `shutdown` * use result instead of optional * rework with results * fix tests * use waitFor in signal handlers * error helper * use `?` and mapFailure where possible * remove unnecesary `=?` * improve empty cid digest initialization Co-authored-by: Tanguy <tanguy@status.im> Co-authored-by: markspanbroek <mark@spanbroek.net>
2022-01-10 15:32:56 +00:00
# should succeed retrieving block from remote
check await blockexc1.getBlock(blk.cid)
.withTimeout(100.millis) # should succede
Poc 2 (#7) * moving protobuf into bitswap * adding block type * reworking bitswap * adding chunker * adding license header * use 1.2.6 * adding fixed size chunker * add blockstore * add iterator to chunker * more bitswap changes * rename ipfs to dagger * rename to dagger * blockstore inherits from BlockProvider * wip - add core block handling logic * smal changes * use proper block store methods * adding asynq heapqueue * wip prepare for bitswap task runner * adding `$` * adding memory store and tests * fixed chunking * extracted desicion engine from bitswap * added helper random funcs * adding testing helpers * only handle seqs * add peer events * cleanup pending blocks on blockstore event * allow nil handlers * move protobuf type helpers * allow initializing block from Cid * testing and fixes * small fixes * expose `<` * spelling * default value * spelling * pending blocks manager * adding stores manager * more tests a wip around bitswap * small changes * merge bitswap and engine for now * for now run only the new poc's tests * add a more complete ci setup * use template in map * remove p2pd * remove go * dont use asyncCheck * few small changes * adding ability to update items * adding multiple task runners * handle cancelation properly * use Result instead of throwing * wip bitswap tests * moving things around * split out engine again * add request and handlers interface * fix tests * wip - engine tests * remove unused imports * fix tests * cleanup block requesting logic * add block request tests * more block requests * add support for max heap * don't use result * use max heap & send block presence in task handler * add task handler tests * rename store to localStore * cleanup & logging * cancel task on stop * don't depend on local store for events * dont use heap queue for wants * add chronicles * fix issue with peer wants * add test for delayed block sends * remove obsolete tests * wip chunker * run all tests * add todo * misc * remove irrelevant files * removing more files * adding helpers for bitswap tests * moved bitswap file * misc * make blocks timeout longer * adjust block timeout * speedup test * compile with threads * import missing crypto * misc * disable threads for now * fix 32 bit platforms * re-enable threads support in tests
2021-02-26 00:23:22 +00:00
test "receives payments for blocks that were sent":
Node setup (#32) * settup basic nim node * adding http utils * adding confutils * rough rest api proto * adding missing deps * turn tls emulation off * adding toml serialization * wip * adding missing deps * make sure to clean old state in teardown * adding file upload rest endpoint * renaming blockexchange to networkstore * updating nim-presto * updating libp2p * wip adding streaming upload * reworked chunking * bump to latest unstable * adding asyncfutures stream * make streamable * deleting unused files * reworking stores api * use new stores api * rework blockset and remove blockstream * don't return option from constructor * rework chunker * wip implement upload * fix tests * move unrelated logic to engine * don't print entire message * logging * basic encode/decode to/from dag-pb * add basic upload/download support * fix tests * renaming blockset to manifest * don't pass config to node * remove config and use new manifest * wip: make endpoints more reliable * wip: adding node tests * include correct manifest test * removing asyncfutures * proper chunking of files * simplify stream reading * test with encoding/decoding with many blocks * add block storing tests * adding retrieval test * add logging * tidy up chunker * tidy up manifest and node * use default chunk size * fix tests * fix tests * make sure Eof is set properly * wip * minor cleanup * add file utils * cleanup config * splitout DaggerServer and "main" * remove events since they are not used * add broadcast method to network peer * add and wire localstore * use localstore in the node * wip * logging * move file utils * use the constant * updating deps * fix memstore * use latest libp2p unstable * fix tests * rework block streaming * don't fail storing if the block already exists * add helper info endpoint * correct comment * rename localstore to fsstore * fix tests * remove unused tests * add test to retrieve one block * move some test files around * consolidate setup * Update dagger/blockexchange/engine.nim Co-authored-by: Tanguy <tanguy@status.im> * typo * better block path handling * don't inherit rootobj * remove useless template * Update tests/dagger/blockexc/testblockexc.nim Co-authored-by: markspanbroek <mark@spanbroek.net> * use isMainModule * use proper flag for starter/stoped * cleanup optional use * wrap in isMainModule * use `cancelAndAwait` * remove unused imports * wip * don't use optional * use functional error api * rework store tests and add fs tests * Block.new() to Block.init() * don't use optional for engine blocks * use result instead of optional for getBlock * remove unused imports * move stopping servers to `shutdown` * use result instead of optional * rework with results * fix tests * use waitFor in signal handlers * error helper * use `?` and mapFailure where possible * remove unnecesary `=?` * improve empty cid digest initialization Co-authored-by: Tanguy <tanguy@status.im> Co-authored-by: markspanbroek <mark@spanbroek.net>
2022-01-10 15:32:56 +00:00
let blocks = await allFinished(
blocks2.mapIt( blockexc1.getBlock(it.cid) ))
await sleepAsync(100.millis)
let channel = !peerCtx1.paymentChannel
check wallet2.balance(channel, Asset) > 0
Node setup (#32) * settup basic nim node * adding http utils * adding confutils * rough rest api proto * adding missing deps * turn tls emulation off * adding toml serialization * wip * adding missing deps * make sure to clean old state in teardown * adding file upload rest endpoint * renaming blockexchange to networkstore * updating nim-presto * updating libp2p * wip adding streaming upload * reworked chunking * bump to latest unstable * adding asyncfutures stream * make streamable * deleting unused files * reworking stores api * use new stores api * rework blockset and remove blockstream * don't return option from constructor * rework chunker * wip implement upload * fix tests * move unrelated logic to engine * don't print entire message * logging * basic encode/decode to/from dag-pb * add basic upload/download support * fix tests * renaming blockset to manifest * don't pass config to node * remove config and use new manifest * wip: make endpoints more reliable * wip: adding node tests * include correct manifest test * removing asyncfutures * proper chunking of files * simplify stream reading * test with encoding/decoding with many blocks * add block storing tests * adding retrieval test * add logging * tidy up chunker * tidy up manifest and node * use default chunk size * fix tests * fix tests * make sure Eof is set properly * wip * minor cleanup * add file utils * cleanup config * splitout DaggerServer and "main" * remove events since they are not used * add broadcast method to network peer * add and wire localstore * use localstore in the node * wip * logging * move file utils * use the constant * updating deps * fix memstore * use latest libp2p unstable * fix tests * rework block streaming * don't fail storing if the block already exists * add helper info endpoint * correct comment * rename localstore to fsstore * fix tests * remove unused tests * add test to retrieve one block * move some test files around * consolidate setup * Update dagger/blockexchange/engine.nim Co-authored-by: Tanguy <tanguy@status.im> * typo * better block path handling * don't inherit rootobj * remove useless template * Update tests/dagger/blockexc/testblockexc.nim Co-authored-by: markspanbroek <mark@spanbroek.net> * use isMainModule * use proper flag for starter/stoped * cleanup optional use * wrap in isMainModule * use `cancelAndAwait` * remove unused imports * wip * don't use optional * use functional error api * rework store tests and add fs tests * Block.new() to Block.init() * don't use optional for engine blocks * use result instead of optional for getBlock * remove unused imports * move stopping servers to `shutdown` * use result instead of optional * rework with results * fix tests * use waitFor in signal handlers * error helper * use `?` and mapFailure where possible * remove unnecesary `=?` * improve empty cid digest initialization Co-authored-by: Tanguy <tanguy@status.im> Co-authored-by: markspanbroek <mark@spanbroek.net>
2022-01-10 15:32:56 +00:00
suite "NetworkStore - multiple nodes":
Poc 2 (#7) * moving protobuf into bitswap * adding block type * reworking bitswap * adding chunker * adding license header * use 1.2.6 * adding fixed size chunker * add blockstore * add iterator to chunker * more bitswap changes * rename ipfs to dagger * rename to dagger * blockstore inherits from BlockProvider * wip - add core block handling logic * smal changes * use proper block store methods * adding asynq heapqueue * wip prepare for bitswap task runner * adding `$` * adding memory store and tests * fixed chunking * extracted desicion engine from bitswap * added helper random funcs * adding testing helpers * only handle seqs * add peer events * cleanup pending blocks on blockstore event * allow nil handlers * move protobuf type helpers * allow initializing block from Cid * testing and fixes * small fixes * expose `<` * spelling * default value * spelling * pending blocks manager * adding stores manager * more tests a wip around bitswap * small changes * merge bitswap and engine for now * for now run only the new poc's tests * add a more complete ci setup * use template in map * remove p2pd * remove go * dont use asyncCheck * few small changes * adding ability to update items * adding multiple task runners * handle cancelation properly * use Result instead of throwing * wip bitswap tests * moving things around * split out engine again * add request and handlers interface * fix tests * wip - engine tests * remove unused imports * fix tests * cleanup block requesting logic * add block request tests * more block requests * add support for max heap * don't use result * use max heap & send block presence in task handler * add task handler tests * rename store to localStore * cleanup & logging * cancel task on stop * don't depend on local store for events * dont use heap queue for wants * add chronicles * fix issue with peer wants * add test for delayed block sends * remove obsolete tests * wip chunker * run all tests * add todo * misc * remove irrelevant files * removing more files * adding helpers for bitswap tests * moved bitswap file * misc * make blocks timeout longer * adjust block timeout * speedup test * compile with threads * import missing crypto * misc * disable threads for now * fix 32 bit platforms * re-enable threads support in tests
2021-02-26 00:23:22 +00:00
let
Node setup (#32) * settup basic nim node * adding http utils * adding confutils * rough rest api proto * adding missing deps * turn tls emulation off * adding toml serialization * wip * adding missing deps * make sure to clean old state in teardown * adding file upload rest endpoint * renaming blockexchange to networkstore * updating nim-presto * updating libp2p * wip adding streaming upload * reworked chunking * bump to latest unstable * adding asyncfutures stream * make streamable * deleting unused files * reworking stores api * use new stores api * rework blockset and remove blockstream * don't return option from constructor * rework chunker * wip implement upload * fix tests * move unrelated logic to engine * don't print entire message * logging * basic encode/decode to/from dag-pb * add basic upload/download support * fix tests * renaming blockset to manifest * don't pass config to node * remove config and use new manifest * wip: make endpoints more reliable * wip: adding node tests * include correct manifest test * removing asyncfutures * proper chunking of files * simplify stream reading * test with encoding/decoding with many blocks * add block storing tests * adding retrieval test * add logging * tidy up chunker * tidy up manifest and node * use default chunk size * fix tests * fix tests * make sure Eof is set properly * wip * minor cleanup * add file utils * cleanup config * splitout DaggerServer and "main" * remove events since they are not used * add broadcast method to network peer * add and wire localstore * use localstore in the node * wip * logging * move file utils * use the constant * updating deps * fix memstore * use latest libp2p unstable * fix tests * rework block streaming * don't fail storing if the block already exists * add helper info endpoint * correct comment * rename localstore to fsstore * fix tests * remove unused tests * add test to retrieve one block * move some test files around * consolidate setup * Update dagger/blockexchange/engine.nim Co-authored-by: Tanguy <tanguy@status.im> * typo * better block path handling * don't inherit rootobj * remove useless template * Update tests/dagger/blockexc/testblockexc.nim Co-authored-by: markspanbroek <mark@spanbroek.net> * use isMainModule * use proper flag for starter/stoped * cleanup optional use * wrap in isMainModule * use `cancelAndAwait` * remove unused imports * wip * don't use optional * use functional error api * rework store tests and add fs tests * Block.new() to Block.init() * don't use optional for engine blocks * use result instead of optional for getBlock * remove unused imports * move stopping servers to `shutdown` * use result instead of optional * rework with results * fix tests * use waitFor in signal handlers * error helper * use `?` and mapFailure where possible * remove unnecesary `=?` * improve empty cid digest initialization Co-authored-by: Tanguy <tanguy@status.im> Co-authored-by: markspanbroek <mark@spanbroek.net>
2022-01-10 15:32:56 +00:00
chunker = RandomChunker.new(Rng.instance(), size = 4096, chunkSize = 256)
Poc 2 (#7) * moving protobuf into bitswap * adding block type * reworking bitswap * adding chunker * adding license header * use 1.2.6 * adding fixed size chunker * add blockstore * add iterator to chunker * more bitswap changes * rename ipfs to dagger * rename to dagger * blockstore inherits from BlockProvider * wip - add core block handling logic * smal changes * use proper block store methods * adding asynq heapqueue * wip prepare for bitswap task runner * adding `$` * adding memory store and tests * fixed chunking * extracted desicion engine from bitswap * added helper random funcs * adding testing helpers * only handle seqs * add peer events * cleanup pending blocks on blockstore event * allow nil handlers * move protobuf type helpers * allow initializing block from Cid * testing and fixes * small fixes * expose `<` * spelling * default value * spelling * pending blocks manager * adding stores manager * more tests a wip around bitswap * small changes * merge bitswap and engine for now * for now run only the new poc's tests * add a more complete ci setup * use template in map * remove p2pd * remove go * dont use asyncCheck * few small changes * adding ability to update items * adding multiple task runners * handle cancelation properly * use Result instead of throwing * wip bitswap tests * moving things around * split out engine again * add request and handlers interface * fix tests * wip - engine tests * remove unused imports * fix tests * cleanup block requesting logic * add block request tests * more block requests * add support for max heap * don't use result * use max heap & send block presence in task handler * add task handler tests * rename store to localStore * cleanup & logging * cancel task on stop * don't depend on local store for events * dont use heap queue for wants * add chronicles * fix issue with peer wants * add test for delayed block sends * remove obsolete tests * wip chunker * run all tests * add todo * misc * remove irrelevant files * removing more files * adding helpers for bitswap tests * moved bitswap file * misc * make blocks timeout longer * adjust block timeout * speedup test * compile with threads * import missing crypto * misc * disable threads for now * fix 32 bit platforms * re-enable threads support in tests
2021-02-26 00:23:22 +00:00
var
switch: seq[Switch]
Node setup (#32) * settup basic nim node * adding http utils * adding confutils * rough rest api proto * adding missing deps * turn tls emulation off * adding toml serialization * wip * adding missing deps * make sure to clean old state in teardown * adding file upload rest endpoint * renaming blockexchange to networkstore * updating nim-presto * updating libp2p * wip adding streaming upload * reworked chunking * bump to latest unstable * adding asyncfutures stream * make streamable * deleting unused files * reworking stores api * use new stores api * rework blockset and remove blockstream * don't return option from constructor * rework chunker * wip implement upload * fix tests * move unrelated logic to engine * don't print entire message * logging * basic encode/decode to/from dag-pb * add basic upload/download support * fix tests * renaming blockset to manifest * don't pass config to node * remove config and use new manifest * wip: make endpoints more reliable * wip: adding node tests * include correct manifest test * removing asyncfutures * proper chunking of files * simplify stream reading * test with encoding/decoding with many blocks * add block storing tests * adding retrieval test * add logging * tidy up chunker * tidy up manifest and node * use default chunk size * fix tests * fix tests * make sure Eof is set properly * wip * minor cleanup * add file utils * cleanup config * splitout DaggerServer and "main" * remove events since they are not used * add broadcast method to network peer * add and wire localstore * use localstore in the node * wip * logging * move file utils * use the constant * updating deps * fix memstore * use latest libp2p unstable * fix tests * rework block streaming * don't fail storing if the block already exists * add helper info endpoint * correct comment * rename localstore to fsstore * fix tests * remove unused tests * add test to retrieve one block * move some test files around * consolidate setup * Update dagger/blockexchange/engine.nim Co-authored-by: Tanguy <tanguy@status.im> * typo * better block path handling * don't inherit rootobj * remove useless template * Update tests/dagger/blockexc/testblockexc.nim Co-authored-by: markspanbroek <mark@spanbroek.net> * use isMainModule * use proper flag for starter/stoped * cleanup optional use * wrap in isMainModule * use `cancelAndAwait` * remove unused imports * wip * don't use optional * use functional error api * rework store tests and add fs tests * Block.new() to Block.init() * don't use optional for engine blocks * use result instead of optional for getBlock * remove unused imports * move stopping servers to `shutdown` * use result instead of optional * rework with results * fix tests * use waitFor in signal handlers * error helper * use `?` and mapFailure where possible * remove unnecesary `=?` * improve empty cid digest initialization Co-authored-by: Tanguy <tanguy@status.im> Co-authored-by: markspanbroek <mark@spanbroek.net>
2022-01-10 15:32:56 +00:00
blockexc: seq[NetworkStore]
blocks: seq[bt.Block]
Poc 2 (#7) * moving protobuf into bitswap * adding block type * reworking bitswap * adding chunker * adding license header * use 1.2.6 * adding fixed size chunker * add blockstore * add iterator to chunker * more bitswap changes * rename ipfs to dagger * rename to dagger * blockstore inherits from BlockProvider * wip - add core block handling logic * smal changes * use proper block store methods * adding asynq heapqueue * wip prepare for bitswap task runner * adding `$` * adding memory store and tests * fixed chunking * extracted desicion engine from bitswap * added helper random funcs * adding testing helpers * only handle seqs * add peer events * cleanup pending blocks on blockstore event * allow nil handlers * move protobuf type helpers * allow initializing block from Cid * testing and fixes * small fixes * expose `<` * spelling * default value * spelling * pending blocks manager * adding stores manager * more tests a wip around bitswap * small changes * merge bitswap and engine for now * for now run only the new poc's tests * add a more complete ci setup * use template in map * remove p2pd * remove go * dont use asyncCheck * few small changes * adding ability to update items * adding multiple task runners * handle cancelation properly * use Result instead of throwing * wip bitswap tests * moving things around * split out engine again * add request and handlers interface * fix tests * wip - engine tests * remove unused imports * fix tests * cleanup block requesting logic * add block request tests * more block requests * add support for max heap * don't use result * use max heap & send block presence in task handler * add task handler tests * rename store to localStore * cleanup & logging * cancel task on stop * don't depend on local store for events * dont use heap queue for wants * add chronicles * fix issue with peer wants * add test for delayed block sends * remove obsolete tests * wip chunker * run all tests * add todo * misc * remove irrelevant files * removing more files * adding helpers for bitswap tests * moved bitswap file * misc * make blocks timeout longer * adjust block timeout * speedup test * compile with threads * import missing crypto * misc * disable threads for now * fix 32 bit platforms * re-enable threads support in tests
2021-02-26 00:23:22 +00:00
setup:
Node setup (#32) * settup basic nim node * adding http utils * adding confutils * rough rest api proto * adding missing deps * turn tls emulation off * adding toml serialization * wip * adding missing deps * make sure to clean old state in teardown * adding file upload rest endpoint * renaming blockexchange to networkstore * updating nim-presto * updating libp2p * wip adding streaming upload * reworked chunking * bump to latest unstable * adding asyncfutures stream * make streamable * deleting unused files * reworking stores api * use new stores api * rework blockset and remove blockstream * don't return option from constructor * rework chunker * wip implement upload * fix tests * move unrelated logic to engine * don't print entire message * logging * basic encode/decode to/from dag-pb * add basic upload/download support * fix tests * renaming blockset to manifest * don't pass config to node * remove config and use new manifest * wip: make endpoints more reliable * wip: adding node tests * include correct manifest test * removing asyncfutures * proper chunking of files * simplify stream reading * test with encoding/decoding with many blocks * add block storing tests * adding retrieval test * add logging * tidy up chunker * tidy up manifest and node * use default chunk size * fix tests * fix tests * make sure Eof is set properly * wip * minor cleanup * add file utils * cleanup config * splitout DaggerServer and "main" * remove events since they are not used * add broadcast method to network peer * add and wire localstore * use localstore in the node * wip * logging * move file utils * use the constant * updating deps * fix memstore * use latest libp2p unstable * fix tests * rework block streaming * don't fail storing if the block already exists * add helper info endpoint * correct comment * rename localstore to fsstore * fix tests * remove unused tests * add test to retrieve one block * move some test files around * consolidate setup * Update dagger/blockexchange/engine.nim Co-authored-by: Tanguy <tanguy@status.im> * typo * better block path handling * don't inherit rootobj * remove useless template * Update tests/dagger/blockexc/testblockexc.nim Co-authored-by: markspanbroek <mark@spanbroek.net> * use isMainModule * use proper flag for starter/stoped * cleanup optional use * wrap in isMainModule * use `cancelAndAwait` * remove unused imports * wip * don't use optional * use functional error api * rework store tests and add fs tests * Block.new() to Block.init() * don't use optional for engine blocks * use result instead of optional for getBlock * remove unused imports * move stopping servers to `shutdown` * use result instead of optional * rework with results * fix tests * use waitFor in signal handlers * error helper * use `?` and mapFailure where possible * remove unnecesary `=?` * improve empty cid digest initialization Co-authored-by: Tanguy <tanguy@status.im> Co-authored-by: markspanbroek <mark@spanbroek.net>
2022-01-10 15:32:56 +00:00
while true:
let chunk = await chunker.getBytes()
if chunk.len <= 0:
break
blocks.add(bt.Block.new(chunk).tryGet())
Node setup (#32) * settup basic nim node * adding http utils * adding confutils * rough rest api proto * adding missing deps * turn tls emulation off * adding toml serialization * wip * adding missing deps * make sure to clean old state in teardown * adding file upload rest endpoint * renaming blockexchange to networkstore * updating nim-presto * updating libp2p * wip adding streaming upload * reworked chunking * bump to latest unstable * adding asyncfutures stream * make streamable * deleting unused files * reworking stores api * use new stores api * rework blockset and remove blockstream * don't return option from constructor * rework chunker * wip implement upload * fix tests * move unrelated logic to engine * don't print entire message * logging * basic encode/decode to/from dag-pb * add basic upload/download support * fix tests * renaming blockset to manifest * don't pass config to node * remove config and use new manifest * wip: make endpoints more reliable * wip: adding node tests * include correct manifest test * removing asyncfutures * proper chunking of files * simplify stream reading * test with encoding/decoding with many blocks * add block storing tests * adding retrieval test * add logging * tidy up chunker * tidy up manifest and node * use default chunk size * fix tests * fix tests * make sure Eof is set properly * wip * minor cleanup * add file utils * cleanup config * splitout DaggerServer and "main" * remove events since they are not used * add broadcast method to network peer * add and wire localstore * use localstore in the node * wip * logging * move file utils * use the constant * updating deps * fix memstore * use latest libp2p unstable * fix tests * rework block streaming * don't fail storing if the block already exists * add helper info endpoint * correct comment * rename localstore to fsstore * fix tests * remove unused tests * add test to retrieve one block * move some test files around * consolidate setup * Update dagger/blockexchange/engine.nim Co-authored-by: Tanguy <tanguy@status.im> * typo * better block path handling * don't inherit rootobj * remove useless template * Update tests/dagger/blockexc/testblockexc.nim Co-authored-by: markspanbroek <mark@spanbroek.net> * use isMainModule * use proper flag for starter/stoped * cleanup optional use * wrap in isMainModule * use `cancelAndAwait` * remove unused imports * wip * don't use optional * use functional error api * rework store tests and add fs tests * Block.new() to Block.init() * don't use optional for engine blocks * use result instead of optional for getBlock * remove unused imports * move stopping servers to `shutdown` * use result instead of optional * rework with results * fix tests * use waitFor in signal handlers * error helper * use `?` and mapFailure where possible * remove unnecesary `=?` * improve empty cid digest initialization Co-authored-by: Tanguy <tanguy@status.im> Co-authored-by: markspanbroek <mark@spanbroek.net>
2022-01-10 15:32:56 +00:00
Poc 2 (#7) * moving protobuf into bitswap * adding block type * reworking bitswap * adding chunker * adding license header * use 1.2.6 * adding fixed size chunker * add blockstore * add iterator to chunker * more bitswap changes * rename ipfs to dagger * rename to dagger * blockstore inherits from BlockProvider * wip - add core block handling logic * smal changes * use proper block store methods * adding asynq heapqueue * wip prepare for bitswap task runner * adding `$` * adding memory store and tests * fixed chunking * extracted desicion engine from bitswap * added helper random funcs * adding testing helpers * only handle seqs * add peer events * cleanup pending blocks on blockstore event * allow nil handlers * move protobuf type helpers * allow initializing block from Cid * testing and fixes * small fixes * expose `<` * spelling * default value * spelling * pending blocks manager * adding stores manager * more tests a wip around bitswap * small changes * merge bitswap and engine for now * for now run only the new poc's tests * add a more complete ci setup * use template in map * remove p2pd * remove go * dont use asyncCheck * few small changes * adding ability to update items * adding multiple task runners * handle cancelation properly * use Result instead of throwing * wip bitswap tests * moving things around * split out engine again * add request and handlers interface * fix tests * wip - engine tests * remove unused imports * fix tests * cleanup block requesting logic * add block request tests * more block requests * add support for max heap * don't use result * use max heap & send block presence in task handler * add task handler tests * rename store to localStore * cleanup & logging * cancel task on stop * don't depend on local store for events * dont use heap queue for wants * add chronicles * fix issue with peer wants * add test for delayed block sends * remove obsolete tests * wip chunker * run all tests * add todo * misc * remove irrelevant files * removing more files * adding helpers for bitswap tests * moved bitswap file * misc * make blocks timeout longer * adjust block timeout * speedup test * compile with threads * import missing crypto * misc * disable threads for now * fix 32 bit platforms * re-enable threads support in tests
2021-02-26 00:23:22 +00:00
for e in generateNodes(5):
switch.add(e.switch)
blockexc.add(e.blockexc)
Node setup (#32) * settup basic nim node * adding http utils * adding confutils * rough rest api proto * adding missing deps * turn tls emulation off * adding toml serialization * wip * adding missing deps * make sure to clean old state in teardown * adding file upload rest endpoint * renaming blockexchange to networkstore * updating nim-presto * updating libp2p * wip adding streaming upload * reworked chunking * bump to latest unstable * adding asyncfutures stream * make streamable * deleting unused files * reworking stores api * use new stores api * rework blockset and remove blockstream * don't return option from constructor * rework chunker * wip implement upload * fix tests * move unrelated logic to engine * don't print entire message * logging * basic encode/decode to/from dag-pb * add basic upload/download support * fix tests * renaming blockset to manifest * don't pass config to node * remove config and use new manifest * wip: make endpoints more reliable * wip: adding node tests * include correct manifest test * removing asyncfutures * proper chunking of files * simplify stream reading * test with encoding/decoding with many blocks * add block storing tests * adding retrieval test * add logging * tidy up chunker * tidy up manifest and node * use default chunk size * fix tests * fix tests * make sure Eof is set properly * wip * minor cleanup * add file utils * cleanup config * splitout DaggerServer and "main" * remove events since they are not used * add broadcast method to network peer * add and wire localstore * use localstore in the node * wip * logging * move file utils * use the constant * updating deps * fix memstore * use latest libp2p unstable * fix tests * rework block streaming * don't fail storing if the block already exists * add helper info endpoint * correct comment * rename localstore to fsstore * fix tests * remove unused tests * add test to retrieve one block * move some test files around * consolidate setup * Update dagger/blockexchange/engine.nim Co-authored-by: Tanguy <tanguy@status.im> * typo * better block path handling * don't inherit rootobj * remove useless template * Update tests/dagger/blockexc/testblockexc.nim Co-authored-by: markspanbroek <mark@spanbroek.net> * use isMainModule * use proper flag for starter/stoped * cleanup optional use * wrap in isMainModule * use `cancelAndAwait` * remove unused imports * wip * don't use optional * use functional error api * rework store tests and add fs tests * Block.new() to Block.init() * don't use optional for engine blocks * use result instead of optional for getBlock * remove unused imports * move stopping servers to `shutdown` * use result instead of optional * rework with results * fix tests * use waitFor in signal handlers * error helper * use `?` and mapFailure where possible * remove unnecesary `=?` * improve empty cid digest initialization Co-authored-by: Tanguy <tanguy@status.im> Co-authored-by: markspanbroek <mark@spanbroek.net>
2022-01-10 15:32:56 +00:00
await e.blockexc.engine.start()
Poc 2 (#7) * moving protobuf into bitswap * adding block type * reworking bitswap * adding chunker * adding license header * use 1.2.6 * adding fixed size chunker * add blockstore * add iterator to chunker * more bitswap changes * rename ipfs to dagger * rename to dagger * blockstore inherits from BlockProvider * wip - add core block handling logic * smal changes * use proper block store methods * adding asynq heapqueue * wip prepare for bitswap task runner * adding `$` * adding memory store and tests * fixed chunking * extracted desicion engine from bitswap * added helper random funcs * adding testing helpers * only handle seqs * add peer events * cleanup pending blocks on blockstore event * allow nil handlers * move protobuf type helpers * allow initializing block from Cid * testing and fixes * small fixes * expose `<` * spelling * default value * spelling * pending blocks manager * adding stores manager * more tests a wip around bitswap * small changes * merge bitswap and engine for now * for now run only the new poc's tests * add a more complete ci setup * use template in map * remove p2pd * remove go * dont use asyncCheck * few small changes * adding ability to update items * adding multiple task runners * handle cancelation properly * use Result instead of throwing * wip bitswap tests * moving things around * split out engine again * add request and handlers interface * fix tests * wip - engine tests * remove unused imports * fix tests * cleanup block requesting logic * add block request tests * more block requests * add support for max heap * don't use result * use max heap & send block presence in task handler * add task handler tests * rename store to localStore * cleanup & logging * cancel task on stop * don't depend on local store for events * dont use heap queue for wants * add chronicles * fix issue with peer wants * add test for delayed block sends * remove obsolete tests * wip chunker * run all tests * add todo * misc * remove irrelevant files * removing more files * adding helpers for bitswap tests * moved bitswap file * misc * make blocks timeout longer * adjust block timeout * speedup test * compile with threads * import missing crypto * misc * disable threads for now * fix 32 bit platforms * re-enable threads support in tests
2021-02-26 00:23:22 +00:00
Node setup (#32) * settup basic nim node * adding http utils * adding confutils * rough rest api proto * adding missing deps * turn tls emulation off * adding toml serialization * wip * adding missing deps * make sure to clean old state in teardown * adding file upload rest endpoint * renaming blockexchange to networkstore * updating nim-presto * updating libp2p * wip adding streaming upload * reworked chunking * bump to latest unstable * adding asyncfutures stream * make streamable * deleting unused files * reworking stores api * use new stores api * rework blockset and remove blockstream * don't return option from constructor * rework chunker * wip implement upload * fix tests * move unrelated logic to engine * don't print entire message * logging * basic encode/decode to/from dag-pb * add basic upload/download support * fix tests * renaming blockset to manifest * don't pass config to node * remove config and use new manifest * wip: make endpoints more reliable * wip: adding node tests * include correct manifest test * removing asyncfutures * proper chunking of files * simplify stream reading * test with encoding/decoding with many blocks * add block storing tests * adding retrieval test * add logging * tidy up chunker * tidy up manifest and node * use default chunk size * fix tests * fix tests * make sure Eof is set properly * wip * minor cleanup * add file utils * cleanup config * splitout DaggerServer and "main" * remove events since they are not used * add broadcast method to network peer * add and wire localstore * use localstore in the node * wip * logging * move file utils * use the constant * updating deps * fix memstore * use latest libp2p unstable * fix tests * rework block streaming * don't fail storing if the block already exists * add helper info endpoint * correct comment * rename localstore to fsstore * fix tests * remove unused tests * add test to retrieve one block * move some test files around * consolidate setup * Update dagger/blockexchange/engine.nim Co-authored-by: Tanguy <tanguy@status.im> * typo * better block path handling * don't inherit rootobj * remove useless template * Update tests/dagger/blockexc/testblockexc.nim Co-authored-by: markspanbroek <mark@spanbroek.net> * use isMainModule * use proper flag for starter/stoped * cleanup optional use * wrap in isMainModule * use `cancelAndAwait` * remove unused imports * wip * don't use optional * use functional error api * rework store tests and add fs tests * Block.new() to Block.init() * don't use optional for engine blocks * use result instead of optional for getBlock * remove unused imports * move stopping servers to `shutdown` * use result instead of optional * rework with results * fix tests * use waitFor in signal handlers * error helper * use `?` and mapFailure where possible * remove unnecesary `=?` * improve empty cid digest initialization Co-authored-by: Tanguy <tanguy@status.im> Co-authored-by: markspanbroek <mark@spanbroek.net>
2022-01-10 15:32:56 +00:00
await allFuturesThrowing(
switch.mapIt( it.start() )
)
Poc 2 (#7) * moving protobuf into bitswap * adding block type * reworking bitswap * adding chunker * adding license header * use 1.2.6 * adding fixed size chunker * add blockstore * add iterator to chunker * more bitswap changes * rename ipfs to dagger * rename to dagger * blockstore inherits from BlockProvider * wip - add core block handling logic * smal changes * use proper block store methods * adding asynq heapqueue * wip prepare for bitswap task runner * adding `$` * adding memory store and tests * fixed chunking * extracted desicion engine from bitswap * added helper random funcs * adding testing helpers * only handle seqs * add peer events * cleanup pending blocks on blockstore event * allow nil handlers * move protobuf type helpers * allow initializing block from Cid * testing and fixes * small fixes * expose `<` * spelling * default value * spelling * pending blocks manager * adding stores manager * more tests a wip around bitswap * small changes * merge bitswap and engine for now * for now run only the new poc's tests * add a more complete ci setup * use template in map * remove p2pd * remove go * dont use asyncCheck * few small changes * adding ability to update items * adding multiple task runners * handle cancelation properly * use Result instead of throwing * wip bitswap tests * moving things around * split out engine again * add request and handlers interface * fix tests * wip - engine tests * remove unused imports * fix tests * cleanup block requesting logic * add block request tests * more block requests * add support for max heap * don't use result * use max heap & send block presence in task handler * add task handler tests * rename store to localStore * cleanup & logging * cancel task on stop * don't depend on local store for events * dont use heap queue for wants * add chronicles * fix issue with peer wants * add test for delayed block sends * remove obsolete tests * wip chunker * run all tests * add todo * misc * remove irrelevant files * removing more files * adding helpers for bitswap tests * moved bitswap file * misc * make blocks timeout longer * adjust block timeout * speedup test * compile with threads * import missing crypto * misc * disable threads for now * fix 32 bit platforms * re-enable threads support in tests
2021-02-26 00:23:22 +00:00
teardown:
await allFuturesThrowing(
switch.mapIt( it.stop() )
)
Node setup (#32) * settup basic nim node * adding http utils * adding confutils * rough rest api proto * adding missing deps * turn tls emulation off * adding toml serialization * wip * adding missing deps * make sure to clean old state in teardown * adding file upload rest endpoint * renaming blockexchange to networkstore * updating nim-presto * updating libp2p * wip adding streaming upload * reworked chunking * bump to latest unstable * adding asyncfutures stream * make streamable * deleting unused files * reworking stores api * use new stores api * rework blockset and remove blockstream * don't return option from constructor * rework chunker * wip implement upload * fix tests * move unrelated logic to engine * don't print entire message * logging * basic encode/decode to/from dag-pb * add basic upload/download support * fix tests * renaming blockset to manifest * don't pass config to node * remove config and use new manifest * wip: make endpoints more reliable * wip: adding node tests * include correct manifest test * removing asyncfutures * proper chunking of files * simplify stream reading * test with encoding/decoding with many blocks * add block storing tests * adding retrieval test * add logging * tidy up chunker * tidy up manifest and node * use default chunk size * fix tests * fix tests * make sure Eof is set properly * wip * minor cleanup * add file utils * cleanup config * splitout DaggerServer and "main" * remove events since they are not used * add broadcast method to network peer * add and wire localstore * use localstore in the node * wip * logging * move file utils * use the constant * updating deps * fix memstore * use latest libp2p unstable * fix tests * rework block streaming * don't fail storing if the block already exists * add helper info endpoint * correct comment * rename localstore to fsstore * fix tests * remove unused tests * add test to retrieve one block * move some test files around * consolidate setup * Update dagger/blockexchange/engine.nim Co-authored-by: Tanguy <tanguy@status.im> * typo * better block path handling * don't inherit rootobj * remove useless template * Update tests/dagger/blockexc/testblockexc.nim Co-authored-by: markspanbroek <mark@spanbroek.net> * use isMainModule * use proper flag for starter/stoped * cleanup optional use * wrap in isMainModule * use `cancelAndAwait` * remove unused imports * wip * don't use optional * use functional error api * rework store tests and add fs tests * Block.new() to Block.init() * don't use optional for engine blocks * use result instead of optional for getBlock * remove unused imports * move stopping servers to `shutdown` * use result instead of optional * rework with results * fix tests * use waitFor in signal handlers * error helper * use `?` and mapFailure where possible * remove unnecesary `=?` * improve empty cid digest initialization Co-authored-by: Tanguy <tanguy@status.im> Co-authored-by: markspanbroek <mark@spanbroek.net>
2022-01-10 15:32:56 +00:00
switch = @[]
blockexc = @[]
Poc 2 (#7) * moving protobuf into bitswap * adding block type * reworking bitswap * adding chunker * adding license header * use 1.2.6 * adding fixed size chunker * add blockstore * add iterator to chunker * more bitswap changes * rename ipfs to dagger * rename to dagger * blockstore inherits from BlockProvider * wip - add core block handling logic * smal changes * use proper block store methods * adding asynq heapqueue * wip prepare for bitswap task runner * adding `$` * adding memory store and tests * fixed chunking * extracted desicion engine from bitswap * added helper random funcs * adding testing helpers * only handle seqs * add peer events * cleanup pending blocks on blockstore event * allow nil handlers * move protobuf type helpers * allow initializing block from Cid * testing and fixes * small fixes * expose `<` * spelling * default value * spelling * pending blocks manager * adding stores manager * more tests a wip around bitswap * small changes * merge bitswap and engine for now * for now run only the new poc's tests * add a more complete ci setup * use template in map * remove p2pd * remove go * dont use asyncCheck * few small changes * adding ability to update items * adding multiple task runners * handle cancelation properly * use Result instead of throwing * wip bitswap tests * moving things around * split out engine again * add request and handlers interface * fix tests * wip - engine tests * remove unused imports * fix tests * cleanup block requesting logic * add block request tests * more block requests * add support for max heap * don't use result * use max heap & send block presence in task handler * add task handler tests * rename store to localStore * cleanup & logging * cancel task on stop * don't depend on local store for events * dont use heap queue for wants * add chronicles * fix issue with peer wants * add test for delayed block sends * remove obsolete tests * wip chunker * run all tests * add todo * misc * remove irrelevant files * removing more files * adding helpers for bitswap tests * moved bitswap file * misc * make blocks timeout longer * adjust block timeout * speedup test * compile with threads * import missing crypto * misc * disable threads for now * fix 32 bit platforms * re-enable threads support in tests
2021-02-26 00:23:22 +00:00
test "should receive haves for own want list":
let
downloader = blockexc[4]
Poc 2 (#7) * moving protobuf into bitswap * adding block type * reworking bitswap * adding chunker * adding license header * use 1.2.6 * adding fixed size chunker * add blockstore * add iterator to chunker * more bitswap changes * rename ipfs to dagger * rename to dagger * blockstore inherits from BlockProvider * wip - add core block handling logic * smal changes * use proper block store methods * adding asynq heapqueue * wip prepare for bitswap task runner * adding `$` * adding memory store and tests * fixed chunking * extracted desicion engine from bitswap * added helper random funcs * adding testing helpers * only handle seqs * add peer events * cleanup pending blocks on blockstore event * allow nil handlers * move protobuf type helpers * allow initializing block from Cid * testing and fixes * small fixes * expose `<` * spelling * default value * spelling * pending blocks manager * adding stores manager * more tests a wip around bitswap * small changes * merge bitswap and engine for now * for now run only the new poc's tests * add a more complete ci setup * use template in map * remove p2pd * remove go * dont use asyncCheck * few small changes * adding ability to update items * adding multiple task runners * handle cancelation properly * use Result instead of throwing * wip bitswap tests * moving things around * split out engine again * add request and handlers interface * fix tests * wip - engine tests * remove unused imports * fix tests * cleanup block requesting logic * add block request tests * more block requests * add support for max heap * don't use result * use max heap & send block presence in task handler * add task handler tests * rename store to localStore * cleanup & logging * cancel task on stop * don't depend on local store for events * dont use heap queue for wants * add chronicles * fix issue with peer wants * add test for delayed block sends * remove obsolete tests * wip chunker * run all tests * add todo * misc * remove irrelevant files * removing more files * adding helpers for bitswap tests * moved bitswap file * misc * make blocks timeout longer * adjust block timeout * speedup test * compile with threads * import missing crypto * misc * disable threads for now * fix 32 bit platforms * re-enable threads support in tests
2021-02-26 00:23:22 +00:00
engine = downloader.engine
# Add blocks from 1st peer to want list
engine.wantList &= blocks[0..3].mapIt( it.cid )
engine.wantList &= blocks[12..15].mapIt( it.cid )
Node setup (#32) * settup basic nim node * adding http utils * adding confutils * rough rest api proto * adding missing deps * turn tls emulation off * adding toml serialization * wip * adding missing deps * make sure to clean old state in teardown * adding file upload rest endpoint * renaming blockexchange to networkstore * updating nim-presto * updating libp2p * wip adding streaming upload * reworked chunking * bump to latest unstable * adding asyncfutures stream * make streamable * deleting unused files * reworking stores api * use new stores api * rework blockset and remove blockstream * don't return option from constructor * rework chunker * wip implement upload * fix tests * move unrelated logic to engine * don't print entire message * logging * basic encode/decode to/from dag-pb * add basic upload/download support * fix tests * renaming blockset to manifest * don't pass config to node * remove config and use new manifest * wip: make endpoints more reliable * wip: adding node tests * include correct manifest test * removing asyncfutures * proper chunking of files * simplify stream reading * test with encoding/decoding with many blocks * add block storing tests * adding retrieval test * add logging * tidy up chunker * tidy up manifest and node * use default chunk size * fix tests * fix tests * make sure Eof is set properly * wip * minor cleanup * add file utils * cleanup config * splitout DaggerServer and "main" * remove events since they are not used * add broadcast method to network peer * add and wire localstore * use localstore in the node * wip * logging * move file utils * use the constant * updating deps * fix memstore * use latest libp2p unstable * fix tests * rework block streaming * don't fail storing if the block already exists * add helper info endpoint * correct comment * rename localstore to fsstore * fix tests * remove unused tests * add test to retrieve one block * move some test files around * consolidate setup * Update dagger/blockexchange/engine.nim Co-authored-by: Tanguy <tanguy@status.im> * typo * better block path handling * don't inherit rootobj * remove useless template * Update tests/dagger/blockexc/testblockexc.nim Co-authored-by: markspanbroek <mark@spanbroek.net> * use isMainModule * use proper flag for starter/stoped * cleanup optional use * wrap in isMainModule * use `cancelAndAwait` * remove unused imports * wip * don't use optional * use functional error api * rework store tests and add fs tests * Block.new() to Block.init() * don't use optional for engine blocks * use result instead of optional for getBlock * remove unused imports * move stopping servers to `shutdown` * use result instead of optional * rework with results * fix tests * use waitFor in signal handlers * error helper * use `?` and mapFailure where possible * remove unnecesary `=?` * improve empty cid digest initialization Co-authored-by: Tanguy <tanguy@status.im> Co-authored-by: markspanbroek <mark@spanbroek.net>
2022-01-10 15:32:56 +00:00
await allFutures(
blocks[0..3].mapIt( blockexc[0].engine.localStore.putBlock(it) ))
await allFutures(
blocks[4..7].mapIt( blockexc[1].engine.localStore.putBlock(it) ))
await allFutures(
blocks[8..11].mapIt( blockexc[2].engine.localStore.putBlock(it) ))
await allFutures(
blocks[12..15].mapIt( blockexc[3].engine.localStore.putBlock(it) ))
Poc 2 (#7) * moving protobuf into bitswap * adding block type * reworking bitswap * adding chunker * adding license header * use 1.2.6 * adding fixed size chunker * add blockstore * add iterator to chunker * more bitswap changes * rename ipfs to dagger * rename to dagger * blockstore inherits from BlockProvider * wip - add core block handling logic * smal changes * use proper block store methods * adding asynq heapqueue * wip prepare for bitswap task runner * adding `$` * adding memory store and tests * fixed chunking * extracted desicion engine from bitswap * added helper random funcs * adding testing helpers * only handle seqs * add peer events * cleanup pending blocks on blockstore event * allow nil handlers * move protobuf type helpers * allow initializing block from Cid * testing and fixes * small fixes * expose `<` * spelling * default value * spelling * pending blocks manager * adding stores manager * more tests a wip around bitswap * small changes * merge bitswap and engine for now * for now run only the new poc's tests * add a more complete ci setup * use template in map * remove p2pd * remove go * dont use asyncCheck * few small changes * adding ability to update items * adding multiple task runners * handle cancelation properly * use Result instead of throwing * wip bitswap tests * moving things around * split out engine again * add request and handlers interface * fix tests * wip - engine tests * remove unused imports * fix tests * cleanup block requesting logic * add block request tests * more block requests * add support for max heap * don't use result * use max heap & send block presence in task handler * add task handler tests * rename store to localStore * cleanup & logging * cancel task on stop * don't depend on local store for events * dont use heap queue for wants * add chronicles * fix issue with peer wants * add test for delayed block sends * remove obsolete tests * wip chunker * run all tests * add todo * misc * remove irrelevant files * removing more files * adding helpers for bitswap tests * moved bitswap file * misc * make blocks timeout longer * adjust block timeout * speedup test * compile with threads * import missing crypto * misc * disable threads for now * fix 32 bit platforms * re-enable threads support in tests
2021-02-26 00:23:22 +00:00
await connectNodes(switch)
await sleepAsync(1.seconds)
check:
engine.peers[0].peerHave.mapIt($it).sorted(cmp[string]) ==
blocks[0..3].mapIt( it.cid ).mapIt($it).sorted(cmp[string])
Node setup (#32) * settup basic nim node * adding http utils * adding confutils * rough rest api proto * adding missing deps * turn tls emulation off * adding toml serialization * wip * adding missing deps * make sure to clean old state in teardown * adding file upload rest endpoint * renaming blockexchange to networkstore * updating nim-presto * updating libp2p * wip adding streaming upload * reworked chunking * bump to latest unstable * adding asyncfutures stream * make streamable * deleting unused files * reworking stores api * use new stores api * rework blockset and remove blockstream * don't return option from constructor * rework chunker * wip implement upload * fix tests * move unrelated logic to engine * don't print entire message * logging * basic encode/decode to/from dag-pb * add basic upload/download support * fix tests * renaming blockset to manifest * don't pass config to node * remove config and use new manifest * wip: make endpoints more reliable * wip: adding node tests * include correct manifest test * removing asyncfutures * proper chunking of files * simplify stream reading * test with encoding/decoding with many blocks * add block storing tests * adding retrieval test * add logging * tidy up chunker * tidy up manifest and node * use default chunk size * fix tests * fix tests * make sure Eof is set properly * wip * minor cleanup * add file utils * cleanup config * splitout DaggerServer and "main" * remove events since they are not used * add broadcast method to network peer * add and wire localstore * use localstore in the node * wip * logging * move file utils * use the constant * updating deps * fix memstore * use latest libp2p unstable * fix tests * rework block streaming * don't fail storing if the block already exists * add helper info endpoint * correct comment * rename localstore to fsstore * fix tests * remove unused tests * add test to retrieve one block * move some test files around * consolidate setup * Update dagger/blockexchange/engine.nim Co-authored-by: Tanguy <tanguy@status.im> * typo * better block path handling * don't inherit rootobj * remove useless template * Update tests/dagger/blockexc/testblockexc.nim Co-authored-by: markspanbroek <mark@spanbroek.net> * use isMainModule * use proper flag for starter/stoped * cleanup optional use * wrap in isMainModule * use `cancelAndAwait` * remove unused imports * wip * don't use optional * use functional error api * rework store tests and add fs tests * Block.new() to Block.init() * don't use optional for engine blocks * use result instead of optional for getBlock * remove unused imports * move stopping servers to `shutdown` * use result instead of optional * rework with results * fix tests * use waitFor in signal handlers * error helper * use `?` and mapFailure where possible * remove unnecesary `=?` * improve empty cid digest initialization Co-authored-by: Tanguy <tanguy@status.im> Co-authored-by: markspanbroek <mark@spanbroek.net>
2022-01-10 15:32:56 +00:00
engine.peers[3].peerHave.mapIt($it).sorted(cmp[string]) ==
blocks[12..15].mapIt( it.cid ).mapIt($it).sorted(cmp[string])
Poc 2 (#7) * moving protobuf into bitswap * adding block type * reworking bitswap * adding chunker * adding license header * use 1.2.6 * adding fixed size chunker * add blockstore * add iterator to chunker * more bitswap changes * rename ipfs to dagger * rename to dagger * blockstore inherits from BlockProvider * wip - add core block handling logic * smal changes * use proper block store methods * adding asynq heapqueue * wip prepare for bitswap task runner * adding `$` * adding memory store and tests * fixed chunking * extracted desicion engine from bitswap * added helper random funcs * adding testing helpers * only handle seqs * add peer events * cleanup pending blocks on blockstore event * allow nil handlers * move protobuf type helpers * allow initializing block from Cid * testing and fixes * small fixes * expose `<` * spelling * default value * spelling * pending blocks manager * adding stores manager * more tests a wip around bitswap * small changes * merge bitswap and engine for now * for now run only the new poc's tests * add a more complete ci setup * use template in map * remove p2pd * remove go * dont use asyncCheck * few small changes * adding ability to update items * adding multiple task runners * handle cancelation properly * use Result instead of throwing * wip bitswap tests * moving things around * split out engine again * add request and handlers interface * fix tests * wip - engine tests * remove unused imports * fix tests * cleanup block requesting logic * add block request tests * more block requests * add support for max heap * don't use result * use max heap & send block presence in task handler * add task handler tests * rename store to localStore * cleanup & logging * cancel task on stop * don't depend on local store for events * dont use heap queue for wants * add chronicles * fix issue with peer wants * add test for delayed block sends * remove obsolete tests * wip chunker * run all tests * add todo * misc * remove irrelevant files * removing more files * adding helpers for bitswap tests * moved bitswap file * misc * make blocks timeout longer * adjust block timeout * speedup test * compile with threads * import missing crypto * misc * disable threads for now * fix 32 bit platforms * re-enable threads support in tests
2021-02-26 00:23:22 +00:00
test "should exchange blocks with multiple nodes":
let
downloader = blockexc[4]
Poc 2 (#7) * moving protobuf into bitswap * adding block type * reworking bitswap * adding chunker * adding license header * use 1.2.6 * adding fixed size chunker * add blockstore * add iterator to chunker * more bitswap changes * rename ipfs to dagger * rename to dagger * blockstore inherits from BlockProvider * wip - add core block handling logic * smal changes * use proper block store methods * adding asynq heapqueue * wip prepare for bitswap task runner * adding `$` * adding memory store and tests * fixed chunking * extracted desicion engine from bitswap * added helper random funcs * adding testing helpers * only handle seqs * add peer events * cleanup pending blocks on blockstore event * allow nil handlers * move protobuf type helpers * allow initializing block from Cid * testing and fixes * small fixes * expose `<` * spelling * default value * spelling * pending blocks manager * adding stores manager * more tests a wip around bitswap * small changes * merge bitswap and engine for now * for now run only the new poc's tests * add a more complete ci setup * use template in map * remove p2pd * remove go * dont use asyncCheck * few small changes * adding ability to update items * adding multiple task runners * handle cancelation properly * use Result instead of throwing * wip bitswap tests * moving things around * split out engine again * add request and handlers interface * fix tests * wip - engine tests * remove unused imports * fix tests * cleanup block requesting logic * add block request tests * more block requests * add support for max heap * don't use result * use max heap & send block presence in task handler * add task handler tests * rename store to localStore * cleanup & logging * cancel task on stop * don't depend on local store for events * dont use heap queue for wants * add chronicles * fix issue with peer wants * add test for delayed block sends * remove obsolete tests * wip chunker * run all tests * add todo * misc * remove irrelevant files * removing more files * adding helpers for bitswap tests * moved bitswap file * misc * make blocks timeout longer * adjust block timeout * speedup test * compile with threads * import missing crypto * misc * disable threads for now * fix 32 bit platforms * re-enable threads support in tests
2021-02-26 00:23:22 +00:00
engine = downloader.engine
# Add blocks from 1st peer to want list
engine.wantList &= blocks[0..3].mapIt( it.cid )
engine.wantList &= blocks[12..15].mapIt( it.cid )
Node setup (#32) * settup basic nim node * adding http utils * adding confutils * rough rest api proto * adding missing deps * turn tls emulation off * adding toml serialization * wip * adding missing deps * make sure to clean old state in teardown * adding file upload rest endpoint * renaming blockexchange to networkstore * updating nim-presto * updating libp2p * wip adding streaming upload * reworked chunking * bump to latest unstable * adding asyncfutures stream * make streamable * deleting unused files * reworking stores api * use new stores api * rework blockset and remove blockstream * don't return option from constructor * rework chunker * wip implement upload * fix tests * move unrelated logic to engine * don't print entire message * logging * basic encode/decode to/from dag-pb * add basic upload/download support * fix tests * renaming blockset to manifest * don't pass config to node * remove config and use new manifest * wip: make endpoints more reliable * wip: adding node tests * include correct manifest test * removing asyncfutures * proper chunking of files * simplify stream reading * test with encoding/decoding with many blocks * add block storing tests * adding retrieval test * add logging * tidy up chunker * tidy up manifest and node * use default chunk size * fix tests * fix tests * make sure Eof is set properly * wip * minor cleanup * add file utils * cleanup config * splitout DaggerServer and "main" * remove events since they are not used * add broadcast method to network peer * add and wire localstore * use localstore in the node * wip * logging * move file utils * use the constant * updating deps * fix memstore * use latest libp2p unstable * fix tests * rework block streaming * don't fail storing if the block already exists * add helper info endpoint * correct comment * rename localstore to fsstore * fix tests * remove unused tests * add test to retrieve one block * move some test files around * consolidate setup * Update dagger/blockexchange/engine.nim Co-authored-by: Tanguy <tanguy@status.im> * typo * better block path handling * don't inherit rootobj * remove useless template * Update tests/dagger/blockexc/testblockexc.nim Co-authored-by: markspanbroek <mark@spanbroek.net> * use isMainModule * use proper flag for starter/stoped * cleanup optional use * wrap in isMainModule * use `cancelAndAwait` * remove unused imports * wip * don't use optional * use functional error api * rework store tests and add fs tests * Block.new() to Block.init() * don't use optional for engine blocks * use result instead of optional for getBlock * remove unused imports * move stopping servers to `shutdown` * use result instead of optional * rework with results * fix tests * use waitFor in signal handlers * error helper * use `?` and mapFailure where possible * remove unnecesary `=?` * improve empty cid digest initialization Co-authored-by: Tanguy <tanguy@status.im> Co-authored-by: markspanbroek <mark@spanbroek.net>
2022-01-10 15:32:56 +00:00
await allFutures(
blocks[0..3].mapIt( blockexc[0].engine.localStore.putBlock(it) ))
await allFutures(
blocks[4..7].mapIt( blockexc[1].engine.localStore.putBlock(it) ))
await allFutures(
blocks[8..11].mapIt( blockexc[2].engine.localStore.putBlock(it) ))
await allFutures(
blocks[12..15].mapIt( blockexc[3].engine.localStore.putBlock(it) ))
Poc 2 (#7) * moving protobuf into bitswap * adding block type * reworking bitswap * adding chunker * adding license header * use 1.2.6 * adding fixed size chunker * add blockstore * add iterator to chunker * more bitswap changes * rename ipfs to dagger * rename to dagger * blockstore inherits from BlockProvider * wip - add core block handling logic * smal changes * use proper block store methods * adding asynq heapqueue * wip prepare for bitswap task runner * adding `$` * adding memory store and tests * fixed chunking * extracted desicion engine from bitswap * added helper random funcs * adding testing helpers * only handle seqs * add peer events * cleanup pending blocks on blockstore event * allow nil handlers * move protobuf type helpers * allow initializing block from Cid * testing and fixes * small fixes * expose `<` * spelling * default value * spelling * pending blocks manager * adding stores manager * more tests a wip around bitswap * small changes * merge bitswap and engine for now * for now run only the new poc's tests * add a more complete ci setup * use template in map * remove p2pd * remove go * dont use asyncCheck * few small changes * adding ability to update items * adding multiple task runners * handle cancelation properly * use Result instead of throwing * wip bitswap tests * moving things around * split out engine again * add request and handlers interface * fix tests * wip - engine tests * remove unused imports * fix tests * cleanup block requesting logic * add block request tests * more block requests * add support for max heap * don't use result * use max heap & send block presence in task handler * add task handler tests * rename store to localStore * cleanup & logging * cancel task on stop * don't depend on local store for events * dont use heap queue for wants * add chronicles * fix issue with peer wants * add test for delayed block sends * remove obsolete tests * wip chunker * run all tests * add todo * misc * remove irrelevant files * removing more files * adding helpers for bitswap tests * moved bitswap file * misc * make blocks timeout longer * adjust block timeout * speedup test * compile with threads * import missing crypto * misc * disable threads for now * fix 32 bit platforms * re-enable threads support in tests
2021-02-26 00:23:22 +00:00
await connectNodes(switch)
Node setup (#32) * settup basic nim node * adding http utils * adding confutils * rough rest api proto * adding missing deps * turn tls emulation off * adding toml serialization * wip * adding missing deps * make sure to clean old state in teardown * adding file upload rest endpoint * renaming blockexchange to networkstore * updating nim-presto * updating libp2p * wip adding streaming upload * reworked chunking * bump to latest unstable * adding asyncfutures stream * make streamable * deleting unused files * reworking stores api * use new stores api * rework blockset and remove blockstream * don't return option from constructor * rework chunker * wip implement upload * fix tests * move unrelated logic to engine * don't print entire message * logging * basic encode/decode to/from dag-pb * add basic upload/download support * fix tests * renaming blockset to manifest * don't pass config to node * remove config and use new manifest * wip: make endpoints more reliable * wip: adding node tests * include correct manifest test * removing asyncfutures * proper chunking of files * simplify stream reading * test with encoding/decoding with many blocks * add block storing tests * adding retrieval test * add logging * tidy up chunker * tidy up manifest and node * use default chunk size * fix tests * fix tests * make sure Eof is set properly * wip * minor cleanup * add file utils * cleanup config * splitout DaggerServer and "main" * remove events since they are not used * add broadcast method to network peer * add and wire localstore * use localstore in the node * wip * logging * move file utils * use the constant * updating deps * fix memstore * use latest libp2p unstable * fix tests * rework block streaming * don't fail storing if the block already exists * add helper info endpoint * correct comment * rename localstore to fsstore * fix tests * remove unused tests * add test to retrieve one block * move some test files around * consolidate setup * Update dagger/blockexchange/engine.nim Co-authored-by: Tanguy <tanguy@status.im> * typo * better block path handling * don't inherit rootobj * remove useless template * Update tests/dagger/blockexc/testblockexc.nim Co-authored-by: markspanbroek <mark@spanbroek.net> * use isMainModule * use proper flag for starter/stoped * cleanup optional use * wrap in isMainModule * use `cancelAndAwait` * remove unused imports * wip * don't use optional * use functional error api * rework store tests and add fs tests * Block.new() to Block.init() * don't use optional for engine blocks * use result instead of optional for getBlock * remove unused imports * move stopping servers to `shutdown` * use result instead of optional * rework with results * fix tests * use waitFor in signal handlers * error helper * use `?` and mapFailure where possible * remove unnecesary `=?` * improve empty cid digest initialization Co-authored-by: Tanguy <tanguy@status.im> Co-authored-by: markspanbroek <mark@spanbroek.net>
2022-01-10 15:32:56 +00:00
await sleepAsync(1.seconds)
let wantListBlocks = await allFinished(
blocks[0..3].mapIt( downloader.getBlock(it.cid) ))
check wantListBlocks.mapIt( !it.read ) == blocks[0..3]