ffa9b624f1
Integrate erasure ( #73 )
...
* wip: adding request for storage endpoint
* wire in erasure coding
* fix tests for erasure coding
* put type definitions into separate file
* integrate erasure coding
* change run/shutdown to start/stop
* temporary sleep, otherwise the fsstore blocks
2022-04-05 18:34:29 -06:00
49c41e27b7
Erasure coding dataset ( #72 )
...
* add erasure coding support
* add leopard support
* erasure coding tests
2022-04-05 13:12:59 -06:00
3222f436cc
Protected manifest ( #69 )
...
* add protected manifes type
* encode protected manifest
* add empty block type handling
* add tests for protected manifest
2022-04-04 18:46:13 -06:00
befebcf325
[sales] Make storage available again when offer expires
2022-04-04 11:29:32 +02:00
e46c9816fa
[sales] Make storage available again when offer declined
2022-04-04 11:29:32 +02:00
11f7b4b4d8
[sales] Clean up tests
2022-04-04 11:29:32 +02:00
3810889fcb
[sales] Do not call onSale() when someone else's offer is selected
2022-04-04 11:29:32 +02:00
71f25d40c7
[market] offerStorage() returns submitted offer
2022-04-04 11:29:32 +02:00
94e34e7d28
[market] requestStorage() returns submitted request
2022-04-04 11:29:32 +02:00
2ffde32772
[sales] call onSale() when offer has been selected
2022-04-04 11:29:32 +02:00
71baedb5ec
[sales] Set offer expiry timestamp
2022-04-04 11:29:32 +02:00
76be728f61
[sales] Availability.new -> Availability.init
2022-04-04 11:29:32 +02:00
291c506e0b
[market] Fix timing of test for slow CI
2022-04-04 11:29:32 +02:00
dd89a55351
[market] Add Market.getTime()
2022-04-04 11:29:32 +02:00
061b32296a
Introduce Sales
2022-04-04 11:29:32 +02:00
75ec8c0bfd
[market] wait until request/offer expires
2022-04-04 10:03:46 +02:00
8ff748eff8
[market] select a storage offer
2022-04-04 10:03:46 +02:00
5ea64522d8
[market] subscribe to offers for a certain request
2022-04-04 10:03:46 +02:00
9f76e714a2
[market] submit storage offers
2022-04-04 10:03:46 +02:00
e52153e2ba
[market] Require a storage contract with a signer
2022-04-04 10:03:46 +02:00
e78a0fb43e
[contracts] Wrap contracts in Market abstraction
2022-04-04 10:03:46 +02:00
bd8f4d5d74
[tests] Extract common examples into separate module
2022-04-04 10:03:46 +02:00
5b5f3335d6
[purchasing] Ignore offers that are about to expire
2022-04-04 10:03:46 +02:00
dcfd6be1c6
[purchasing] Ignore expired offers
2022-04-04 10:03:46 +02:00
fe23cb89d7
[purchasing] Select cheapest offer
2022-04-04 10:03:46 +02:00
04c291b05c
[purchasing] rename requests -> requested
2022-04-04 10:03:46 +02:00
7c9c244836
[purchasing] Remove PurchaseRequest; use StorageRequest
2022-04-04 10:03:46 +02:00
9ade3fdd37
[contracts] StorageRequest/Offer is object instead of tuple
2022-04-04 10:03:46 +02:00
5f10549f19
Introduce Purchasing and Market
2022-04-04 10:03:46 +02:00
a3fdd35f73
[contracts] Split marketplace.nim into requests.nim and offers.nim
2022-04-04 10:03:46 +02:00
43cea1743a
Node storestream ( #62 )
...
* use storestream for retrieval
* use `len` prop on manifest
* remove read pattern
* make blocksize and chunksize consistent
* fix tests
* fix chunker tests - now padded by default
2022-03-29 20:43:35 -06:00
42edbf5076
[contracts] Remove obsolete marketplace tests
2022-03-28 11:59:01 +02:00
efb4f5c375
[contracts] Update to new marketplace design
2022-03-28 11:59:01 +02:00
87bd947d3d
[contracts] Update to new collateral design
2022-03-28 11:59:01 +02:00
d07a5cd19e
[contracts] Run contract tests in CI
...
Introduce new nimble 'test' task that runs *all* tests.
Renames testAll to testDagger, because testAll didn't
run all tests, it forgot to run the contract tests.
2022-03-28 11:59:01 +02:00
e965f5e0de
Seekable stream ( #58 )
...
* cleanup manifest
* adding a "seekable" base stream
2022-03-21 12:09:59 -06:00
d204b518e8
Make block a ref ( #56 )
...
* cleanup manifest
* make block a ref type
2022-03-18 13:50:53 -06:00
472e1e6fc3
Store stream ( #55 )
...
* make manifest a ref object
* capitalize tests
* add storestream and tests
* fix tests
* run storestream tests
2022-03-17 07:56:46 -06:00
f2aa945991
Manifest block size ( #53 )
...
* add `blockSize` and `size` to manifest
* fix tests
* fix decoding/encoding
* re-export coders
* style
2022-03-15 12:47:31 -06:00
f414c695be
Cleanup-manifest ( #52 )
...
* adding nim-blscurve
* adding taskpools
* cleanup manifest
2022-03-14 10:06:36 -06:00
70cbdff901
feat: introduce LRU cache ( #50 )
...
* feat: introduce LRU cache
Replace `MemoryStore` with LRU caching mechanism.
`lrucache` library was forked to https://github.com/status-im/lrucache.nim .
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
# Conflicts:
# dagger/dagger.nim
# dagger/stores.nim
# dagger/stores/manager.nim
# tests/dagger/blockexc/testengine.nim
# tests/dagger/helpers/nodeutils.nim
# tests/dagger/testnode.nim
# tests/dagger/teststores.nim
* feat: introduce --cache-size CLI option
Allow for a value of `0` to disable the cache.
# Conflicts:
# dagger/dagger.nim
* allow dynamic block size in cache
allow block size to be variable/dynamic
update lrucache to use updated lrucache dep
Using removeLru proc, added tests
* Refactor CacheStore init block
Co-authored-by: Michael Bradley, Jr <michaelsbradleyjr@gmail.com>
2022-03-02 10:30:42 -06:00
2e5c28781c
feat: integrate dagger contracts
...
Integrate dagger contracts from `nim-dagger-contracts` repo.
Add `dagger-contracts`, `nim-web3`, and all of `nim-web3`’s transitive deps as submodule deps to `nim-dagger`. Note: `nim-web3` and its transitive deps may no longer be needed when we switch to `nim-ethers`.
Add a `testContracts` nimble task to test all of the contracts functionality. Namely, this spins up an ethereum simulator, deploys the contracts (in `dagger-contracts`), runs the contract tests, and finally, regardless of success/error, kills the ethereum sim processes. The nimble task can be run with `./env.sh nimble testContracts`.
We also tested `nim-dagger-contracts` as a submodule dep of `nim-dagger`, and while the tests run as expected, the preference is to merge `nim-dagger-contracts` inside of `nim-dagger` for ease of parallel development. There’s also a high probability that `nim-dagger-contracts` is not being used as a dep by other projects. Are there any strong objections to this?
Co-authored-by: Michael Bradley <michaelsbradleyjr@gmail.com>
2022-02-04 15:34:56 +11:00
5f48de6a44
cleanup manifest ( #44 )
...
* cleanup manifest
* Attach decode to `Manifest`
* move manifest container to it's own file
* rebase main
* update header license year
2022-01-12 19:55:51 -06:00
68a45d8f57
use tryGet, which is more informative than get ( #45 )
2022-01-12 18:42:18 -06:00
2188f69273
Minor cleanup and update deps ( #42 )
...
* abort download if atleast one block failed
* make chunker a regular object
* updating questionable
* updating vendor deps
* cleanup unused imports
* Remove mention of Waku
2022-01-12 16:19:37 -06:00
15ef6c78eb
fix treehash generation ( #39 )
2022-01-10 21:04:00 -06:00
47a0d4ef7a
Move empty digest ( #38 )
...
* improve empty digest initialization
* cleanup warnings
2022-01-10 20:25:27 -06:00
a9721dd65f
Block.init should return Result ( #37 )
...
* Block.init should return Result
* use get() instead of tryGet()
2022-01-10 20:25:13 -06:00
fbe161a073
Node setup ( #32 )
...
* settup basic nim node
* adding http utils
* adding confutils
* rough rest api proto
* adding missing deps
* turn tls emulation off
* adding toml serialization
* wip
* adding missing deps
* make sure to clean old state in teardown
* adding file upload rest endpoint
* renaming blockexchange to networkstore
* updating nim-presto
* updating libp2p
* wip adding streaming upload
* reworked chunking
* bump to latest unstable
* adding asyncfutures stream
* make streamable
* deleting unused files
* reworking stores api
* use new stores api
* rework blockset and remove blockstream
* don't return option from constructor
* rework chunker
* wip implement upload
* fix tests
* move unrelated logic to engine
* don't print entire message
* logging
* basic encode/decode to/from dag-pb
* add basic upload/download support
* fix tests
* renaming blockset to manifest
* don't pass config to node
* remove config and use new manifest
* wip: make endpoints more reliable
* wip: adding node tests
* include correct manifest test
* removing asyncfutures
* proper chunking of files
* simplify stream reading
* test with encoding/decoding with many blocks
* add block storing tests
* adding retrieval test
* add logging
* tidy up chunker
* tidy up manifest and node
* use default chunk size
* fix tests
* fix tests
* make sure Eof is set properly
* wip
* minor cleanup
* add file utils
* cleanup config
* splitout DaggerServer and "main"
* remove events since they are not used
* add broadcast method to network peer
* add and wire localstore
* use localstore in the node
* wip
* logging
* move file utils
* use the constant
* updating deps
* fix memstore
* use latest libp2p unstable
* fix tests
* rework block streaming
* don't fail storing if the block already exists
* add helper info endpoint
* correct comment
* rename localstore to fsstore
* fix tests
* remove unused tests
* add test to retrieve one block
* move some test files around
* consolidate setup
* Update dagger/blockexchange/engine.nim
Co-authored-by: Tanguy <tanguy@status.im>
* typo
* better block path handling
* don't inherit rootobj
* remove useless template
* Update tests/dagger/blockexc/testblockexc.nim
Co-authored-by: markspanbroek <mark@spanbroek.net>
* use isMainModule
* use proper flag for starter/stoped
* cleanup optional use
* wrap in isMainModule
* use `cancelAndAwait`
* remove unused imports
* wip
* don't use optional
* use functional error api
* rework store tests and add fs tests
* Block.new() to Block.init()
* don't use optional for engine blocks
* use result instead of optional for getBlock
* remove unused imports
* move stopping servers to `shutdown`
* use result instead of optional
* rework with results
* fix tests
* use waitFor in signal handlers
* error helper
* use `?` and mapFailure where possible
* remove unnecesary `=?`
* improve empty cid digest initialization
Co-authored-by: Tanguy <tanguy@status.im>
Co-authored-by: markspanbroek <mark@spanbroek.net>
2022-01-10 09:32:56 -06:00
90a6131d7d
Fix occasional test failure due to timeout
2021-11-16 16:51:40 +01:00
Dmitriy Ryajov
Mark Spanbroek
Eric Mastro