8afad9843f
bug, nodes should not have blocks in local store
2022-04-20 12:12:30 -06:00
104d5d095e
don't pass entire blocks to list blocks calback
2022-04-19 20:44:25 -06:00
a9cd7982fd
rework discovery with methods
2022-04-19 19:42:38 -06:00
29d829d51c
don't force moc discovery on all tests
2022-04-19 19:25:37 -06:00
69b7dff09b
move discovery moc to disc dir
2022-04-19 19:22:55 -06:00
8fafc3edf3
rework moc discovery
2022-04-19 19:22:08 -06:00
6b0e5f95e7
add discovery tests
2022-04-19 19:21:55 -06:00
4f5f142a37
add list block tests
2022-04-19 19:21:32 -06:00
9b729e0d76
don't auto mount network
2022-04-19 19:19:51 -06:00
c3df822724
fix tests and split out discovery tests
2022-04-19 19:19:17 -06:00
b88561e090
Subscribe to proof submissions ( #83 )
...
* Update dagger-contracts
* [proving] rename ProofTiming -> Proofs
* Update nim-ethers to 0.1.4
* [proving] Subscribe to proof submissions
* [proving] support proof submission through the Proving abstraction
2022-04-13 10:41:48 -06:00
4d681102e5
Add DHT ( #75 )
...
* First implem
* Add persistent net key option
* Working DHT setup
* Bootstrap nodes
* Implement DaggerNode.findPeer
* Remove unrelevant comment
* Added discovery to blockexchange requestBlock
* add FSStore.blockList
* Block advertisement
* Tests compiles
* Green tests
* toDiscoveryId instead of toNodeId
* remove stopAdvertisingBlock
* Removed nim-eth dependency
* Move discovery stuff to discovery.nim
* Add missing file, start of discovery tests
* Better discovery logic
* Add tests
* Address comment
* Better E2E test
2022-04-13 10:32:35 -06:00
1a3003e043
Update to latest dagger-contracts ( #78 )
...
- StorageRequest struct has changed
- StorageRequested event has changed,
it no longer returns the entire request,
but only the id and the ask
2022-04-11 12:03:55 -06:00
03fa370624
Proving ( #66 )
...
* Add Proving object, which maintains contract id's to watch
* [proving] invoke callback when proof is required
# Conflicts:
# dagger/por/timing/periods.nim
# dagger/por/timing/prooftiming.nim
* [proving] check proof requirements for all our contracts
# Conflicts:
# tests/dagger/helpers/mockprooftiming.nim
* Update vendor/dagger-contracts
* [proving] call onProofRequired() when proof is required soon
* [proving] stop checking contracts that have ended
* [proving] Remove duplicated funcs
* [proving] Implement ProofTiming on top of smart contract
* [proving] Fix race condition in waitUntilNextPeriod()
Sometimes waitUntilNextPeriod would take a while to
determine the current period, leading to unexpected results.
Splits waitUntilNextPeriod() into getCurrentPeriod()
and waitUntilPeriod(), to ensure that we're really waiting
for the period that we think we're waiting for.
2022-04-08 15:58:16 -06:00
2e8b39cf7c
fix random failing test ( #77 )
2022-04-07 17:08:43 -06:00
ffa9b624f1
Integrate erasure ( #73 )
...
* wip: adding request for storage endpoint
* wire in erasure coding
* fix tests for erasure coding
* put type definitions into separate file
* integrate erasure coding
* change run/shutdown to start/stop
* temporary sleep, otherwise the fsstore blocks
2022-04-05 18:34:29 -06:00
49c41e27b7
Erasure coding dataset ( #72 )
...
* add erasure coding support
* add leopard support
* erasure coding tests
2022-04-05 13:12:59 -06:00
3222f436cc
Protected manifest ( #69 )
...
* add protected manifes type
* encode protected manifest
* add empty block type handling
* add tests for protected manifest
2022-04-04 18:46:13 -06:00
befebcf325
[sales] Make storage available again when offer expires
2022-04-04 11:29:32 +02:00
e46c9816fa
[sales] Make storage available again when offer declined
2022-04-04 11:29:32 +02:00
11f7b4b4d8
[sales] Clean up tests
2022-04-04 11:29:32 +02:00
3810889fcb
[sales] Do not call onSale() when someone else's offer is selected
2022-04-04 11:29:32 +02:00
71f25d40c7
[market] offerStorage() returns submitted offer
2022-04-04 11:29:32 +02:00
94e34e7d28
[market] requestStorage() returns submitted request
2022-04-04 11:29:32 +02:00
2ffde32772
[sales] call onSale() when offer has been selected
2022-04-04 11:29:32 +02:00
71baedb5ec
[sales] Set offer expiry timestamp
2022-04-04 11:29:32 +02:00
76be728f61
[sales] Availability.new -> Availability.init
2022-04-04 11:29:32 +02:00
dd89a55351
[market] Add Market.getTime()
2022-04-04 11:29:32 +02:00
061b32296a
Introduce Sales
2022-04-04 11:29:32 +02:00
bd8f4d5d74
[tests] Extract common examples into separate module
2022-04-04 10:03:46 +02:00
5b5f3335d6
[purchasing] Ignore offers that are about to expire
2022-04-04 10:03:46 +02:00
dcfd6be1c6
[purchasing] Ignore expired offers
2022-04-04 10:03:46 +02:00
fe23cb89d7
[purchasing] Select cheapest offer
2022-04-04 10:03:46 +02:00
04c291b05c
[purchasing] rename requests -> requested
2022-04-04 10:03:46 +02:00
7c9c244836
[purchasing] Remove PurchaseRequest; use StorageRequest
2022-04-04 10:03:46 +02:00
5f10549f19
Introduce Purchasing and Market
2022-04-04 10:03:46 +02:00
43cea1743a
Node storestream ( #62 )
...
* use storestream for retrieval
* use `len` prop on manifest
* remove read pattern
* make blocksize and chunksize consistent
* fix tests
* fix chunker tests - now padded by default
2022-03-29 20:43:35 -06:00
e965f5e0de
Seekable stream ( #58 )
...
* cleanup manifest
* adding a "seekable" base stream
2022-03-21 12:09:59 -06:00
d204b518e8
Make block a ref ( #56 )
...
* cleanup manifest
* make block a ref type
2022-03-18 13:50:53 -06:00
472e1e6fc3
Store stream ( #55 )
...
* make manifest a ref object
* capitalize tests
* add storestream and tests
* fix tests
* run storestream tests
2022-03-17 07:56:46 -06:00
f2aa945991
Manifest block size ( #53 )
...
* add `blockSize` and `size` to manifest
* fix tests
* fix decoding/encoding
* re-export coders
* style
2022-03-15 12:47:31 -06:00
f414c695be
Cleanup-manifest ( #52 )
...
* adding nim-blscurve
* adding taskpools
* cleanup manifest
2022-03-14 10:06:36 -06:00
70cbdff901
feat: introduce LRU cache ( #50 )
...
* feat: introduce LRU cache
Replace `MemoryStore` with LRU caching mechanism.
`lrucache` library was forked to https://github.com/status-im/lrucache.nim .
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
# Conflicts:
# dagger/dagger.nim
# dagger/stores.nim
# dagger/stores/manager.nim
# tests/dagger/blockexc/testengine.nim
# tests/dagger/helpers/nodeutils.nim
# tests/dagger/testnode.nim
# tests/dagger/teststores.nim
* feat: introduce --cache-size CLI option
Allow for a value of `0` to disable the cache.
# Conflicts:
# dagger/dagger.nim
* allow dynamic block size in cache
allow block size to be variable/dynamic
update lrucache to use updated lrucache dep
Using removeLru proc, added tests
* Refactor CacheStore init block
Co-authored-by: Michael Bradley, Jr <michaelsbradleyjr@gmail.com>
2022-03-02 10:30:42 -06:00
5f48de6a44
cleanup manifest ( #44 )
...
* cleanup manifest
* Attach decode to `Manifest`
* move manifest container to it's own file
* rebase main
* update header license year
2022-01-12 19:55:51 -06:00
68a45d8f57
use tryGet, which is more informative than get ( #45 )
2022-01-12 18:42:18 -06:00
2188f69273
Minor cleanup and update deps ( #42 )
...
* abort download if atleast one block failed
* make chunker a regular object
* updating questionable
* updating vendor deps
* cleanup unused imports
* Remove mention of Waku
2022-01-12 16:19:37 -06:00
15ef6c78eb
fix treehash generation ( #39 )
2022-01-10 21:04:00 -06:00
47a0d4ef7a
Move empty digest ( #38 )
...
* improve empty digest initialization
* cleanup warnings
2022-01-10 20:25:27 -06:00
a9721dd65f
Block.init should return Result ( #37 )
...
* Block.init should return Result
* use get() instead of tryGet()
2022-01-10 20:25:13 -06:00
fbe161a073
Node setup ( #32 )
...
* settup basic nim node
* adding http utils
* adding confutils
* rough rest api proto
* adding missing deps
* turn tls emulation off
* adding toml serialization
* wip
* adding missing deps
* make sure to clean old state in teardown
* adding file upload rest endpoint
* renaming blockexchange to networkstore
* updating nim-presto
* updating libp2p
* wip adding streaming upload
* reworked chunking
* bump to latest unstable
* adding asyncfutures stream
* make streamable
* deleting unused files
* reworking stores api
* use new stores api
* rework blockset and remove blockstream
* don't return option from constructor
* rework chunker
* wip implement upload
* fix tests
* move unrelated logic to engine
* don't print entire message
* logging
* basic encode/decode to/from dag-pb
* add basic upload/download support
* fix tests
* renaming blockset to manifest
* don't pass config to node
* remove config and use new manifest
* wip: make endpoints more reliable
* wip: adding node tests
* include correct manifest test
* removing asyncfutures
* proper chunking of files
* simplify stream reading
* test with encoding/decoding with many blocks
* add block storing tests
* adding retrieval test
* add logging
* tidy up chunker
* tidy up manifest and node
* use default chunk size
* fix tests
* fix tests
* make sure Eof is set properly
* wip
* minor cleanup
* add file utils
* cleanup config
* splitout DaggerServer and "main"
* remove events since they are not used
* add broadcast method to network peer
* add and wire localstore
* use localstore in the node
* wip
* logging
* move file utils
* use the constant
* updating deps
* fix memstore
* use latest libp2p unstable
* fix tests
* rework block streaming
* don't fail storing if the block already exists
* add helper info endpoint
* correct comment
* rename localstore to fsstore
* fix tests
* remove unused tests
* add test to retrieve one block
* move some test files around
* consolidate setup
* Update dagger/blockexchange/engine.nim
Co-authored-by: Tanguy <tanguy@status.im>
* typo
* better block path handling
* don't inherit rootobj
* remove useless template
* Update tests/dagger/blockexc/testblockexc.nim
Co-authored-by: markspanbroek <mark@spanbroek.net>
* use isMainModule
* use proper flag for starter/stoped
* cleanup optional use
* wrap in isMainModule
* use `cancelAndAwait`
* remove unused imports
* wip
* don't use optional
* use functional error api
* rework store tests and add fs tests
* Block.new() to Block.init()
* don't use optional for engine blocks
* use result instead of optional for getBlock
* remove unused imports
* move stopping servers to `shutdown`
* use result instead of optional
* rework with results
* fix tests
* use waitFor in signal handlers
* error helper
* use `?` and mapFailure where possible
* remove unnecesary `=?`
* improve empty cid digest initialization
Co-authored-by: Tanguy <tanguy@status.im>
Co-authored-by: markspanbroek <mark@spanbroek.net>
2022-01-10 09:32:56 -06:00
Dmitriy Ryajov
markspanbroek
Tanguy
Eric Mastro