4d681102e5
Add DHT ( #75 )
...
* First implem
* Add persistent net key option
* Working DHT setup
* Bootstrap nodes
* Implement DaggerNode.findPeer
* Remove unrelevant comment
* Added discovery to blockexchange requestBlock
* add FSStore.blockList
* Block advertisement
* Tests compiles
* Green tests
* toDiscoveryId instead of toNodeId
* remove stopAdvertisingBlock
* Removed nim-eth dependency
* Move discovery stuff to discovery.nim
* Add missing file, start of discovery tests
* Better discovery logic
* Add tests
* Address comment
* Better E2E test
2022-04-13 10:32:35 -06:00
1a3003e043
Update to latest dagger-contracts ( #78 )
...
- StorageRequest struct has changed
- StorageRequested event has changed,
it no longer returns the entire request,
but only the id and the ask
2022-04-11 12:03:55 -06:00
03fa370624
Proving ( #66 )
...
* Add Proving object, which maintains contract id's to watch
* [proving] invoke callback when proof is required
# Conflicts:
# dagger/por/timing/periods.nim
# dagger/por/timing/prooftiming.nim
* [proving] check proof requirements for all our contracts
# Conflicts:
# tests/dagger/helpers/mockprooftiming.nim
* Update vendor/dagger-contracts
* [proving] call onProofRequired() when proof is required soon
* [proving] stop checking contracts that have ended
* [proving] Remove duplicated funcs
* [proving] Implement ProofTiming on top of smart contract
* [proving] Fix race condition in waitUntilNextPeriod()
Sometimes waitUntilNextPeriod would take a while to
determine the current period, leading to unexpected results.
Splits waitUntilNextPeriod() into getCurrentPeriod()
and waitUntilPeriod(), to ensure that we're really waiting
for the period that we think we're waiting for.
2022-04-08 15:58:16 -06:00
2e8b39cf7c
fix random failing test ( #77 )
2022-04-07 17:08:43 -06:00
ffa9b624f1
Integrate erasure ( #73 )
...
* wip: adding request for storage endpoint
* wire in erasure coding
* fix tests for erasure coding
* put type definitions into separate file
* integrate erasure coding
* change run/shutdown to start/stop
* temporary sleep, otherwise the fsstore blocks
2022-04-05 18:34:29 -06:00
49c41e27b7
Erasure coding dataset ( #72 )
...
* add erasure coding support
* add leopard support
* erasure coding tests
2022-04-05 13:12:59 -06:00
3222f436cc
Protected manifest ( #69 )
...
* add protected manifes type
* encode protected manifest
* add empty block type handling
* add tests for protected manifest
2022-04-04 18:46:13 -06:00
befebcf325
[sales] Make storage available again when offer expires
2022-04-04 11:29:32 +02:00
e46c9816fa
[sales] Make storage available again when offer declined
2022-04-04 11:29:32 +02:00
11f7b4b4d8
[sales] Clean up tests
2022-04-04 11:29:32 +02:00
3810889fcb
[sales] Do not call onSale() when someone else's offer is selected
2022-04-04 11:29:32 +02:00
71f25d40c7
[market] offerStorage() returns submitted offer
2022-04-04 11:29:32 +02:00
94e34e7d28
[market] requestStorage() returns submitted request
2022-04-04 11:29:32 +02:00
2ffde32772
[sales] call onSale() when offer has been selected
2022-04-04 11:29:32 +02:00
71baedb5ec
[sales] Set offer expiry timestamp
2022-04-04 11:29:32 +02:00
76be728f61
[sales] Availability.new -> Availability.init
2022-04-04 11:29:32 +02:00
dd89a55351
[market] Add Market.getTime()
2022-04-04 11:29:32 +02:00
061b32296a
Introduce Sales
2022-04-04 11:29:32 +02:00
bd8f4d5d74
[tests] Extract common examples into separate module
2022-04-04 10:03:46 +02:00
5b5f3335d6
[purchasing] Ignore offers that are about to expire
2022-04-04 10:03:46 +02:00
dcfd6be1c6
[purchasing] Ignore expired offers
2022-04-04 10:03:46 +02:00
fe23cb89d7
[purchasing] Select cheapest offer
2022-04-04 10:03:46 +02:00
04c291b05c
[purchasing] rename requests -> requested
2022-04-04 10:03:46 +02:00
7c9c244836
[purchasing] Remove PurchaseRequest; use StorageRequest
2022-04-04 10:03:46 +02:00
5f10549f19
Introduce Purchasing and Market
2022-04-04 10:03:46 +02:00
43cea1743a
Node storestream ( #62 )
...
* use storestream for retrieval
* use `len` prop on manifest
* remove read pattern
* make blocksize and chunksize consistent
* fix tests
* fix chunker tests - now padded by default
2022-03-29 20:43:35 -06:00
e965f5e0de
Seekable stream ( #58 )
...
* cleanup manifest
* adding a "seekable" base stream
2022-03-21 12:09:59 -06:00
d204b518e8
Make block a ref ( #56 )
...
* cleanup manifest
* make block a ref type
2022-03-18 13:50:53 -06:00
472e1e6fc3
Store stream ( #55 )
...
* make manifest a ref object
* capitalize tests
* add storestream and tests
* fix tests
* run storestream tests
2022-03-17 07:56:46 -06:00
f2aa945991
Manifest block size ( #53 )
...
* add `blockSize` and `size` to manifest
* fix tests
* fix decoding/encoding
* re-export coders
* style
2022-03-15 12:47:31 -06:00
f414c695be
Cleanup-manifest ( #52 )
...
* adding nim-blscurve
* adding taskpools
* cleanup manifest
2022-03-14 10:06:36 -06:00
70cbdff901
feat: introduce LRU cache ( #50 )
...
* feat: introduce LRU cache
Replace `MemoryStore` with LRU caching mechanism.
`lrucache` library was forked to https://github.com/status-im/lrucache.nim .
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
# Conflicts:
# dagger/dagger.nim
# dagger/stores.nim
# dagger/stores/manager.nim
# tests/dagger/blockexc/testengine.nim
# tests/dagger/helpers/nodeutils.nim
# tests/dagger/testnode.nim
# tests/dagger/teststores.nim
* feat: introduce --cache-size CLI option
Allow for a value of `0` to disable the cache.
# Conflicts:
# dagger/dagger.nim
* allow dynamic block size in cache
allow block size to be variable/dynamic
update lrucache to use updated lrucache dep
Using removeLru proc, added tests
* Refactor CacheStore init block
Co-authored-by: Michael Bradley, Jr <michaelsbradleyjr@gmail.com>
2022-03-02 10:30:42 -06:00
5f48de6a44
cleanup manifest ( #44 )
...
* cleanup manifest
* Attach decode to `Manifest`
* move manifest container to it's own file
* rebase main
* update header license year
2022-01-12 19:55:51 -06:00
68a45d8f57
use tryGet, which is more informative than get ( #45 )
2022-01-12 18:42:18 -06:00
2188f69273
Minor cleanup and update deps ( #42 )
...
* abort download if atleast one block failed
* make chunker a regular object
* updating questionable
* updating vendor deps
* cleanup unused imports
* Remove mention of Waku
2022-01-12 16:19:37 -06:00
15ef6c78eb
fix treehash generation ( #39 )
2022-01-10 21:04:00 -06:00
47a0d4ef7a
Move empty digest ( #38 )
...
* improve empty digest initialization
* cleanup warnings
2022-01-10 20:25:27 -06:00
a9721dd65f
Block.init should return Result ( #37 )
...
* Block.init should return Result
* use get() instead of tryGet()
2022-01-10 20:25:13 -06:00
fbe161a073
Node setup ( #32 )
...
* settup basic nim node
* adding http utils
* adding confutils
* rough rest api proto
* adding missing deps
* turn tls emulation off
* adding toml serialization
* wip
* adding missing deps
* make sure to clean old state in teardown
* adding file upload rest endpoint
* renaming blockexchange to networkstore
* updating nim-presto
* updating libp2p
* wip adding streaming upload
* reworked chunking
* bump to latest unstable
* adding asyncfutures stream
* make streamable
* deleting unused files
* reworking stores api
* use new stores api
* rework blockset and remove blockstream
* don't return option from constructor
* rework chunker
* wip implement upload
* fix tests
* move unrelated logic to engine
* don't print entire message
* logging
* basic encode/decode to/from dag-pb
* add basic upload/download support
* fix tests
* renaming blockset to manifest
* don't pass config to node
* remove config and use new manifest
* wip: make endpoints more reliable
* wip: adding node tests
* include correct manifest test
* removing asyncfutures
* proper chunking of files
* simplify stream reading
* test with encoding/decoding with many blocks
* add block storing tests
* adding retrieval test
* add logging
* tidy up chunker
* tidy up manifest and node
* use default chunk size
* fix tests
* fix tests
* make sure Eof is set properly
* wip
* minor cleanup
* add file utils
* cleanup config
* splitout DaggerServer and "main"
* remove events since they are not used
* add broadcast method to network peer
* add and wire localstore
* use localstore in the node
* wip
* logging
* move file utils
* use the constant
* updating deps
* fix memstore
* use latest libp2p unstable
* fix tests
* rework block streaming
* don't fail storing if the block already exists
* add helper info endpoint
* correct comment
* rename localstore to fsstore
* fix tests
* remove unused tests
* add test to retrieve one block
* move some test files around
* consolidate setup
* Update dagger/blockexchange/engine.nim
Co-authored-by: Tanguy <tanguy@status.im>
* typo
* better block path handling
* don't inherit rootobj
* remove useless template
* Update tests/dagger/blockexc/testblockexc.nim
Co-authored-by: markspanbroek <mark@spanbroek.net>
* use isMainModule
* use proper flag for starter/stoped
* cleanup optional use
* wrap in isMainModule
* use `cancelAndAwait`
* remove unused imports
* wip
* don't use optional
* use functional error api
* rework store tests and add fs tests
* Block.new() to Block.init()
* don't use optional for engine blocks
* use result instead of optional for getBlock
* remove unused imports
* move stopping servers to `shutdown`
* use result instead of optional
* rework with results
* fix tests
* use waitFor in signal handlers
* error helper
* use `?` and mapFailure where possible
* remove unnecesary `=?`
* improve empty cid digest initialization
Co-authored-by: Tanguy <tanguy@status.im>
Co-authored-by: markspanbroek <mark@spanbroek.net>
2022-01-10 09:32:56 -06:00
90a6131d7d
Fix occasional test failure due to timeout
2021-11-16 16:51:40 +01:00
ef5fa96c7d
Fix unused import warnings
2021-11-16 16:51:24 +01:00
ce66e43440
fix compilation issues with latest libp2p ( #27 )
...
* fix compilation issues with latest libp2p
* change package description
* fix CI compiler
* try with nim 1.4.2
* try with 1.4.2
* use more standardized ci config
* set reasonable log level
* remove nim version dependency
* reverting to 1.4.6 as otherwise it fails on amd64
2021-10-29 13:30:52 -06:00
2fb39ca4a3
Rename bitswap ( #25 )
...
* use PeerInfo in event handlers
* use CidV1 and raw multicodec as default
* add block stream abstraction
* raises defect
* adding dataset abstraction
* move blockstream into own dir
* reorg files and fix tests
* rename dataset to blockset
* wip
* wip
* adding basic test for treehash algo
* run blockset tests along with with the rest
* remove obsolete contents
* fix chunker tests
* rename bitswap and move to stores
* rename bitwsap to blockexc and move to stores
* moare project structure reorg
2021-08-30 13:25:20 -06:00
6bd4260911
Replace pricing exchange by account exchange
...
Rationale: price is no longer set per peer, but per chunk.
Only the Ethereum accounts of the peers needs to be exchanged.
2021-06-07 09:50:17 +02:00
d4dd15734e
Remove double bookkeeping in peerHave and peerPrices
2021-06-07 09:50:17 +02:00
e3b68f96d8
Remove debt ratio
...
Rationale: selecting peers based on debt ratio in bytes
is no longer needed when the bandwidth for each chunk is
compensated.
2021-06-07 09:50:17 +02:00
fa05bffeac
Pay per-block price instead of per-peer price
2021-06-07 09:50:17 +02:00
f69827d582
Send block prices
2021-06-07 09:50:17 +02:00
0a28338136
Simplify test
2021-06-07 09:50:17 +02:00
0e34b69d7d
Add block prices to peer context
2021-06-07 09:50:17 +02:00
Tanguy
markspanbroek
Dmitriy Ryajov
Eric Mastro