Commit Graph

112 Commits

Author SHA1 Message Date
Dmitriy Ryajov 4740ffc144
Cleanup engine and rework discovery (#87)
* rework discovery with async queues

* misc style changes

* increase max message size for large manifests

* use upraises and avoid exceptions on key access

* increase sleep time to 100 millis

* pass config

* make list blocks trigger a callback on each block

* check for nil on start/stop

* fix tests and split out discovery tests

* don't auto mount network

* add list block tests

* add discovery tests

* rework moc discovery

* move discovery moc to disc dir

* don't force logging syncs

* don't force moc discovery on all tests

* rework discovery with methods

* add top level utils file

* don't use asyncCheck

* don't pass entire blocks to list blocks calback

* spelling
2022-04-19 21:46:44 -06:00
markspanbroek b88561e090
Subscribe to proof submissions (#83)
* Update dagger-contracts

* [proving] rename ProofTiming -> Proofs

* Update nim-ethers to 0.1.4

* [proving] Subscribe to proof submissions

* [proving] support proof submission through the Proving abstraction
2022-04-13 10:41:48 -06:00
Tanguy 4d681102e5
Add DHT (#75)
* First implem

* Add persistent net key option

* Working DHT setup

* Bootstrap nodes

* Implement DaggerNode.findPeer

* Remove unrelevant comment

* Added discovery to blockexchange requestBlock

* add FSStore.blockList

* Block advertisement

* Tests compiles

* Green tests

* toDiscoveryId instead of toNodeId

* remove stopAdvertisingBlock

* Removed nim-eth dependency

* Move discovery stuff to discovery.nim

* Add missing file, start of discovery tests

* Better discovery logic

* Add tests

* Address comment

* Better E2E test
2022-04-13 10:32:35 -06:00
Zahary Karadjov b658192540
Nimble lock file 2022-04-12 16:59:00 +03:00
markspanbroek 1a3003e043
Update to latest dagger-contracts (#78)
- StorageRequest struct has changed

- StorageRequested event has changed,
  it no longer returns the entire request,
  but only the id and the ask
2022-04-11 12:03:55 -06:00
markspanbroek 03fa370624
Proving (#66)
* Add Proving object, which maintains contract id's to watch

* [proving] invoke callback when proof is required

# Conflicts:
#	dagger/por/timing/periods.nim
#	dagger/por/timing/prooftiming.nim

* [proving] check proof requirements for all our contracts

# Conflicts:
#	tests/dagger/helpers/mockprooftiming.nim

* Update vendor/dagger-contracts

* [proving] call onProofRequired() when proof is required soon

* [proving] stop checking contracts that have ended

* [proving] Remove duplicated funcs

* [proving] Implement ProofTiming on top of smart contract

* [proving] Fix race condition in waitUntilNextPeriod()

Sometimes waitUntilNextPeriod would take a while to
determine the current period, leading to unexpected results.

Splits waitUntilNextPeriod() into getCurrentPeriod()
and waitUntilPeriod(), to ensure that we're really waiting
for the period that we think we're waiting for.
2022-04-08 15:58:16 -06:00
Dmitriy Ryajov 2e8b39cf7c
fix random failing test (#77) 2022-04-07 17:08:43 -06:00
Tanguy aa50127c14
Compiler dagger.nim in the CI (#71) 2022-04-06 14:00:25 +02:00
Dmitriy Ryajov ffa9b624f1
Integrate erasure (#73)
* wip: adding request for storage endpoint

* wire in erasure coding

* fix tests for erasure coding

* put type definitions into separate file

* integrate erasure coding

* change run/shutdown to start/stop

* temporary sleep, otherwise the fsstore blocks
2022-04-05 18:34:29 -06:00
Dmitriy Ryajov 49c41e27b7
Erasure coding dataset (#72)
* add erasure coding support

* add leopard support

* erasure coding tests
2022-04-05 13:12:59 -06:00
Dmitriy Ryajov 3222f436cc
Protected manifest (#69)
* add protected manifes type

* encode protected manifest

* add empty block type handling

* add tests for protected manifest
2022-04-04 18:46:13 -06:00
Mark Spanbroek befebcf325 [sales] Make storage available again when offer expires 2022-04-04 11:29:32 +02:00
Mark Spanbroek e46c9816fa [sales] Make storage available again when offer declined 2022-04-04 11:29:32 +02:00
Mark Spanbroek 11f7b4b4d8 [sales] Clean up tests 2022-04-04 11:29:32 +02:00
Mark Spanbroek 3810889fcb [sales] Do not call onSale() when someone else's offer is selected 2022-04-04 11:29:32 +02:00
Mark Spanbroek 71f25d40c7 [market] offerStorage() returns submitted offer 2022-04-04 11:29:32 +02:00
Mark Spanbroek 94e34e7d28 [market] requestStorage() returns submitted request 2022-04-04 11:29:32 +02:00
Mark Spanbroek 2ffde32772 [sales] call onSale() when offer has been selected 2022-04-04 11:29:32 +02:00
Mark Spanbroek 71baedb5ec [sales] Set offer expiry timestamp 2022-04-04 11:29:32 +02:00
Mark Spanbroek 76be728f61 [sales] Availability.new -> Availability.init 2022-04-04 11:29:32 +02:00
Mark Spanbroek 291c506e0b [market] Fix timing of test for slow CI 2022-04-04 11:29:32 +02:00
Mark Spanbroek dd89a55351 [market] Add Market.getTime() 2022-04-04 11:29:32 +02:00
Mark Spanbroek 061b32296a Introduce Sales 2022-04-04 11:29:32 +02:00
Mark Spanbroek 75ec8c0bfd [market] wait until request/offer expires 2022-04-04 10:03:46 +02:00
Mark Spanbroek 8ff748eff8 [market] select a storage offer 2022-04-04 10:03:46 +02:00
Mark Spanbroek 5ea64522d8 [market] subscribe to offers for a certain request 2022-04-04 10:03:46 +02:00
Mark Spanbroek 9f76e714a2 [market] submit storage offers 2022-04-04 10:03:46 +02:00
Mark Spanbroek e52153e2ba [market] Require a storage contract with a signer 2022-04-04 10:03:46 +02:00
Mark Spanbroek e78a0fb43e [contracts] Wrap contracts in Market abstraction 2022-04-04 10:03:46 +02:00
Mark Spanbroek bd8f4d5d74 [tests] Extract common examples into separate module 2022-04-04 10:03:46 +02:00
Mark Spanbroek 5b5f3335d6 [purchasing] Ignore offers that are about to expire 2022-04-04 10:03:46 +02:00
Mark Spanbroek dcfd6be1c6 [purchasing] Ignore expired offers 2022-04-04 10:03:46 +02:00
Mark Spanbroek fe23cb89d7 [purchasing] Select cheapest offer 2022-04-04 10:03:46 +02:00
Mark Spanbroek 04c291b05c [purchasing] rename requests -> requested 2022-04-04 10:03:46 +02:00
Mark Spanbroek 7c9c244836 [purchasing] Remove PurchaseRequest; use StorageRequest 2022-04-04 10:03:46 +02:00
Mark Spanbroek 9ade3fdd37 [contracts] StorageRequest/Offer is object instead of tuple 2022-04-04 10:03:46 +02:00
Mark Spanbroek 5f10549f19 Introduce Purchasing and Market 2022-04-04 10:03:46 +02:00
Mark Spanbroek a3fdd35f73 [contracts] Split marketplace.nim into requests.nim and offers.nim 2022-04-04 10:03:46 +02:00
Dmitriy Ryajov 43cea1743a
Node storestream (#62)
* use storestream for retrieval

* use `len` prop on manifest

* remove read pattern

* make blocksize and chunksize consistent

* fix tests

* fix chunker tests - now padded by default
2022-03-29 20:43:35 -06:00
Mark Spanbroek 42edbf5076 [contracts] Remove obsolete marketplace tests 2022-03-28 11:59:01 +02:00
Mark Spanbroek efb4f5c375 [contracts] Update to new marketplace design 2022-03-28 11:59:01 +02:00
Mark Spanbroek 87bd947d3d [contracts] Update to new collateral design 2022-03-28 11:59:01 +02:00
Mark Spanbroek d07a5cd19e [contracts] Run contract tests in CI
Introduce new nimble 'test' task that runs *all* tests.
Renames testAll to testDagger, because testAll didn't
run all tests, it forgot to run the contract tests.
2022-03-28 11:59:01 +02:00
Dmitriy Ryajov e965f5e0de
Seekable stream (#58)
* cleanup manifest

* adding a "seekable" base stream
2022-03-21 12:09:59 -06:00
Dmitriy Ryajov d204b518e8
Make block a ref (#56)
* cleanup manifest

* make block a ref type
2022-03-18 13:50:53 -06:00
Dmitriy Ryajov 472e1e6fc3
Store stream (#55)
* make manifest a ref object

* capitalize tests

* add storestream and tests

* fix tests

* run storestream tests
2022-03-17 07:56:46 -06:00
Dmitriy Ryajov f2aa945991
Manifest block size (#53)
* add `blockSize` and `size` to manifest

* fix tests

* fix decoding/encoding

* re-export coders

* style
2022-03-15 12:47:31 -06:00
Dmitriy Ryajov f414c695be
Cleanup-manifest (#52)
* adding nim-blscurve

* adding taskpools

* cleanup manifest
2022-03-14 10:06:36 -06:00
Eric Mastro 70cbdff901
feat: introduce LRU cache (#50)
* feat: introduce LRU cache

Replace `MemoryStore` with LRU caching mechanism.

`lrucache` library was forked to https://github.com/status-im/lrucache.nim.

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>

# Conflicts:
#	dagger/dagger.nim
#	dagger/stores.nim
#	dagger/stores/manager.nim
#	tests/dagger/blockexc/testengine.nim
#	tests/dagger/helpers/nodeutils.nim
#	tests/dagger/testnode.nim
#	tests/dagger/teststores.nim

* feat: introduce --cache-size CLI option

Allow for a value of `0` to disable the cache.

# Conflicts:
#	dagger/dagger.nim

* allow dynamic block size in cache

allow block size to be variable/dynamic

update lrucache to use updated lrucache dep

Using removeLru proc, added tests

* Refactor CacheStore init block

Co-authored-by: Michael Bradley, Jr <michaelsbradleyjr@gmail.com>
2022-03-02 10:30:42 -06:00
Eric Mastro 2e5c28781c feat: integrate dagger contracts
Integrate dagger contracts from `nim-dagger-contracts` repo.

Add `dagger-contracts`, `nim-web3`, and all of `nim-web3`’s transitive deps as submodule deps to `nim-dagger`. Note: `nim-web3` and its transitive deps may no longer be needed when we switch to `nim-ethers`.

Add a `testContracts` nimble task to test all of the contracts functionality. Namely, this spins up an ethereum simulator, deploys the contracts (in `dagger-contracts`), runs the contract tests, and finally, regardless of success/error, kills the ethereum sim processes. The nimble task can be run with `./env.sh nimble testContracts`.

We also tested `nim-dagger-contracts` as a submodule dep of `nim-dagger`, and while the tests run as expected, the preference is to merge `nim-dagger-contracts` inside of `nim-dagger` for ease of parallel development. There’s also a high probability that `nim-dagger-contracts` is not being used as a dep by other projects. Are there any strong objections to this?

Co-authored-by: Michael Bradley <michaelsbradleyjr@gmail.com>
2022-02-04 15:34:56 +11:00