Commit Graph

224 Commits

Author SHA1 Message Date
Mark Spanbroek adcb91a5d5 [node] Store datasets locally when sales asks for it 2022-07-20 13:43:20 +02:00
Mark Spanbroek 96ca1de768 Cleanup: remove StorageOffer everywhere 2022-07-20 13:43:20 +02:00
Mark Spanbroek bd2fba50c7 [purchasing] Fix integration test 2022-07-20 13:43:20 +02:00
Mark Spanbroek f14eeb688b [WIP sales] Handle request expiry 2022-07-20 13:43:20 +02:00
Mark Spanbroek 9438aba5d2 [WIP sales] Handle other host fulfilling request 2022-07-20 13:43:20 +02:00
Mark Spanbroek 7f864570bd [WIP sales] call onSale() 2022-07-20 13:43:20 +02:00
Mark Spanbroek a2570f9380 [WIP sales] fulfill request 2022-07-20 13:43:20 +02:00
Mark Spanbroek 5abd5dd674 [WIP sales] make storage available again upon failure 2022-07-20 13:43:20 +02:00
Mark Spanbroek a7f3ead44e [WIP sales] refactor tests 2022-07-20 13:43:20 +02:00
Mark Spanbroek dcb8f8a7f7 [WIP sales] Generate proof of storage 2022-07-20 13:43:20 +02:00
Mark Spanbroek 07dbb6ae18 [WIP sales] Retrieve data 2022-07-20 13:43:20 +02:00
Mark Spanbroek 3e6d51754d [WIP sales] Make tests pass by commenting out code 2022-07-20 13:43:20 +02:00
Mark Spanbroek 0636c94b74 [market] Add market.getSigner() 2022-07-20 13:43:20 +02:00
Mark Spanbroek 03b92a2067 [market] Add market.getHost() 2022-07-20 13:43:20 +02:00
Mark Spanbroek b414ecd67e [market] Add market.getRequest() 2022-07-20 13:43:20 +02:00
Mark Spanbroek 4b5cfe8e41 [purchasing] Update to latest dagger-contracts 2022-07-20 13:43:20 +02:00
Mark Spanbroek b7ab9481d9 [market] Update to latest dagger-contracts 2022-07-20 13:43:20 +02:00
Mark Spanbroek e352a181e4 [contracts] Update to latest dagger-contracts 2022-07-20 13:43:20 +02:00
Michael Bradley, Jr 8f0bc54fba test: make `proc finish` async in "nodes negotiate ..." integration test
`waitFor` has been moved into the main body of integration test
`nodes negotiate ...` to reduce blockage on the main thread. That change, along
with widening the expiry in `when defined(windows)` (and bumping up
`timeout-minutes` in our CI workflow) allows Windows CI builds to succeed more
consistently, i.e. without timing out.
2022-07-06 19:03:10 -05:00
Bulat-Ziganshin 15ae157e61
BlockStore: delBlock got new type (#123)
Now every store traces only its own operations and failures (currently implemented only for delBlock)
2022-06-28 19:10:05 +03:00
Dmitriy Ryajov 77733725bf
fix failing test after nim-ethers bump (#112) 2022-06-09 09:36:59 -06:00
Dmitriy Ryajov 6ce7e23767
Upload authenticators (#108)
* initial implementation of storage proofs upload

* make sure proof verifies with after deserializing

* add por store

* rename por store to stp store

* rename porstore to stpstore

* add support for host discovery to discovery mock

* add tags upload network tests
2022-05-25 20:29:31 -06:00
Dmitriy Ryajov 738738c3c6
Adding storage proofs store (#109)
* add por store

* rename por store to stp store

* add stp tests
2022-05-24 18:42:49 -06:00
Dmitriy Ryajov 56b80d6f6d
Por serialize (#106)
* move por into storage proofs

* use SeekableStream

* adding serialization for por

* remove leftovers

* add empty block support

* add basic por test

* rename block exchange for consistency

* add storageproofstests

* moving timing to storageproofs

* fix imports

* fix imports

* fix imports

* add top level exports

* move delete blocks helper to helpers

* more import/export fixes

* cleanup

* more import fixes

* fix unused warnings

* detect corrupt blocks tests

* add serialization tests

* move init method around

* bump asynctest

* fix CID version

* get rid of warning

* wip: fix CI

* increase CI timeout
2022-05-23 23:24:15 -06:00
Dmitriy Ryajov d4f3ebc867
reorg engine directory structure (#104) 2022-05-19 16:28:53 -06:00
Michael Bradley 209343087c
Change every dagger to codex (#102) 2022-05-19 13:56:03 -06:00
Mark Spanbroek 15a571387f [test] Give integration test a bit more time to complete
For slow windows CI runners.
2022-05-19 10:38:19 +02:00
Mark Spanbroek 49e60d6a5c [clock] Fix test failure when sleepAsync takes a bit longer than expected 2022-05-19 10:38:19 +02:00
Mark Spanbroek d6393de324 [test] Add integration test for sales and purchasing 2022-05-19 10:38:19 +02:00
Mark Spanbroek 8086dda3f2 [purchasing] set selected offer on purchase 2022-05-19 10:38:19 +02:00
Mark Spanbroek cdcab43590 [proving] Use clock instead of getPeriod() and waitUntilPeriod() 2022-05-19 10:38:19 +02:00
Mark Spanbroek 09a7aa3eed [clock] Replace Market.waitUntil() with Clock.waitUntil() 2022-05-19 10:38:19 +02:00
Mark Spanbroek dcf3d485c4 [tests] Ensure that tests don't hang on closing process 2022-05-19 10:38:19 +02:00
Mark Spanbroek 3a8ce485e4 [sales] Use Clock instead of getTime() 2022-05-19 10:38:19 +02:00
Mark Spanbroek 1e44ed5fd3 [purchasing] Use Clock instead of getTime() 2022-05-19 10:38:19 +02:00
Mark Spanbroek 4210cca6e7 [proving] Use Clock instead of getTime() 2022-05-19 10:38:19 +02:00
Mark Spanbroek 8d19476a4c [sales] Ensure that Sales is completely stopped 2022-05-19 10:38:19 +02:00
Mark Spanbroek 3f627790f8 [proving] Ensure that proving is completely stopped 2022-05-19 10:38:19 +02:00
Mark Spanbroek ea72d99798 [clock] Add Clock interface that abstracts away ethereum details 2022-05-19 10:38:19 +02:00
Mark Spanbroek fa84861d86 [contracts] Add Clock that synchronizes time with EVM 2022-05-19 10:38:19 +02:00
Mark Spanbroek 43c0a48245 [tests] move node management into separate module
And add a debug option that writes the output of
the node during integration tests to stdout.
2022-05-19 10:38:19 +02:00
Mark Spanbroek 7f9fa0a183 [tests] move ethertest module up one level
It's going to be used by both contract and integration tests.
2022-05-19 10:38:19 +02:00
Mark Spanbroek b31ef8356b [purchasing] retrieve purchase details through REST API 2022-05-19 10:38:19 +02:00
Mark Spanbroek fb56f60d77 [purchasing] keep track of all purchases 2022-05-19 10:38:19 +02:00
Mark Spanbroek 7d03e83bfc [rest api] submit storage request parameters as json 2022-05-19 10:38:19 +02:00
Mark Spanbroek fec9ad8141 [erasure] remove duplication in tests 2022-05-19 10:38:19 +02:00
Mark Spanbroek 4d00814f9e [erasure] Handle edge case of 0 parity blocks 2022-05-19 10:38:19 +02:00
Mark Spanbroek 11581fd12c [sales] REST endpoint that lists available storage 2022-05-19 09:36:47 +02:00
Mark Spanbroek d564cc2375 [sales] Use UInt256 for size and duration
Use the same data types for availability and requests
for simplicity.
2022-05-19 09:36:47 +02:00
Mark Spanbroek ef80871984 [sales] Perform POST with JSON body instead of URL params 2022-05-19 09:36:47 +02:00
Mark Spanbroek 45c55ab2a8 [sales] Test sales rest api 2022-05-19 09:36:47 +02:00
Mark Spanbroek cc9a4e2af6 Add integration tests 2022-05-19 09:36:47 +02:00
Dmitriy Ryajov d3dbbc75fa
Extract Discovery engine (#99)
* don't force logging syncs

* Add failing test

* wip discovery engine

* re-add chronicles sinks

* wip

* move network related stuff to own folder

* move peer related stuff to own folder

* extract discovery into it's own engine

* update imports

* move pending blocks into engine module

* add top level exports

* update imports

* update import paths

* update imports

* support for inflight request filtering and tests

* use `remove` instead of `del`

* fix sorting in `selectCheapest`

* re-org test file structure

* fix to use discovery engine

* file re-org

* fix compilation

* fixup discovery to use async handlers

* more re-org

* rework with support for discovery engine

* add logging

* use defaults

* wip: reworking with discoveryengine

* wip: more test fixes

* more logging

* use ordered table

* use `bt` for blocktype Block

* fix tests

* make tests work with discovery engine

* expose all node components

* fix to work with discovery engine

* wip

* propagate cancellation in listBlocks

* start/stop disc engine in blockexc engine

* remove disc engine start/stop

* wire up discovery engine

* misc comments and imports

* pass discovery to dagger node

* set sleep timers

* unused imports

* misc

* don't spawn a task, await it

* don't await handlers

* trace logging

* reduce default sleep time

Co-authored-by: Tanguy <tanguy@status.im>
2022-05-18 20:29:15 -06:00
Dmitriy Ryajov d669e344bc
Engine (#89)
* rework discovery with async queues

* increase max message size for large manifests

* increase sleep time to 100 millis

* pass config

* check for nil on start/stop

* fix tests and split out discovery tests

* don't auto mount network

* add discovery tests

* rework moc discovery

* move discovery moc to disc dir

* don't force logging syncs

* don't force moc discovery on all tests

* rework discovery with methods

* add top level utils file

* don't use asyncCheck

* don't pass entire blocks to list blocks calback

* spelling

* - don't send want reqs to peers reporting the cid

- Don't request blocks directly on presense update, use `requestBlock`

* bug, nodes should not have blocks in local store

* Add failing test

* prefetch blocks so that download isn't serial

* if request already pending, return the handle

* fire discovery if no peers report block as have

* only query discovery if not enough nodes for cid

* wrap async req in template

* use non awaiting version of queue routines

* rework E2E tests as unittest

* re-add chronicles sinks

Co-authored-by: Tanguy <tanguy@status.im>
2022-05-12 15:52:03 -06:00
Dmitriy Ryajov 46197f957b
add onblock callback to list blocks visitor (#96)
* add onblock callback to list blocks visitor

* tests for listBlocks

* remove legacy blockList
2022-05-12 14:02:30 -06:00
markspanbroek d8ed4257df
[contracts] Make contract interactions optional in the node (#91)
Adds --eth-deployment parameter.
Does not crash when deployment json is missing or incorrect.
2022-04-25 15:12:37 +02:00
Mark Spanbroek c93e015e43 Disable logging in tests
Logger output interferes with the test runner output.
2022-04-21 14:04:07 +02:00
Mark Spanbroek 59f75fc540 [contracts] Start contract interactions when node starts 2022-04-21 14:04:07 +02:00
Mark Spanbroek 142ddd0fd3 [contracts] Use command line parameters for ContractInteractions 2022-04-21 14:04:07 +02:00
Mark Spanbroek 0738606303 [contracts] Add ContractInteractions to dagger node 2022-04-21 14:04:07 +02:00
Mark Spanbroek 9cbf6d0b6c [contracts] ContractInteractions wraps purchasing, sales and proving 2022-04-21 14:04:07 +02:00
Tanguy ce59dbd4a2
Revert "Cleanup engine and rework discovery (#87)"
This reverts commit 4740ffc144.
2022-04-20 14:28:11 +02:00
Dmitriy Ryajov 4740ffc144
Cleanup engine and rework discovery (#87)
* rework discovery with async queues

* misc style changes

* increase max message size for large manifests

* use upraises and avoid exceptions on key access

* increase sleep time to 100 millis

* pass config

* make list blocks trigger a callback on each block

* check for nil on start/stop

* fix tests and split out discovery tests

* don't auto mount network

* add list block tests

* add discovery tests

* rework moc discovery

* move discovery moc to disc dir

* don't force logging syncs

* don't force moc discovery on all tests

* rework discovery with methods

* add top level utils file

* don't use asyncCheck

* don't pass entire blocks to list blocks calback

* spelling
2022-04-19 21:46:44 -06:00
markspanbroek b88561e090
Subscribe to proof submissions (#83)
* Update dagger-contracts

* [proving] rename ProofTiming -> Proofs

* Update nim-ethers to 0.1.4

* [proving] Subscribe to proof submissions

* [proving] support proof submission through the Proving abstraction
2022-04-13 10:41:48 -06:00
Tanguy 4d681102e5
Add DHT (#75)
* First implem

* Add persistent net key option

* Working DHT setup

* Bootstrap nodes

* Implement DaggerNode.findPeer

* Remove unrelevant comment

* Added discovery to blockexchange requestBlock

* add FSStore.blockList

* Block advertisement

* Tests compiles

* Green tests

* toDiscoveryId instead of toNodeId

* remove stopAdvertisingBlock

* Removed nim-eth dependency

* Move discovery stuff to discovery.nim

* Add missing file, start of discovery tests

* Better discovery logic

* Add tests

* Address comment

* Better E2E test
2022-04-13 10:32:35 -06:00
Zahary Karadjov b658192540
Nimble lock file 2022-04-12 16:59:00 +03:00
markspanbroek 1a3003e043
Update to latest dagger-contracts (#78)
- StorageRequest struct has changed

- StorageRequested event has changed,
  it no longer returns the entire request,
  but only the id and the ask
2022-04-11 12:03:55 -06:00
markspanbroek 03fa370624
Proving (#66)
* Add Proving object, which maintains contract id's to watch

* [proving] invoke callback when proof is required

# Conflicts:
#	dagger/por/timing/periods.nim
#	dagger/por/timing/prooftiming.nim

* [proving] check proof requirements for all our contracts

# Conflicts:
#	tests/dagger/helpers/mockprooftiming.nim

* Update vendor/dagger-contracts

* [proving] call onProofRequired() when proof is required soon

* [proving] stop checking contracts that have ended

* [proving] Remove duplicated funcs

* [proving] Implement ProofTiming on top of smart contract

* [proving] Fix race condition in waitUntilNextPeriod()

Sometimes waitUntilNextPeriod would take a while to
determine the current period, leading to unexpected results.

Splits waitUntilNextPeriod() into getCurrentPeriod()
and waitUntilPeriod(), to ensure that we're really waiting
for the period that we think we're waiting for.
2022-04-08 15:58:16 -06:00
Dmitriy Ryajov 2e8b39cf7c
fix random failing test (#77) 2022-04-07 17:08:43 -06:00
Tanguy aa50127c14
Compiler dagger.nim in the CI (#71) 2022-04-06 14:00:25 +02:00
Dmitriy Ryajov ffa9b624f1
Integrate erasure (#73)
* wip: adding request for storage endpoint

* wire in erasure coding

* fix tests for erasure coding

* put type definitions into separate file

* integrate erasure coding

* change run/shutdown to start/stop

* temporary sleep, otherwise the fsstore blocks
2022-04-05 18:34:29 -06:00
Dmitriy Ryajov 49c41e27b7
Erasure coding dataset (#72)
* add erasure coding support

* add leopard support

* erasure coding tests
2022-04-05 13:12:59 -06:00
Dmitriy Ryajov 3222f436cc
Protected manifest (#69)
* add protected manifes type

* encode protected manifest

* add empty block type handling

* add tests for protected manifest
2022-04-04 18:46:13 -06:00
Mark Spanbroek befebcf325 [sales] Make storage available again when offer expires 2022-04-04 11:29:32 +02:00
Mark Spanbroek e46c9816fa [sales] Make storage available again when offer declined 2022-04-04 11:29:32 +02:00
Mark Spanbroek 11f7b4b4d8 [sales] Clean up tests 2022-04-04 11:29:32 +02:00
Mark Spanbroek 3810889fcb [sales] Do not call onSale() when someone else's offer is selected 2022-04-04 11:29:32 +02:00
Mark Spanbroek 71f25d40c7 [market] offerStorage() returns submitted offer 2022-04-04 11:29:32 +02:00
Mark Spanbroek 94e34e7d28 [market] requestStorage() returns submitted request 2022-04-04 11:29:32 +02:00
Mark Spanbroek 2ffde32772 [sales] call onSale() when offer has been selected 2022-04-04 11:29:32 +02:00
Mark Spanbroek 71baedb5ec [sales] Set offer expiry timestamp 2022-04-04 11:29:32 +02:00
Mark Spanbroek 76be728f61 [sales] Availability.new -> Availability.init 2022-04-04 11:29:32 +02:00
Mark Spanbroek 291c506e0b [market] Fix timing of test for slow CI 2022-04-04 11:29:32 +02:00
Mark Spanbroek dd89a55351 [market] Add Market.getTime() 2022-04-04 11:29:32 +02:00
Mark Spanbroek 061b32296a Introduce Sales 2022-04-04 11:29:32 +02:00
Mark Spanbroek 75ec8c0bfd [market] wait until request/offer expires 2022-04-04 10:03:46 +02:00
Mark Spanbroek 8ff748eff8 [market] select a storage offer 2022-04-04 10:03:46 +02:00
Mark Spanbroek 5ea64522d8 [market] subscribe to offers for a certain request 2022-04-04 10:03:46 +02:00
Mark Spanbroek 9f76e714a2 [market] submit storage offers 2022-04-04 10:03:46 +02:00
Mark Spanbroek e52153e2ba [market] Require a storage contract with a signer 2022-04-04 10:03:46 +02:00
Mark Spanbroek e78a0fb43e [contracts] Wrap contracts in Market abstraction 2022-04-04 10:03:46 +02:00
Mark Spanbroek bd8f4d5d74 [tests] Extract common examples into separate module 2022-04-04 10:03:46 +02:00
Mark Spanbroek 5b5f3335d6 [purchasing] Ignore offers that are about to expire 2022-04-04 10:03:46 +02:00
Mark Spanbroek dcfd6be1c6 [purchasing] Ignore expired offers 2022-04-04 10:03:46 +02:00
Mark Spanbroek fe23cb89d7 [purchasing] Select cheapest offer 2022-04-04 10:03:46 +02:00
Mark Spanbroek 04c291b05c [purchasing] rename requests -> requested 2022-04-04 10:03:46 +02:00
Mark Spanbroek 7c9c244836 [purchasing] Remove PurchaseRequest; use StorageRequest 2022-04-04 10:03:46 +02:00
Mark Spanbroek 9ade3fdd37 [contracts] StorageRequest/Offer is object instead of tuple 2022-04-04 10:03:46 +02:00
Mark Spanbroek 5f10549f19 Introduce Purchasing and Market 2022-04-04 10:03:46 +02:00
Mark Spanbroek a3fdd35f73 [contracts] Split marketplace.nim into requests.nim and offers.nim 2022-04-04 10:03:46 +02:00