388 Commits

Author SHA1 Message Date
Mark Spanbroek
a0f6ef54bb
[PoR] Test PoR network with a small block size
To get a reasonable test execution time.
2022-11-09 16:38:29 +01:00
Mark Spanbroek
72f67abfb1
[PoR] Test PoR store with a small block size
To get a reasonable test execution time.
2022-11-09 16:38:29 +01:00
Mark Spanbroek
51f25c7655
[erasure] fix typo 2022-11-09 16:38:28 +01:00
Mark Spanbroek
5c2041c0d7
[erasure] Test erasure coding with a small block size
To get a reasonable test execution time.
2022-11-09 16:38:28 +01:00
Mark Spanbroek
d5a8d110a5
[build] update to latest nimbus-build-system
Requires an update to latest nim-nat-traversal
2022-11-09 16:38:28 +01:00
Mark Spanbroek
cf6d5c1640
[ci] Set cache_nonce for all jobs, not just those in the matrix 2022-11-09 16:38:28 +01:00
Mark Spanbroek
b890b9d387
[ci] Update caches on every run
Will automatically update caches when a new minor version
of Nim is released.

See also:
https://github.com/actions/cache/blob/main/workarounds.md#update-a-cache
2022-11-09 16:38:28 +01:00
Mark Spanbroek
e19cc548c9
[ci] Checkout submodules recursively
So that nimbus-build-system doesn't have to
do it afterwards.
2022-11-09 16:38:28 +01:00
Mark Spanbroek
9df32625d9
[ci] Set NIM_COMMIT environment variable
To prevent subsequent `make` command to build a
different version of Nim.
2022-11-09 16:38:27 +01:00
Mark Spanbroek
67832a39e7
[PoR] Test proofs of retrievability with a small block size
To get a reasonable test execution time.
2022-11-09 16:38:27 +01:00
Mark Spanbroek
e31f405b90
[ci] Calculate coverage on unit tests only
This enables faster development cycles at
the expense of code coverage accuracy.
2022-11-09 10:35:32 +01:00
Mark Spanbroek
8962e14cea
[ci] Separate build job for Nim 1.2 2022-11-09 10:35:32 +01:00
Mark Spanbroek
c031fbd2a4
[ci] Separate build job for coverage that is allowed to fail 2022-11-09 10:35:32 +01:00
Mark Spanbroek
91e184aef8
[ci] Separate steps for unit, contract and integration tests 2022-11-09 10:35:32 +01:00
Mark Spanbroek
e9681fcaae
[ci] Use amd64 by default 2022-11-09 10:35:31 +01:00
Mark Spanbroek
8a98ea4a3d
[ci] Cleanup NodeJS setup 2022-11-09 10:35:05 +01:00
Mark Spanbroek
fa2fa3574f
[ci] Extract setup of nimbus build system into separate file 2022-11-08 16:10:34 +01:00
markspanbroek
4175689745
Load purchase state from chain (#283)
* [purchasing] Simplify test

* [utils] Move StorageRequest.example up one level

* [purchasing] Load purchases from market

* [purchasing] load purchase states

* Implement myRequest() and getState() methods for OnChainMarket

* [proofs] Fix intermittently failing tests

Ensures that examples of proofs in tests are never of length 0;
these are considered invalid proofs by the smart contract logic.

* [contracts] Fix failing test

With the new solidity contracts update, a contract can only
be paid out after it started.

* [market] Add method to get request end time

* [purchasing] wait until purchase is finished

Purchase.wait() would previously wait until purchase
was started, now we wait until it is finished.

* [purchasing] Handle 'finished' and 'failed' states

* [marketplace] move to failed state once request fails

- Add support for subscribing to request failure events.
- Add supporting contract tests for subscribing to request failure events.
- Allow the PurchaseStarted state to move to PurchaseFailure once a request failure event is emitted
- Add supporting tests for moving from PurchaseStarted to PurchaseFailure
- Add state transition tests for PurchaseUnknown.

* [marketplace] Fix test with longer sleepAsync

* [integration] Add function to restart a codex node

* [purchasing] Set client address before requesting storage

To prevent the purchase id (which equals the request id)
from changing once it's been submitted.

* [contracts] Fix: OnChainMarket.getState()

Had the wrong method signature before

* [purchasing] Load purchases on node start

* [purchasing] Rename state 'PurchaseError' to 'PurchaseErrored'

Allows for an exception type called 'PurchaseError'

* [purchasing] Load purchases in background

No longer calls market.getRequest() for every purchase
on node start.

* [contracts] Add `$` for RequestId, SlotId and Nonce

To aid with debugging

* [purchasing] Add Purchasing.stop()

To ensure that all contract interactions have both a
start() and a stop() for

* [tests] Remove sleepAsync where possible

Use `eventually` loop instead, to make sure that we're
not waiting unnecessarily.

* [integration] Fix: handle non-json response in test

* [purchasing] Add purchase state to json

* [integration] Ensure that purchase is submitted before restart

Fixes test failure on slower CI

* [purchasing] re-implement `description` as method

Allows description to be set in the same module where the
state type is defined.

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>

* [contracts] fix typo

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>

* [market] Use more generic error type

Should we decide to change the provider type later

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
2022-11-08 08:10:17 +01:00
markspanbroek
be32b9619b
Fixes loading of private key on Windows (#299)
Unix permissions don't work on Windows; 
adds check for correct ACL settings in Windows.
2022-11-07 15:54:24 +01:00
Michael Bradley
c893b1f0cb
track latest nim-libp2p's master branch (#248)
* [build] track nim-libp2p's unstable branch

* cid formatIt change

* track nim-libp2p-unstable

* don't stringify cid

* Fixed testblockexc.nim

1. Assign price to blocks
2. Delete on node1 cached blocks from node2 before buying them again

* add trace logging

* init pricing info

* remove duplicate price setting

* cid serialization in trace logs

* bumping dht to latest main

* bump

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Co-authored-by: Bulat-Ziganshin <bulat.ziganshin@gmail.com>
2022-11-02 11:40:28 -06:00
Dmitriy Ryajov
0ecbfcec9f
setup and persist private key (#292)
* setup and persist private key

* return dht record spr

* helper to remap multiaddr ip and port

* set/update discovery and announce addrs

* add nat and discovery IPs

* allow for announce and DHT addresses separatelly

* update tests

* check for nat or discoveryIp

* fix integration tests

* misc align

* don't share data dirs and and set bootstrap node

* add log scope

* remap announceAddrs after node start

* simplify discovery initialization

* make nat and disc-ip required

* add log scope don't init dht spr in constructor

* bump dht

* dissallow `0.0.0.0` for `--nat`
2022-11-01 18:58:41 -06:00
Dmitriy Ryajov
ae46f4dc2f
add support for logging topics (#293)
* add support for logging topics

* add top level codex topic
2022-11-01 14:05:40 -06:00
Dmitriy Ryajov
6e4a8b86ab
Rework discovery (#288)
* use multiaddrs instead of ip/port

* rework to support updating spr

* fix tests

* fix compilation

* use base for base methods
2022-10-27 07:44:56 -06:00
Dmitriy Ryajov
e50ea88411
Remove protobuf serialization (#289)
* add format for cid

* cid formatIt change

* track nim-libp2p-unstable

* rework probuf serialization for por

* add missing include

* removing nim protobuf serialization

* rollback to dht to main

* remove protobuf serialization import
2022-10-27 07:41:34 -06:00
Mark Spanbroek
92eecb0702 [purchasing] Rework purchasing as a state machine 2022-10-25 15:10:35 +11:00
Mark Spanbroek
7841a9692c [utils] Add AsyncState to state machine 2022-10-25 15:10:35 +11:00
Mark Spanbroek
b52d291785 [utils] Add state machine implementation 2022-10-25 15:10:35 +11:00
Mark Spanbroek
e648c26340 [utils] Add operator as for optional casting 2022-10-25 15:10:35 +11:00
Mark Spanbroek
bc3734b1a8 [purchasing] move Purchase and PurchaseId into separate modules 2022-10-25 15:10:35 +11:00
Eric Mastro
16024f21e9 bump nim-ethers (to main) 2022-10-25 15:10:35 +11:00
Eric Mastro
1f50612b68 bump dagger-contracts (to main) 2022-10-25 15:10:35 +11:00
Eric Mastro
25df599c08 Update to new reverts API, add tests
1. use `ProviderError` to catch contract errors instaed of `JsonRpcProviderError`.
2. Add tests for `isProofRequired`, `willProofBeRequired`, and `getProofEnd` when slot is empty (and is call is reverted)
3. bump nim-ethers to branch that has new `reverts` API and `ProviderError` support.
4. Update to new `reverts` API.
2022-10-25 15:10:35 +11:00
Eric Mastro
a9cea802ff bump dagger-contracts 2022-10-25 15:10:35 +11:00
Eric Mastro
a7277e90fb update market test 2022-10-25 15:10:35 +11:00
Eric Mastro
38643c9709 fix revert check on storage test 2022-10-25 15:10:35 +11:00
Eric Mastro
5eb65336fd catch and check revert reasons
Ensure specific contract reverts are handled and returned as booleans or default values

# Conflicts:
#	codex/contracts/proofs.nim
2022-10-25 15:10:35 +11:00
Eric Mastro
10d4dcd70d bump nim-ethers, fix tests
# Conflicts:
#	vendor/nim-ethers
2022-10-25 15:10:35 +11:00
Eric Mastro
dab65eb6bd catch contract errors to return bool
# Conflicts:
#	codex/contracts/proofs.nim
#	vendor/nim-ethers
2022-10-25 15:10:35 +11:00
Eric Mastro
ae9467b780 simplify getRequest 2022-10-25 15:10:35 +11:00
Eric Mastro
eb14ca6166 bump dagger-contracts
# Conflicts:
#	vendor/dagger-contracts
2022-10-25 15:10:35 +11:00
Eric Mastro
b9e53e182f [marketplace] support freeing of slots and failed state
Support storage contract parameters to free slots after too many proofs missed, as well as fail a contract after too many slots have been freed.

# Conflicts:
#	vendor/dagger-contracts
2022-10-25 15:10:35 +11:00
Eric Mastro
8dd110765f [marketplace] Catch slot empty, return default values 2022-10-25 15:10:35 +11:00
Eric Mastro
9d218c88a5 [marketplace] Remove isSlotCancelled and isCancelled
Remove `isSlotCancelled` and `isCancelled` from proving, as it did not fit in the module.

Update the proving module to not rely on checking the contract to understand if a request was cancelled. Instead, `proofEnd` was modified in `dagger-contracts` such that it returns a past timestamp when the contract is cancelled. This successfully removes
2022-10-25 15:10:35 +11:00
Eric Mastro
b4a14e00f7 [marketplace] address more PR comments
- remove RequestState as no longer being used
- remove RequestState check on purchase timeout
- add tests for `withdrawFunds` and `subscribeRequestCancelled`
- update `dagger-contracts` so that `RequestCancelled.requestId` property is indexed in the event
2022-10-25 15:10:35 +11:00
Eric Mastro
2db0131fa4 [fix] fix tests 2022-10-25 15:10:35 +11:00
Eric Mastro
5cb74b6638 [chore] additional clean up of array[32, byte] types
- rename `ContractId` to `SlotId`
- rename `Proving.contracts` to `Proving.slots`
- change signatures of `isSlotCancelled` and `isCancelled` to use `SlotId` and `RequestId` types, respectively.
- change all references to `RequestId`, `SlotId`
2022-10-25 15:10:35 +11:00
Eric Mastro
1039e27524 [marketplace] bump dagger contracts after rebase 2022-10-25 15:10:35 +11:00
Eric Mastro
372f827982 [marketplace] add/remove proofs for contract state
Add or remove proof requirements when a request contract’s state changes. When a request sale has completed (for a slot), the host who purchased that slot now must provide regular proofs for the data they are contracted to hold. This is now enforced by adding the slotId to the HashSet of Ids for which to require proofs. When a request has been cancelled (not all slots were filled before the request expired), proofs no longer need to be provided and the slotId is removed from teh HashSet.

Add `isCancelled` and `isSlotCancelled` checks to query the contract state without relying the on the state context variable in the contract. Because contract state can only be updated in a transaction, and the client withdrawing funds is responsible for changing the contract state to “Cancelled”, the `isCancelled` and `isSlotCancelled` functions were introduced to check the state regardless of whether or not the client had already withdrawn their funds.
2022-10-25 15:10:35 +11:00
Eric Mastro
0c3fbad470 [purchasing] Withdraw funds when request times out
When a request for storage times out (not enough slots filled), the client will initiate a withdraw request to retrieve its funds out of the contract, setting the state of the request to RequestState.Cancelled. The client will also emit a RequestCancelled event for others to listen to (ie hosts will need to listen for this event to withdraw its collateral).

Add unit test that checks for emission of RequestCancelled after request is purchased request expires.

Update dagger-contracts dependency to commit that holds the changes supporting withdrawing of funds.
2022-10-25 15:10:35 +11:00
Bulat-Ziganshin
9939d85b74
Replace protobuf serialization for Block exchange with minprotobuf (#271)
* minprotobuf serialization for Block exchange

* Handle decoding errors by stopping peer connection
2022-10-13 18:58:57 -05:00