Commit Graph

459 Commits

Author SHA1 Message Date
Eric Mastro 16024f21e9 bump nim-ethers (to main) 2022-10-25 15:10:35 +11:00
Eric Mastro 1f50612b68 bump dagger-contracts (to main) 2022-10-25 15:10:35 +11:00
Eric Mastro 25df599c08 Update to new reverts API, add tests
1. use `ProviderError` to catch contract errors instaed of `JsonRpcProviderError`.
2. Add tests for `isProofRequired`, `willProofBeRequired`, and `getProofEnd` when slot is empty (and is call is reverted)
3. bump nim-ethers to branch that has new `reverts` API and `ProviderError` support.
4. Update to new `reverts` API.
2022-10-25 15:10:35 +11:00
Eric Mastro a9cea802ff bump dagger-contracts 2022-10-25 15:10:35 +11:00
Eric Mastro a7277e90fb update market test 2022-10-25 15:10:35 +11:00
Eric Mastro 38643c9709 fix revert check on storage test 2022-10-25 15:10:35 +11:00
Eric Mastro 5eb65336fd catch and check revert reasons
Ensure specific contract reverts are handled and returned as booleans or default values

# Conflicts:
#	codex/contracts/proofs.nim
2022-10-25 15:10:35 +11:00
Eric Mastro 10d4dcd70d bump nim-ethers, fix tests
# Conflicts:
#	vendor/nim-ethers
2022-10-25 15:10:35 +11:00
Eric Mastro dab65eb6bd catch contract errors to return bool
# Conflicts:
#	codex/contracts/proofs.nim
#	vendor/nim-ethers
2022-10-25 15:10:35 +11:00
Eric Mastro ae9467b780 simplify getRequest 2022-10-25 15:10:35 +11:00
Eric Mastro eb14ca6166 bump dagger-contracts
# Conflicts:
#	vendor/dagger-contracts
2022-10-25 15:10:35 +11:00
Eric Mastro b9e53e182f [marketplace] support freeing of slots and failed state
Support storage contract parameters to free slots after too many proofs missed, as well as fail a contract after too many slots have been freed.

# Conflicts:
#	vendor/dagger-contracts
2022-10-25 15:10:35 +11:00
Eric Mastro 8dd110765f [marketplace] Catch slot empty, return default values 2022-10-25 15:10:35 +11:00
Eric Mastro 9d218c88a5 [marketplace] Remove `isSlotCancelled` and `isCancelled`
Remove `isSlotCancelled` and `isCancelled` from proving, as it did not fit in the module.

Update the proving module to not rely on checking the contract to understand if a request was cancelled. Instead, `proofEnd` was modified in `dagger-contracts` such that it returns a past timestamp when the contract is cancelled. This successfully removes
2022-10-25 15:10:35 +11:00
Eric Mastro b4a14e00f7 [marketplace] address more PR comments
- remove RequestState as no longer being used
- remove RequestState check on purchase timeout
- add tests for `withdrawFunds` and `subscribeRequestCancelled`
- update `dagger-contracts` so that `RequestCancelled.requestId` property is indexed in the event
2022-10-25 15:10:35 +11:00
Eric Mastro 2db0131fa4 [fix] fix tests 2022-10-25 15:10:35 +11:00
Eric Mastro 5cb74b6638 [chore] additional clean up of `array[32, byte]` types
- rename `ContractId` to `SlotId`
- rename `Proving.contracts` to `Proving.slots`
- change signatures of `isSlotCancelled` and `isCancelled` to use `SlotId` and `RequestId` types, respectively.
- change all references to `RequestId`, `SlotId`
2022-10-25 15:10:35 +11:00
Eric Mastro 1039e27524 [marketplace] bump dagger contracts after rebase 2022-10-25 15:10:35 +11:00
Eric Mastro 372f827982 [marketplace] add/remove proofs for contract state
Add or remove proof requirements when a request contract’s state changes. When a request sale has completed (for a slot), the host who purchased that slot now must provide regular proofs for the data they are contracted to hold. This is now enforced by adding the slotId to the HashSet of Ids for which to require proofs. When a request has been cancelled (not all slots were filled before the request expired), proofs no longer need to be provided and the slotId is removed from teh HashSet.

Add `isCancelled` and `isSlotCancelled` checks to query the contract state without relying the on the state context variable in the contract. Because contract state can only be updated in a transaction, and the client withdrawing funds is responsible for changing the contract state to “Cancelled”, the `isCancelled` and `isSlotCancelled` functions were introduced to check the state regardless of whether or not the client had already withdrawn their funds.
2022-10-25 15:10:35 +11:00
Eric Mastro 0c3fbad470 [purchasing] Withdraw funds when request times out
When a request for storage times out (not enough slots filled), the client will initiate a withdraw request to retrieve its funds out of the contract, setting the state of the request to RequestState.Cancelled. The client will also emit a RequestCancelled event for others to listen to (ie hosts will need to listen for this event to withdraw its collateral).

Add unit test that checks for emission of RequestCancelled after request is purchased request expires.

Update dagger-contracts dependency to commit that holds the changes supporting withdrawing of funds.
2022-10-25 15:10:35 +11:00
Bulat-Ziganshin 9939d85b74
Replace protobuf serialization for Block exchange with minprotobuf (#271)
* minprotobuf serialization for Block exchange

* Handle decoding errors by stopping peer connection
2022-10-13 18:58:57 -05:00
Bulat-Ziganshin 6e6f40016c
Changes required for reasonable upload/download speed (#265)
Increase blocksize from ~8KiB to ~64KiB and remove useless unbounded prefetching on download
* increased upload/download speed on my Win10 Zen3 box to 50/150 MB/s (with FSStore)
* made manifest files 8x smaller
* two more changes in the tests to make them pass
Closes #263
2022-10-06 23:01:25 +03:00
Dmitriy Ryajov 106b04340a
Remove local providers (#268)
* add remove provider method

* remove failed providers

* bumping dht to latest master
2022-10-05 10:01:21 -06:00
Mark Spanbroek 58f0439f37 [contracts] Use Duration instead of float for clock offset
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
2022-10-03 15:17:03 +02:00
Mark Spanbroek 4e6d84850a [contracts] Fix intermittently failing clock test
Use a sub-second offset to avoid off-by-one errors in the
test at the second boundary.

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
2022-10-03 15:17:03 +02:00
Dmitriy Ryajov 8d315b38b8
Dht maintenance (#267)
* protocol start is now async

* bumping to maintenance branch
2022-10-01 11:08:44 -06:00
Dmitriy Ryajov f190bfb96e
bump datastore (#266)
* updating datastore to batched version

* bumping datastore
2022-09-30 17:57:57 -06:00
Dmitriy Ryajov 69bd359287
Persist Dht providers (#257)
* providers store integration
2022-09-29 20:16:59 -06:00
Mark Spanbroek 90c818bdda [build] update questionable to 0.10.6
Includes fix for double evaluations of options and results
2022-09-29 10:16:41 +02:00
Eric Mastro 28c73e18a0 bump bearssl to latest 2022-09-27 12:56:07 +10:00
Mark Spanbroek 14cfe05ef6 Update questionable to 0.10.5 2022-09-19 15:51:03 +02:00
Dmitriy Ryajov 7e2f917c84
Dht providers lru (#247)
* upgrading nimbus build system to latest

* use new `updateRecord`

* bump nim-libp2p-dht

* bump nimbus build system

* bumping deps

* update to latest dht
2022-09-13 13:37:49 -06:00
Bulat-Ziganshin df1522c9f8
Fixed message.proto to match IPFS specs (#245)
We incorrectly assigned code 2 to the payload field instead of code 3:

Our code: 8c59392526/codex/blockexchange/protobuf/message.proto (L50-L52)

While original code: 0fa397581c/message/pb/message.proto (L43-L45)

And [documentation](https://github.com/ipfs/specs/blob/main/BITSWAP.md#bitswap-120-wire-format).
2022-09-12 21:32:12 -06:00
Michael Bradley 6b2adb7017
Fix vendor/nim-eth (#246)
* [build] bump vendor/nim-eth and drop vendor/vendor/nim-eth

* [build] bump vendor/nim-bearssl and drop vendor/bearssl
2022-09-12 21:30:16 -06:00
Dmitriy Ryajov bb6e812923
Update curl upload command (#240) 2022-09-12 11:03:27 -06:00
markspanbroek 5ec2e40a73
[build] Update asynctest to version 0.3.2 (#244)
Includes a fix for Nil exception on unthandled exception
2022-09-12 09:59:39 -06:00
Eric Mastro 014207c87f
bump nim-ethers to 0.2.2, add nim-eth dependency (#236)
Support latest version of nim-ethers which relies on nim-eth.

NOTE: This commit points to PR #31 which is a branch of nim-ethers. That branch should probably be merged first, and this PR updated to point to the commit on main.
2022-09-08 16:56:41 -06:00
Dmitriy Ryajov fd56875667
updating bug report template environment section 2022-09-08 12:35:07 -06:00
Dmitriy Ryajov d074d3b351
Adding bug report template 2022-09-08 12:32:32 -06:00
Michael Bradley, Jr ba42533ad6 [build] pass NIM_PARAMS to nim compiler in exec target of Makefile
with this change `make exec` runs a release build instead of a debug build
2022-09-01 20:44:29 -05:00
Eric Mastro b843118076 bump nim-ethers to use better compile time check 2022-08-26 13:29:09 +10:00
Eric Mastro 58f3335cc1 [fix] Fix C-compilation error failing tests 2022-08-26 13:29:09 +10:00
Eric Mastro 01419d3fc3 [chore] bump nim-ethers for distinct indexed event field support 2022-08-26 13:29:09 +10:00
Eric Mastro 8f116cd01e [chore] make RequestId, SlotId, Nonce, PurcahseId distinct
Make RequestId, SlotId, Nonce, PurcahseId distinct types.

Add/modify conversions to support the distinct type (ABI encoding/decoding, JSON encoding, REST decoding).

Update tests
2022-08-26 13:29:09 +10:00
Eric Mastro 6df5a7cf54 [chore] clean up `array[32, byte]` types
- rename `ContractId` to `SlotId`
- add `RequestId`, `PurchaseId`, `Nonce` types as aliases of `array[32, byte]`
- rename `Proving.contracts` to `Proving.slots`
- change signatures of `isSlotCancelled` and `isCancelled` to use `SlotId` and `RequestId` types, respectively.
- change all references to `RequestId`, `SlotId`, and `PurchaseId`
2022-08-26 13:29:09 +10:00
Michael Bradley, Jr 4258734a81 [docs] add BUILDING.md and link from root README.md
and make a few other touch ups in BUILDING.md and README.md, e.g. updating the
CLI options section in README.md

closes #216
2022-08-24 17:14:22 -05:00
Bulat-Ziganshin f24ded0f76
Download files without padding (#218)
The initial goal of this patch was to allow to download of a file via REST API in exactly the same size as it was uploaded, which required adding fields Chunker.offset and Manifest.originalBytes to keep that size. On top of that, we added more integrity checks to operations on Manifest, and reorganized TestNode.nim to test the actual interaction between node.store and node.retrieve operations.

Note that the wire format of Manifest was changed, so we need to recreate all BlockStores.

* Download without padding
* Fixed chunker tests
* Chunker: get rid of RabinChunker
* Verify offset in the chunker tests
* Use manifest.originalBytesPadded in StoreStream.size
* StoreStream: replace emptyBlock with zeroMem
* Manifest.bytes: compute how many bytes corresponding StoreStream(Manifest, pad) will return
* Manifest: verify originalBytes and originalLen on new/encode/decode
Also set originalBytes in each Manifest creation/update scenario
* Manifest: comments, split code into sections
* Reordered parameters to deal with int64 size in 32-bit builds
* TestNode.nim: combine Store and Retrieve tests
1. Instead of copy-pasting code from node.nim, new test calls node.store() and node.retrieve() in order to check that they can correctly store and then retrieve data
2. New test compares only file contents, manifest contents considered an implementation detail
3. New test chunks at odd chunkSize=BlockSize/1.618 in order to ensure that data retrieved correctly even when buffer sizes mismatch
* TestNode.nim: code refactoring
* Manifest.add: one more test
* Manifest.verify: return Result instead of raising Defect
* Node.store: added blockSize parameter
2022-08-24 15:15:59 +03:00
Dmitriy Ryajov 4bc701652f
Adding metrics (#203)
* adding inflight discovery metric

* adding metrics instructions and default dashboard

* spelling fixes
2022-08-23 10:11:21 -06:00
Michael Bradley, Jr 3d823dcbc6 [stores] update getBlock return type
change return type for `method getBlock` from `Future[?!(?Block)]` to
`Future[?!Block]`

use `type BlockNotFoundError = object of CodexError` to differentiate between
"block not found in the store" and other errors

also make some logic and error handling/messages more consistent across
BlockStore implementations

closes #177
closes #182
closes #210

alternative to #205, #209
2022-08-22 20:52:51 -05:00
Michael Bradley, Jr e0e2d7b583 [build] bump questionable 2022-08-22 20:52:51 -05:00