Commit Graph

60 Commits

Author SHA1 Message Date
Michael Bradley c893b1f0cb
track latest nim-libp2p's master branch (#248)
* [build] track nim-libp2p's unstable branch

* cid formatIt change

* track nim-libp2p-unstable

* don't stringify cid

* Fixed testblockexc.nim

1. Assign price to blocks
2. Delete on node1 cached blocks from node2 before buying them again

* add trace logging

* init pricing info

* remove duplicate price setting

* cid serialization in trace logs

* bumping dht to latest main

* bump

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Co-authored-by: Bulat-Ziganshin <bulat.ziganshin@gmail.com>
2022-11-02 11:40:28 -06:00
Dmitriy Ryajov 0ecbfcec9f
setup and persist private key (#292)
* setup and persist private key

* return dht record spr

* helper to remap multiaddr ip and port

* set/update discovery and announce addrs

* add nat and discovery IPs

* allow for announce and DHT addresses separatelly

* update tests

* check for nat or discoveryIp

* fix integration tests

* misc align

* don't share data dirs and and set bootstrap node

* add log scope

* remap announceAddrs after node start

* simplify discovery initialization

* make nat and disc-ip required

* add log scope don't init dht spr in constructor

* bump dht

* dissallow `0.0.0.0` for `--nat`
2022-11-01 18:58:41 -06:00
Dmitriy Ryajov e50ea88411
Remove protobuf serialization (#289)
* add format for cid

* cid formatIt change

* track nim-libp2p-unstable

* rework probuf serialization for por

* add missing include

* removing nim protobuf serialization

* rollback to dht to main

* remove protobuf serialization import
2022-10-27 07:41:34 -06:00
Eric Mastro 16024f21e9 bump nim-ethers (to main) 2022-10-25 15:10:35 +11:00
Eric Mastro 1f50612b68 bump dagger-contracts (to main) 2022-10-25 15:10:35 +11:00
Eric Mastro 25df599c08 Update to new reverts API, add tests
1. use `ProviderError` to catch contract errors instaed of `JsonRpcProviderError`.
2. Add tests for `isProofRequired`, `willProofBeRequired`, and `getProofEnd` when slot is empty (and is call is reverted)
3. bump nim-ethers to branch that has new `reverts` API and `ProviderError` support.
4. Update to new `reverts` API.
2022-10-25 15:10:35 +11:00
Eric Mastro a9cea802ff bump dagger-contracts 2022-10-25 15:10:35 +11:00
Eric Mastro eb14ca6166 bump dagger-contracts
# Conflicts:
#	vendor/dagger-contracts
2022-10-25 15:10:35 +11:00
Eric Mastro b9e53e182f [marketplace] support freeing of slots and failed state
Support storage contract parameters to free slots after too many proofs missed, as well as fail a contract after too many slots have been freed.

# Conflicts:
#	vendor/dagger-contracts
2022-10-25 15:10:35 +11:00
Eric Mastro 9d218c88a5 [marketplace] Remove `isSlotCancelled` and `isCancelled`
Remove `isSlotCancelled` and `isCancelled` from proving, as it did not fit in the module.

Update the proving module to not rely on checking the contract to understand if a request was cancelled. Instead, `proofEnd` was modified in `dagger-contracts` such that it returns a past timestamp when the contract is cancelled. This successfully removes
2022-10-25 15:10:35 +11:00
Eric Mastro b4a14e00f7 [marketplace] address more PR comments
- remove RequestState as no longer being used
- remove RequestState check on purchase timeout
- add tests for `withdrawFunds` and `subscribeRequestCancelled`
- update `dagger-contracts` so that `RequestCancelled.requestId` property is indexed in the event
2022-10-25 15:10:35 +11:00
Eric Mastro 1039e27524 [marketplace] bump dagger contracts after rebase 2022-10-25 15:10:35 +11:00
Eric Mastro 0c3fbad470 [purchasing] Withdraw funds when request times out
When a request for storage times out (not enough slots filled), the client will initiate a withdraw request to retrieve its funds out of the contract, setting the state of the request to RequestState.Cancelled. The client will also emit a RequestCancelled event for others to listen to (ie hosts will need to listen for this event to withdraw its collateral).

Add unit test that checks for emission of RequestCancelled after request is purchased request expires.

Update dagger-contracts dependency to commit that holds the changes supporting withdrawing of funds.
2022-10-25 15:10:35 +11:00
Dmitriy Ryajov 106b04340a
Remove local providers (#268)
* add remove provider method

* remove failed providers

* bumping dht to latest master
2022-10-05 10:01:21 -06:00
Dmitriy Ryajov 8d315b38b8
Dht maintenance (#267)
* protocol start is now async

* bumping to maintenance branch
2022-10-01 11:08:44 -06:00
Dmitriy Ryajov f190bfb96e
bump datastore (#266)
* updating datastore to batched version

* bumping datastore
2022-09-30 17:57:57 -06:00
Dmitriy Ryajov 69bd359287
Persist Dht providers (#257)
* providers store integration
2022-09-29 20:16:59 -06:00
Mark Spanbroek 90c818bdda [build] update questionable to 0.10.6
Includes fix for double evaluations of options and results
2022-09-29 10:16:41 +02:00
Eric Mastro 28c73e18a0 bump bearssl to latest 2022-09-27 12:56:07 +10:00
Mark Spanbroek 14cfe05ef6 Update questionable to 0.10.5 2022-09-19 15:51:03 +02:00
Dmitriy Ryajov 7e2f917c84
Dht providers lru (#247)
* upgrading nimbus build system to latest

* use new `updateRecord`

* bump nim-libp2p-dht

* bump nimbus build system

* bumping deps

* update to latest dht
2022-09-13 13:37:49 -06:00
Michael Bradley 6b2adb7017
Fix vendor/nim-eth (#246)
* [build] bump vendor/nim-eth and drop vendor/vendor/nim-eth

* [build] bump vendor/nim-bearssl and drop vendor/bearssl
2022-09-12 21:30:16 -06:00
markspanbroek 5ec2e40a73
[build] Update asynctest to version 0.3.2 (#244)
Includes a fix for Nil exception on unthandled exception
2022-09-12 09:59:39 -06:00
Eric Mastro 014207c87f
bump nim-ethers to 0.2.2, add nim-eth dependency (#236)
Support latest version of nim-ethers which relies on nim-eth.

NOTE: This commit points to PR #31 which is a branch of nim-ethers. That branch should probably be merged first, and this PR updated to point to the commit on main.
2022-09-08 16:56:41 -06:00
Eric Mastro b843118076 bump nim-ethers to use better compile time check 2022-08-26 13:29:09 +10:00
Eric Mastro 01419d3fc3 [chore] bump nim-ethers for distinct indexed event field support 2022-08-26 13:29:09 +10:00
Bulat-Ziganshin f24ded0f76
Download files without padding (#218)
The initial goal of this patch was to allow to download of a file via REST API in exactly the same size as it was uploaded, which required adding fields Chunker.offset and Manifest.originalBytes to keep that size. On top of that, we added more integrity checks to operations on Manifest, and reorganized TestNode.nim to test the actual interaction between node.store and node.retrieve operations.

Note that the wire format of Manifest was changed, so we need to recreate all BlockStores.

* Download without padding
* Fixed chunker tests
* Chunker: get rid of RabinChunker
* Verify offset in the chunker tests
* Use manifest.originalBytesPadded in StoreStream.size
* StoreStream: replace emptyBlock with zeroMem
* Manifest.bytes: compute how many bytes corresponding StoreStream(Manifest, pad) will return
* Manifest: verify originalBytes and originalLen on new/encode/decode
Also set originalBytes in each Manifest creation/update scenario
* Manifest: comments, split code into sections
* Reordered parameters to deal with int64 size in 32-bit builds
* TestNode.nim: combine Store and Retrieve tests
1. Instead of copy-pasting code from node.nim, new test calls node.store() and node.retrieve() in order to check that they can correctly store and then retrieve data
2. New test compares only file contents, manifest contents considered an implementation detail
3. New test chunks at odd chunkSize=BlockSize/1.618 in order to ensure that data retrieved correctly even when buffer sizes mismatch
* TestNode.nim: code refactoring
* Manifest.add: one more test
* Manifest.verify: return Result instead of raising Defect
* Node.store: added blockSize parameter
2022-08-24 15:15:59 +03:00
Michael Bradley, Jr e0e2d7b583 [build] bump questionable 2022-08-22 20:52:51 -05:00
Michael Bradley, Jr 5aa42541b3 [build] update nim-datastore
also make changes to `codex/codex.nim` and tests for SQLiteStore re: changes in
nim-datastore

closes #178
2022-08-10 13:13:07 -05:00
Mark Spanbroek b7df2d151c [build] Update dagger-contracts 2022-08-08 14:00:55 +02:00
Mark Spanbroek 6ec2d3bf8f Update dagger-contracts 2022-08-08 14:00:55 +02:00
Mark Spanbroek 5a00eddfba Update dagger-contracts 2022-08-08 14:00:55 +02:00
Michael Bradley, Jr 92d83bb1d2 [build] add github.com/arnetheduck/nim-sqlite3-abi to vendor 2022-08-08 02:12:43 -05:00
Michael Bradley, Jr 679df23b6f [build] add github.com/status-im/nim-datastore to vendor 2022-08-08 02:12:43 -05:00
Mark Spanbroek 7c50d9f739 Update ethers to version 0.2.0 2022-07-20 13:43:20 +02:00
Mark Spanbroek e352a181e4 [contracts] Update to latest dagger-contracts 2022-07-20 13:43:20 +02:00
Ivan Yonchovski 9c2224e306 Bump libbacktrace and secp256k1 2022-06-28 11:04:18 +03:00
Eric Mastro 3b0e93eab4
[chore] bump nim-ethers to 0.1.7 (#107)
* [chore] bump nim-ethers to 0.1.7

* [chore] make block handler callback async

nim-ethers 0.1.7 makes the subscription callbacks async.
2022-06-02 16:09:53 -06:00
Dmitriy Ryajov 56b80d6f6d
Por serialize (#106)
* move por into storage proofs

* use SeekableStream

* adding serialization for por

* remove leftovers

* add empty block support

* add basic por test

* rename block exchange for consistency

* add storageproofstests

* moving timing to storageproofs

* fix imports

* fix imports

* fix imports

* add top level exports

* move delete blocks helper to helpers

* more import/export fixes

* cleanup

* more import fixes

* fix unused warnings

* detect corrupt blocks tests

* add serialization tests

* move init method around

* bump asynctest

* fix CID version

* get rid of warning

* wip: fix CI

* increase CI timeout
2022-05-23 23:24:15 -06:00
Csaba Kiraly aec9a6eb92 enable build on Mac M1
- On Mac, -march=native is only supported in newest Clang versions,
better disable it for now.

- Bumping Leopard version to one that support arm64

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
2022-05-19 11:34:49 +02:00
Mark Spanbroek ce38a38849 Update nim-ethers to version 0.1.6 2022-05-19 10:38:19 +02:00
Mark Spanbroek a43e3d6d94 Fix version conflict on nimble install
Resolves version conflict between
questionable 0.9.x and 0.10.x.
2022-05-19 09:36:47 +02:00
Mark Spanbroek d2da225f21 Update nim-ethers to 0.1.5 2022-04-21 14:04:07 +02:00
Tanguy c06c9b578c
Various dev-ops things (#85)
* Add metrics server
* Add --version
* Fixes for chronicles=trace + ci
2022-04-14 12:49:03 +02:00
markspanbroek b88561e090
Subscribe to proof submissions (#83)
* Update dagger-contracts

* [proving] rename ProofTiming -> Proofs

* Update nim-ethers to 0.1.4

* [proving] Subscribe to proof submissions

* [proving] support proof submission through the Proving abstraction
2022-04-13 10:41:48 -06:00
Tanguy 4d681102e5
Add DHT (#75)
* First implem

* Add persistent net key option

* Working DHT setup

* Bootstrap nodes

* Implement DaggerNode.findPeer

* Remove unrelevant comment

* Added discovery to blockexchange requestBlock

* add FSStore.blockList

* Block advertisement

* Tests compiles

* Green tests

* toDiscoveryId instead of toNodeId

* remove stopAdvertisingBlock

* Removed nim-eth dependency

* Move discovery stuff to discovery.nim

* Add missing file, start of discovery tests

* Better discovery logic

* Add tests

* Address comment

* Better E2E test
2022-04-13 10:32:35 -06:00
markspanbroek 1a3003e043
Update to latest dagger-contracts (#78)
- StorageRequest struct has changed

- StorageRequested event has changed,
  it no longer returns the entire request,
  but only the id and the ask
2022-04-11 12:03:55 -06:00
markspanbroek 03fa370624
Proving (#66)
* Add Proving object, which maintains contract id's to watch

* [proving] invoke callback when proof is required

# Conflicts:
#	dagger/por/timing/periods.nim
#	dagger/por/timing/prooftiming.nim

* [proving] check proof requirements for all our contracts

# Conflicts:
#	tests/dagger/helpers/mockprooftiming.nim

* Update vendor/dagger-contracts

* [proving] call onProofRequired() when proof is required soon

* [proving] stop checking contracts that have ended

* [proving] Remove duplicated funcs

* [proving] Implement ProofTiming on top of smart contract

* [proving] Fix race condition in waitUntilNextPeriod()

Sometimes waitUntilNextPeriod would take a while to
determine the current period, leading to unexpected results.

Splits waitUntilNextPeriod() into getCurrentPeriod()
and waitUntilPeriod(), to ensure that we're really waiting
for the period that we think we're waiting for.
2022-04-08 15:58:16 -06:00
Mark Spanbroek 0cee79b1a2 Update nim-ethers 2022-04-04 10:03:46 +02:00
Michael Bradley 1e5b41a447
add github.com/status-im/nim-leopard to vendor (#61)
* [leopard] add github.com/status-im/nim-leopard to vendor

* [leopard] specify absolute path for nimcache dir

* [leopard] make adjustments to nim-dagger CI GHA workflow
2022-03-30 09:34:47 -06:00