Commit Graph

339 Commits

Author SHA1 Message Date
Mark Spanbroek 291c506e0b [market] Fix timing of test for slow CI 2022-04-04 11:29:32 +02:00
Mark Spanbroek dd89a55351 [market] Add Market.getTime() 2022-04-04 11:29:32 +02:00
Mark Spanbroek 061b32296a Introduce Sales 2022-04-04 11:29:32 +02:00
Mark Spanbroek 75ec8c0bfd [market] wait until request/offer expires 2022-04-04 10:03:46 +02:00
Mark Spanbroek 8ff748eff8 [market] select a storage offer 2022-04-04 10:03:46 +02:00
Mark Spanbroek 5ea64522d8 [market] subscribe to offers for a certain request 2022-04-04 10:03:46 +02:00
Mark Spanbroek 0cee79b1a2 Update nim-ethers 2022-04-04 10:03:46 +02:00
Mark Spanbroek 9f76e714a2 [market] submit storage offers 2022-04-04 10:03:46 +02:00
Mark Spanbroek e52153e2ba [market] Require a storage contract with a signer 2022-04-04 10:03:46 +02:00
Mark Spanbroek e78a0fb43e [contracts] Wrap contracts in Market abstraction 2022-04-04 10:03:46 +02:00
Mark Spanbroek bd8f4d5d74 [tests] Extract common examples into separate module 2022-04-04 10:03:46 +02:00
Mark Spanbroek 5b5f3335d6 [purchasing] Ignore offers that are about to expire 2022-04-04 10:03:46 +02:00
Mark Spanbroek dcfd6be1c6 [purchasing] Ignore expired offers 2022-04-04 10:03:46 +02:00
Mark Spanbroek fe23cb89d7 [purchasing] Select cheapest offer 2022-04-04 10:03:46 +02:00
Mark Spanbroek 03140fdf49 [purchasing] Add state to track purchase progress 2022-04-04 10:03:46 +02:00
Mark Spanbroek 04c291b05c [purchasing] rename requests -> requested 2022-04-04 10:03:46 +02:00
Mark Spanbroek 7c9c244836 [purchasing] Remove PurchaseRequest; use StorageRequest 2022-04-04 10:03:46 +02:00
Mark Spanbroek 9ade3fdd37 [contracts] StorageRequest/Offer is object instead of tuple 2022-04-04 10:03:46 +02:00
Mark Spanbroek 5f10549f19 Introduce Purchasing and Market 2022-04-04 10:03:46 +02:00
Mark Spanbroek a3fdd35f73 [contracts] Split marketplace.nim into requests.nim and offers.nim 2022-04-04 10:03:46 +02:00
Michael Bradley e92fbd4ad1
[ci] use matrix.branch to specify NIM_COMMIT make var of nimbus-build-system (#65)
See: https://github.com/status-im/nimbus-build-system#nim_commit
2022-04-01 10:22:10 -06:00
Michael Bradley 1e5b41a447
add github.com/status-im/nim-leopard to vendor (#61)
* [leopard] add github.com/status-im/nim-leopard to vendor

* [leopard] specify absolute path for nimcache dir

* [leopard] make adjustments to nim-dagger CI GHA workflow
2022-03-30 09:34:47 -06:00
Dmitriy Ryajov 43cea1743a
Node storestream (#62)
* use storestream for retrieval

* use `len` prop on manifest

* remove read pattern

* make blocksize and chunksize consistent

* fix tests

* fix chunker tests - now padded by default
2022-03-29 20:43:35 -06:00
Mark Spanbroek 03c8ceccf9 [contracts] Separate nimble task to run all tests
A simple "nimble test" won't require you to run a hardhat
node with the contracts, but a "nimble testAll" will.
2022-03-28 11:59:01 +02:00
Mark Spanbroek 28294d431a [contracts] Tests now also work in Windows 2022-03-28 11:59:01 +02:00
Mark Spanbroek b8e013e439 [contracts] Add Readme 2022-03-28 11:59:01 +02:00
Mark Spanbroek 4fd92b52eb [contracts] Install NodeJS in CI environment 2022-03-28 11:59:01 +02:00
Mark Spanbroek 42edbf5076 [contracts] Remove obsolete marketplace tests 2022-03-28 11:59:01 +02:00
Mark Spanbroek efb4f5c375 [contracts] Update to new marketplace design 2022-03-28 11:59:01 +02:00
Mark Spanbroek 48ebae9ae5 Update nim-ethers 2022-03-28 11:59:01 +02:00
Mark Spanbroek 87bd947d3d [contracts] Update to new collateral design 2022-03-28 11:59:01 +02:00
Mark Spanbroek 8d0f9775b5 [contracts] Simplify hardhat node startup 2022-03-28 11:59:01 +02:00
Mark Spanbroek 8853105920 Update dagger-contracts 2022-03-28 11:59:01 +02:00
Mark Spanbroek d07a5cd19e [contracts] Run contract tests in CI
Introduce new nimble 'test' task that runs *all* tests.
Renames testAll to testDagger, because testAll didn't
run all tests, it forgot to run the contract tests.
2022-03-28 11:59:01 +02:00
Dmitriy Ryajov e965f5e0de
Seekable stream (#58)
* cleanup manifest

* adding a "seekable" base stream
2022-03-21 12:09:59 -06:00
Dmitriy Ryajov 2c4b1e6906
Upraises (#57)
* cleanup manifest

* make block a ref type

* use upraises

* more upraises
2022-03-18 16:17:51 -06:00
Dmitriy Ryajov d204b518e8
Make block a ref (#56)
* cleanup manifest

* make block a ref type
2022-03-18 13:50:53 -06:00
Dmitriy Ryajov 472e1e6fc3
Store stream (#55)
* make manifest a ref object

* capitalize tests

* add storestream and tests

* fix tests

* run storestream tests
2022-03-17 07:56:46 -06:00
Dmitriy Ryajov f2aa945991
Manifest block size (#53)
* add `blockSize` and `size` to manifest

* fix tests

* fix decoding/encoding

* re-export coders

* style
2022-03-15 12:47:31 -06:00
Dmitriy Ryajov f414c695be
Cleanup-manifest (#52)
* adding nim-blscurve

* adding taskpools

* cleanup manifest
2022-03-14 10:06:36 -06:00
Csaba Kiraly d8162b1e16
initial commit of the Shacham BLS-based and RSA-based public schemes (#26)
* initial commit of the Shacham RSA-based public scheme

Minimal working version with lots of error checks and corrections
still needed.
- using Bearssl RSA code through libp2p
- with selecteble BigInt library for experimentation

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* better proc names

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* separating demo code from library

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* using normal file io instead of memfiles

mmap has serveral potential issues and we do not really need it, so
changing to use the normal system file interface is better.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* draft version of bls proofs

Implementation of the BLS-based public PoS scheme from
Shacham H., Waters B., "Compact Proofs of Retrievability"
using pairing over BLS12-381 ECC

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* draft test and benchmark code for BLS PoS

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* improve documentation of BLS scheme

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* fix getSector

* fixing DST tag in hashToG1

The DST tag should be unique to achieve domain separation
of hash functions as defined in:
https://tools.ietf.org/id/draft-irtf-cfrg-hash-to-curve-06.html#domain-separation

Changed DST tag to one that indicates the PoC status of this code.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* add verifyPairings abstraction

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* change random number generator to a secure one

Use Rng based on BrHmacDrbgContext

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* fix benchmark template

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* exchange parameter order in pairing

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* add optimized verifyPairing implementation

When verifying two pairings, one final exponentiation
can be spared through the use of cneg.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* speed up tag generation by a factor of s

Scalar multiplications in tag generation can be rearranged
to benefit from the way random points are being generated.
Since random points are themselves generated using scalar
multiplication and the base is common, the sum of multiplications
becomes a single multiplication with the scalar sum, resulting in
a nice speedup.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* use blst_p1_add_or_double instead of blst_p1_add

blst exposes two add functions: one that works for the corner case
of doubling, and one that isn't. It seems safer to use the one that
works, even if it is highly improbable in these cases that doubling
would occur.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* sectorsperblock should be an external parameter

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* parametrize sectorsblock and querylen

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* improving benchmark messages

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* rebasing main

* generateAuthenticator: remove unused ubase parameter from naive impl

No need to have the same interface on the two implementations, so
we can remove this parameter.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* generateAuthenticator: add some more explanation

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* renaming pos.nim to rsa.nim

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* sign and verify metadata in Tau

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* adding more comments

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* remove code of slow RSA based version

Removed RSA-based version to ease maintenance, as it is
highly unlikely we would use it.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* formatting: use just one type section

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* more comments added

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* make `namelen` a const

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* generalize hashToG1

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* hashNameI: switch to faster implementation

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

Co-authored-by: Tanguy <tanguy@status.im>
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2022-03-08 15:04:42 -06:00
Eric Mastro 70cbdff901
feat: introduce LRU cache (#50)
* feat: introduce LRU cache

Replace `MemoryStore` with LRU caching mechanism.

`lrucache` library was forked to https://github.com/status-im/lrucache.nim.

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>

# Conflicts:
#	dagger/dagger.nim
#	dagger/stores.nim
#	dagger/stores/manager.nim
#	tests/dagger/blockexc/testengine.nim
#	tests/dagger/helpers/nodeutils.nim
#	tests/dagger/testnode.nim
#	tests/dagger/teststores.nim

* feat: introduce --cache-size CLI option

Allow for a value of `0` to disable the cache.

# Conflicts:
#	dagger/dagger.nim

* allow dynamic block size in cache

allow block size to be variable/dynamic

update lrucache to use updated lrucache dep

Using removeLru proc, added tests

* Refactor CacheStore init block

Co-authored-by: Michael Bradley, Jr <michaelsbradleyjr@gmail.com>
2022-03-02 10:30:42 -06:00
Eric Mastro 26ead9726d [rest api]: improve exception handling for `/connect` and add `content-type` header for `/download`
This PR achieves the following:
1. Improves the exception handling when dialling a peer fails or an unknown error occurs.
2. Add a `Content-Type` header to the `/download` endpoint of `application/octet-stream`, which is [defined by MDN](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types#types) as meant to be used for "generic binary data (or binary data whose true type is unknown)".

Co-authored-by: Michael Bradley <michaelsbradleyjr@gmail.com>
2022-02-04 15:38:39 +11:00
Eric Mastro 2e5c28781c feat: integrate dagger contracts
Integrate dagger contracts from `nim-dagger-contracts` repo.

Add `dagger-contracts`, `nim-web3`, and all of `nim-web3`’s transitive deps as submodule deps to `nim-dagger`. Note: `nim-web3` and its transitive deps may no longer be needed when we switch to `nim-ethers`.

Add a `testContracts` nimble task to test all of the contracts functionality. Namely, this spins up an ethereum simulator, deploys the contracts (in `dagger-contracts`), runs the contract tests, and finally, regardless of success/error, kills the ethereum sim processes. The nimble task can be run with `./env.sh nimble testContracts`.

We also tested `nim-dagger-contracts` as a submodule dep of `nim-dagger`, and while the tests run as expected, the preference is to merge `nim-dagger-contracts` inside of `nim-dagger` for ease of parallel development. There’s also a high probability that `nim-dagger-contracts` is not being used as a dep by other projects. Are there any strong objections to this?

Co-authored-by: Michael Bradley <michaelsbradleyjr@gmail.com>
2022-02-04 15:34:56 +11:00
Eric Mastro ec66e42e73
fix(rest): download unknown cid (#48)
Downloading a CID that is unknown to the network would wait for the blockexchange to try to get the data. However, a subsequent API call, eg upload for that CID, or a blockexchange timeout would cause the dagger node to crash, due to an attempt to complete a future that had already been completed.

We were able to narrow this down to an attempt to call  `resp.finish()` when the download block had already returned an `RestApiResponse` (and never sent any chunks to the response from the libp2p stream).

Change the HTTP response code for an unknown cid from 400 to 404.

Fix spelling mistake “cunk”.

Co-authored-by: Michael Bradley <michaelsbradleyjr@gmail.com>

Co-authored-by: Michael Bradley <michaelsbradleyjr@gmail.com>
2022-02-03 08:02:18 -06:00
Dmitriy Ryajov 5f48de6a44
cleanup manifest (#44)
* cleanup manifest

* Attach decode to `Manifest`

* move manifest container to it's own file

* rebase main

* update header license year
2022-01-12 19:55:51 -06:00
Dmitriy Ryajov 68a45d8f57
use tryGet, which is more informative than get (#45) 2022-01-12 18:42:18 -06:00
Dmitriy Ryajov b315c759bb
refreshing stew (#43) 2022-01-12 16:52:09 -06:00
Dmitriy Ryajov 2188f69273
Minor cleanup and update deps (#42)
* abort download if atleast one block failed

* make chunker a regular object

* updating questionable

* updating vendor deps

* cleanup unused imports

* Remove mention of Waku
2022-01-12 16:19:37 -06:00
Dmitriy Ryajov 15ef6c78eb
fix treehash generation (#39) 2022-01-10 21:04:00 -06:00