Commit Graph

685 Commits

Author SHA1 Message Date
Mark Spanbroek 8ff748eff8 [market] select a storage offer 2022-04-04 10:03:46 +02:00
Mark Spanbroek 5ea64522d8 [market] subscribe to offers for a certain request 2022-04-04 10:03:46 +02:00
Mark Spanbroek 0cee79b1a2 Update nim-ethers 2022-04-04 10:03:46 +02:00
Mark Spanbroek 9f76e714a2 [market] submit storage offers 2022-04-04 10:03:46 +02:00
Mark Spanbroek e52153e2ba [market] Require a storage contract with a signer 2022-04-04 10:03:46 +02:00
Mark Spanbroek e78a0fb43e [contracts] Wrap contracts in Market abstraction 2022-04-04 10:03:46 +02:00
Mark Spanbroek bd8f4d5d74 [tests] Extract common examples into separate module 2022-04-04 10:03:46 +02:00
Mark Spanbroek 5b5f3335d6 [purchasing] Ignore offers that are about to expire 2022-04-04 10:03:46 +02:00
Mark Spanbroek dcfd6be1c6 [purchasing] Ignore expired offers 2022-04-04 10:03:46 +02:00
Mark Spanbroek fe23cb89d7 [purchasing] Select cheapest offer 2022-04-04 10:03:46 +02:00
Mark Spanbroek 03140fdf49 [purchasing] Add state to track purchase progress 2022-04-04 10:03:46 +02:00
Mark Spanbroek 04c291b05c [purchasing] rename requests -> requested 2022-04-04 10:03:46 +02:00
Mark Spanbroek 7c9c244836 [purchasing] Remove PurchaseRequest; use StorageRequest 2022-04-04 10:03:46 +02:00
Mark Spanbroek 9ade3fdd37 [contracts] StorageRequest/Offer is object instead of tuple 2022-04-04 10:03:46 +02:00
Mark Spanbroek 5f10549f19 Introduce Purchasing and Market 2022-04-04 10:03:46 +02:00
Mark Spanbroek a3fdd35f73 [contracts] Split marketplace.nim into requests.nim and offers.nim 2022-04-04 10:03:46 +02:00
Michael Bradley e92fbd4ad1
[ci] use matrix.branch to specify NIM_COMMIT make var of nimbus-build-system (#65)
See: https://github.com/status-im/nimbus-build-system#nim_commit
2022-04-01 10:22:10 -06:00
Michael Bradley 1e5b41a447
add github.com/status-im/nim-leopard to vendor (#61)
* [leopard] add github.com/status-im/nim-leopard to vendor

* [leopard] specify absolute path for nimcache dir

* [leopard] make adjustments to nim-dagger CI GHA workflow
2022-03-30 09:34:47 -06:00
Dmitriy Ryajov 43cea1743a
Node storestream (#62)
* use storestream for retrieval

* use `len` prop on manifest

* remove read pattern

* make blocksize and chunksize consistent

* fix tests

* fix chunker tests - now padded by default
2022-03-29 20:43:35 -06:00
Mark Spanbroek 03c8ceccf9 [contracts] Separate nimble task to run all tests
A simple "nimble test" won't require you to run a hardhat
node with the contracts, but a "nimble testAll" will.
2022-03-28 11:59:01 +02:00
Mark Spanbroek 28294d431a [contracts] Tests now also work in Windows 2022-03-28 11:59:01 +02:00
Mark Spanbroek b8e013e439 [contracts] Add Readme 2022-03-28 11:59:01 +02:00
Mark Spanbroek 4fd92b52eb [contracts] Install NodeJS in CI environment 2022-03-28 11:59:01 +02:00
Mark Spanbroek 42edbf5076 [contracts] Remove obsolete marketplace tests 2022-03-28 11:59:01 +02:00
Mark Spanbroek efb4f5c375 [contracts] Update to new marketplace design 2022-03-28 11:59:01 +02:00
Mark Spanbroek 48ebae9ae5 Update nim-ethers 2022-03-28 11:59:01 +02:00
Mark Spanbroek 87bd947d3d [contracts] Update to new collateral design 2022-03-28 11:59:01 +02:00
Mark Spanbroek 8d0f9775b5 [contracts] Simplify hardhat node startup 2022-03-28 11:59:01 +02:00
Mark Spanbroek 8853105920 Update dagger-contracts 2022-03-28 11:59:01 +02:00
Mark Spanbroek d07a5cd19e [contracts] Run contract tests in CI
Introduce new nimble 'test' task that runs *all* tests.
Renames testAll to testDagger, because testAll didn't
run all tests, it forgot to run the contract tests.
2022-03-28 11:59:01 +02:00
Dmitriy Ryajov e965f5e0de
Seekable stream (#58)
* cleanup manifest

* adding a "seekable" base stream
2022-03-21 12:09:59 -06:00
Dmitriy Ryajov 2c4b1e6906
Upraises (#57)
* cleanup manifest

* make block a ref type

* use upraises

* more upraises
2022-03-18 16:17:51 -06:00
Dmitriy Ryajov d204b518e8
Make block a ref (#56)
* cleanup manifest

* make block a ref type
2022-03-18 13:50:53 -06:00
Dmitriy Ryajov 472e1e6fc3
Store stream (#55)
* make manifest a ref object

* capitalize tests

* add storestream and tests

* fix tests

* run storestream tests
2022-03-17 07:56:46 -06:00
Dmitriy Ryajov f2aa945991
Manifest block size (#53)
* add `blockSize` and `size` to manifest

* fix tests

* fix decoding/encoding

* re-export coders

* style
2022-03-15 12:47:31 -06:00
Dmitriy Ryajov f414c695be
Cleanup-manifest (#52)
* adding nim-blscurve

* adding taskpools

* cleanup manifest
2022-03-14 10:06:36 -06:00
Csaba Kiraly d8162b1e16
initial commit of the Shacham BLS-based and RSA-based public schemes (#26)
* initial commit of the Shacham RSA-based public scheme

Minimal working version with lots of error checks and corrections
still needed.
- using Bearssl RSA code through libp2p
- with selecteble BigInt library for experimentation

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* better proc names

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* separating demo code from library

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* using normal file io instead of memfiles

mmap has serveral potential issues and we do not really need it, so
changing to use the normal system file interface is better.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* draft version of bls proofs

Implementation of the BLS-based public PoS scheme from
Shacham H., Waters B., "Compact Proofs of Retrievability"
using pairing over BLS12-381 ECC

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* draft test and benchmark code for BLS PoS

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* improve documentation of BLS scheme

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* fix getSector

* fixing DST tag in hashToG1

The DST tag should be unique to achieve domain separation
of hash functions as defined in:
https://tools.ietf.org/id/draft-irtf-cfrg-hash-to-curve-06.html#domain-separation

Changed DST tag to one that indicates the PoC status of this code.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* add verifyPairings abstraction

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* change random number generator to a secure one

Use Rng based on BrHmacDrbgContext

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* fix benchmark template

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* exchange parameter order in pairing

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* add optimized verifyPairing implementation

When verifying two pairings, one final exponentiation
can be spared through the use of cneg.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* speed up tag generation by a factor of s

Scalar multiplications in tag generation can be rearranged
to benefit from the way random points are being generated.
Since random points are themselves generated using scalar
multiplication and the base is common, the sum of multiplications
becomes a single multiplication with the scalar sum, resulting in
a nice speedup.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* use blst_p1_add_or_double instead of blst_p1_add

blst exposes two add functions: one that works for the corner case
of doubling, and one that isn't. It seems safer to use the one that
works, even if it is highly improbable in these cases that doubling
would occur.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* sectorsperblock should be an external parameter

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* parametrize sectorsblock and querylen

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* improving benchmark messages

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* rebasing main

* generateAuthenticator: remove unused ubase parameter from naive impl

No need to have the same interface on the two implementations, so
we can remove this parameter.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* generateAuthenticator: add some more explanation

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* renaming pos.nim to rsa.nim

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* sign and verify metadata in Tau

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* adding more comments

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* remove code of slow RSA based version

Removed RSA-based version to ease maintenance, as it is
highly unlikely we would use it.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* formatting: use just one type section

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* more comments added

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* make `namelen` a const

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* generalize hashToG1

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* hashNameI: switch to faster implementation

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

Co-authored-by: Tanguy <tanguy@status.im>
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2022-03-08 15:04:42 -06:00
Eric Mastro 70cbdff901
feat: introduce LRU cache (#50)
* feat: introduce LRU cache

Replace `MemoryStore` with LRU caching mechanism.

`lrucache` library was forked to https://github.com/status-im/lrucache.nim.

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>

# Conflicts:
#	dagger/dagger.nim
#	dagger/stores.nim
#	dagger/stores/manager.nim
#	tests/dagger/blockexc/testengine.nim
#	tests/dagger/helpers/nodeutils.nim
#	tests/dagger/testnode.nim
#	tests/dagger/teststores.nim

* feat: introduce --cache-size CLI option

Allow for a value of `0` to disable the cache.

# Conflicts:
#	dagger/dagger.nim

* allow dynamic block size in cache

allow block size to be variable/dynamic

update lrucache to use updated lrucache dep

Using removeLru proc, added tests

* Refactor CacheStore init block

Co-authored-by: Michael Bradley, Jr <michaelsbradleyjr@gmail.com>
2022-03-02 10:30:42 -06:00
Eric Mastro 26ead9726d [rest api]: improve exception handling for `/connect` and add `content-type` header for `/download`
This PR achieves the following:
1. Improves the exception handling when dialling a peer fails or an unknown error occurs.
2. Add a `Content-Type` header to the `/download` endpoint of `application/octet-stream`, which is [defined by MDN](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types#types) as meant to be used for "generic binary data (or binary data whose true type is unknown)".

Co-authored-by: Michael Bradley <michaelsbradleyjr@gmail.com>
2022-02-04 15:38:39 +11:00
Eric Mastro 2e5c28781c feat: integrate dagger contracts
Integrate dagger contracts from `nim-dagger-contracts` repo.

Add `dagger-contracts`, `nim-web3`, and all of `nim-web3`’s transitive deps as submodule deps to `nim-dagger`. Note: `nim-web3` and its transitive deps may no longer be needed when we switch to `nim-ethers`.

Add a `testContracts` nimble task to test all of the contracts functionality. Namely, this spins up an ethereum simulator, deploys the contracts (in `dagger-contracts`), runs the contract tests, and finally, regardless of success/error, kills the ethereum sim processes. The nimble task can be run with `./env.sh nimble testContracts`.

We also tested `nim-dagger-contracts` as a submodule dep of `nim-dagger`, and while the tests run as expected, the preference is to merge `nim-dagger-contracts` inside of `nim-dagger` for ease of parallel development. There’s also a high probability that `nim-dagger-contracts` is not being used as a dep by other projects. Are there any strong objections to this?

Co-authored-by: Michael Bradley <michaelsbradleyjr@gmail.com>
2022-02-04 15:34:56 +11:00
Eric Mastro ec66e42e73
fix(rest): download unknown cid (#48)
Downloading a CID that is unknown to the network would wait for the blockexchange to try to get the data. However, a subsequent API call, eg upload for that CID, or a blockexchange timeout would cause the dagger node to crash, due to an attempt to complete a future that had already been completed.

We were able to narrow this down to an attempt to call  `resp.finish()` when the download block had already returned an `RestApiResponse` (and never sent any chunks to the response from the libp2p stream).

Change the HTTP response code for an unknown cid from 400 to 404.

Fix spelling mistake “cunk”.

Co-authored-by: Michael Bradley <michaelsbradleyjr@gmail.com>

Co-authored-by: Michael Bradley <michaelsbradleyjr@gmail.com>
2022-02-03 08:02:18 -06:00
Dmitriy Ryajov 5f48de6a44
cleanup manifest (#44)
* cleanup manifest

* Attach decode to `Manifest`

* move manifest container to it's own file

* rebase main

* update header license year
2022-01-12 19:55:51 -06:00
Dmitriy Ryajov 68a45d8f57
use tryGet, which is more informative than get (#45) 2022-01-12 18:42:18 -06:00
Dmitriy Ryajov b315c759bb
refreshing stew (#43) 2022-01-12 16:52:09 -06:00
Dmitriy Ryajov 2188f69273
Minor cleanup and update deps (#42)
* abort download if atleast one block failed

* make chunker a regular object

* updating questionable

* updating vendor deps

* cleanup unused imports

* Remove mention of Waku
2022-01-12 16:19:37 -06:00
Dmitriy Ryajov 15ef6c78eb
fix treehash generation (#39) 2022-01-10 21:04:00 -06:00
Dmitriy Ryajov 47a0d4ef7a
Move empty digest (#38)
* improve empty digest initialization

* cleanup warnings
2022-01-10 20:25:27 -06:00
Dmitriy Ryajov a9721dd65f
Block.init should return Result (#37)
* Block.init should return Result

* use get() instead of tryGet()
2022-01-10 20:25:13 -06:00
Dmitriy Ryajov 9d17c91d4c
adding basic comments to rest api (#36)
* adding basic comments to rest api

* Update dagger/rest/api.nim

Co-authored-by: Michael Bradley <michaelsbradleyjr@gmail.com>

Co-authored-by: Michael Bradley <michaelsbradleyjr@gmail.com>
2022-01-10 14:35:52 -06:00
Dmitriy Ryajov fbe161a073
Node setup (#32)
* settup basic nim node

* adding http utils

* adding confutils

* rough rest api proto

* adding missing deps

* turn tls emulation off

* adding toml serialization

* wip

* adding missing deps

* make sure to clean old state in teardown

* adding file upload rest endpoint

* renaming blockexchange to networkstore

* updating nim-presto

* updating libp2p

* wip adding streaming upload

* reworked chunking

* bump to latest unstable

* adding asyncfutures stream

* make streamable

* deleting unused files

* reworking stores api

* use new stores api

* rework blockset and remove blockstream

* don't return option from constructor

* rework chunker

* wip implement upload

* fix tests

* move unrelated logic to engine

* don't print entire message

* logging

* basic encode/decode to/from dag-pb

* add basic upload/download support

* fix tests

* renaming blockset to manifest

* don't pass config to node

* remove config and use new manifest

* wip: make endpoints more reliable

* wip: adding node tests

* include correct manifest test

* removing asyncfutures

* proper chunking of files

* simplify stream reading

* test with encoding/decoding with many blocks

* add block storing tests

* adding retrieval test

* add logging

* tidy up chunker

* tidy up manifest and node

* use default chunk size

* fix tests

* fix tests

* make sure Eof is set properly

* wip

* minor cleanup

* add file utils

* cleanup config

* splitout DaggerServer and "main"

* remove events since they are not used

* add broadcast method to network peer

* add and wire localstore

* use localstore in the node

* wip

* logging

* move file utils

* use the constant

* updating deps

* fix memstore

* use latest libp2p unstable

* fix tests

* rework block streaming

* don't fail storing if the block already exists

* add helper info endpoint

* correct comment

* rename localstore to fsstore

* fix tests

* remove unused tests

* add test to retrieve one block

* move some test files around

* consolidate setup

* Update dagger/blockexchange/engine.nim

Co-authored-by: Tanguy <tanguy@status.im>

* typo

* better block path handling

* don't inherit rootobj

* remove useless template

* Update tests/dagger/blockexc/testblockexc.nim

Co-authored-by: markspanbroek <mark@spanbroek.net>

* use isMainModule

* use proper flag for starter/stoped

* cleanup optional use

* wrap in isMainModule

* use `cancelAndAwait`

* remove unused imports

* wip

* don't use optional

* use functional error api

* rework store tests and add fs tests

* Block.new() to Block.init()

* don't use optional for engine blocks

* use result instead of optional for getBlock

* remove unused imports

* move stopping servers to `shutdown`

* use result instead of optional

* rework with results

* fix tests

* use waitFor in signal handlers

* error helper

* use `?` and mapFailure where possible

* remove unnecesary `=?`

* improve empty cid digest initialization

Co-authored-by: Tanguy <tanguy@status.im>
Co-authored-by: markspanbroek <mark@spanbroek.net>
2022-01-10 09:32:56 -06:00