Commit Graph

57 Commits

Author SHA1 Message Date
Dmitriy Ryajov b1264a9c87
add storestream and tests 2022-03-16 15:48:37 -06:00
Dmitriy Ryajov 13dec3ac52
make manifest a ref object 2022-03-16 15:48:33 -06:00
Dmitriy Ryajov f2aa945991
Manifest block size (#53)
* add `blockSize` and `size` to manifest

* fix tests

* fix decoding/encoding

* re-export coders

* style
2022-03-15 12:47:31 -06:00
Dmitriy Ryajov f414c695be
Cleanup-manifest (#52)
* adding nim-blscurve

* adding taskpools

* cleanup manifest
2022-03-14 10:06:36 -06:00
Csaba Kiraly d8162b1e16
initial commit of the Shacham BLS-based and RSA-based public schemes (#26)
* initial commit of the Shacham RSA-based public scheme

Minimal working version with lots of error checks and corrections
still needed.
- using Bearssl RSA code through libp2p
- with selecteble BigInt library for experimentation

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* better proc names

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* separating demo code from library

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* using normal file io instead of memfiles

mmap has serveral potential issues and we do not really need it, so
changing to use the normal system file interface is better.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* draft version of bls proofs

Implementation of the BLS-based public PoS scheme from
Shacham H., Waters B., "Compact Proofs of Retrievability"
using pairing over BLS12-381 ECC

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* draft test and benchmark code for BLS PoS

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* improve documentation of BLS scheme

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* fix getSector

* fixing DST tag in hashToG1

The DST tag should be unique to achieve domain separation
of hash functions as defined in:
https://tools.ietf.org/id/draft-irtf-cfrg-hash-to-curve-06.html#domain-separation

Changed DST tag to one that indicates the PoC status of this code.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* add verifyPairings abstraction

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* change random number generator to a secure one

Use Rng based on BrHmacDrbgContext

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* fix benchmark template

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* exchange parameter order in pairing

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* add optimized verifyPairing implementation

When verifying two pairings, one final exponentiation
can be spared through the use of cneg.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* speed up tag generation by a factor of s

Scalar multiplications in tag generation can be rearranged
to benefit from the way random points are being generated.
Since random points are themselves generated using scalar
multiplication and the base is common, the sum of multiplications
becomes a single multiplication with the scalar sum, resulting in
a nice speedup.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* use blst_p1_add_or_double instead of blst_p1_add

blst exposes two add functions: one that works for the corner case
of doubling, and one that isn't. It seems safer to use the one that
works, even if it is highly improbable in these cases that doubling
would occur.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* sectorsperblock should be an external parameter

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* parametrize sectorsblock and querylen

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* improving benchmark messages

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* rebasing main

* generateAuthenticator: remove unused ubase parameter from naive impl

No need to have the same interface on the two implementations, so
we can remove this parameter.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* generateAuthenticator: add some more explanation

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* renaming pos.nim to rsa.nim

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* sign and verify metadata in Tau

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* adding more comments

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* remove code of slow RSA based version

Removed RSA-based version to ease maintenance, as it is
highly unlikely we would use it.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* formatting: use just one type section

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* more comments added

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* make `namelen` a const

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* generalize hashToG1

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* hashNameI: switch to faster implementation

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

Co-authored-by: Tanguy <tanguy@status.im>
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2022-03-08 15:04:42 -06:00
Eric Mastro 70cbdff901
feat: introduce LRU cache (#50)
* feat: introduce LRU cache

Replace `MemoryStore` with LRU caching mechanism.

`lrucache` library was forked to https://github.com/status-im/lrucache.nim.

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>

# Conflicts:
#	dagger/dagger.nim
#	dagger/stores.nim
#	dagger/stores/manager.nim
#	tests/dagger/blockexc/testengine.nim
#	tests/dagger/helpers/nodeutils.nim
#	tests/dagger/testnode.nim
#	tests/dagger/teststores.nim

* feat: introduce --cache-size CLI option

Allow for a value of `0` to disable the cache.

# Conflicts:
#	dagger/dagger.nim

* allow dynamic block size in cache

allow block size to be variable/dynamic

update lrucache to use updated lrucache dep

Using removeLru proc, added tests

* Refactor CacheStore init block

Co-authored-by: Michael Bradley, Jr <michaelsbradleyjr@gmail.com>
2022-03-02 10:30:42 -06:00
Eric Mastro 26ead9726d [rest api]: improve exception handling for `/connect` and add `content-type` header for `/download`
This PR achieves the following:
1. Improves the exception handling when dialling a peer fails or an unknown error occurs.
2. Add a `Content-Type` header to the `/download` endpoint of `application/octet-stream`, which is [defined by MDN](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types#types) as meant to be used for "generic binary data (or binary data whose true type is unknown)".

Co-authored-by: Michael Bradley <michaelsbradleyjr@gmail.com>
2022-02-04 15:38:39 +11:00
Eric Mastro 2e5c28781c feat: integrate dagger contracts
Integrate dagger contracts from `nim-dagger-contracts` repo.

Add `dagger-contracts`, `nim-web3`, and all of `nim-web3`’s transitive deps as submodule deps to `nim-dagger`. Note: `nim-web3` and its transitive deps may no longer be needed when we switch to `nim-ethers`.

Add a `testContracts` nimble task to test all of the contracts functionality. Namely, this spins up an ethereum simulator, deploys the contracts (in `dagger-contracts`), runs the contract tests, and finally, regardless of success/error, kills the ethereum sim processes. The nimble task can be run with `./env.sh nimble testContracts`.

We also tested `nim-dagger-contracts` as a submodule dep of `nim-dagger`, and while the tests run as expected, the preference is to merge `nim-dagger-contracts` inside of `nim-dagger` for ease of parallel development. There’s also a high probability that `nim-dagger-contracts` is not being used as a dep by other projects. Are there any strong objections to this?

Co-authored-by: Michael Bradley <michaelsbradleyjr@gmail.com>
2022-02-04 15:34:56 +11:00
Eric Mastro ec66e42e73
fix(rest): download unknown cid (#48)
Downloading a CID that is unknown to the network would wait for the blockexchange to try to get the data. However, a subsequent API call, eg upload for that CID, or a blockexchange timeout would cause the dagger node to crash, due to an attempt to complete a future that had already been completed.

We were able to narrow this down to an attempt to call  `resp.finish()` when the download block had already returned an `RestApiResponse` (and never sent any chunks to the response from the libp2p stream).

Change the HTTP response code for an unknown cid from 400 to 404.

Fix spelling mistake “cunk”.

Co-authored-by: Michael Bradley <michaelsbradleyjr@gmail.com>

Co-authored-by: Michael Bradley <michaelsbradleyjr@gmail.com>
2022-02-03 08:02:18 -06:00
Dmitriy Ryajov 5f48de6a44
cleanup manifest (#44)
* cleanup manifest

* Attach decode to `Manifest`

* move manifest container to it's own file

* rebase main

* update header license year
2022-01-12 19:55:51 -06:00
Dmitriy Ryajov 2188f69273
Minor cleanup and update deps (#42)
* abort download if atleast one block failed

* make chunker a regular object

* updating questionable

* updating vendor deps

* cleanup unused imports

* Remove mention of Waku
2022-01-12 16:19:37 -06:00
Dmitriy Ryajov 15ef6c78eb
fix treehash generation (#39) 2022-01-10 21:04:00 -06:00
Dmitriy Ryajov 47a0d4ef7a
Move empty digest (#38)
* improve empty digest initialization

* cleanup warnings
2022-01-10 20:25:27 -06:00
Dmitriy Ryajov a9721dd65f
Block.init should return Result (#37)
* Block.init should return Result

* use get() instead of tryGet()
2022-01-10 20:25:13 -06:00
Dmitriy Ryajov 9d17c91d4c
adding basic comments to rest api (#36)
* adding basic comments to rest api

* Update dagger/rest/api.nim

Co-authored-by: Michael Bradley <michaelsbradleyjr@gmail.com>

Co-authored-by: Michael Bradley <michaelsbradleyjr@gmail.com>
2022-01-10 14:35:52 -06:00
Dmitriy Ryajov fbe161a073
Node setup (#32)
* settup basic nim node

* adding http utils

* adding confutils

* rough rest api proto

* adding missing deps

* turn tls emulation off

* adding toml serialization

* wip

* adding missing deps

* make sure to clean old state in teardown

* adding file upload rest endpoint

* renaming blockexchange to networkstore

* updating nim-presto

* updating libp2p

* wip adding streaming upload

* reworked chunking

* bump to latest unstable

* adding asyncfutures stream

* make streamable

* deleting unused files

* reworking stores api

* use new stores api

* rework blockset and remove blockstream

* don't return option from constructor

* rework chunker

* wip implement upload

* fix tests

* move unrelated logic to engine

* don't print entire message

* logging

* basic encode/decode to/from dag-pb

* add basic upload/download support

* fix tests

* renaming blockset to manifest

* don't pass config to node

* remove config and use new manifest

* wip: make endpoints more reliable

* wip: adding node tests

* include correct manifest test

* removing asyncfutures

* proper chunking of files

* simplify stream reading

* test with encoding/decoding with many blocks

* add block storing tests

* adding retrieval test

* add logging

* tidy up chunker

* tidy up manifest and node

* use default chunk size

* fix tests

* fix tests

* make sure Eof is set properly

* wip

* minor cleanup

* add file utils

* cleanup config

* splitout DaggerServer and "main"

* remove events since they are not used

* add broadcast method to network peer

* add and wire localstore

* use localstore in the node

* wip

* logging

* move file utils

* use the constant

* updating deps

* fix memstore

* use latest libp2p unstable

* fix tests

* rework block streaming

* don't fail storing if the block already exists

* add helper info endpoint

* correct comment

* rename localstore to fsstore

* fix tests

* remove unused tests

* add test to retrieve one block

* move some test files around

* consolidate setup

* Update dagger/blockexchange/engine.nim

Co-authored-by: Tanguy <tanguy@status.im>

* typo

* better block path handling

* don't inherit rootobj

* remove useless template

* Update tests/dagger/blockexc/testblockexc.nim

Co-authored-by: markspanbroek <mark@spanbroek.net>

* use isMainModule

* use proper flag for starter/stoped

* cleanup optional use

* wrap in isMainModule

* use `cancelAndAwait`

* remove unused imports

* wip

* don't use optional

* use functional error api

* rework store tests and add fs tests

* Block.new() to Block.init()

* don't use optional for engine blocks

* use result instead of optional for getBlock

* remove unused imports

* move stopping servers to `shutdown`

* use result instead of optional

* rework with results

* fix tests

* use waitFor in signal handlers

* error helper

* use `?` and mapFailure where possible

* remove unnecesary `=?`

* improve empty cid digest initialization

Co-authored-by: Tanguy <tanguy@status.im>
Co-authored-by: markspanbroek <mark@spanbroek.net>
2022-01-10 09:32:56 -06:00
Dmitriy Ryajov 8522f516df remove unused file 2021-11-25 09:37:12 +01:00
Mark Spanbroek ef5fa96c7d Fix unused import warnings 2021-11-16 16:51:24 +01:00
Dmitriy Ryajov ce66e43440
fix compilation issues with latest libp2p (#27)
* fix compilation issues with latest libp2p

* change package description

* fix CI compiler

* try with nim 1.4.2

* try with 1.4.2

* use more standardized ci config

* set reasonable log level

* remove nim version dependency

* reverting to 1.4.6 as otherwise it fails on amd64
2021-10-29 13:30:52 -06:00
Dmitriy Ryajov 2fb39ca4a3
Rename bitswap (#25)
* use PeerInfo in event handlers

* use CidV1 and raw multicodec as default

* add block stream abstraction

* raises defect

* adding dataset abstraction

* move blockstream into own dir

* reorg files and fix tests

* rename dataset to blockset

* wip

* wip

* adding basic test for treehash algo

* run blockset tests along with with the rest

* remove obsolete contents

* fix chunker tests

* rename bitswap and move to stores

* rename bitwsap to blockexc and move to stores

* moare project structure reorg
2021-08-30 13:25:20 -06:00
Mark Spanbroek 6bd4260911 Replace pricing exchange by account exchange
Rationale: price is no longer set per peer, but per chunk.
Only the Ethereum accounts of the peers needs to be exchanged.
2021-06-07 09:50:17 +02:00
Mark Spanbroek d4dd15734e Remove double bookkeeping in peerHave and peerPrices 2021-06-07 09:50:17 +02:00
Mark Spanbroek e3b68f96d8 Remove debt ratio
Rationale: selecting peers based on debt ratio in bytes
is no longer needed when the bandwidth for each chunk is
compensated.
2021-06-07 09:50:17 +02:00
Mark Spanbroek fa05bffeac Pay per-block price instead of per-peer price 2021-06-07 09:50:17 +02:00
Mark Spanbroek f69827d582 Send block prices 2021-06-07 09:50:17 +02:00
Mark Spanbroek 98d8571ceb Simplify 2021-06-07 09:50:17 +02:00
Mark Spanbroek c57ecde046 Update to version 0.9.1 of questionable 2021-06-07 09:50:17 +02:00
Mark Spanbroek 0e34b69d7d Add block prices to peer context 2021-06-07 09:50:17 +02:00
Mark Spanbroek 0ac876dbc3 Add pricing to block presence messages 2021-06-07 09:50:17 +02:00
Mark Spanbroek 11da2339de Hard-code asset address
We're only going to support a single asset for now.
2021-06-07 09:50:17 +02:00
Mark Spanbroek 811b29fb5e Receive payments for blocks that were sent 2021-06-07 09:50:17 +02:00
Mark Spanbroek 5160b267b6 Use reference semantics for wallets 2021-06-07 09:50:17 +02:00
Mark Spanbroek 2d4a735e5f Update to latest versions of nitro and questionable 2021-06-07 09:50:17 +02:00
Mark Spanbroek 753bde7775 Fix build failure on 32 bit platforms 2021-06-07 09:50:17 +02:00
Mark Spanbroek ee978f354e Engine sends payments for received blocks 2021-06-07 09:50:17 +02:00
Mark Spanbroek 9512bbc50b Add proc to engine that pays peers for bytes 2021-06-07 09:50:17 +02:00
Mark Spanbroek d9e5bc934f Move peer context into its own module 2021-06-07 09:50:17 +02:00
Mark Spanbroek 4ce3f6d3da Add nitro wallet to BitswapEngine 2021-06-07 09:50:17 +02:00
Mark Spanbroek 22f4c277dd Revert "Add nitro wallet to BitswapNetwork"
This reverts commit e55ac4e9de.
2021-06-07 09:50:17 +02:00
Mark Spanbroek 6fbe37eb55 Remove dead code 2021-06-07 09:50:17 +02:00
Mark Spanbroek 3c4d4dc29f Exchange pricing when connecting to new peer 2021-06-07 09:50:17 +02:00
Mark Spanbroek 0897e615e6 Broadcast payment 2021-06-07 09:50:17 +02:00
Mark Spanbroek 6e7220efbb Broadcast pricing 2021-06-07 09:50:17 +02:00
Mark Spanbroek e02c40db80 Add handler for payment messages 2021-06-07 09:50:17 +02:00
Mark Spanbroek 5f9215a371 Add handler for pricing messages 2021-06-07 09:50:17 +02:00
Mark Spanbroek 02ef50ab46 Move pricing and payments into bitswap protobuf message
Only a single message type is forwarded to the rpc handler,
so we add pricing and payment info to that message type.
2021-06-07 09:50:17 +02:00
Mark Spanbroek c819ba1be1 protobuf message for state channel updates 2021-06-07 09:50:17 +02:00
Mark Spanbroek f60df42c88 Ensure that encoding of pricing doesn't raise 2021-06-07 09:50:17 +02:00
Mark Spanbroek 1f2a9f90d5 protobuf message for bandwidth pricing 2021-06-07 09:50:17 +02:00
Mark Spanbroek 8adea94196 Add nitro wallet to BitswapNetwork 2021-06-07 09:50:17 +02:00