Commit Graph

101 Commits

Author SHA1 Message Date
Tanguy 4d681102e5
Add DHT (#75)
* First implem

* Add persistent net key option

* Working DHT setup

* Bootstrap nodes

* Implement DaggerNode.findPeer

* Remove unrelevant comment

* Added discovery to blockexchange requestBlock

* add FSStore.blockList

* Block advertisement

* Tests compiles

* Green tests

* toDiscoveryId instead of toNodeId

* remove stopAdvertisingBlock

* Removed nim-eth dependency

* Move discovery stuff to discovery.nim

* Add missing file, start of discovery tests

* Better discovery logic

* Add tests

* Address comment

* Better E2E test
2022-04-13 10:32:35 -06:00
Tanguy bade020d67
Propper logging options (#81) 2022-04-12 15:21:07 +02:00
markspanbroek 1a3003e043
Update to latest dagger-contracts (#78)
- StorageRequest struct has changed

- StorageRequested event has changed,
  it no longer returns the entire request,
  but only the id and the ask
2022-04-11 12:03:55 -06:00
markspanbroek 03fa370624
Proving (#66)
* Add Proving object, which maintains contract id's to watch

* [proving] invoke callback when proof is required

# Conflicts:
#	dagger/por/timing/periods.nim
#	dagger/por/timing/prooftiming.nim

* [proving] check proof requirements for all our contracts

# Conflicts:
#	tests/dagger/helpers/mockprooftiming.nim

* Update vendor/dagger-contracts

* [proving] call onProofRequired() when proof is required soon

* [proving] stop checking contracts that have ended

* [proving] Remove duplicated funcs

* [proving] Implement ProofTiming on top of smart contract

* [proving] Fix race condition in waitUntilNextPeriod()

Sometimes waitUntilNextPeriod would take a while to
determine the current period, leading to unexpected results.

Splits waitUntilNextPeriod() into getCurrentPeriod()
and waitUntilPeriod(), to ensure that we're really waiting
for the period that we think we're waiting for.
2022-04-08 15:58:16 -06:00
Dmitriy Ryajov ffa9b624f1
Integrate erasure (#73)
* wip: adding request for storage endpoint

* wire in erasure coding

* fix tests for erasure coding

* put type definitions into separate file

* integrate erasure coding

* change run/shutdown to start/stop

* temporary sleep, otherwise the fsstore blocks
2022-04-05 18:34:29 -06:00
Dmitriy Ryajov 22c6705312
rename storageproofs to por (#74) 2022-04-05 14:44:53 -06:00
Dmitriy Ryajov 49c41e27b7
Erasure coding dataset (#72)
* add erasure coding support

* add leopard support

* erasure coding tests
2022-04-05 13:12:59 -06:00
Dmitriy Ryajov 70e4b2e5eb
handle empty blocks (#70) 2022-04-05 08:24:48 -06:00
Dmitriy Ryajov 8ef8cfcd74
add sample proc with exclude array (#68) 2022-04-05 08:24:39 -06:00
Dmitriy Ryajov 3222f436cc
Protected manifest (#69)
* add protected manifes type

* encode protected manifest

* add empty block type handling

* add tests for protected manifest
2022-04-04 18:46:13 -06:00
Mark Spanbroek befebcf325 [sales] Make storage available again when offer expires 2022-04-04 11:29:32 +02:00
Mark Spanbroek 55e326b467 [sales] Introduce Negotiation object 2022-04-04 11:29:32 +02:00
Mark Spanbroek e46c9816fa [sales] Make storage available again when offer declined 2022-04-04 11:29:32 +02:00
Mark Spanbroek 5d434f20d3 [sales] onSale() callback is optional 2022-04-04 11:29:32 +02:00
Mark Spanbroek 3810889fcb [sales] Do not call onSale() when someone else's offer is selected 2022-04-04 11:29:32 +02:00
Mark Spanbroek 71f25d40c7 [market] offerStorage() returns submitted offer 2022-04-04 11:29:32 +02:00
Mark Spanbroek 94e34e7d28 [market] requestStorage() returns submitted request 2022-04-04 11:29:32 +02:00
Mark Spanbroek 2ffde32772 [sales] call onSale() when offer has been selected 2022-04-04 11:29:32 +02:00
Mark Spanbroek 71baedb5ec [sales] Set offer expiry timestamp 2022-04-04 11:29:32 +02:00
Mark Spanbroek 76be728f61 [sales] Availability.new -> Availability.init 2022-04-04 11:29:32 +02:00
Mark Spanbroek dd89a55351 [market] Add Market.getTime() 2022-04-04 11:29:32 +02:00
Mark Spanbroek 061b32296a Introduce Sales 2022-04-04 11:29:32 +02:00
Mark Spanbroek 75ec8c0bfd [market] wait until request/offer expires 2022-04-04 10:03:46 +02:00
Mark Spanbroek 8ff748eff8 [market] select a storage offer 2022-04-04 10:03:46 +02:00
Mark Spanbroek 5ea64522d8 [market] subscribe to offers for a certain request 2022-04-04 10:03:46 +02:00
Mark Spanbroek 9f76e714a2 [market] submit storage offers 2022-04-04 10:03:46 +02:00
Mark Spanbroek e52153e2ba [market] Require a storage contract with a signer 2022-04-04 10:03:46 +02:00
Mark Spanbroek e78a0fb43e [contracts] Wrap contracts in Market abstraction 2022-04-04 10:03:46 +02:00
Mark Spanbroek 5b5f3335d6 [purchasing] Ignore offers that are about to expire 2022-04-04 10:03:46 +02:00
Mark Spanbroek dcfd6be1c6 [purchasing] Ignore expired offers 2022-04-04 10:03:46 +02:00
Mark Spanbroek fe23cb89d7 [purchasing] Select cheapest offer 2022-04-04 10:03:46 +02:00
Mark Spanbroek 03140fdf49 [purchasing] Add state to track purchase progress 2022-04-04 10:03:46 +02:00
Mark Spanbroek 7c9c244836 [purchasing] Remove PurchaseRequest; use StorageRequest 2022-04-04 10:03:46 +02:00
Mark Spanbroek 9ade3fdd37 [contracts] StorageRequest/Offer is object instead of tuple 2022-04-04 10:03:46 +02:00
Mark Spanbroek 5f10549f19 Introduce Purchasing and Market 2022-04-04 10:03:46 +02:00
Mark Spanbroek a3fdd35f73 [contracts] Split marketplace.nim into requests.nim and offers.nim 2022-04-04 10:03:46 +02:00
Dmitriy Ryajov 43cea1743a
Node storestream (#62)
* use storestream for retrieval

* use `len` prop on manifest

* remove read pattern

* make blocksize and chunksize consistent

* fix tests

* fix chunker tests - now padded by default
2022-03-29 20:43:35 -06:00
Mark Spanbroek 28294d431a [contracts] Tests now also work in Windows 2022-03-28 11:59:01 +02:00
Mark Spanbroek b8e013e439 [contracts] Add Readme 2022-03-28 11:59:01 +02:00
Mark Spanbroek efb4f5c375 [contracts] Update to new marketplace design 2022-03-28 11:59:01 +02:00
Mark Spanbroek 87bd947d3d [contracts] Update to new collateral design 2022-03-28 11:59:01 +02:00
Mark Spanbroek 8d0f9775b5 [contracts] Simplify hardhat node startup 2022-03-28 11:59:01 +02:00
Dmitriy Ryajov e965f5e0de
Seekable stream (#58)
* cleanup manifest

* adding a "seekable" base stream
2022-03-21 12:09:59 -06:00
Dmitriy Ryajov 2c4b1e6906
Upraises (#57)
* cleanup manifest

* make block a ref type

* use upraises

* more upraises
2022-03-18 16:17:51 -06:00
Dmitriy Ryajov d204b518e8
Make block a ref (#56)
* cleanup manifest

* make block a ref type
2022-03-18 13:50:53 -06:00
Dmitriy Ryajov 472e1e6fc3
Store stream (#55)
* make manifest a ref object

* capitalize tests

* add storestream and tests

* fix tests

* run storestream tests
2022-03-17 07:56:46 -06:00
Dmitriy Ryajov f2aa945991
Manifest block size (#53)
* add `blockSize` and `size` to manifest

* fix tests

* fix decoding/encoding

* re-export coders

* style
2022-03-15 12:47:31 -06:00
Dmitriy Ryajov f414c695be
Cleanup-manifest (#52)
* adding nim-blscurve

* adding taskpools

* cleanup manifest
2022-03-14 10:06:36 -06:00
Csaba Kiraly d8162b1e16
initial commit of the Shacham BLS-based and RSA-based public schemes (#26)
* initial commit of the Shacham RSA-based public scheme

Minimal working version with lots of error checks and corrections
still needed.
- using Bearssl RSA code through libp2p
- with selecteble BigInt library for experimentation

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* better proc names

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* separating demo code from library

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* using normal file io instead of memfiles

mmap has serveral potential issues and we do not really need it, so
changing to use the normal system file interface is better.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* draft version of bls proofs

Implementation of the BLS-based public PoS scheme from
Shacham H., Waters B., "Compact Proofs of Retrievability"
using pairing over BLS12-381 ECC

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* draft test and benchmark code for BLS PoS

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* improve documentation of BLS scheme

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* fix getSector

* fixing DST tag in hashToG1

The DST tag should be unique to achieve domain separation
of hash functions as defined in:
https://tools.ietf.org/id/draft-irtf-cfrg-hash-to-curve-06.html#domain-separation

Changed DST tag to one that indicates the PoC status of this code.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* add verifyPairings abstraction

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* change random number generator to a secure one

Use Rng based on BrHmacDrbgContext

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* fix benchmark template

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* exchange parameter order in pairing

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* add optimized verifyPairing implementation

When verifying two pairings, one final exponentiation
can be spared through the use of cneg.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* speed up tag generation by a factor of s

Scalar multiplications in tag generation can be rearranged
to benefit from the way random points are being generated.
Since random points are themselves generated using scalar
multiplication and the base is common, the sum of multiplications
becomes a single multiplication with the scalar sum, resulting in
a nice speedup.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* use blst_p1_add_or_double instead of blst_p1_add

blst exposes two add functions: one that works for the corner case
of doubling, and one that isn't. It seems safer to use the one that
works, even if it is highly improbable in these cases that doubling
would occur.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* sectorsperblock should be an external parameter

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* parametrize sectorsblock and querylen

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* improving benchmark messages

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* rebasing main

* generateAuthenticator: remove unused ubase parameter from naive impl

No need to have the same interface on the two implementations, so
we can remove this parameter.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* generateAuthenticator: add some more explanation

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* renaming pos.nim to rsa.nim

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* sign and verify metadata in Tau

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* adding more comments

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* remove code of slow RSA based version

Removed RSA-based version to ease maintenance, as it is
highly unlikely we would use it.

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* formatting: use just one type section

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* more comments added

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* make `namelen` a const

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* generalize hashToG1

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* hashNameI: switch to faster implementation

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

Co-authored-by: Tanguy <tanguy@status.im>
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2022-03-08 15:04:42 -06:00
Eric Mastro 70cbdff901
feat: introduce LRU cache (#50)
* feat: introduce LRU cache

Replace `MemoryStore` with LRU caching mechanism.

`lrucache` library was forked to https://github.com/status-im/lrucache.nim.

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>

# Conflicts:
#	dagger/dagger.nim
#	dagger/stores.nim
#	dagger/stores/manager.nim
#	tests/dagger/blockexc/testengine.nim
#	tests/dagger/helpers/nodeutils.nim
#	tests/dagger/testnode.nim
#	tests/dagger/teststores.nim

* feat: introduce --cache-size CLI option

Allow for a value of `0` to disable the cache.

# Conflicts:
#	dagger/dagger.nim

* allow dynamic block size in cache

allow block size to be variable/dynamic

update lrucache to use updated lrucache dep

Using removeLru proc, added tests

* Refactor CacheStore init block

Co-authored-by: Michael Bradley, Jr <michaelsbradleyjr@gmail.com>
2022-03-02 10:30:42 -06:00