Commit Graph

85 Commits

Author SHA1 Message Date
Dmitriy Ryajov 46197f957b
add onblock callback to list blocks visitor (#96)
* add onblock callback to list blocks visitor

* tests for listBlocks

* remove legacy blockList
2022-05-12 14:02:30 -06:00
markspanbroek d8ed4257df
[contracts] Make contract interactions optional in the node (#91)
Adds --eth-deployment parameter.
Does not crash when deployment json is missing or incorrect.
2022-04-25 15:12:37 +02:00
Mark Spanbroek 59f75fc540 [contracts] Start contract interactions when node starts 2022-04-21 14:04:07 +02:00
Mark Spanbroek 0738606303 [contracts] Add ContractInteractions to dagger node 2022-04-21 14:04:07 +02:00
Tanguy ce59dbd4a2
Revert "Cleanup engine and rework discovery (#87)"
This reverts commit 4740ffc144.
2022-04-20 14:28:11 +02:00
Dmitriy Ryajov 4740ffc144
Cleanup engine and rework discovery (#87)
* rework discovery with async queues

* misc style changes

* increase max message size for large manifests

* use upraises and avoid exceptions on key access

* increase sleep time to 100 millis

* pass config

* make list blocks trigger a callback on each block

* check for nil on start/stop

* fix tests and split out discovery tests

* don't auto mount network

* add list block tests

* add discovery tests

* rework moc discovery

* move discovery moc to disc dir

* don't force logging syncs

* don't force moc discovery on all tests

* rework discovery with methods

* add top level utils file

* don't use asyncCheck

* don't pass entire blocks to list blocks calback

* spelling
2022-04-19 21:46:44 -06:00
markspanbroek b88561e090
Subscribe to proof submissions (#83)
* Update dagger-contracts

* [proving] rename ProofTiming -> Proofs

* Update nim-ethers to 0.1.4

* [proving] Subscribe to proof submissions

* [proving] support proof submission through the Proving abstraction
2022-04-13 10:41:48 -06:00
Tanguy 4d681102e5
Add DHT (#75)
* First implem

* Add persistent net key option

* Working DHT setup

* Bootstrap nodes

* Implement DaggerNode.findPeer

* Remove unrelevant comment

* Added discovery to blockexchange requestBlock

* add FSStore.blockList

* Block advertisement

* Tests compiles

* Green tests

* toDiscoveryId instead of toNodeId

* remove stopAdvertisingBlock

* Removed nim-eth dependency

* Move discovery stuff to discovery.nim

* Add missing file, start of discovery tests

* Better discovery logic

* Add tests

* Address comment

* Better E2E test
2022-04-13 10:32:35 -06:00
markspanbroek 1a3003e043
Update to latest dagger-contracts (#78)
- StorageRequest struct has changed

- StorageRequested event has changed,
  it no longer returns the entire request,
  but only the id and the ask
2022-04-11 12:03:55 -06:00
markspanbroek 03fa370624
Proving (#66)
* Add Proving object, which maintains contract id's to watch

* [proving] invoke callback when proof is required

# Conflicts:
#	dagger/por/timing/periods.nim
#	dagger/por/timing/prooftiming.nim

* [proving] check proof requirements for all our contracts

# Conflicts:
#	tests/dagger/helpers/mockprooftiming.nim

* Update vendor/dagger-contracts

* [proving] call onProofRequired() when proof is required soon

* [proving] stop checking contracts that have ended

* [proving] Remove duplicated funcs

* [proving] Implement ProofTiming on top of smart contract

* [proving] Fix race condition in waitUntilNextPeriod()

Sometimes waitUntilNextPeriod would take a while to
determine the current period, leading to unexpected results.

Splits waitUntilNextPeriod() into getCurrentPeriod()
and waitUntilPeriod(), to ensure that we're really waiting
for the period that we think we're waiting for.
2022-04-08 15:58:16 -06:00
Dmitriy Ryajov 2e8b39cf7c
fix random failing test (#77) 2022-04-07 17:08:43 -06:00
Dmitriy Ryajov ffa9b624f1
Integrate erasure (#73)
* wip: adding request for storage endpoint

* wire in erasure coding

* fix tests for erasure coding

* put type definitions into separate file

* integrate erasure coding

* change run/shutdown to start/stop

* temporary sleep, otherwise the fsstore blocks
2022-04-05 18:34:29 -06:00
Dmitriy Ryajov 49c41e27b7
Erasure coding dataset (#72)
* add erasure coding support

* add leopard support

* erasure coding tests
2022-04-05 13:12:59 -06:00
Dmitriy Ryajov 3222f436cc
Protected manifest (#69)
* add protected manifes type

* encode protected manifest

* add empty block type handling

* add tests for protected manifest
2022-04-04 18:46:13 -06:00
Mark Spanbroek befebcf325 [sales] Make storage available again when offer expires 2022-04-04 11:29:32 +02:00
Mark Spanbroek e46c9816fa [sales] Make storage available again when offer declined 2022-04-04 11:29:32 +02:00
Mark Spanbroek 11f7b4b4d8 [sales] Clean up tests 2022-04-04 11:29:32 +02:00
Mark Spanbroek 3810889fcb [sales] Do not call onSale() when someone else's offer is selected 2022-04-04 11:29:32 +02:00
Mark Spanbroek 71f25d40c7 [market] offerStorage() returns submitted offer 2022-04-04 11:29:32 +02:00
Mark Spanbroek 94e34e7d28 [market] requestStorage() returns submitted request 2022-04-04 11:29:32 +02:00
Mark Spanbroek 2ffde32772 [sales] call onSale() when offer has been selected 2022-04-04 11:29:32 +02:00
Mark Spanbroek 71baedb5ec [sales] Set offer expiry timestamp 2022-04-04 11:29:32 +02:00
Mark Spanbroek 76be728f61 [sales] Availability.new -> Availability.init 2022-04-04 11:29:32 +02:00
Mark Spanbroek dd89a55351 [market] Add Market.getTime() 2022-04-04 11:29:32 +02:00
Mark Spanbroek 061b32296a Introduce Sales 2022-04-04 11:29:32 +02:00
Mark Spanbroek bd8f4d5d74 [tests] Extract common examples into separate module 2022-04-04 10:03:46 +02:00
Mark Spanbroek 5b5f3335d6 [purchasing] Ignore offers that are about to expire 2022-04-04 10:03:46 +02:00
Mark Spanbroek dcfd6be1c6 [purchasing] Ignore expired offers 2022-04-04 10:03:46 +02:00
Mark Spanbroek fe23cb89d7 [purchasing] Select cheapest offer 2022-04-04 10:03:46 +02:00
Mark Spanbroek 04c291b05c [purchasing] rename requests -> requested 2022-04-04 10:03:46 +02:00
Mark Spanbroek 7c9c244836 [purchasing] Remove PurchaseRequest; use StorageRequest 2022-04-04 10:03:46 +02:00
Mark Spanbroek 5f10549f19 Introduce Purchasing and Market 2022-04-04 10:03:46 +02:00
Dmitriy Ryajov 43cea1743a
Node storestream (#62)
* use storestream for retrieval

* use `len` prop on manifest

* remove read pattern

* make blocksize and chunksize consistent

* fix tests

* fix chunker tests - now padded by default
2022-03-29 20:43:35 -06:00
Dmitriy Ryajov e965f5e0de
Seekable stream (#58)
* cleanup manifest

* adding a "seekable" base stream
2022-03-21 12:09:59 -06:00
Dmitriy Ryajov d204b518e8
Make block a ref (#56)
* cleanup manifest

* make block a ref type
2022-03-18 13:50:53 -06:00
Dmitriy Ryajov 472e1e6fc3
Store stream (#55)
* make manifest a ref object

* capitalize tests

* add storestream and tests

* fix tests

* run storestream tests
2022-03-17 07:56:46 -06:00
Dmitriy Ryajov f2aa945991
Manifest block size (#53)
* add `blockSize` and `size` to manifest

* fix tests

* fix decoding/encoding

* re-export coders

* style
2022-03-15 12:47:31 -06:00
Dmitriy Ryajov f414c695be
Cleanup-manifest (#52)
* adding nim-blscurve

* adding taskpools

* cleanup manifest
2022-03-14 10:06:36 -06:00
Eric Mastro 70cbdff901
feat: introduce LRU cache (#50)
* feat: introduce LRU cache

Replace `MemoryStore` with LRU caching mechanism.

`lrucache` library was forked to https://github.com/status-im/lrucache.nim.

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>

# Conflicts:
#	dagger/dagger.nim
#	dagger/stores.nim
#	dagger/stores/manager.nim
#	tests/dagger/blockexc/testengine.nim
#	tests/dagger/helpers/nodeutils.nim
#	tests/dagger/testnode.nim
#	tests/dagger/teststores.nim

* feat: introduce --cache-size CLI option

Allow for a value of `0` to disable the cache.

# Conflicts:
#	dagger/dagger.nim

* allow dynamic block size in cache

allow block size to be variable/dynamic

update lrucache to use updated lrucache dep

Using removeLru proc, added tests

* Refactor CacheStore init block

Co-authored-by: Michael Bradley, Jr <michaelsbradleyjr@gmail.com>
2022-03-02 10:30:42 -06:00
Dmitriy Ryajov 5f48de6a44
cleanup manifest (#44)
* cleanup manifest

* Attach decode to `Manifest`

* move manifest container to it's own file

* rebase main

* update header license year
2022-01-12 19:55:51 -06:00
Dmitriy Ryajov 68a45d8f57
use tryGet, which is more informative than get (#45) 2022-01-12 18:42:18 -06:00
Dmitriy Ryajov 2188f69273
Minor cleanup and update deps (#42)
* abort download if atleast one block failed

* make chunker a regular object

* updating questionable

* updating vendor deps

* cleanup unused imports

* Remove mention of Waku
2022-01-12 16:19:37 -06:00
Dmitriy Ryajov 15ef6c78eb
fix treehash generation (#39) 2022-01-10 21:04:00 -06:00
Dmitriy Ryajov 47a0d4ef7a
Move empty digest (#38)
* improve empty digest initialization

* cleanup warnings
2022-01-10 20:25:27 -06:00
Dmitriy Ryajov a9721dd65f
Block.init should return Result (#37)
* Block.init should return Result

* use get() instead of tryGet()
2022-01-10 20:25:13 -06:00
Dmitriy Ryajov fbe161a073
Node setup (#32)
* settup basic nim node

* adding http utils

* adding confutils

* rough rest api proto

* adding missing deps

* turn tls emulation off

* adding toml serialization

* wip

* adding missing deps

* make sure to clean old state in teardown

* adding file upload rest endpoint

* renaming blockexchange to networkstore

* updating nim-presto

* updating libp2p

* wip adding streaming upload

* reworked chunking

* bump to latest unstable

* adding asyncfutures stream

* make streamable

* deleting unused files

* reworking stores api

* use new stores api

* rework blockset and remove blockstream

* don't return option from constructor

* rework chunker

* wip implement upload

* fix tests

* move unrelated logic to engine

* don't print entire message

* logging

* basic encode/decode to/from dag-pb

* add basic upload/download support

* fix tests

* renaming blockset to manifest

* don't pass config to node

* remove config and use new manifest

* wip: make endpoints more reliable

* wip: adding node tests

* include correct manifest test

* removing asyncfutures

* proper chunking of files

* simplify stream reading

* test with encoding/decoding with many blocks

* add block storing tests

* adding retrieval test

* add logging

* tidy up chunker

* tidy up manifest and node

* use default chunk size

* fix tests

* fix tests

* make sure Eof is set properly

* wip

* minor cleanup

* add file utils

* cleanup config

* splitout DaggerServer and "main"

* remove events since they are not used

* add broadcast method to network peer

* add and wire localstore

* use localstore in the node

* wip

* logging

* move file utils

* use the constant

* updating deps

* fix memstore

* use latest libp2p unstable

* fix tests

* rework block streaming

* don't fail storing if the block already exists

* add helper info endpoint

* correct comment

* rename localstore to fsstore

* fix tests

* remove unused tests

* add test to retrieve one block

* move some test files around

* consolidate setup

* Update dagger/blockexchange/engine.nim

Co-authored-by: Tanguy <tanguy@status.im>

* typo

* better block path handling

* don't inherit rootobj

* remove useless template

* Update tests/dagger/blockexc/testblockexc.nim

Co-authored-by: markspanbroek <mark@spanbroek.net>

* use isMainModule

* use proper flag for starter/stoped

* cleanup optional use

* wrap in isMainModule

* use `cancelAndAwait`

* remove unused imports

* wip

* don't use optional

* use functional error api

* rework store tests and add fs tests

* Block.new() to Block.init()

* don't use optional for engine blocks

* use result instead of optional for getBlock

* remove unused imports

* move stopping servers to `shutdown`

* use result instead of optional

* rework with results

* fix tests

* use waitFor in signal handlers

* error helper

* use `?` and mapFailure where possible

* remove unnecesary `=?`

* improve empty cid digest initialization

Co-authored-by: Tanguy <tanguy@status.im>
Co-authored-by: markspanbroek <mark@spanbroek.net>
2022-01-10 09:32:56 -06:00
Mark Spanbroek 90a6131d7d Fix occasional test failure due to timeout 2021-11-16 16:51:40 +01:00
Mark Spanbroek ef5fa96c7d Fix unused import warnings 2021-11-16 16:51:24 +01:00
Dmitriy Ryajov ce66e43440
fix compilation issues with latest libp2p (#27)
* fix compilation issues with latest libp2p

* change package description

* fix CI compiler

* try with nim 1.4.2

* try with 1.4.2

* use more standardized ci config

* set reasonable log level

* remove nim version dependency

* reverting to 1.4.6 as otherwise it fails on amd64
2021-10-29 13:30:52 -06:00
Dmitriy Ryajov 2fb39ca4a3
Rename bitswap (#25)
* use PeerInfo in event handlers

* use CidV1 and raw multicodec as default

* add block stream abstraction

* raises defect

* adding dataset abstraction

* move blockstream into own dir

* reorg files and fix tests

* rename dataset to blockset

* wip

* wip

* adding basic test for treehash algo

* run blockset tests along with with the rest

* remove obsolete contents

* fix chunker tests

* rename bitswap and move to stores

* rename bitwsap to blockexc and move to stores

* moare project structure reorg
2021-08-30 13:25:20 -06:00