Commit Graph

364 Commits

Author SHA1 Message Date
Eric Mastro 70cbdff901
feat: introduce LRU cache (#50)
* feat: introduce LRU cache

Replace `MemoryStore` with LRU caching mechanism.

`lrucache` library was forked to https://github.com/status-im/lrucache.nim.

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>

# Conflicts:
#	dagger/dagger.nim
#	dagger/stores.nim
#	dagger/stores/manager.nim
#	tests/dagger/blockexc/testengine.nim
#	tests/dagger/helpers/nodeutils.nim
#	tests/dagger/testnode.nim
#	tests/dagger/teststores.nim

* feat: introduce --cache-size CLI option

Allow for a value of `0` to disable the cache.

# Conflicts:
#	dagger/dagger.nim

* allow dynamic block size in cache

allow block size to be variable/dynamic

update lrucache to use updated lrucache dep

Using removeLru proc, added tests

* Refactor CacheStore init block

Co-authored-by: Michael Bradley, Jr <michaelsbradleyjr@gmail.com>
2022-03-02 10:30:42 -06:00
Eric Mastro 2e5c28781c feat: integrate dagger contracts
Integrate dagger contracts from `nim-dagger-contracts` repo.

Add `dagger-contracts`, `nim-web3`, and all of `nim-web3`’s transitive deps as submodule deps to `nim-dagger`. Note: `nim-web3` and its transitive deps may no longer be needed when we switch to `nim-ethers`.

Add a `testContracts` nimble task to test all of the contracts functionality. Namely, this spins up an ethereum simulator, deploys the contracts (in `dagger-contracts`), runs the contract tests, and finally, regardless of success/error, kills the ethereum sim processes. The nimble task can be run with `./env.sh nimble testContracts`.

We also tested `nim-dagger-contracts` as a submodule dep of `nim-dagger`, and while the tests run as expected, the preference is to merge `nim-dagger-contracts` inside of `nim-dagger` for ease of parallel development. There’s also a high probability that `nim-dagger-contracts` is not being used as a dep by other projects. Are there any strong objections to this?

Co-authored-by: Michael Bradley <michaelsbradleyjr@gmail.com>
2022-02-04 15:34:56 +11:00
Dmitriy Ryajov 5f48de6a44
cleanup manifest (#44)
* cleanup manifest

* Attach decode to `Manifest`

* move manifest container to it's own file

* rebase main

* update header license year
2022-01-12 19:55:51 -06:00
Dmitriy Ryajov 68a45d8f57
use tryGet, which is more informative than get (#45) 2022-01-12 18:42:18 -06:00
Dmitriy Ryajov 2188f69273
Minor cleanup and update deps (#42)
* abort download if atleast one block failed

* make chunker a regular object

* updating questionable

* updating vendor deps

* cleanup unused imports

* Remove mention of Waku
2022-01-12 16:19:37 -06:00
Dmitriy Ryajov 15ef6c78eb
fix treehash generation (#39) 2022-01-10 21:04:00 -06:00
Dmitriy Ryajov 47a0d4ef7a
Move empty digest (#38)
* improve empty digest initialization

* cleanup warnings
2022-01-10 20:25:27 -06:00
Dmitriy Ryajov a9721dd65f
Block.init should return Result (#37)
* Block.init should return Result

* use get() instead of tryGet()
2022-01-10 20:25:13 -06:00
Dmitriy Ryajov fbe161a073
Node setup (#32)
* settup basic nim node

* adding http utils

* adding confutils

* rough rest api proto

* adding missing deps

* turn tls emulation off

* adding toml serialization

* wip

* adding missing deps

* make sure to clean old state in teardown

* adding file upload rest endpoint

* renaming blockexchange to networkstore

* updating nim-presto

* updating libp2p

* wip adding streaming upload

* reworked chunking

* bump to latest unstable

* adding asyncfutures stream

* make streamable

* deleting unused files

* reworking stores api

* use new stores api

* rework blockset and remove blockstream

* don't return option from constructor

* rework chunker

* wip implement upload

* fix tests

* move unrelated logic to engine

* don't print entire message

* logging

* basic encode/decode to/from dag-pb

* add basic upload/download support

* fix tests

* renaming blockset to manifest

* don't pass config to node

* remove config and use new manifest

* wip: make endpoints more reliable

* wip: adding node tests

* include correct manifest test

* removing asyncfutures

* proper chunking of files

* simplify stream reading

* test with encoding/decoding with many blocks

* add block storing tests

* adding retrieval test

* add logging

* tidy up chunker

* tidy up manifest and node

* use default chunk size

* fix tests

* fix tests

* make sure Eof is set properly

* wip

* minor cleanup

* add file utils

* cleanup config

* splitout DaggerServer and "main"

* remove events since they are not used

* add broadcast method to network peer

* add and wire localstore

* use localstore in the node

* wip

* logging

* move file utils

* use the constant

* updating deps

* fix memstore

* use latest libp2p unstable

* fix tests

* rework block streaming

* don't fail storing if the block already exists

* add helper info endpoint

* correct comment

* rename localstore to fsstore

* fix tests

* remove unused tests

* add test to retrieve one block

* move some test files around

* consolidate setup

* Update dagger/blockexchange/engine.nim

Co-authored-by: Tanguy <tanguy@status.im>

* typo

* better block path handling

* don't inherit rootobj

* remove useless template

* Update tests/dagger/blockexc/testblockexc.nim

Co-authored-by: markspanbroek <mark@spanbroek.net>

* use isMainModule

* use proper flag for starter/stoped

* cleanup optional use

* wrap in isMainModule

* use `cancelAndAwait`

* remove unused imports

* wip

* don't use optional

* use functional error api

* rework store tests and add fs tests

* Block.new() to Block.init()

* don't use optional for engine blocks

* use result instead of optional for getBlock

* remove unused imports

* move stopping servers to `shutdown`

* use result instead of optional

* rework with results

* fix tests

* use waitFor in signal handlers

* error helper

* use `?` and mapFailure where possible

* remove unnecesary `=?`

* improve empty cid digest initialization

Co-authored-by: Tanguy <tanguy@status.im>
Co-authored-by: markspanbroek <mark@spanbroek.net>
2022-01-10 09:32:56 -06:00
Mark Spanbroek 90a6131d7d Fix occasional test failure due to timeout 2021-11-16 16:51:40 +01:00
Mark Spanbroek ef5fa96c7d Fix unused import warnings 2021-11-16 16:51:24 +01:00
Dmitriy Ryajov ce66e43440
fix compilation issues with latest libp2p (#27)
* fix compilation issues with latest libp2p

* change package description

* fix CI compiler

* try with nim 1.4.2

* try with 1.4.2

* use more standardized ci config

* set reasonable log level

* remove nim version dependency

* reverting to 1.4.6 as otherwise it fails on amd64
2021-10-29 13:30:52 -06:00
Dmitriy Ryajov 2fb39ca4a3
Rename bitswap (#25)
* use PeerInfo in event handlers

* use CidV1 and raw multicodec as default

* add block stream abstraction

* raises defect

* adding dataset abstraction

* move blockstream into own dir

* reorg files and fix tests

* rename dataset to blockset

* wip

* wip

* adding basic test for treehash algo

* run blockset tests along with with the rest

* remove obsolete contents

* fix chunker tests

* rename bitswap and move to stores

* rename bitwsap to blockexc and move to stores

* moare project structure reorg
2021-08-30 13:25:20 -06:00
Mark Spanbroek 6bd4260911 Replace pricing exchange by account exchange
Rationale: price is no longer set per peer, but per chunk.
Only the Ethereum accounts of the peers needs to be exchanged.
2021-06-07 09:50:17 +02:00
Mark Spanbroek d4dd15734e Remove double bookkeeping in peerHave and peerPrices 2021-06-07 09:50:17 +02:00
Mark Spanbroek e3b68f96d8 Remove debt ratio
Rationale: selecting peers based on debt ratio in bytes
is no longer needed when the bandwidth for each chunk is
compensated.
2021-06-07 09:50:17 +02:00
Mark Spanbroek fa05bffeac Pay per-block price instead of per-peer price 2021-06-07 09:50:17 +02:00
Mark Spanbroek f69827d582 Send block prices 2021-06-07 09:50:17 +02:00
Mark Spanbroek 0a28338136 Simplify test 2021-06-07 09:50:17 +02:00
Mark Spanbroek 0e34b69d7d Add block prices to peer context 2021-06-07 09:50:17 +02:00
Mark Spanbroek 0ac876dbc3 Add pricing to block presence messages 2021-06-07 09:50:17 +02:00
Mark Spanbroek 11da2339de Hard-code asset address
We're only going to support a single asset for now.
2021-06-07 09:50:17 +02:00
Mark Spanbroek 811b29fb5e Receive payments for blocks that were sent 2021-06-07 09:50:17 +02:00
Mark Spanbroek 5160b267b6 Use reference semantics for wallets 2021-06-07 09:50:17 +02:00
Mark Spanbroek 2d4a735e5f Update to latest versions of nitro and questionable 2021-06-07 09:50:17 +02:00
Mark Spanbroek ee978f354e Engine sends payments for received blocks 2021-06-07 09:50:17 +02:00
Mark Spanbroek 9512bbc50b Add proc to engine that pays peers for bytes 2021-06-07 09:50:17 +02:00
Mark Spanbroek ad48f55aaa Update to latest versions of nitro and questionable 2021-06-07 09:50:17 +02:00
Mark Spanbroek 4ce3f6d3da Add nitro wallet to BitswapEngine 2021-06-07 09:50:17 +02:00
Mark Spanbroek 22f4c277dd Revert "Add nitro wallet to BitswapNetwork"
This reverts commit e55ac4e9de.
2021-06-07 09:50:17 +02:00
Mark Spanbroek 3c4d4dc29f Exchange pricing when connecting to new peer 2021-06-07 09:50:17 +02:00
Mark Spanbroek 0897e615e6 Broadcast payment 2021-06-07 09:50:17 +02:00
Mark Spanbroek 6e7220efbb Broadcast pricing 2021-06-07 09:50:17 +02:00
Mark Spanbroek e02c40db80 Add handler for payment messages 2021-06-07 09:50:17 +02:00
Mark Spanbroek 5f9215a371 Add handler for pricing messages 2021-06-07 09:50:17 +02:00
Mark Spanbroek 364d3a9277 Fix typo 2021-06-07 09:50:17 +02:00
Mark Spanbroek 9635f26b90 Fix tests on 32 bit platforms 2021-06-07 09:50:17 +02:00
Mark Spanbroek c819ba1be1 protobuf message for state channel updates 2021-06-07 09:50:17 +02:00
Mark Spanbroek 1f2a9f90d5 protobuf message for bandwidth pricing 2021-06-07 09:50:17 +02:00
Mark Spanbroek 8adea94196 Add nitro wallet to BitswapNetwork 2021-06-07 09:50:17 +02:00
Mark Spanbroek 5cd714ea83 Remove deprecated libp2p import 2021-06-07 09:50:17 +02:00
Mark Spanbroek 339f7f04bc Fix warnings 2021-06-07 09:50:17 +02:00
Mark Spanbroek 5bb87c4ecc Add missing test 2021-06-07 09:50:17 +02:00
Dmitriy Ryajov 6c92b3dc25
Poc 2 (#7)
* moving protobuf into bitswap

* adding block type

* reworking bitswap

* adding chunker

* adding license header

* use 1.2.6

* adding fixed size chunker

* add blockstore

* add iterator to chunker

* more bitswap changes

* rename ipfs to dagger

* rename to dagger

* blockstore inherits from BlockProvider

* wip - add core block handling logic

* smal changes

* use proper block store methods

* adding asynq heapqueue

* wip prepare for bitswap task runner

* adding `$`

* adding memory store and tests

* fixed chunking

* extracted desicion engine from bitswap

* added helper random funcs

* adding testing helpers

* only handle seqs

* add peer events

* cleanup pending blocks on blockstore event

* allow nil handlers

* move protobuf type helpers

* allow initializing block from Cid

* testing and fixes

* small fixes

* expose `<`

* spelling

* default value

* spelling

* pending blocks manager

* adding stores manager

* more tests a wip around bitswap

* small changes

* merge bitswap and engine for now

* for now run only the new poc's tests

* add a more complete ci setup

* use template in map

* remove p2pd

* remove go

* dont use asyncCheck

* few small changes

* adding ability to update items

* adding multiple task runners

* handle cancelation properly

* use Result instead of throwing

* wip bitswap tests

* moving things around

* split out engine again

* add request and handlers interface

* fix tests

* wip - engine tests

* remove unused imports

* fix tests

* cleanup block requesting logic

* add block request tests

* more block requests

* add support for max heap

* don't use result

* use max heap & send block presence in task handler

* add task handler tests

* rename store to localStore

* cleanup & logging

* cancel task on stop

* don't depend on local store for events

* dont use heap queue for wants

* add chronicles

* fix issue with peer wants

* add test for delayed block sends

* remove obsolete tests

* wip chunker

* run all tests

* add todo

* misc

* remove irrelevant files

* removing more files

* adding helpers for bitswap tests

* moved bitswap file

* misc

* make blocks timeout longer

* adjust block timeout

* speedup test

* compile with threads

* import missing crypto

* misc

* disable threads for now

* fix 32 bit platforms

* re-enable threads support in tests
2021-02-25 18:23:22 -06:00
Mark Spanbroek 8e76ecfa9f Fix: adapt to latest libp2p api
Fixes compilation error.
2021-02-08 14:39:27 +01:00
Mark Spanbroek b0a1166c3c Bitswap: retrieve() waits until IPFS object has been retrieved
Workaround that always waited for exactly 1 second has been removed.
2021-02-02 19:29:52 +01:00
Mark Spanbroek b54c3d9585 Implement Ipfs API 2021-02-02 19:29:52 +01:00
Mark Spanbroek a63000f602 Exchange objects between peers using bitswap 2021-02-02 19:29:52 +01:00
Mark Spanbroek 28d1ddf7e3 Repo: use Option to signal retrieval success or failure 2021-02-02 19:29:52 +01:00
Mark Spanbroek 520f3f3bc9 Add Bitswap protocol for libp2p 2021-02-02 19:29:52 +01:00
Mark Spanbroek e6d84b9c0d Add initializers for bitswap messages 2021-02-02 19:29:52 +01:00
Mark Spanbroek fa15c92158 Serialize bitswap messages using protobuf 2021-02-02 19:29:52 +01:00
Mark Spanbroek b3f5599925 Add absolutely minimal implementation of a DHT routing table 2021-02-02 19:29:52 +01:00
Mark Spanbroek 7b15a6ae83 Disable failing test for now 2021-02-02 19:29:52 +01:00
Mark Spanbroek dc06106ae1 Change MerkleDag to IpfsObject to better match IPFS API 2021-02-02 19:29:52 +01:00
Mark Spanbroek 5a36e26e7c Change type and proc names to better match IPFS API 2021-02-02 19:29:52 +01:00
Mark Spanbroek 05a7f47113 Rename dagger->ipfs
This is a storage engine based on IPFS, which is only a part of Dagger
2021-02-02 19:29:52 +01:00
Mark Spanbroek dd18f50ec6 Replace root hash by root content id 2021-02-02 19:29:52 +01:00
Mark Spanbroek 096fe0e40d Repo for storing and retrieving Merkle DAGs 2021-02-02 19:29:52 +01:00
Mark Spanbroek ecc4e50d28 Add root hash to Merkle DAG 2021-02-02 19:29:52 +01:00
Mark Spanbroek c795c698fe Compile all tests in one go 2021-02-02 19:29:52 +01:00
Mark Spanbroek 5f5153820f Add absolutely minimal implementation of chunking 2021-02-02 19:29:52 +01:00
Mark Spanbroek 2471423197 Added (failing) test for transferring file from one peer to another 2021-02-02 19:29:52 +01:00
Mark Spanbroek bd29797aba Project setup 2021-02-02 19:29:52 +01:00