status-im
/
nim-codex
mirror of https://github.com/status-im/nim-codex.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
204 Commits 167 Branches 9 Tags 197 MiB
Commit Graph

10 Commits

Author SHA1 Message Date
Dmitriy Ryajov d3dbbc75fa
Extract Discovery engine (#99) 
* don't force logging syncs

* Add failing test

* wip discovery engine

* re-add chronicles sinks

* wip

* move network related stuff to own folder

* move peer related stuff to own folder

* extract discovery into it's own engine

* update imports

* move pending blocks into engine module

* add top level exports

* update imports

* update import paths

* update imports

* support for inflight request filtering and tests

* use `remove` instead of `del`

* fix sorting in `selectCheapest`

* re-org test file structure

* fix to use discovery engine

* file re-org

* fix compilation

* fixup discovery to use async handlers

* more re-org

* rework with support for discovery engine

* add logging

* use defaults

* wip: reworking with discoveryengine

* wip: more test fixes

* more logging

* use ordered table

* use `bt` for blocktype Block

* fix tests

* make tests work with discovery engine

* expose all node components

* fix to work with discovery engine

* wip

* propagate cancellation in listBlocks

* start/stop disc engine in blockexc engine

* remove disc engine start/stop

* wire up discovery engine

* misc comments and imports

* pass discovery to dagger node

* set sleep timers

* unused imports

* misc

* don't spawn a task, await it

* don't await handlers

* trace logging

* reduce default sleep time

Co-authored-by: Tanguy <tanguy@status.im>
 2022-05-18 20:29:15 -06:00
Dmitriy Ryajov 9ca4f90cf3
safe tables access (#95) 
* safer table access

* use withValue
 2022-05-12 14:23:05 -06:00
Tanguy ce59dbd4a2
Revert "Cleanup engine and rework discovery (#87)" 
This reverts commit 4740ffc144.
 2022-04-20 14:28:11 +02:00
Dmitriy Ryajov 4740ffc144
Cleanup engine and rework discovery (#87) 
* rework discovery with async queues

* misc style changes

* increase max message size for large manifests

* use upraises and avoid exceptions on key access

* increase sleep time to 100 millis

* pass config

* make list blocks trigger a callback on each block

* check for nil on start/stop

* fix tests and split out discovery tests

* don't auto mount network

* add list block tests

* add discovery tests

* rework moc discovery

* move discovery moc to disc dir

* don't force logging syncs

* don't force moc discovery on all tests

* rework discovery with methods

* add top level utils file

* don't use asyncCheck

* don't pass entire blocks to list blocks calback

* spelling
 2022-04-19 21:46:44 -06:00
Tanguy 4d681102e5
Add DHT (#75) 
* First implem

* Add persistent net key option

* Working DHT setup

* Bootstrap nodes

* Implement DaggerNode.findPeer

* Remove unrelevant comment

* Added discovery to blockexchange requestBlock

* add FSStore.blockList

* Block advertisement

* Tests compiles

* Green tests

* toDiscoveryId instead of toNodeId

* remove stopAdvertisingBlock

* Removed nim-eth dependency

* Move discovery stuff to discovery.nim

* Add missing file, start of discovery tests

* Better discovery logic

* Add tests

* Address comment

* Better E2E test
 2022-04-13 10:32:35 -06:00
Dmitriy Ryajov d204b518e8
Make block a ref (#56) 
* cleanup manifest

* make block a ref type
 2022-03-18 13:50:53 -06:00
Dmitriy Ryajov a9721dd65f
Block.init should return Result (#37) 
* Block.init should return Result

* use get() instead of tryGet()
 2022-01-10 20:25:13 -06:00
Dmitriy Ryajov fbe161a073
Node setup (#32) 
* settup basic nim node

* adding http utils

* adding confutils

* rough rest api proto

* adding missing deps

* turn tls emulation off

* adding toml serialization

* wip

* adding missing deps

* make sure to clean old state in teardown

* adding file upload rest endpoint

* renaming blockexchange to networkstore

* updating nim-presto

* updating libp2p

* wip adding streaming upload

* reworked chunking

* bump to latest unstable

* adding asyncfutures stream

* make streamable

* deleting unused files

* reworking stores api

* use new stores api

* rework blockset and remove blockstream

* don't return option from constructor

* rework chunker

* wip implement upload

* fix tests

* move unrelated logic to engine

* don't print entire message

* logging

* basic encode/decode to/from dag-pb

* add basic upload/download support

* fix tests

* renaming blockset to manifest

* don't pass config to node

* remove config and use new manifest

* wip: make endpoints more reliable

* wip: adding node tests

* include correct manifest test

* removing asyncfutures

* proper chunking of files

* simplify stream reading

* test with encoding/decoding with many blocks

* add block storing tests

* adding retrieval test

* add logging

* tidy up chunker

* tidy up manifest and node

* use default chunk size

* fix tests

* fix tests

* make sure Eof is set properly

* wip

* minor cleanup

* add file utils

* cleanup config

* splitout DaggerServer and "main"

* remove events since they are not used

* add broadcast method to network peer

* add and wire localstore

* use localstore in the node

* wip

* logging

* move file utils

* use the constant

* updating deps

* fix memstore

* use latest libp2p unstable

* fix tests

* rework block streaming

* don't fail storing if the block already exists

* add helper info endpoint

* correct comment

* rename localstore to fsstore

* fix tests

* remove unused tests

* add test to retrieve one block

* move some test files around

* consolidate setup

* Update dagger/blockexchange/engine.nim

Co-authored-by: Tanguy <tanguy@status.im>

* typo

* better block path handling

* don't inherit rootobj

* remove useless template

* Update tests/dagger/blockexc/testblockexc.nim

Co-authored-by: markspanbroek <mark@spanbroek.net>

* use isMainModule

* use proper flag for starter/stoped

* cleanup optional use

* wrap in isMainModule

* use `cancelAndAwait`

* remove unused imports

* wip

* don't use optional

* use functional error api

* rework store tests and add fs tests

* Block.new() to Block.init()

* don't use optional for engine blocks

* use result instead of optional for getBlock

* remove unused imports

* move stopping servers to `shutdown`

* use result instead of optional

* rework with results

* fix tests

* use waitFor in signal handlers

* error helper

* use `?` and mapFailure where possible

* remove unnecesary `=?`

* improve empty cid digest initialization

Co-authored-by: Tanguy <tanguy@status.im>
Co-authored-by: markspanbroek <mark@spanbroek.net>
 2022-01-10 09:32:56 -06:00
Dmitriy Ryajov ce66e43440
fix compilation issues with latest libp2p (#27) 
* fix compilation issues with latest libp2p

* change package description

* fix CI compiler

* try with nim 1.4.2

* try with 1.4.2

* use more standardized ci config

* set reasonable log level

* remove nim version dependency

* reverting to 1.4.6 as otherwise it fails on amd64
 2021-10-29 13:30:52 -06:00
Dmitriy Ryajov 2fb39ca4a3
Rename bitswap (#25) 
* use PeerInfo in event handlers

* use CidV1 and raw multicodec as default

* add block stream abstraction

* raises defect

* adding dataset abstraction

* move blockstream into own dir

* reorg files and fix tests

* rename dataset to blockset

* wip

* wip

* adding basic test for treehash algo

* run blockset tests along with with the rest

* remove obsolete contents

* fix chunker tests

* rename bitswap and move to stores

* rename bitwsap to blockexc and move to stores

* moare project structure reorg
 2021-08-30 13:25:20 -06:00