codex-storage
/
nim-codex
mirror of https://github.com/codex-storage/nim-codex.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
393 Commits 167 Branches 9 Tags 198 MiB
Commit Graph

13 Commits

Author SHA1 Message Date
Dmitriy Ryajov 0beeefd760
Repo limits (#319) 
* initial implementation of repo store

* allow isManifest on multicodec

* rework with new blockstore

* add raw codec

* rework listBlocks

* remove fsstore

* reworking with repostore

* bump datastore

* fix listBlocks iterator

* adding store's common tests

* run common store tests

* remove fsstore backend tests

* bump datastore

* add `listBlocks` tests

* listBlocks filter based on block type

* disabling tests in need of rewriting

* allow passing block type

* move BlockNotFoundError definition

* fix tests

* increase default advertise loop sleep to 10 mins

* use `self`

* add cache quota functionality

* pass meta store and start repo

* add `CacheQuotaNamespace`

* pass meta store

* bump datastore to latest master

* don't use os `/` as key separator

* Added quota limits support

* tests for quota limits

* add block expiration key

* remove unnesesary space

* use idleAsync in listBlocks

* proper test name

* re-add contrlC try/except

* add storage quota and block ttl config options

* clarify comments

* change expires key format

* check for block presence before storing

* bump datastore

* use dht with fixed datastore `has`

* bump datastore to latest master

* bump dht to latest master
 2022-12-02 18:00:55 -06:00
Dmitriy Ryajov 5abf80cc69
Block download (#304) 
* track inflight requests

* preperly handle precense updates

* trace number of of scheduled blocks

* invoke `payForBlocks` at the correct time

* reduntant block info on want list updates

* don't update prices in task handler

* PeerID -> PeerId

* cleanup

* proper log topic

* better chronicles topic filtering

* more trace logging

* sort want blocks

* wip - fix tests

* wip - fix tests, presence changes

* fix small test issue

* return price

* payment related changes

* misc

* re-enable payment tests

* fix warn wording

* fix `u256` conversion

* minor misc changes

* don't idle for so long on `encode`

* logging

* move buff

* disable cache by default

* disable cache by default

* fix streamOneBlock

* log node stopping/exiting

* trace logging

* don't stringify cid

* use `self`

* quick cleanup

* rename enums

* rename enums

* turns out we don't needs this test

* fix wording
 2022-11-15 09:46:21 -06:00
Dmitriy Ryajov 5f9507cfcd
Blocks leak and inflight for pending blocks (#315) 
* attach `wait` to handle fut

* fix crash when can't find a CID and timeout hits

the exception was not raised, thus a Nil was returned that was
than wrapped in an option, leading to crash.


Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* add inFlight flag

* adding pending blocks tests

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
Co-authored-by: Csaba Kiraly <csaba.kiraly@gmail.com>
 2022-11-15 00:12:05 -06:00
Dmitriy Ryajov f235cc6621
don't spawn queue requests (#313) 
* don't spawn queue requests

* adjust list blocks idle timer

* increase timeout on failing test...
 2022-11-14 18:01:05 -06:00
Michael Bradley c893b1f0cb
track latest nim-libp2p's master branch (#248) 
* [build] track nim-libp2p's unstable branch

* cid formatIt change

* track nim-libp2p-unstable

* don't stringify cid

* Fixed testblockexc.nim

1. Assign price to blocks
2. Delete on node1 cached blocks from node2 before buying them again

* add trace logging

* init pricing info

* remove duplicate price setting

* cid serialization in trace logs

* bumping dht to latest main

* bump

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Co-authored-by: Bulat-Ziganshin <bulat.ziganshin@gmail.com>
 2022-11-02 11:40:28 -06:00
Bulat-Ziganshin 9939d85b74
Replace protobuf serialization for Block exchange with minprotobuf (#271) 
* minprotobuf serialization for Block exchange

* Handle decoding errors by stopping peer connection
 2022-10-13 18:58:57 -05:00
Michael Bradley, Jr 3d823dcbc6 [stores] update getBlock return type 
change return type for `method getBlock` from `Future[?!(?Block)]` to
`Future[?!Block]`

use `type BlockNotFoundError = object of CodexError` to differentiate between
"block not found in the store" and other errors

also make some logic and error handling/messages more consistent across
BlockStore implementations

closes #177
closes #182
closes #210

alternative to #205, #209
 2022-08-22 20:52:51 -05:00
Dmitriy Ryajov 48368893c9
Add limits for inflight requests (#169) 
* convert network to async

* use async network api

* test with async network

* test concurrent send limits
 2022-07-29 10:19:34 -06:00
Bulat-Ziganshin 0bfe26440e
WIP: Harmonize BlockStore API (remaining functions) (#123) (#130) 
BlockStore API got new return types (rationale in https://github.com/status-im/nim-codex/issues/123#issuecomment-1163797753):
- getBlock: Future[?! (?Block)]
- putBlock/delBlock/listBlocks: Future[?!void]
- hasBlock: Future[?!bool]

Plus refactored readOnce(StoreStream) and check received data in its tests.

And replaced local use of AsyncHeapQueue with seq.sort.
 2022-07-28 03:39:17 +03:00
Mark Spanbroek 13bbf2b052 Fix test that was failing intermittently 
Give test more time to complete but do not increase
sleep time more than necessary, by introducing
waiting loop for test purposes.
 2022-07-20 13:43:20 +02:00
Dmitriy Ryajov 6ce7e23767
Upload authenticators (#108) 
* initial implementation of storage proofs upload

* make sure proof verifies with after deserializing

* add por store

* rename por store to stp store

* rename porstore to stpstore

* add support for host discovery to discovery mock

* add tags upload network tests
 2022-05-25 20:29:31 -06:00
Dmitriy Ryajov 56b80d6f6d
Por serialize (#106) 
* move por into storage proofs

* use SeekableStream

* adding serialization for por

* remove leftovers

* add empty block support

* add basic por test

* rename block exchange for consistency

* add storageproofstests

* moving timing to storageproofs

* fix imports

* fix imports

* fix imports

* add top level exports

* move delete blocks helper to helpers

* more import/export fixes

* cleanup

* more import fixes

* fix unused warnings

* detect corrupt blocks tests

* add serialization tests

* move init method around

* bump asynctest

* fix CID version

* get rid of warning

* wip: fix CI

* increase CI timeout
 2022-05-23 23:24:15 -06:00
Dmitriy Ryajov d4f3ebc867
reorg engine directory structure (#104) 2022-05-19 16:28:53 -06:00