Commit Graph

3 Commits

Author SHA1 Message Date
Ben Bierens 9c8a59d150
Blockstore maintenance (#347)
* setting up

* Implements timer utility

* applies async and cancellation to timer loop

* Sets up mocktimer and mockblockstore to set up first maintenance module test

* wip: first test that calls blockChecker

* wip: adding user type to timer callback

* Chronicles doesn't like type-arguments? Disables logging in timer module for now

* Implementing block check test for blockMaintainer module

* Sets up additional tests for blockmaintainer

* Removes generic from timer module. Implements numberOfBlocks per interval in blockMaintainer.

* Implements blockMaintainer

* Sets up tests for blockChecker

* Some comments by Mark

* Cleanup repostore tests

* Setting up the use of std/times for block TTL tracking

* repostore adds expiration timestamp

* Defaults the repostore clock to the system clock

* Applies updates to repostore interface.

* Implements retrieving of block expiration information from repostore

* Sets up tests for maintenance module behavior

* Implements block maintenance module

* Wires maintenance module into codex. Sets up integration tests for block expiration

* Sets up test for missing behavior: removing timestamp metadata on block delete

* Implements removing of expiration metadata in repoStore

* Fixes integration tests for block expiration

* Adds block expiration tests to integration test run

* Handled some comments by Dmitriy

* Review comment by Dmitriy: Removes seq[cid] from runBlockCheck

* Review comment by Dmitriy: Moves key formatting methods to keyutils.

* Review comment by Dmitriy: Encodes durations using chronos

* Fixes conversion of TTL type in conf.

* Review comments by Dmitriy

* Adds unit tests for keyUtils.

* Adds test coverage for exception in maintenance module
2023-03-08 16:04:54 +01:00
Dmitriy Ryajov 8481e301d5
add missed repo store test (#324) 2022-12-05 09:00:13 -06:00
Dmitriy Ryajov 0beeefd760
Repo limits (#319)
* initial implementation of repo store

* allow isManifest on multicodec

* rework with new blockstore

* add raw codec

* rework listBlocks

* remove fsstore

* reworking with repostore

* bump datastore

* fix listBlocks iterator

* adding store's common tests

* run common store tests

* remove fsstore backend tests

* bump datastore

* add `listBlocks` tests

* listBlocks filter based on block type

* disabling tests in need of rewriting

* allow passing block type

* move BlockNotFoundError definition

* fix tests

* increase default advertise loop sleep to 10 mins

* use `self`

* add cache quota functionality

* pass meta store and start repo

* add `CacheQuotaNamespace`

* pass meta store

* bump datastore to latest master

* don't use os `/` as key separator

* Added quota limits support

* tests for quota limits

* add block expiration key

* remove unnesesary space

* use idleAsync in listBlocks

* proper test name

* re-add contrlC try/except

* add storage quota and block ttl config options

* clarify comments

* change expires key format

* check for block presence before storing

* bump datastore

* use dht with fixed datastore `has`

* bump datastore to latest master

* bump dht to latest master
2022-12-02 18:00:55 -06:00