Commit Graph

12 Commits

Author SHA1 Message Date
Tomasz Bekas 4cf5cd001d
Expiry per dataset 2024-06-18 15:43:35 +02:00
Tomasz Bekas 5d4b971f03
Block deletion with ref count & repostore refactor 2024-06-18 15:41:04 +02:00
markspanbroek 3046b7636c
Cleanup integration tests (#757)
* integration: move REST API tests into their own module

* integration: move upload and download tests into their own module

* integration: move purchasing tests into their own module

* integration: move marketplace tests to the right module

* integration: mine a block *after* starting nodes

To ensure that tests involving multiple nodes do
not start with out-of-sync clocks

* Fix: do not swallow CancellationErrors

* integration: avoid underflow in UInt256

* network: remove unnecessary error handling

No Exceptions can occur, only Defects, because everything
is asyncSpawned.

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>

* network: do not raise in asyncSpawned proc

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>

---------

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2024-05-23 15:29:30 +00:00
Eric 61ec4275c8
Revert "remove call to `waitFor` from clock.now and make it async (#691)" (#707)
This reverts commit 591be9446a.
2024-02-16 22:12:16 +00:00
Giuliano Mega 591be9446a
remove call to `waitFor` from clock.now and make it async (#691) 2024-02-05 08:36:28 -03:00
Eric de88fd2c53
feat: create logging proxy (#663)
* implement a logging proxy

The logging proxy:
- prevents the need to import chronicles (as well as export except toJson),
- prevents the need to override `writeValue` or use or import nim-json-seralization elsewhere in the codebase, allowing for sole use of utils/json for de/serialization,
- and handles json formatting correctly in chronicles json sinks

* Rename logging -> logutils to avoid ambiguity with common names

* clean up

* add setProperty for JsonRecord, remove nim-json-serialization conflict

* Allow specifying textlines and json format separately

Not specifying a LogFormat will apply the formatting to both textlines and json sinks.

Specifying a LogFormat will apply the formatting to only that sink.

* remove unneeded usages of std/json

We only need to import utils/json instead of std/json

* move serialization from rest/json to utils/json so it can be shared

* fix NoColors ambiguity

Was causing unit tests to fail on Windows.

* Remove nre usage to fix Windows error

Windows was erroring with `could not load: pcre64.dll`. Instead of fixing that error, remove the pcre usage :)

* Add logutils module doc

* Shorten logutils.formatIt for `NBytes`

Both json and textlines formatIt were not needed, and could be combined into one formatIt

* remove debug integration test config

debug output and logformat of json for integration test logs

* Use ## module doc to support docgen

* bump nim-poseidon2 to export fromBytes

Before the changes in this branch, fromBytes was likely being resolved by nim-stew, or other dependency. With the changes in this branch, that dependency was removed and fromBytes could no longer be resolved. By exporting fromBytes from nim-poseidon, the correct resolution is now happening.

* fixes to get compiling after rebasing master

* Add support for Result types being logged using formatIt
2024-01-22 23:35:03 -08:00
Eric f293082ae9
Reverts logging-proxy, commit 27f585eb6f (#660) 2023-12-20 13:24:40 +11:00
Eric 27f585eb6f
feat: create logging proxy (#653)
* implement a logging proxy

The logging proxy:
- prevents the need to import chronicles (as well as export except toJson),
- prevents the need to override `writeValue` or use or import nim-json-seralization elsewhere in the codebase, allowing for sole use of utils/json for de/serialization,
- and handles json formatting correctly in chronicles json sinks

* Rename logging -> logutils to avoid ambiguity with common names

* clean up

* add setProperty for JsonRecord, remove nim-json-serialization conflict

* Allow specifying textlines and json format separately

Not specifying a LogFormat will apply the formatting to both textlines and json sinks.

Specifying a LogFormat will apply the formatting to only that sink.

* remove unneeded usages of std/json

We only need to import utils/json instead of std/json

* move serialization from rest/json to utils/json so it can be shared

* fix NoColors ambiguity

Was causing unit tests to fail on Windows.

* Remove nre usage to fix Windows error

Windows was erroring with `could not load: pcre64.dll`. Instead of fixing that error, remove the pcre usage :)

* Add logutils module doc

* Shorten logutils.formatIt for `NBytes`

Both json and textlines formatIt were not needed, and could be combined into one formatIt

* remove debug integration test config

debug output and logformat of json for integration test logs

* Use ## module doc to support docgen
2023-12-19 22:12:47 +00:00
Tomasz Bekas 2396c4d76d
Blockexchange uses merkle root and index to fetch blocks (#566)
* Blockexchange uses merkle root and index to fetch blocks

* Links the network store getTree to the local store.

* Update codex/stores/repostore.nim

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Signed-off-by: Tomasz Bekas <tomasz.bekas@gmail.com>

* Rework erasure.nim to include recent cleanup

* Revert accidential changes to lib versions

* Addressing review comments

* Storing proofs instead of trees

* Fix a comment

* Fix broken tests

* Fix for broken testerasure.nim

* Addressing PR comments

---------

Signed-off-by: Tomasz Bekas <tomasz.bekas@gmail.com>
Co-authored-by: benbierens <thatbenbierens@gmail.com>
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2023-11-14 13:02:17 +01:00
Ben Bierens 545e0d47e1
Slows down block maintenance iteration (#543)
* Slows down block maintenance iteration

* Fixes unstable integration test.
2023-09-04 11:12:14 +02:00
Jaremy Creechley e47b38af11
Improving proc/func formatting consistency (#454)
* Fixes/workarounds for nimsuggest failures in codex.nim.

* remove rng prefix - it appears to work now

* format new's to be more consistent

* making proc formatting a bit more consistent
2023-06-22 08:11:18 -07:00
Ben Bierens 9c8a59d150
Blockstore maintenance (#347)
* setting up

* Implements timer utility

* applies async and cancellation to timer loop

* Sets up mocktimer and mockblockstore to set up first maintenance module test

* wip: first test that calls blockChecker

* wip: adding user type to timer callback

* Chronicles doesn't like type-arguments? Disables logging in timer module for now

* Implementing block check test for blockMaintainer module

* Sets up additional tests for blockmaintainer

* Removes generic from timer module. Implements numberOfBlocks per interval in blockMaintainer.

* Implements blockMaintainer

* Sets up tests for blockChecker

* Some comments by Mark

* Cleanup repostore tests

* Setting up the use of std/times for block TTL tracking

* repostore adds expiration timestamp

* Defaults the repostore clock to the system clock

* Applies updates to repostore interface.

* Implements retrieving of block expiration information from repostore

* Sets up tests for maintenance module behavior

* Implements block maintenance module

* Wires maintenance module into codex. Sets up integration tests for block expiration

* Sets up test for missing behavior: removing timestamp metadata on block delete

* Implements removing of expiration metadata in repoStore

* Fixes integration tests for block expiration

* Adds block expiration tests to integration test run

* Handled some comments by Dmitriy

* Review comment by Dmitriy: Removes seq[cid] from runBlockCheck

* Review comment by Dmitriy: Moves key formatting methods to keyutils.

* Review comment by Dmitriy: Encodes durations using chronos

* Fixes conversion of TTL type in conf.

* Review comments by Dmitriy

* Adds unit tests for keyUtils.

* Adds test coverage for exception in maintenance module
2023-03-08 16:04:54 +01:00