2022-05-19 19:56:03 +00:00
|
|
|
## Nim-Codex
|
2022-01-10 15:32:56 +00:00
|
|
|
## Copyright (c) 2021 Status Research & Development GmbH
|
|
|
|
## Licensed under either of
|
|
|
|
## * Apache License, version 2.0, ([LICENSE-APACHE](LICENSE-APACHE))
|
|
|
|
## * MIT license ([LICENSE-MIT](LICENSE-MIT))
|
|
|
|
## at your option.
|
|
|
|
## This file may not be copied, modified, or distributed except according to
|
|
|
|
## those terms.
|
|
|
|
|
2023-12-20 02:24:40 +00:00
|
|
|
|
feat: create logging proxy (#663)
* implement a logging proxy
The logging proxy:
- prevents the need to import chronicles (as well as export except toJson),
- prevents the need to override `writeValue` or use or import nim-json-seralization elsewhere in the codebase, allowing for sole use of utils/json for de/serialization,
- and handles json formatting correctly in chronicles json sinks
* Rename logging -> logutils to avoid ambiguity with common names
* clean up
* add setProperty for JsonRecord, remove nim-json-serialization conflict
* Allow specifying textlines and json format separately
Not specifying a LogFormat will apply the formatting to both textlines and json sinks.
Specifying a LogFormat will apply the formatting to only that sink.
* remove unneeded usages of std/json
We only need to import utils/json instead of std/json
* move serialization from rest/json to utils/json so it can be shared
* fix NoColors ambiguity
Was causing unit tests to fail on Windows.
* Remove nre usage to fix Windows error
Windows was erroring with `could not load: pcre64.dll`. Instead of fixing that error, remove the pcre usage :)
* Add logutils module doc
* Shorten logutils.formatIt for `NBytes`
Both json and textlines formatIt were not needed, and could be combined into one formatIt
* remove debug integration test config
debug output and logformat of json for integration test logs
* Use ## module doc to support docgen
* bump nim-poseidon2 to export fromBytes
Before the changes in this branch, fromBytes was likely being resolved by nim-stew, or other dependency. With the changes in this branch, that dependency was removed and fromBytes could no longer be resolved. By exporting fromBytes from nim-poseidon, the correct resolution is now happening.
* fixes to get compiling after rebasing master
* Add support for Result types being logged using formatIt
2024-01-23 07:35:03 +00:00
|
|
|
import pkg/upraises
|
2022-03-18 22:17:51 +00:00
|
|
|
push: {.upraises: [].}
|
2022-01-10 15:32:56 +00:00
|
|
|
|
|
|
|
import pkg/chronos
|
|
|
|
import pkg/libp2p
|
2023-11-27 18:25:53 +00:00
|
|
|
import pkg/questionable/results
|
2022-01-10 15:32:56 +00:00
|
|
|
|
2023-11-06 08:10:30 +00:00
|
|
|
import ../clock
|
2023-11-27 18:25:53 +00:00
|
|
|
import ../blocktype
|
2022-05-19 02:29:15 +00:00
|
|
|
import ../blockexchange
|
feat: create logging proxy (#663)
* implement a logging proxy
The logging proxy:
- prevents the need to import chronicles (as well as export except toJson),
- prevents the need to override `writeValue` or use or import nim-json-seralization elsewhere in the codebase, allowing for sole use of utils/json for de/serialization,
- and handles json formatting correctly in chronicles json sinks
* Rename logging -> logutils to avoid ambiguity with common names
* clean up
* add setProperty for JsonRecord, remove nim-json-serialization conflict
* Allow specifying textlines and json format separately
Not specifying a LogFormat will apply the formatting to both textlines and json sinks.
Specifying a LogFormat will apply the formatting to only that sink.
* remove unneeded usages of std/json
We only need to import utils/json instead of std/json
* move serialization from rest/json to utils/json so it can be shared
* fix NoColors ambiguity
Was causing unit tests to fail on Windows.
* Remove nre usage to fix Windows error
Windows was erroring with `could not load: pcre64.dll`. Instead of fixing that error, remove the pcre usage :)
* Add logutils module doc
* Shorten logutils.formatIt for `NBytes`
Both json and textlines formatIt were not needed, and could be combined into one formatIt
* remove debug integration test config
debug output and logformat of json for integration test logs
* Use ## module doc to support docgen
* bump nim-poseidon2 to export fromBytes
Before the changes in this branch, fromBytes was likely being resolved by nim-stew, or other dependency. With the changes in this branch, that dependency was removed and fromBytes could no longer be resolved. By exporting fromBytes from nim-poseidon, the correct resolution is now happening.
* fixes to get compiling after rebasing master
* Add support for Result types being logged using formatIt
2024-01-23 07:35:03 +00:00
|
|
|
import ../logutils
|
2023-11-14 12:02:17 +00:00
|
|
|
import ../merkletree
|
feat: create logging proxy (#663)
* implement a logging proxy
The logging proxy:
- prevents the need to import chronicles (as well as export except toJson),
- prevents the need to override `writeValue` or use or import nim-json-seralization elsewhere in the codebase, allowing for sole use of utils/json for de/serialization,
- and handles json formatting correctly in chronicles json sinks
* Rename logging -> logutils to avoid ambiguity with common names
* clean up
* add setProperty for JsonRecord, remove nim-json-serialization conflict
* Allow specifying textlines and json format separately
Not specifying a LogFormat will apply the formatting to both textlines and json sinks.
Specifying a LogFormat will apply the formatting to only that sink.
* remove unneeded usages of std/json
We only need to import utils/json instead of std/json
* move serialization from rest/json to utils/json so it can be shared
* fix NoColors ambiguity
Was causing unit tests to fail on Windows.
* Remove nre usage to fix Windows error
Windows was erroring with `could not load: pcre64.dll`. Instead of fixing that error, remove the pcre usage :)
* Add logutils module doc
* Shorten logutils.formatIt for `NBytes`
Both json and textlines formatIt were not needed, and could be combined into one formatIt
* remove debug integration test config
debug output and logformat of json for integration test logs
* Use ## module doc to support docgen
* bump nim-poseidon2 to export fromBytes
Before the changes in this branch, fromBytes was likely being resolved by nim-stew, or other dependency. With the changes in this branch, that dependency was removed and fromBytes could no longer be resolved. By exporting fromBytes from nim-poseidon, the correct resolution is now happening.
* fixes to get compiling after rebasing master
* Add support for Result types being logged using formatIt
2024-01-23 07:35:03 +00:00
|
|
|
import ../utils/asyncheapqueue
|
|
|
|
import ../utils/asynciter
|
|
|
|
import ./blockstore
|
2022-01-10 15:32:56 +00:00
|
|
|
|
2022-05-19 02:29:15 +00:00
|
|
|
export blockstore, blockexchange, asyncheapqueue
|
2022-01-10 15:32:56 +00:00
|
|
|
|
|
|
|
logScope:
|
2022-05-19 19:56:03 +00:00
|
|
|
topics = "codex networkstore"
|
2022-01-10 15:32:56 +00:00
|
|
|
|
|
|
|
type
|
|
|
|
NetworkStore* = ref object of BlockStore
|
|
|
|
engine*: BlockExcEngine # blockexc decision engine
|
|
|
|
localStore*: BlockStore # local block store
|
|
|
|
|
2023-11-14 12:02:17 +00:00
|
|
|
method getBlock*(self: NetworkStore, address: BlockAddress): Future[?!Block] {.async.} =
|
|
|
|
trace "Getting block from local store or network", address
|
2022-07-28 00:39:17 +00:00
|
|
|
|
2023-11-14 12:02:17 +00:00
|
|
|
without blk =? await self.localStore.getBlock(address), error:
|
2022-08-19 00:56:36 +00:00
|
|
|
if not (error of BlockNotFoundError): return failure error
|
2023-11-14 12:02:17 +00:00
|
|
|
trace "Block not in local store", address
|
2023-07-18 05:50:47 +00:00
|
|
|
|
2023-11-14 12:02:17 +00:00
|
|
|
without newBlock =? (await self.engine.requestBlock(address)).catch, error:
|
|
|
|
trace "Unable to get block from exchange engine", address
|
2023-07-18 05:50:47 +00:00
|
|
|
return failure error
|
|
|
|
|
|
|
|
return success newBlock
|
2022-08-19 00:56:36 +00:00
|
|
|
|
|
|
|
return success blk
|
2022-07-28 00:39:17 +00:00
|
|
|
|
2023-11-14 12:02:17 +00:00
|
|
|
method getBlock*(self: NetworkStore, cid: Cid): Future[?!Block] =
|
|
|
|
## Get a block from the blockstore
|
|
|
|
##
|
|
|
|
|
|
|
|
self.getBlock(BlockAddress.init(cid))
|
|
|
|
|
|
|
|
method getBlock*(self: NetworkStore, treeCid: Cid, index: Natural): Future[?!Block] =
|
|
|
|
## Get a block from the blockstore
|
|
|
|
##
|
|
|
|
|
|
|
|
self.getBlock(BlockAddress.init(treeCid, index))
|
|
|
|
|
2022-12-03 00:00:55 +00:00
|
|
|
method putBlock*(
|
2023-11-27 18:25:53 +00:00
|
|
|
self: NetworkStore,
|
|
|
|
blk: Block,
|
|
|
|
ttl = Duration.none): Future[?!void] {.async.} =
|
2022-07-28 00:39:17 +00:00
|
|
|
## Store block locally and notify the network
|
2022-04-05 14:24:48 +00:00
|
|
|
##
|
|
|
|
|
2023-11-27 18:25:53 +00:00
|
|
|
trace "Putting block into network store", cid = blk.cid
|
2022-01-10 15:32:56 +00:00
|
|
|
|
2022-12-03 00:00:55 +00:00
|
|
|
let res = await self.localStore.putBlock(blk, ttl)
|
2022-07-28 00:39:17 +00:00
|
|
|
if res.isErr:
|
|
|
|
return res
|
2022-01-10 15:32:56 +00:00
|
|
|
|
2022-07-28 00:39:17 +00:00
|
|
|
await self.engine.resolveBlocks(@[blk])
|
|
|
|
return success()
|
2022-01-10 15:32:56 +00:00
|
|
|
|
2024-01-08 22:52:46 +00:00
|
|
|
method putCidAndProof*(
|
2023-11-14 12:02:17 +00:00
|
|
|
self: NetworkStore,
|
|
|
|
treeCid: Cid,
|
|
|
|
index: Natural,
|
|
|
|
blockCid: Cid,
|
2023-12-21 06:41:43 +00:00
|
|
|
proof: CodexProof): Future[?!void] =
|
2024-01-08 22:52:46 +00:00
|
|
|
self.localStore.putCidAndProof(treeCid, index, blockCid, proof)
|
2023-11-14 12:02:17 +00:00
|
|
|
|
2024-01-17 19:24:34 +00:00
|
|
|
method getCidAndProof*(
|
|
|
|
self: NetworkStore,
|
|
|
|
treeCid: Cid,
|
|
|
|
index: Natural): Future[?!(Cid, CodexProof)] =
|
|
|
|
## Get a block proof from the blockstore
|
|
|
|
##
|
|
|
|
|
|
|
|
self.localStore.getCidAndProof(treeCid, index)
|
|
|
|
|
2023-11-06 08:10:30 +00:00
|
|
|
method ensureExpiry*(
|
2023-11-27 18:25:53 +00:00
|
|
|
self: NetworkStore,
|
|
|
|
cid: Cid,
|
|
|
|
expiry: SecondsSince1970): Future[?!void] {.async.} =
|
2023-11-06 08:10:30 +00:00
|
|
|
## Ensure that block's assosicated expiry is at least given timestamp
|
|
|
|
## If the current expiry is lower then it is updated to the given one, otherwise it is left intact
|
|
|
|
##
|
|
|
|
|
2023-11-22 10:09:12 +00:00
|
|
|
without blockCheck =? await self.localStore.hasBlock(cid), err:
|
|
|
|
return failure(err)
|
|
|
|
|
|
|
|
if blockCheck:
|
2023-11-06 08:10:30 +00:00
|
|
|
return await self.localStore.ensureExpiry(cid, expiry)
|
|
|
|
else:
|
|
|
|
trace "Updating expiry - block not in local store", cid
|
|
|
|
|
|
|
|
return success()
|
|
|
|
|
2023-11-22 10:09:12 +00:00
|
|
|
method ensureExpiry*(
|
2023-11-27 18:25:53 +00:00
|
|
|
self: NetworkStore,
|
|
|
|
treeCid: Cid,
|
|
|
|
index: Natural,
|
|
|
|
expiry: SecondsSince1970): Future[?!void] {.async.} =
|
2023-11-22 10:09:12 +00:00
|
|
|
## Ensure that block's associated expiry is at least given timestamp
|
|
|
|
## If the current expiry is lower then it is updated to the given one, otherwise it is left intact
|
|
|
|
##
|
|
|
|
|
|
|
|
without blockCheck =? await self.localStore.hasBlock(treeCid, index), err:
|
|
|
|
return failure(err)
|
|
|
|
|
|
|
|
if blockCheck:
|
|
|
|
return await self.localStore.ensureExpiry(treeCid, index, expiry)
|
|
|
|
else:
|
|
|
|
trace "Updating expiry - block not in local store", treeCid, index
|
|
|
|
|
|
|
|
return success()
|
|
|
|
|
2023-11-09 08:47:09 +00:00
|
|
|
method listBlocks*(
|
|
|
|
self: NetworkStore,
|
2023-11-14 12:02:17 +00:00
|
|
|
blockType = BlockType.Manifest): Future[?!AsyncIter[?Cid]] =
|
2023-11-09 08:47:09 +00:00
|
|
|
self.localStore.listBlocks(blockType)
|
|
|
|
|
2022-07-28 00:39:17 +00:00
|
|
|
method delBlock*(self: NetworkStore, cid: Cid): Future[?!void] =
|
2022-06-28 16:10:05 +00:00
|
|
|
## Delete a block from the blockstore
|
2022-01-10 15:32:56 +00:00
|
|
|
##
|
|
|
|
|
2022-07-28 00:39:17 +00:00
|
|
|
trace "Deleting block from network store", cid
|
2022-06-28 16:10:05 +00:00
|
|
|
return self.localStore.delBlock(cid)
|
2022-01-10 15:32:56 +00:00
|
|
|
|
|
|
|
{.pop.}
|
|
|
|
|
2022-07-28 00:39:17 +00:00
|
|
|
method hasBlock*(self: NetworkStore, cid: Cid): Future[?!bool] {.async.} =
|
2022-01-10 15:32:56 +00:00
|
|
|
## Check if the block exists in the blockstore
|
|
|
|
##
|
|
|
|
|
2022-07-28 00:39:17 +00:00
|
|
|
trace "Checking network store for block existence", cid
|
|
|
|
return await self.localStore.hasBlock(cid)
|
2022-01-10 15:32:56 +00:00
|
|
|
|
2022-07-22 23:38:49 +00:00
|
|
|
method close*(self: NetworkStore): Future[void] {.async.} =
|
|
|
|
## Close the underlying local blockstore
|
|
|
|
##
|
|
|
|
|
2022-12-03 00:00:55 +00:00
|
|
|
if not self.localStore.isNil:
|
|
|
|
await self.localStore.close
|
2022-07-22 23:38:49 +00:00
|
|
|
|
2022-01-10 15:32:56 +00:00
|
|
|
proc new*(
|
|
|
|
T: type NetworkStore,
|
|
|
|
engine: BlockExcEngine,
|
2023-06-22 15:11:18 +00:00
|
|
|
localStore: BlockStore
|
|
|
|
): NetworkStore =
|
2023-11-06 08:10:30 +00:00
|
|
|
## Create new instance of a NetworkStore
|
|
|
|
##
|
2023-06-22 15:11:18 +00:00
|
|
|
NetworkStore(
|
2022-12-03 00:00:55 +00:00
|
|
|
localStore: localStore,
|
|
|
|
engine: engine)
|