logos-storage/logos-storage-nim
1
0
Fork 0
mirror of https://github.com/logos-storage/logos-storage-nim.git synced 2026-01-16 04:13:09 +00:00
Code Issues Packages Projects Releases Wiki Activity
logos-storage-nim/codex/utils/json.nim

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

49 lines
1.1 KiB
Nim
Raw Normal View History

Improve integration testing client (CodexClient) and json serialization (#514) * Improve integration testing client (CodexClient) and json serialization The current client used for integration testing against the REST endpoints for Codex accepts and passes primitive types. This caused a hard to diagnose bug where a `uint` was not being deserialized correctly. In addition, the json de/serializing done between the CodexClient and REST client was not easy to read and was not tested. These changes bring non-primitive types to most of the CodexClient functions, allowing us to lean on the compiler to ensure we're providing correct typings. More importantly, a json de/serialization util was created as a drop-in replacement for the std/json lib, with the main two differences being that field serialization is opt-in (instead of opt-out as in the case of json_serialization) and serialization errors are captured and logged, making debugging serialization issues much easier. * Update integration test to use nodes=2 and tolerance=1 * clean up
2023-09-01 15:44:41 +10:00
import std/options
import std/typetraits
feat: create logging proxy (#663) * implement a logging proxy The logging proxy: - prevents the need to import chronicles (as well as export except toJson), - prevents the need to override `writeValue` or use or import nim-json-seralization elsewhere in the codebase, allowing for sole use of utils/json for de/serialization, - and handles json formatting correctly in chronicles json sinks * Rename logging -> logutils to avoid ambiguity with common names * clean up * add setProperty for JsonRecord, remove nim-json-serialization conflict * Allow specifying textlines and json format separately Not specifying a LogFormat will apply the formatting to both textlines and json sinks. Specifying a LogFormat will apply the formatting to only that sink. * remove unneeded usages of std/json We only need to import utils/json instead of std/json * move serialization from rest/json to utils/json so it can be shared * fix NoColors ambiguity Was causing unit tests to fail on Windows. * Remove nre usage to fix Windows error Windows was erroring with `could not load: pcre64.dll`. Instead of fixing that error, remove the pcre usage :) * Add logutils module doc * Shorten logutils.formatIt for `NBytes` Both json and textlines formatIt were not needed, and could be combined into one formatIt * remove debug integration test config debug output and logformat of json for integration test logs * Use ## module doc to support docgen * bump nim-poseidon2 to export fromBytes Before the changes in this branch, fromBytes was likely being resolved by nim-stew, or other dependency. With the changes in this branch, that dependency was removed and fromBytes could no longer be resolved. By exporting fromBytes from nim-poseidon, the correct resolution is now happening. * fixes to get compiling after rebasing master * Add support for Result types being logged using formatIt
2024-01-23 18:35:03 +11:00
from pkg/ethers import Address
from pkg/libp2p import
adds JSON serialization to BitTorrent manifest
2025-03-12 02:59:02 +01:00
Cid, PeerId, SignedPeerRecord, MultiAddress, AddressInfo, MultiHash, init, hex, `$`
import pkg/stew/byteutils
Improve integration testing client (CodexClient) and json serialization (#514) * Improve integration testing client (CodexClient) and json serialization The current client used for integration testing against the REST endpoints for Codex accepts and passes primitive types. This caused a hard to diagnose bug where a `uint` was not being deserialized correctly. In addition, the json de/serializing done between the CodexClient and REST client was not easy to read and was not tested. These changes bring non-primitive types to most of the CodexClient functions, allowing us to lean on the compiler to ensure we're providing correct typings. More importantly, a json de/serialization util was created as a drop-in replacement for the std/json lib, with the main two differences being that field serialization is opt-in (instead of opt-out as in the case of json_serialization) and serialization errors are captured and logged, making debugging serialization issues much easier. * Update integration test to use nodes=2 and tolerance=1 * clean up
2023-09-01 15:44:41 +10:00
import pkg/contractabi
feat: create logging proxy (#663) * implement a logging proxy The logging proxy: - prevents the need to import chronicles (as well as export except toJson), - prevents the need to override `writeValue` or use or import nim-json-seralization elsewhere in the codebase, allowing for sole use of utils/json for de/serialization, - and handles json formatting correctly in chronicles json sinks * Rename logging -> logutils to avoid ambiguity with common names * clean up * add setProperty for JsonRecord, remove nim-json-serialization conflict * Allow specifying textlines and json format separately Not specifying a LogFormat will apply the formatting to both textlines and json sinks. Specifying a LogFormat will apply the formatting to only that sink. * remove unneeded usages of std/json We only need to import utils/json instead of std/json * move serialization from rest/json to utils/json so it can be shared * fix NoColors ambiguity Was causing unit tests to fail on Windows. * Remove nre usage to fix Windows error Windows was erroring with `could not load: pcre64.dll`. Instead of fixing that error, remove the pcre usage :) * Add logutils module doc * Shorten logutils.formatIt for `NBytes` Both json and textlines formatIt were not needed, and could be combined into one formatIt * remove debug integration test config debug output and logformat of json for integration test logs * Use ## module doc to support docgen * bump nim-poseidon2 to export fromBytes Before the changes in this branch, fromBytes was likely being resolved by nim-stew, or other dependency. With the changes in this branch, that dependency was removed and fromBytes could no longer be resolved. By exporting fromBytes from nim-poseidon, the correct resolution is now happening. * fixes to get compiling after rebasing master * Add support for Result types being logged using formatIt
2024-01-23 18:35:03 +11:00
import pkg/codexdht/discv5/node as dn
refactor: use serde for json de/serialization instead of utils/json (#704) * json > nim-serde bump Should wait until serde is integrated into nim-ethers before making these changes as there will be less import exceptions required. * bump nim-serde * change func to proc due to chronicles side effects * import serde into utils/json, use as proxy import nim-serde into utils/json and use utils/json as a proxy for serde functions, including overloading `%` and `fromJson` for application types. * update tests to use serde * bump serde to latest * remove testjson -- no longer needed * bump serde in nimble * updates to reconcile rebase with master
2024-03-19 14:25:13 +11:00
import pkg/serde/json
Improve integration testing client (CodexClient) and json serialization (#514) * Improve integration testing client (CodexClient) and json serialization The current client used for integration testing against the REST endpoints for Codex accepts and passes primitive types. This caused a hard to diagnose bug where a `uint` was not being deserialized correctly. In addition, the json de/serializing done between the CodexClient and REST client was not easy to read and was not tested. These changes bring non-primitive types to most of the CodexClient functions, allowing us to lean on the compiler to ensure we're providing correct typings. More importantly, a json de/serialization util was created as a drop-in replacement for the std/json lib, with the main two differences being that field serialization is opt-in (instead of opt-out as in the case of json_serialization) and serialization errors are captured and logged, making debugging serialization issues much easier. * Update integration test to use nodes=2 and tolerance=1 * clean up
2023-09-01 15:44:41 +10:00
import pkg/questionable/results
import ../errors
refactor: use serde for json de/serialization instead of utils/json (#704) * json > nim-serde bump Should wait until serde is integrated into nim-ethers before making these changes as there will be less import exceptions required. * bump nim-serde * change func to proc due to chronicles side effects * import serde into utils/json, use as proxy import nim-serde into utils/json and use utils/json as a proxy for serde functions, including overloading `%` and `fromJson` for application types. * update tests to use serde * bump serde to latest * remove testjson -- no longer needed * bump serde in nimble * updates to reconcile rebase with master
2024-03-19 14:25:13 +11:00
export json
Improve integration testing client (CodexClient) and json serialization (#514) * Improve integration testing client (CodexClient) and json serialization The current client used for integration testing against the REST endpoints for Codex accepts and passes primitive types. This caused a hard to diagnose bug where a `uint` was not being deserialized correctly. In addition, the json de/serializing done between the CodexClient and REST client was not easy to read and was not tested. These changes bring non-primitive types to most of the CodexClient functions, allowing us to lean on the compiler to ensure we're providing correct typings. More importantly, a json de/serialization util was created as a drop-in replacement for the std/json lib, with the main two differences being that field serialization is opt-in (instead of opt-out as in the case of json_serialization) and serialization errors are captured and logged, making debugging serialization issues much easier. * Update integration test to use nodes=2 and tolerance=1 * clean up
2023-09-01 15:44:41 +10:00
proc fromJson*(_: type Cid, json: JsonNode): ?!Cid =
expectJsonKind(Cid, JString, json)
Fix json serialization for cid and enums (#615) * use str on JString types, `$` will preserve `"` * Adding enum support * deserialize cid test * make enum descerializer public * unify fromJson for objects and refs * add enum descerialization testing
2023-11-14 09:23:50 -06:00
Cid.init(json.str).mapFailure
Improve integration testing client (CodexClient) and json serialization (#514) * Improve integration testing client (CodexClient) and json serialization The current client used for integration testing against the REST endpoints for Codex accepts and passes primitive types. This caused a hard to diagnose bug where a `uint` was not being deserialized correctly. In addition, the json de/serializing done between the CodexClient and REST client was not easy to read and was not tested. These changes bring non-primitive types to most of the CodexClient functions, allowing us to lean on the compiler to ensure we're providing correct typings. More importantly, a json de/serialization util was created as a drop-in replacement for the std/json lib, with the main two differences being that field serialization is opt-in (instead of opt-out as in the case of json_serialization) and serialization errors are captured and logged, making debugging serialization issues much easier. * Update integration test to use nodes=2 and tolerance=1 * clean up
2023-09-01 15:44:41 +10:00
feat: create logging proxy (#663) * implement a logging proxy The logging proxy: - prevents the need to import chronicles (as well as export except toJson), - prevents the need to override `writeValue` or use or import nim-json-seralization elsewhere in the codebase, allowing for sole use of utils/json for de/serialization, - and handles json formatting correctly in chronicles json sinks * Rename logging -> logutils to avoid ambiguity with common names * clean up * add setProperty for JsonRecord, remove nim-json-serialization conflict * Allow specifying textlines and json format separately Not specifying a LogFormat will apply the formatting to both textlines and json sinks. Specifying a LogFormat will apply the formatting to only that sink. * remove unneeded usages of std/json We only need to import utils/json instead of std/json * move serialization from rest/json to utils/json so it can be shared * fix NoColors ambiguity Was causing unit tests to fail on Windows. * Remove nre usage to fix Windows error Windows was erroring with `could not load: pcre64.dll`. Instead of fixing that error, remove the pcre usage :) * Add logutils module doc * Shorten logutils.formatIt for `NBytes` Both json and textlines formatIt were not needed, and could be combined into one formatIt * remove debug integration test config debug output and logformat of json for integration test logs * Use ## module doc to support docgen * bump nim-poseidon2 to export fromBytes Before the changes in this branch, fromBytes was likely being resolved by nim-stew, or other dependency. With the changes in this branch, that dependency was removed and fromBytes could no longer be resolved. By exporting fromBytes from nim-poseidon, the correct resolution is now happening. * fixes to get compiling after rebasing master * Add support for Result types being logged using formatIt
2024-01-23 18:35:03 +11:00
func `%`*(cid: Cid): JsonNode =
% $cid
func `%`*(obj: PeerId): JsonNode =
% $obj
func `%`*(obj: SignedPeerRecord): JsonNode =
% $obj
func `%`*(obj: dn.Address): JsonNode =
% $obj
func `%`*(obj: AddressInfo): JsonNode =
% $obj.address
func `%`*(obj: MultiAddress): JsonNode =
% $obj
func `%`*(address: ethers.Address): JsonNode =
% $address
adds JSON serialization to BitTorrent manifest
2025-03-12 02:59:02 +01:00
proc fromJson*(_: type MultiHash, json: JsonNode): ?!MultiHash =
expectJsonKind(MultiHash, JString, json)
echo "[MultiHash.fromJson] json.str: ", json.str
without bytes =? json.str.hexToSeqByte.catch, err:
return failure(err.msg)
MultiHash.init(bytes).mapFailure
func `%`*(multiHash: MultiHash): JsonNode =
%multiHash.hex
Reference in New Issue Copy Permalink