236 Commits

Author SHA1 Message Date
Dmitriy Ryajov
a52c5c3409
adding codex merkle and coders tests 2023-12-22 17:25:03 -06:00
Dmitriy Ryajov
a87c8cbfb8
style 2023-12-22 17:25:03 -06:00
Dmitriy Ryajov
3902f7e695
treed and proof encoding/decoding 2023-12-22 17:25:03 -06:00
Dmitriy Ryajov
818ecf852f
rename merkletree -> codexmerkletree 2023-12-22 17:25:03 -06:00
Dmitriy Ryajov
2736d78619
rework merkle tree support 2023-12-22 17:25:02 -06:00
benbierens
c0f4512f62
Fixes issue where ecK was used as numberOfSlots. 2023-12-22 17:25:02 -06:00
benbierens
4bdf0dfd33
Review comments by Tomasz 2023-12-22 17:25:02 -06:00
benbierens
2b63cb0e08
Cleanup slotbuilder 2023-12-22 17:25:02 -06:00
benbierens
fd156e9cf8
Adds check that block size is a multiple of cell size 2023-12-22 17:25:02 -06:00
benbierens
eead39ba1c
finishes implementing slot builder 2023-12-22 17:25:02 -06:00
benbierens
e7a39dcb6e
Implements building slot tree 2023-12-22 17:25:01 -06:00
benbierens
00e376aee7
finding number of pad cells 2023-12-22 17:25:01 -06:00
benbierens
d883e6e95f
Can select slot blocks 2023-12-22 17:25:01 -06:00
benbierens
fc41545006
debugs indexing strategies 2023-12-22 17:25:01 -06:00
benbierens
6a3102246d
Fixes issue where indexing strategy stepped gives wrong values for smallest of ranges 2023-12-22 17:25:01 -06:00
benbierens
2c2e42d363
Swaps in indexing strategy into erasure. 2023-12-22 17:25:01 -06:00
benbierens
46612123d8
Implements indexing strategies 2023-12-22 17:25:00 -06:00
benbierens
fabd2e20a8
wip 2023-12-22 17:25:00 -06:00
benbierens
c871102d98
ensures blocks are devisable by number of slots 2023-12-22 17:25:00 -06:00
benbierens
8d17fb40b4
Getting cids in slot 2023-12-22 17:25:00 -06:00
benbierens
e7996bf058
setting up slot builder 2023-12-22 17:25:00 -06:00
Dmitriy Ryajov
75a75446bb
fix merge issues and cleanup a few warnings 2023-12-22 17:25:00 -06:00
Dmitriy Ryajov
5e528756a2
root returns a result 2023-12-22 17:22:59 -06:00
Dmitriy Ryajov
eaf052fea9
wip poseidon tree 2023-12-22 17:22:59 -06:00
Dmitriy Ryajov
80de04889f
shorte file names 2023-12-22 17:22:59 -06:00
Dmitriy Ryajov
3d3f6e023e
proof size changed 2023-12-22 17:22:58 -06:00
Dmitriy Ryajov
8f0102f0eb
use default hash codec 2023-12-22 17:22:58 -06:00
Dmitriy Ryajov
ff17551e32
adding codex merkle and coders tests 2023-12-22 17:22:58 -06:00
Dmitriy Ryajov
d032149924
style 2023-12-22 17:22:58 -06:00
Dmitriy Ryajov
75976996ed
treed and proof encoding/decoding 2023-12-22 17:22:58 -06:00
Dmitriy Ryajov
1437df6847
rename merkletree -> codexmerkletree 2023-12-22 17:22:58 -06:00
Dmitriy Ryajov
fc4fb1b203
rework merkle tree support 2023-12-22 17:22:58 -06:00
Dmitriy Ryajov
b8ee2ac71e
Update multicodecs (#665)
* rework merkle tree support

* rename merkletree -> codexmerkletree

* treed and proof encoding/decoding

* style

* adding codex merkle and coders tests

* use default hash codec

* proof size changed

* add from nodes test

* shorte file names

* wip poseidon tree

* shorten file names

* root returns a result

* import poseidon tests

* update multicodecs

* consolidating codex types and adding new codecs

* update codec

* remove temp codecs constants

* move codecs related stuff out

* updating codecs

* misc

* updating sizes since block size was adjusted to 64kb

* fix merge issues and cleanup a few warnings
2023-12-22 13:04:01 +01:00
Dmitriy Ryajov
52c5578c46
Rework merkle tree (#654)
* rework merkle tree support

* deps

* rename merkletree -> codexmerkletree

* treed and proof encoding/decoding

* small change to invoke proof verification

* rename merkletree to codexmerkletree

* style

* adding codex merkle and coders tests

* fixup imports

* remove new codecs for now

* bump deps

* adding trace statement

* properly serde of manifest block codecs

* use default hash codec

* add more trace logging to aid debugging

* misc

* remove double import

* revert un-needded change

* proof size changed

* bump poseidon2

* add from nodes test

* shorte file names

* remove upraises

* wip poseidon tree

* adjust file names

* misc

* shorten file names

* fix bad `elements` iter

* don't do asserts

* add fromNodes and converters

* root and getProof now return result

* add poseidon2 tree tests

* root now returns result

* misc

* had to make merkletree a ref, because nim blows up otherwise

* root returns a result

* root returns a result

* import poseidon tests

* bump

* merkle poseidon2 digest

* misc

* add merkle digest tests

* bump

* don't use checksuite

* Update tests/codex/merkletree/generictreetests.nim

Co-authored-by: markspanbroek <mark@spanbroek.net>
Signed-off-by: Dmitriy Ryajov <dryajov@gmail.com>

* Update codex/merkletree/merkletree.nim

Co-authored-by: markspanbroek <mark@spanbroek.net>
Signed-off-by: Dmitriy Ryajov <dryajov@gmail.com>

* Update codex/merkletree/merkletree.nim

Co-authored-by: markspanbroek <mark@spanbroek.net>
Signed-off-by: Dmitriy Ryajov <dryajov@gmail.com>

* Update tests/codex/merkletree/generictreetests.nim

Co-authored-by: markspanbroek <mark@spanbroek.net>
Signed-off-by: Dmitriy Ryajov <dryajov@gmail.com>

* missing return

* make toBool private (it's still needed otherwise comparison won't work)

* added `digestTree` that returns a tree and `digest` for root

* test against both poseidon trees - codex and poseidon2

* shorten merkle tree names

* don't compare trees - it's going to be too slow

* move comparison to mekrle helper

* remove merkle utils

---------

Signed-off-by: Dmitriy Ryajov <dryajov@gmail.com>
Co-authored-by: markspanbroek <mark@spanbroek.net>
2023-12-21 06:41:43 +00:00
Eric
f293082ae9
Reverts logging-proxy, commit 27f585eb6f380a6de2fbbef721fdf1a16bd5f600 (#660) 2023-12-20 13:24:40 +11:00
Eric
27f585eb6f
feat: create logging proxy (#653)
* implement a logging proxy

The logging proxy:
- prevents the need to import chronicles (as well as export except toJson),
- prevents the need to override `writeValue` or use or import nim-json-seralization elsewhere in the codebase, allowing for sole use of utils/json for de/serialization,
- and handles json formatting correctly in chronicles json sinks

* Rename logging -> logutils to avoid ambiguity with common names

* clean up

* add setProperty for JsonRecord, remove nim-json-serialization conflict

* Allow specifying textlines and json format separately

Not specifying a LogFormat will apply the formatting to both textlines and json sinks.

Specifying a LogFormat will apply the formatting to only that sink.

* remove unneeded usages of std/json

We only need to import utils/json instead of std/json

* move serialization from rest/json to utils/json so it can be shared

* fix NoColors ambiguity

Was causing unit tests to fail on Windows.

* Remove nre usage to fix Windows error

Windows was erroring with `could not load: pcre64.dll`. Instead of fixing that error, remove the pcre usage :)

* Add logutils module doc

* Shorten logutils.formatIt for `NBytes`

Both json and textlines formatIt were not needed, and could be combined into one formatIt

* remove debug integration test config

debug output and logformat of json for integration test logs

* Use ## module doc to support docgen
2023-12-19 22:12:47 +00:00
Eric
df6b9c6760
Workaround for Hardhat last block timestamp bug (#644)
* Workaround for Hardhat timestamp bug

Likely due to a Hardhat bug in which the callbacks for subscription events are called and awaited before updating its local understanding of the last block time, Hardhat will report a block time in the `newHeads` event that is generally 1 second before the time reported from `getLatestBlock.timestamp`. This was causing issues with the OnChainClock's offset and therefore the `now()` used by the `OnChainClock` would sometimes be off by a second (or more), causing tests to fail.

This commit introduce a `codex_use_hardhat` compilation flag, that when set, will always get the latest block timestamp from Hardhat via the `getLatestBlock.timestamp` RPC call for `OnChainClock.now` calls. Otherwise, the last block timestamp reported in the `newHeads` event will be used.

Update the docker dist tests compilation flag for simulated proof failures (it was not correct), and explicitly add the `codex_use_hardhat=false` for clarity.

* enable simulated proof failures for coverage

* comment out failing test on linux -- will be replaced

* bump codex contracts eth

* add back clock offset for non-hardhat cases

* bump codex-contracts-eth

increases pointer by 67 blocks each period increase

* Add `codex_use_hardhat` flag to coverage tests
2023-12-20 08:06:24 +11:00
Eric
2c621e0421
chore: improve log information (#656)
* Add proof downtime to validator logs

* Add previous state to state machine transition log
2023-12-19 04:29:18 +00:00
Eric
39554c8ea9
sales was never marked as running and caused a hang (#655) 2023-12-19 01:28:17 +00:00
Ben Bierens
f1b1fe9152
Adds endpoint to expose quota information (#652) 2023-12-14 10:57:16 +00:00
Adam Uhlíř
e62cb96bde
feat: cleanup returning availability (#650)
Co-authored-by: markspanbroek <mark@spanbroek.net>
2023-12-13 20:58:17 +01:00
Ben Bierens
0c3d1dd563
Verifiable manifests (#642)
* Adds test for encoding/decoding protected manifest

* Setting up verifiable manifest

* mysterious mysteries

* Successful encoding test for verifiable manifests

* extracts toF out of users of manifest code

* Update codex/manifest/coders.nim

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Signed-off-by: Ben Bierens <39762930+benbierens@users.noreply.github.com>

* Review comments by Dmitriy

* Adds missing verifiable print to $ method.

* Replace poseidon2 F type with int as temporary stand-in for verification hashes

* Replaces verification hash placeholder with CID

---------

Signed-off-by: Ben Bierens <39762930+benbierens@users.noreply.github.com>
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2023-12-12 08:11:54 +00:00
Ben Bierens
a9f8090bb4
Add challenge to prove callback (#649)
* Sets up passing proof challenge to onProve callback

* Implements tests for proof challenge

* Removes unused import
2023-12-11 11:29:15 +01:00
Eric
3907ca4095
Add get active slot /slots/{slotId} to REST api, use utils/json (#645)
* Add get active slot /slots/{slotId} to REST api, use utils/json

- Add endpoint /slots/{slotId} to get an active SalesAgent from the Sales module. Used in integration tests to test when a sale has reached a certain state. Those integration test changes will be included in a larger PR, coming later.
- Add OpenAPI changes for new endpoint and associated components
- Use utils/json instead of nim-json-serialization. Required exemption of imports from several packages that export nim-json-serialization by default.

* Only except `toJson` from import/export of chronicles
2023-12-07 12:16:36 +11:00
Eric
86a6ef9215
remove waitFor cancelAndWait in proving state (#643)
In proving.onCancelled, the `waitFor state.loop.cancelAndWait()` was never completing. Turns out this was not needed, because when changing states, the current state's run is cancelled, which automatically cancels the state prove loop, because it is a child of proving.run. Therefore, the logic to cancelAndWait the prove loop was removed as it was not needed.
2023-12-06 14:20:14 +11:00
Adam Uhlíř
b38146d3f7
fix: check expiration is before request end (#641) 2023-12-05 14:25:28 +01:00
Adam Uhlíř
0e4387d1b3
refactor: move expiry update from fetchBatched (#634) 2023-11-28 22:04:11 +01:00
Dmitriy Ryajov
22c31046a7
Dont dial self (#633)
* don't dial self

* revert style changes

* filter out self on dial

* add helpers to `peerId` and `isSelf`

* don't fire up discovery eaguerly

* allow excluding multiple peers in sendWantHave

* revert style changes

* move self check to discovery.find

* readd eaguer dht lookup is required in some cases

* revert style changes

* misc

* drop peer first, before queueing a dht lookup

* moar style changes

* use isSelf
2023-11-27 10:25:53 -08:00
markspanbroek
b62ccf5a8a
Update Request: remove PoR, add merkle root (#590)
* [build] update codex-contracts-eth

* [contracts] update request parameters

- Remove PoR parameters and totalChunks
- Add merkleRoot
2023-11-27 12:25:01 +00:00
Adam Uhlíř
79fce39dbf
fix: require expiry from api (#629)
Co-authored-by: markspanbroek <mark@spanbroek.net>
2023-11-22 11:35:26 +00:00