332 Commits

Author SHA1 Message Date
Dmitriy Ryajov
b8a859d5a5
add helper to derive EC block count 2023-12-22 17:25:07 -06:00
Dmitriy Ryajov
f15f3f5f27
fix tests 2023-12-22 17:25:06 -06:00
Dmitriy Ryajov
9c9730d23a
reworking slotbuilder 2023-12-22 17:25:05 -06:00
Dmitriy Ryajov
98f60411a6
wip 2023-12-22 17:25:04 -06:00
Dmitriy Ryajov
6f4bde6907
shorten file names 2023-12-22 17:25:04 -06:00
Dmitriy Ryajov
f4bc1187ba
add from nodes test 2023-12-22 17:25:03 -06:00
Dmitriy Ryajov
2736d78619
rework merkle tree support 2023-12-22 17:25:02 -06:00
benbierens
4bdf0dfd33
Review comments by Tomasz 2023-12-22 17:25:02 -06:00
benbierens
fd156e9cf8
Adds check that block size is a multiple of cell size 2023-12-22 17:25:02 -06:00
benbierens
eead39ba1c
finishes implementing slot builder 2023-12-22 17:25:02 -06:00
benbierens
e7a39dcb6e
Implements building slot tree 2023-12-22 17:25:01 -06:00
benbierens
00e376aee7
finding number of pad cells 2023-12-22 17:25:01 -06:00
benbierens
d883e6e95f
Can select slot blocks 2023-12-22 17:25:01 -06:00
benbierens
fc41545006
debugs indexing strategies 2023-12-22 17:25:01 -06:00
benbierens
6a3102246d
Fixes issue where indexing strategy stepped gives wrong values for smallest of ranges 2023-12-22 17:25:01 -06:00
benbierens
d4ed43bc05
wires slot and indexing tests up 2023-12-22 17:25:01 -06:00
benbierens
2c2e42d363
Swaps in indexing strategy into erasure. 2023-12-22 17:25:01 -06:00
benbierens
46612123d8
Implements indexing strategies 2023-12-22 17:25:00 -06:00
benbierens
fabd2e20a8
wip 2023-12-22 17:25:00 -06:00
benbierens
c871102d98
ensures blocks are devisable by number of slots 2023-12-22 17:25:00 -06:00
benbierens
8d17fb40b4
Getting cids in slot 2023-12-22 17:25:00 -06:00
benbierens
e7996bf058
setting up slot builder 2023-12-22 17:25:00 -06:00
Dmitriy Ryajov
1b0b024ca8
import poseidon tests 2023-12-22 17:22:59 -06:00
Dmitriy Ryajov
befe9775be
shorten file names 2023-12-22 17:22:59 -06:00
Dmitriy Ryajov
b110a63002
add from nodes test 2023-12-22 17:22:59 -06:00
Dmitriy Ryajov
fc4fb1b203
rework merkle tree support 2023-12-22 17:22:58 -06:00
Dmitriy Ryajov
b8ee2ac71e
Update multicodecs (#665)
* rework merkle tree support

* rename merkletree -> codexmerkletree

* treed and proof encoding/decoding

* style

* adding codex merkle and coders tests

* use default hash codec

* proof size changed

* add from nodes test

* shorte file names

* wip poseidon tree

* shorten file names

* root returns a result

* import poseidon tests

* update multicodecs

* consolidating codex types and adding new codecs

* update codec

* remove temp codecs constants

* move codecs related stuff out

* updating codecs

* misc

* updating sizes since block size was adjusted to 64kb

* fix merge issues and cleanup a few warnings
2023-12-22 13:04:01 +01:00
Dmitriy Ryajov
52c5578c46
Rework merkle tree (#654)
* rework merkle tree support

* deps

* rename merkletree -> codexmerkletree

* treed and proof encoding/decoding

* small change to invoke proof verification

* rename merkletree to codexmerkletree

* style

* adding codex merkle and coders tests

* fixup imports

* remove new codecs for now

* bump deps

* adding trace statement

* properly serde of manifest block codecs

* use default hash codec

* add more trace logging to aid debugging

* misc

* remove double import

* revert un-needded change

* proof size changed

* bump poseidon2

* add from nodes test

* shorte file names

* remove upraises

* wip poseidon tree

* adjust file names

* misc

* shorten file names

* fix bad `elements` iter

* don't do asserts

* add fromNodes and converters

* root and getProof now return result

* add poseidon2 tree tests

* root now returns result

* misc

* had to make merkletree a ref, because nim blows up otherwise

* root returns a result

* root returns a result

* import poseidon tests

* bump

* merkle poseidon2 digest

* misc

* add merkle digest tests

* bump

* don't use checksuite

* Update tests/codex/merkletree/generictreetests.nim

Co-authored-by: markspanbroek <mark@spanbroek.net>
Signed-off-by: Dmitriy Ryajov <dryajov@gmail.com>

* Update codex/merkletree/merkletree.nim

Co-authored-by: markspanbroek <mark@spanbroek.net>
Signed-off-by: Dmitriy Ryajov <dryajov@gmail.com>

* Update codex/merkletree/merkletree.nim

Co-authored-by: markspanbroek <mark@spanbroek.net>
Signed-off-by: Dmitriy Ryajov <dryajov@gmail.com>

* Update tests/codex/merkletree/generictreetests.nim

Co-authored-by: markspanbroek <mark@spanbroek.net>
Signed-off-by: Dmitriy Ryajov <dryajov@gmail.com>

* missing return

* make toBool private (it's still needed otherwise comparison won't work)

* added `digestTree` that returns a tree and `digest` for root

* test against both poseidon trees - codex and poseidon2

* shorten merkle tree names

* don't compare trees - it's going to be too slow

* move comparison to mekrle helper

* remove merkle utils

---------

Signed-off-by: Dmitriy Ryajov <dryajov@gmail.com>
Co-authored-by: markspanbroek <mark@spanbroek.net>
2023-12-21 06:41:43 +00:00
Eric
954c8edf76
comment out, instead of skipping, intermittently failing test (#661)
`skip` seems to working only sometimes. It intermittently fails on Windows in CI. The reason for skipping the test in the first place was to prevent intermittent failures of the test itself, so adding `skip` did not improve the situation, unfortunately.
2023-12-20 22:12:24 +00:00
Eric
f293082ae9
Reverts logging-proxy, commit 27f585eb6f380a6de2fbbef721fdf1a16bd5f600 (#660) 2023-12-20 13:24:40 +11:00
Eric
27f585eb6f
feat: create logging proxy (#653)
* implement a logging proxy

The logging proxy:
- prevents the need to import chronicles (as well as export except toJson),
- prevents the need to override `writeValue` or use or import nim-json-seralization elsewhere in the codebase, allowing for sole use of utils/json for de/serialization,
- and handles json formatting correctly in chronicles json sinks

* Rename logging -> logutils to avoid ambiguity with common names

* clean up

* add setProperty for JsonRecord, remove nim-json-serialization conflict

* Allow specifying textlines and json format separately

Not specifying a LogFormat will apply the formatting to both textlines and json sinks.

Specifying a LogFormat will apply the formatting to only that sink.

* remove unneeded usages of std/json

We only need to import utils/json instead of std/json

* move serialization from rest/json to utils/json so it can be shared

* fix NoColors ambiguity

Was causing unit tests to fail on Windows.

* Remove nre usage to fix Windows error

Windows was erroring with `could not load: pcre64.dll`. Instead of fixing that error, remove the pcre usage :)

* Add logutils module doc

* Shorten logutils.formatIt for `NBytes`

Both json and textlines formatIt were not needed, and could be combined into one formatIt

* remove debug integration test config

debug output and logformat of json for integration test logs

* Use ## module doc to support docgen
2023-12-19 22:12:47 +00:00
Eric
c54ee9ea1a
temporarily remove failing integration test (#659)
* temporarily remove failing integration test

Temporarily removing this test for now as it occasionally fails on macos/linux. This test has been completely refactored in PR #607 in which it will be retintroduced.

* skip instead of comment out
2023-12-19 21:39:58 +00:00
Eric
df6b9c6760
Workaround for Hardhat last block timestamp bug (#644)
* Workaround for Hardhat timestamp bug

Likely due to a Hardhat bug in which the callbacks for subscription events are called and awaited before updating its local understanding of the last block time, Hardhat will report a block time in the `newHeads` event that is generally 1 second before the time reported from `getLatestBlock.timestamp`. This was causing issues with the OnChainClock's offset and therefore the `now()` used by the `OnChainClock` would sometimes be off by a second (or more), causing tests to fail.

This commit introduce a `codex_use_hardhat` compilation flag, that when set, will always get the latest block timestamp from Hardhat via the `getLatestBlock.timestamp` RPC call for `OnChainClock.now` calls. Otherwise, the last block timestamp reported in the `newHeads` event will be used.

Update the docker dist tests compilation flag for simulated proof failures (it was not correct), and explicitly add the `codex_use_hardhat=false` for clarity.

* enable simulated proof failures for coverage

* comment out failing test on linux -- will be replaced

* bump codex contracts eth

* add back clock offset for non-hardhat cases

* bump codex-contracts-eth

increases pointer by 67 blocks each period increase

* Add `codex_use_hardhat` flag to coverage tests
2023-12-20 08:06:24 +11:00
Eric
2c621e0421
chore: improve log information (#656)
* Add proof downtime to validator logs

* Add previous state to state machine transition log
2023-12-19 04:29:18 +00:00
Eric
54c6258e8d
Rename provider to ethProvider in tests (#657) 2023-12-18 09:34:04 +00:00
Ben Bierens
f1b1fe9152
Adds endpoint to expose quota information (#652) 2023-12-14 10:57:16 +00:00
Adam Uhlíř
e62cb96bde
feat: cleanup returning availability (#650)
Co-authored-by: markspanbroek <mark@spanbroek.net>
2023-12-13 20:58:17 +01:00
Ben Bierens
0c3d1dd563
Verifiable manifests (#642)
* Adds test for encoding/decoding protected manifest

* Setting up verifiable manifest

* mysterious mysteries

* Successful encoding test for verifiable manifests

* extracts toF out of users of manifest code

* Update codex/manifest/coders.nim

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Signed-off-by: Ben Bierens <39762930+benbierens@users.noreply.github.com>

* Review comments by Dmitriy

* Adds missing verifiable print to $ method.

* Replace poseidon2 F type with int as temporary stand-in for verification hashes

* Replaces verification hash placeholder with CID

---------

Signed-off-by: Ben Bierens <39762930+benbierens@users.noreply.github.com>
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2023-12-12 08:11:54 +00:00
Ben Bierens
a9f8090bb4
Add challenge to prove callback (#649)
* Sets up passing proof challenge to onProve callback

* Implements tests for proof challenge

* Removes unused import
2023-12-11 11:29:15 +01:00
Eric
3907ca4095
Add get active slot /slots/{slotId} to REST api, use utils/json (#645)
* Add get active slot /slots/{slotId} to REST api, use utils/json

- Add endpoint /slots/{slotId} to get an active SalesAgent from the Sales module. Used in integration tests to test when a sale has reached a certain state. Those integration test changes will be included in a larger PR, coming later.
- Add OpenAPI changes for new endpoint and associated components
- Use utils/json instead of nim-json-serialization. Required exemption of imports from several packages that export nim-json-serialization by default.

* Only except `toJson` from import/export of chronicles
2023-12-07 12:16:36 +11:00
Adam Uhlíř
b38146d3f7
fix: check expiration is before request end (#641) 2023-12-05 14:25:28 +01:00
Adam Uhlíř
0e4387d1b3
refactor: move expiry update from fetchBatched (#634) 2023-11-28 22:04:11 +01:00
Dmitriy Ryajov
22c31046a7
Dont dial self (#633)
* don't dial self

* revert style changes

* filter out self on dial

* add helpers to `peerId` and `isSelf`

* don't fire up discovery eaguerly

* allow excluding multiple peers in sendWantHave

* revert style changes

* move self check to discovery.find

* readd eaguer dht lookup is required in some cases

* revert style changes

* misc

* drop peer first, before queueing a dht lookup

* moar style changes

* use isSelf
2023-11-27 10:25:53 -08:00
markspanbroek
b62ccf5a8a
Update Request: remove PoR, add merkle root (#590)
* [build] update codex-contracts-eth

* [contracts] update request parameters

- Remove PoR parameters and totalChunks
- Add merkleRoot
2023-11-27 12:25:01 +00:00
Adam Uhlíř
79fce39dbf
fix: require expiry from api (#629)
Co-authored-by: markspanbroek <mark@spanbroek.net>
2023-11-22 11:35:26 +00:00
Adam Uhlíř
8681a40ee7
feat: update expiry when data downloaded and slot filled (#619)
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
Co-authored-by: markspanbroek <mark@spanbroek.net>
2023-11-22 10:09:12 +00:00
Tomasz Bekas
4d546f9ace
Bump questionable and test for tree init failure (#630) 2023-11-21 19:30:14 +01:00
Dmitriy Ryajov
ec8d0c98b2
Fix REST endpoints semantics (#612)
* Fix REST endpoints semantics

* update endpoint description

* update, operation id

* Adding enum support

* make enum descerializer public

* add support for listing manifests

* test `/data` endpoint to list local manifests

* debug leftovers

* remove commented out line
2023-11-20 16:14:06 -08:00
Ben Bierens
bece1b88a1
Feat/bump questionable (#627)
* Bumps questionable version to 0.10.12

* removes unnecessary questionnable bindings.

* Fixes tests

* unnecessary whitespaces
2023-11-17 13:49:45 +01:00
Dmitriy Ryajov
06bb21bfc7
backport changes from fix-erasure-tests (#618) 2023-11-14 08:53:06 -08:00