567 Commits

Author SHA1 Message Date
Dmitriy Ryajov
30dae32ff3
remove new codecs for now 2023-12-22 17:25:03 -06:00
Dmitriy Ryajov
a52c5c3409
adding codex merkle and coders tests 2023-12-22 17:25:03 -06:00
Dmitriy Ryajov
a87c8cbfb8
style 2023-12-22 17:25:03 -06:00
Dmitriy Ryajov
3902f7e695
treed and proof encoding/decoding 2023-12-22 17:25:03 -06:00
Dmitriy Ryajov
818ecf852f
rename merkletree -> codexmerkletree 2023-12-22 17:25:03 -06:00
Dmitriy Ryajov
10cee0e35a
deps 2023-12-22 17:25:02 -06:00
Dmitriy Ryajov
2736d78619
rework merkle tree support 2023-12-22 17:25:02 -06:00
benbierens
c0f4512f62
Fixes issue where ecK was used as numberOfSlots. 2023-12-22 17:25:02 -06:00
benbierens
4bdf0dfd33
Review comments by Tomasz 2023-12-22 17:25:02 -06:00
benbierens
2b63cb0e08
Cleanup slotbuilder 2023-12-22 17:25:02 -06:00
benbierens
fd156e9cf8
Adds check that block size is a multiple of cell size 2023-12-22 17:25:02 -06:00
benbierens
eead39ba1c
finishes implementing slot builder 2023-12-22 17:25:02 -06:00
benbierens
e7a39dcb6e
Implements building slot tree 2023-12-22 17:25:01 -06:00
benbierens
00e376aee7
finding number of pad cells 2023-12-22 17:25:01 -06:00
benbierens
d883e6e95f
Can select slot blocks 2023-12-22 17:25:01 -06:00
benbierens
fc41545006
debugs indexing strategies 2023-12-22 17:25:01 -06:00
benbierens
6a3102246d
Fixes issue where indexing strategy stepped gives wrong values for smallest of ranges 2023-12-22 17:25:01 -06:00
benbierens
d4ed43bc05
wires slot and indexing tests up 2023-12-22 17:25:01 -06:00
benbierens
2c2e42d363
Swaps in indexing strategy into erasure. 2023-12-22 17:25:01 -06:00
benbierens
46612123d8
Implements indexing strategies 2023-12-22 17:25:00 -06:00
benbierens
fabd2e20a8
wip 2023-12-22 17:25:00 -06:00
benbierens
c871102d98
ensures blocks are devisable by number of slots 2023-12-22 17:25:00 -06:00
benbierens
8d17fb40b4
Getting cids in slot 2023-12-22 17:25:00 -06:00
benbierens
e7996bf058
setting up slot builder 2023-12-22 17:25:00 -06:00
Dmitriy Ryajov
75a75446bb
fix merge issues and cleanup a few warnings 2023-12-22 17:25:00 -06:00
Dmitriy Ryajov
1b0b024ca8
import poseidon tests 2023-12-22 17:22:59 -06:00
Dmitriy Ryajov
5e528756a2
root returns a result 2023-12-22 17:22:59 -06:00
Dmitriy Ryajov
befe9775be
shorten file names 2023-12-22 17:22:59 -06:00
Dmitriy Ryajov
eaf052fea9
wip poseidon tree 2023-12-22 17:22:59 -06:00
Dmitriy Ryajov
80de04889f
shorte file names 2023-12-22 17:22:59 -06:00
Dmitriy Ryajov
b110a63002
add from nodes test 2023-12-22 17:22:59 -06:00
Dmitriy Ryajov
3d3f6e023e
proof size changed 2023-12-22 17:22:58 -06:00
Dmitriy Ryajov
8f0102f0eb
use default hash codec 2023-12-22 17:22:58 -06:00
Dmitriy Ryajov
ff17551e32
adding codex merkle and coders tests 2023-12-22 17:22:58 -06:00
Dmitriy Ryajov
d032149924
style 2023-12-22 17:22:58 -06:00
Dmitriy Ryajov
75976996ed
treed and proof encoding/decoding 2023-12-22 17:22:58 -06:00
Dmitriy Ryajov
1437df6847
rename merkletree -> codexmerkletree 2023-12-22 17:22:58 -06:00
Dmitriy Ryajov
fc4fb1b203
rework merkle tree support 2023-12-22 17:22:58 -06:00
Dmitriy Ryajov
b8ee2ac71e
Update multicodecs (#665)
* rework merkle tree support

* rename merkletree -> codexmerkletree

* treed and proof encoding/decoding

* style

* adding codex merkle and coders tests

* use default hash codec

* proof size changed

* add from nodes test

* shorte file names

* wip poseidon tree

* shorten file names

* root returns a result

* import poseidon tests

* update multicodecs

* consolidating codex types and adding new codecs

* update codec

* remove temp codecs constants

* move codecs related stuff out

* updating codecs

* misc

* updating sizes since block size was adjusted to 64kb

* fix merge issues and cleanup a few warnings
2023-12-22 13:04:01 +01:00
Dmitriy Ryajov
52c5578c46
Rework merkle tree (#654)
* rework merkle tree support

* deps

* rename merkletree -> codexmerkletree

* treed and proof encoding/decoding

* small change to invoke proof verification

* rename merkletree to codexmerkletree

* style

* adding codex merkle and coders tests

* fixup imports

* remove new codecs for now

* bump deps

* adding trace statement

* properly serde of manifest block codecs

* use default hash codec

* add more trace logging to aid debugging

* misc

* remove double import

* revert un-needded change

* proof size changed

* bump poseidon2

* add from nodes test

* shorte file names

* remove upraises

* wip poseidon tree

* adjust file names

* misc

* shorten file names

* fix bad `elements` iter

* don't do asserts

* add fromNodes and converters

* root and getProof now return result

* add poseidon2 tree tests

* root now returns result

* misc

* had to make merkletree a ref, because nim blows up otherwise

* root returns a result

* root returns a result

* import poseidon tests

* bump

* merkle poseidon2 digest

* misc

* add merkle digest tests

* bump

* don't use checksuite

* Update tests/codex/merkletree/generictreetests.nim

Co-authored-by: markspanbroek <mark@spanbroek.net>
Signed-off-by: Dmitriy Ryajov <dryajov@gmail.com>

* Update codex/merkletree/merkletree.nim

Co-authored-by: markspanbroek <mark@spanbroek.net>
Signed-off-by: Dmitriy Ryajov <dryajov@gmail.com>

* Update codex/merkletree/merkletree.nim

Co-authored-by: markspanbroek <mark@spanbroek.net>
Signed-off-by: Dmitriy Ryajov <dryajov@gmail.com>

* Update tests/codex/merkletree/generictreetests.nim

Co-authored-by: markspanbroek <mark@spanbroek.net>
Signed-off-by: Dmitriy Ryajov <dryajov@gmail.com>

* missing return

* make toBool private (it's still needed otherwise comparison won't work)

* added `digestTree` that returns a tree and `digest` for root

* test against both poseidon trees - codex and poseidon2

* shorten merkle tree names

* don't compare trees - it's going to be too slow

* move comparison to mekrle helper

* remove merkle utils

---------

Signed-off-by: Dmitriy Ryajov <dryajov@gmail.com>
Co-authored-by: markspanbroek <mark@spanbroek.net>
2023-12-21 06:41:43 +00:00
Eric
954c8edf76
comment out, instead of skipping, intermittently failing test (#661)
`skip` seems to working only sometimes. It intermittently fails on Windows in CI. The reason for skipping the test in the first place was to prevent intermittent failures of the test itself, so adding `skip` did not improve the situation, unfortunately.
2023-12-20 22:12:24 +00:00
Eric
f293082ae9
Reverts logging-proxy, commit 27f585eb6f380a6de2fbbef721fdf1a16bd5f600 (#660) 2023-12-20 13:24:40 +11:00
Eric
27f585eb6f
feat: create logging proxy (#653)
* implement a logging proxy

The logging proxy:
- prevents the need to import chronicles (as well as export except toJson),
- prevents the need to override `writeValue` or use or import nim-json-seralization elsewhere in the codebase, allowing for sole use of utils/json for de/serialization,
- and handles json formatting correctly in chronicles json sinks

* Rename logging -> logutils to avoid ambiguity with common names

* clean up

* add setProperty for JsonRecord, remove nim-json-serialization conflict

* Allow specifying textlines and json format separately

Not specifying a LogFormat will apply the formatting to both textlines and json sinks.

Specifying a LogFormat will apply the formatting to only that sink.

* remove unneeded usages of std/json

We only need to import utils/json instead of std/json

* move serialization from rest/json to utils/json so it can be shared

* fix NoColors ambiguity

Was causing unit tests to fail on Windows.

* Remove nre usage to fix Windows error

Windows was erroring with `could not load: pcre64.dll`. Instead of fixing that error, remove the pcre usage :)

* Add logutils module doc

* Shorten logutils.formatIt for `NBytes`

Both json and textlines formatIt were not needed, and could be combined into one formatIt

* remove debug integration test config

debug output and logformat of json for integration test logs

* Use ## module doc to support docgen
2023-12-19 22:12:47 +00:00
Eric
c54ee9ea1a
temporarily remove failing integration test (#659)
* temporarily remove failing integration test

Temporarily removing this test for now as it occasionally fails on macos/linux. This test has been completely refactored in PR #607 in which it will be retintroduced.

* skip instead of comment out
2023-12-19 21:39:58 +00:00
Eric
df6b9c6760
Workaround for Hardhat last block timestamp bug (#644)
* Workaround for Hardhat timestamp bug

Likely due to a Hardhat bug in which the callbacks for subscription events are called and awaited before updating its local understanding of the last block time, Hardhat will report a block time in the `newHeads` event that is generally 1 second before the time reported from `getLatestBlock.timestamp`. This was causing issues with the OnChainClock's offset and therefore the `now()` used by the `OnChainClock` would sometimes be off by a second (or more), causing tests to fail.

This commit introduce a `codex_use_hardhat` compilation flag, that when set, will always get the latest block timestamp from Hardhat via the `getLatestBlock.timestamp` RPC call for `OnChainClock.now` calls. Otherwise, the last block timestamp reported in the `newHeads` event will be used.

Update the docker dist tests compilation flag for simulated proof failures (it was not correct), and explicitly add the `codex_use_hardhat=false` for clarity.

* enable simulated proof failures for coverage

* comment out failing test on linux -- will be replaced

* bump codex contracts eth

* add back clock offset for non-hardhat cases

* bump codex-contracts-eth

increases pointer by 67 blocks each period increase

* Add `codex_use_hardhat` flag to coverage tests
2023-12-20 08:06:24 +11:00
Eric
1186616381
Updates reference to continuous tests workflow in cs-codex-dist-tests (#658) 2023-12-19 06:47:23 +00:00
Eric
2c621e0421
chore: improve log information (#656)
* Add proof downtime to validator logs

* Add previous state to state machine transition log
2023-12-19 04:29:18 +00:00
Eric
39554c8ea9
sales was never marked as running and caused a hang (#655) 2023-12-19 01:28:17 +00:00
Eric
54c6258e8d
Rename provider to ethProvider in tests (#657) 2023-12-18 09:34:04 +00:00
Ben Bierens
f1b1fe9152
Adds endpoint to expose quota information (#652) 2023-12-14 10:57:16 +00:00