248 Commits

Author SHA1 Message Date
Dmitriy Ryajov
0ebbdfda0e
misc 2024-01-16 19:43:14 -06:00
Dmitriy Ryajov
b3d3274765
cleanup 2024-01-16 19:43:13 -06:00
Dmitriy Ryajov
b0ba10aa8f
only return slot cell indices and no duplicates 2024-01-16 19:43:13 -06:00
Dmitriy Ryajov
7e74a13bd8
use correct block index 2024-01-16 19:43:13 -06:00
benbierens
964fe47ea2 Fix: test for correct verify root in slot builder. 2024-01-16 19:42:27 -06:00
benbierens
9c1ea719d8 Restores tests for sampler 2024-01-16 19:42:27 -06:00
benbierens
4e19384a86 Restores tests for proof sampler utils 2024-01-16 19:42:27 -06:00
benbierens
d80ddd41df Fixes issue where sampler takes proof from dataset/verify tree for slot proof. 2024-01-16 19:42:27 -06:00
benbierens
622239ffbd Cleanup 2024-01-16 19:42:27 -06:00
benbierens
4afec65d00 Applies result-type to onProve callback. 2024-01-16 19:42:27 -06:00
benbierens
2ba0977214 Fixes log scope compilation 2024-01-16 19:42:27 -06:00
benbierens
70541921c4 Fix: ProofInputSample data is cell bytes instead of block bytes 2024-01-16 19:42:27 -06:00
benbierens
901d816f56 Fix: slotIdx used as nSamples 2024-01-16 19:42:27 -06:00
Dmitriy Ryajov
a169e7fe24
misc 2024-01-15 21:47:50 -06:00
Dmitriy Ryajov
b88f767f08
wip sampler integration 2024-01-15 21:47:40 -06:00
Dmitriy Ryajov
98181a72b0
don't use upraises 2024-01-15 21:47:21 -06:00
Dmitriy Ryajov
e8fbb6d755
wip sampler implementation 2024-01-15 21:47:06 -06:00
Dmitriy Ryajov
cfc4632efc
various changes to add support for the sampler 2024-01-15 21:46:48 -06:00
Dmitriy Ryajov
8665af1df3
fix paths 2024-01-15 14:06:06 -06:00
Dmitriy Ryajov
76181407e9
move sampler to own dir 2024-01-15 13:59:55 -06:00
Dmitriy Ryajov
877e19cfcf
move builder to own dir 2024-01-15 13:59:24 -06:00
benbierens
b4654ff938
Updates proof checking test to match new format 2024-01-15 10:50:40 -06:00
benbierens
24be5b2788
setting up 2024-01-15 10:50:40 -06:00
benbierens
03c0f9914e
Lining up output type with updated reference code. 2024-01-15 10:50:40 -06:00
benbierens
54b0fe6f42
Cleanup 2024-01-15 10:50:40 -06:00
benbierens
84993e8984
Refactoring to object-oriented 2024-01-15 10:50:39 -06:00
benbierens
26f238409c
almost there 2024-01-15 10:50:39 -06:00
benbierens
0cc4563eba
splits up indexing 2024-01-15 10:50:39 -06:00
benbierens
4642174c6d
Adds method to get dataset block index from slot block index 2024-01-15 10:50:39 -06:00
benbierens
97350cd890
Implements building a minitree for block cells 2024-01-15 10:50:39 -06:00
benbierens
07ed066b1e
Implements selecting a cell sample from a block 2024-01-15 10:50:38 -06:00
benbierens
b031fb9e4a
Sets up getting sample from block 2024-01-15 10:50:38 -06:00
benbierens
742f84a24d
cleanup 2024-01-15 10:50:38 -06:00
benbierens
975d4ce65f
Implements out-of-range check for slot index 2024-01-15 10:50:38 -06:00
benbierens
b17202c8a7
implements getting slot blocks by index 2024-01-15 10:50:38 -06:00
benbierens
f767d2b4d0
Implements getting treeCID from slot 2024-01-15 10:50:38 -06:00
benbierens
7a47fb6e5d
setting up slot blocks module 2024-01-15 10:50:38 -06:00
benbierens
d395328ed4
Implements cell index collection 2024-01-15 10:50:37 -06:00
benbierens
c320dd2e7d
Sets up tests for bitwise modulo 2024-01-15 10:50:37 -06:00
benbierens
fc6ce6491c
Sets up calculating number of cells in a slot 2024-01-15 10:50:37 -06:00
Dmitriy Ryajov
2fc7c75fd2
Contracts handler (#672)
* get rid of unneeded files

* don't reuse batch callback

* move out storage contract handlers

* wip

* add storage handler, to handle storage contracts interactions

* split out node tests

* a bit more cleanup

* remove storage handler, move back into node

* add missing raises:

* wip: add support for rebuilding slot

* split out tests

* wip

* rework indexing strategy to return an iterator

* convert to seq

* minor api change (meh, might revert)

* steping strategy should not die

* allow fetching batches for manifests and indicies

* restored expiry update

* restored expiry update functionality

* avoid closing datastores twice

* correct wrong rename

* fixes sigsegv
2024-01-15 08:45:04 -08:00
Dmitriy Ryajov
8b12934fe2
Build slots (#668)
Wiring in slots builder functionality into `requestStorage`
2024-01-11 08:45:23 -08:00
Eric
b186afbc4f
bump ethers, remove upraises from callbacks (#664) 2024-01-09 15:51:34 +11:00
Dmitriy Ryajov
fffb674bba
Integrate slot builder (#666)
* rework merkle tree support

* rename merkletree -> codexmerkletree

* treed and proof encoding/decoding

* style

* adding codex merkle and coders tests

* use default hash codec

* proof size changed

* add from nodes test

* shorte file names

* wip poseidon tree

* shorten file names

* root returns a result

* import poseidon tests

* fix merge issues and cleanup a few warnings

* setting up slot builder

* Getting cids in slot

* ensures blocks are devisable by number of slots

* wip

* Implements indexing strategies

* Swaps in indexing strategy into erasure.

* wires slot and indexing tests up

* Fixes issue where indexing strategy stepped gives wrong values for smallest of ranges

* debugs indexing strategies

* Can select slot blocks

* finding number of pad cells

* Implements building slot tree

* finishes implementing slot builder

* Adds check that block size is a multiple of cell size

* Cleanup slotbuilder

* Review comments by Tomasz

* Fixes issue where ecK was used as numberOfSlots.

* rework merkle tree support

* deps

* rename merkletree -> codexmerkletree

* treed and proof encoding/decoding

* style

* adding codex merkle and coders tests

* remove new codecs for now

* proof size changed

* add from nodes test

* shorte file names

* wip poseidon tree

* shorten file names

* fix bad `elements` iter

* bump

* bump

* wip

* reworking slotbuilder

* move out of manifest

* expose getCidAndProof

* import index strat...

* remove getMHash

* remove unused artifacts

* alias zero

* add digest for multihash

* merge issues

* remove unused hashes

* add option to result converter

* misc

* fix tests

* add helper to derive EC block count

* rename method

* misc

* bump

* extract slot root building into own proc

* revert to manifest to accessor

---------

Co-authored-by: benbierens <thatbenbierens@gmail.com>
2024-01-08 14:52:46 -08:00
Dmitriy Ryajov
b8ee2ac71e
Update multicodecs (#665)
* rework merkle tree support

* rename merkletree -> codexmerkletree

* treed and proof encoding/decoding

* style

* adding codex merkle and coders tests

* use default hash codec

* proof size changed

* add from nodes test

* shorte file names

* wip poseidon tree

* shorten file names

* root returns a result

* import poseidon tests

* update multicodecs

* consolidating codex types and adding new codecs

* update codec

* remove temp codecs constants

* move codecs related stuff out

* updating codecs

* misc

* updating sizes since block size was adjusted to 64kb

* fix merge issues and cleanup a few warnings
2023-12-22 13:04:01 +01:00
Dmitriy Ryajov
52c5578c46
Rework merkle tree (#654)
* rework merkle tree support

* deps

* rename merkletree -> codexmerkletree

* treed and proof encoding/decoding

* small change to invoke proof verification

* rename merkletree to codexmerkletree

* style

* adding codex merkle and coders tests

* fixup imports

* remove new codecs for now

* bump deps

* adding trace statement

* properly serde of manifest block codecs

* use default hash codec

* add more trace logging to aid debugging

* misc

* remove double import

* revert un-needded change

* proof size changed

* bump poseidon2

* add from nodes test

* shorte file names

* remove upraises

* wip poseidon tree

* adjust file names

* misc

* shorten file names

* fix bad `elements` iter

* don't do asserts

* add fromNodes and converters

* root and getProof now return result

* add poseidon2 tree tests

* root now returns result

* misc

* had to make merkletree a ref, because nim blows up otherwise

* root returns a result

* root returns a result

* import poseidon tests

* bump

* merkle poseidon2 digest

* misc

* add merkle digest tests

* bump

* don't use checksuite

* Update tests/codex/merkletree/generictreetests.nim

Co-authored-by: markspanbroek <mark@spanbroek.net>
Signed-off-by: Dmitriy Ryajov <dryajov@gmail.com>

* Update codex/merkletree/merkletree.nim

Co-authored-by: markspanbroek <mark@spanbroek.net>
Signed-off-by: Dmitriy Ryajov <dryajov@gmail.com>

* Update codex/merkletree/merkletree.nim

Co-authored-by: markspanbroek <mark@spanbroek.net>
Signed-off-by: Dmitriy Ryajov <dryajov@gmail.com>

* Update tests/codex/merkletree/generictreetests.nim

Co-authored-by: markspanbroek <mark@spanbroek.net>
Signed-off-by: Dmitriy Ryajov <dryajov@gmail.com>

* missing return

* make toBool private (it's still needed otherwise comparison won't work)

* added `digestTree` that returns a tree and `digest` for root

* test against both poseidon trees - codex and poseidon2

* shorten merkle tree names

* don't compare trees - it's going to be too slow

* move comparison to mekrle helper

* remove merkle utils

---------

Signed-off-by: Dmitriy Ryajov <dryajov@gmail.com>
Co-authored-by: markspanbroek <mark@spanbroek.net>
2023-12-21 06:41:43 +00:00
Eric
f293082ae9
Reverts logging-proxy, commit 27f585eb6f380a6de2fbbef721fdf1a16bd5f600 (#660) 2023-12-20 13:24:40 +11:00
Eric
27f585eb6f
feat: create logging proxy (#653)
* implement a logging proxy

The logging proxy:
- prevents the need to import chronicles (as well as export except toJson),
- prevents the need to override `writeValue` or use or import nim-json-seralization elsewhere in the codebase, allowing for sole use of utils/json for de/serialization,
- and handles json formatting correctly in chronicles json sinks

* Rename logging -> logutils to avoid ambiguity with common names

* clean up

* add setProperty for JsonRecord, remove nim-json-serialization conflict

* Allow specifying textlines and json format separately

Not specifying a LogFormat will apply the formatting to both textlines and json sinks.

Specifying a LogFormat will apply the formatting to only that sink.

* remove unneeded usages of std/json

We only need to import utils/json instead of std/json

* move serialization from rest/json to utils/json so it can be shared

* fix NoColors ambiguity

Was causing unit tests to fail on Windows.

* Remove nre usage to fix Windows error

Windows was erroring with `could not load: pcre64.dll`. Instead of fixing that error, remove the pcre usage :)

* Add logutils module doc

* Shorten logutils.formatIt for `NBytes`

Both json and textlines formatIt were not needed, and could be combined into one formatIt

* remove debug integration test config

debug output and logformat of json for integration test logs

* Use ## module doc to support docgen
2023-12-19 22:12:47 +00:00
Eric
df6b9c6760
Workaround for Hardhat last block timestamp bug (#644)
* Workaround for Hardhat timestamp bug

Likely due to a Hardhat bug in which the callbacks for subscription events are called and awaited before updating its local understanding of the last block time, Hardhat will report a block time in the `newHeads` event that is generally 1 second before the time reported from `getLatestBlock.timestamp`. This was causing issues with the OnChainClock's offset and therefore the `now()` used by the `OnChainClock` would sometimes be off by a second (or more), causing tests to fail.

This commit introduce a `codex_use_hardhat` compilation flag, that when set, will always get the latest block timestamp from Hardhat via the `getLatestBlock.timestamp` RPC call for `OnChainClock.now` calls. Otherwise, the last block timestamp reported in the `newHeads` event will be used.

Update the docker dist tests compilation flag for simulated proof failures (it was not correct), and explicitly add the `codex_use_hardhat=false` for clarity.

* enable simulated proof failures for coverage

* comment out failing test on linux -- will be replaced

* bump codex contracts eth

* add back clock offset for non-hardhat cases

* bump codex-contracts-eth

increases pointer by 67 blocks each period increase

* Add `codex_use_hardhat` flag to coverage tests
2023-12-20 08:06:24 +11:00
Eric
2c621e0421
chore: improve log information (#656)
* Add proof downtime to validator logs

* Add previous state to state machine transition log
2023-12-19 04:29:18 +00:00