526 Commits

Author SHA1 Message Date
Eric
e762860e6b
speed up tests
- advance to next period and mine every 500ms
- update eventually checks in tests to check for number of periods, not time
- change log file format to include test name
2023-12-06 15:39:08 +11:00
Eric
045f770128
All tests working, with no time advancement
- works with hardhat already running
- works with ethersuite to revert snapshot (ethProvider instead of provider)
- OnChainClock.now() gets latest block timestamp
2023-12-06 15:39:08 +11:00
Eric
3d9eb19855
All integration tests working 2023-12-06 15:39:08 +11:00
Eric
ca4181fe00
WIP proofs tests 2023-12-06 15:39:08 +11:00
Eric
93f9dbba08
WIP getting integration tests to work with marketplacesuite 2023-12-06 15:39:08 +11:00
Eric
eda4a85c3e
DRYing, renaming 2023-12-06 15:39:08 +11:00
Eric
935929461f
fix hardhat stop
- allow multiple multinode suites in one module
- remove need to rely on npmstart script, use eval command instead
- properly override stop method to shutdown hardhat node
2023-12-06 15:39:08 +11:00
Eric
4dffb643a4
comment out other integration tests for now 2023-12-06 15:39:08 +11:00
Eric
3964f868f5
Remove cancelTransaction, add period cadence to testproofs 2023-12-06 15:39:08 +11:00
Eric
bc0652aeb6
Get balance for each provider, bump ethers 2023-12-06 15:39:08 +11:00
Eric
10decd2f28
allow integration tests to start hardhat locally so logs can be captured 2023-12-06 15:39:08 +11:00
Eric
40ee931d6d
remove cancellation of txs in market, handled by ethers now 2023-12-06 15:39:08 +11:00
Adam Uhlíř
3b520fc0c6
fix: unknown state goes to payout when slot state is finished (#555)
Support logging to file

Log the entire config and fix build error

Process slot queue on reservation callback onMarkUnused

Add Reservation object, rename reserve > create

refactor Reservations api to include Reservation CRUD

All tests that use the Reservation module updated

- add requestId and slotIndex to Reservation (hopefully these will prove to be useful when we persist Reservations until request are completed, to add back bytes to Availability)
- add querying of all reservations, with accompanying tests
- change from find to findAvailabilities
- move onCleanUp from SalesContext to SalesAgent as it was getting overwritten for each slot processed
- remove sales agent AFTER deleting reservation, as this was causing some SIGSEGVs
- retrofit testsales and testslotqueue to match updated Reservations module API

Add deletion of inactive reservations on sales load

clean up

add exception message detail util

Apply to onStore errors as we are seeing undetailed errors in the dist tests logs

add missing file

change slotsize to reflect current implemenation

Fix slotSize to reduce by one block

Revert change to slotSize that reduces it by one block

Add additional test check for querying reservations/availabilities

filter past requests based on availability

Because availability filtering on push was removed, when availability is added and past storage request events are queried, those requests need to be filtered by availability before being added to the queue.

Revert "filter past requests based on availability"

This reverts commit 0c2362658b523e0de425794b1fa30ebd53bd30ae.

Add debugging for dist tests

Add cancel on error during filling state

When calling fillSlot, any transaction errors that occur (possibly during estimate gas) will cause that tx to be replaced with a cancellation transaction (a 0-valued tx to ourselves).

more debug logging

fix build

wait for fillSlot to be mined (so that a revert error can be extracted)

fix confirmation of fillSlot

add more debugging

moar debugging

debugging: change echo to trace

bump ethers to add chronicles

fix contracts tests

switch to earlier nim-ethers commit

bump json-rpc and nim-ethers

bump nim-ethers to prevent parsing newHeads log from crashing sales state machine

moar debugging

moar debugging

moar debugging

bump ethers to fix "key not found: data" error

ethers debug logging

bump ethers - better Transaction object deserialization

bump ethers to replay tx with past tx format

bump ethers to add serialization for PastTransaction

ethers bump: better logging, separate logic for revert reason string

fix build

ethers: update revert reason retreival

remove unneeded confirm

ethers: try replay without decrementing blockNumber

ethers: include type and chainId in replayed txs

ethers: add gas into and remove type from replayed txs

ensure gas is being serialized in Transaction

ethers: fix build error

bump ethers: rebased on top of cancel tx due to estimateGas error PR

Fix proving with latest ethers bump (tested on master)

Update multinode suite for better simulateFailedProofs enabling, add proofs test

Improve multinode suite for better debug options, including logging to file

There is a 503 "sales unavailable" error

improve multinode test suite, add logging to file
2023-12-06 15:39:03 +11:00
Eric
86a6ef9215
remove waitFor cancelAndWait in proving state (#643)
In proving.onCancelled, the `waitFor state.loop.cancelAndWait()` was never completing. Turns out this was not needed, because when changing states, the current state's run is cancelled, which automatically cancels the state prove loop, because it is a child of proving.run. Therefore, the logic to cancelAndWait the prove loop was removed as it was not needed.
2023-12-06 14:20:14 +11:00
Adam Uhlíř
b38146d3f7
fix: check expiration is before request end (#641) 2023-12-05 14:25:28 +01:00
Adam Uhlíř
0e4387d1b3
refactor: move expiry update from fetchBatched (#634) 2023-11-28 22:04:11 +01:00
Dmitriy Ryajov
22c31046a7
Dont dial self (#633)
* don't dial self

* revert style changes

* filter out self on dial

* add helpers to `peerId` and `isSelf`

* don't fire up discovery eaguerly

* allow excluding multiple peers in sendWantHave

* revert style changes

* move self check to discovery.find

* readd eaguer dht lookup is required in some cases

* revert style changes

* misc

* drop peer first, before queueing a dht lookup

* moar style changes

* use isSelf
2023-11-27 10:25:53 -08:00
Dmitriy Ryajov
e871859a96
Bump dht (#628)
* rename nim-libp2p-dht to nim-codex-dht

* bump dht

* update .gitmodules

* bump

---------

Co-authored-by: Jaremy Creechley <creechley@gmail.com>
2023-11-27 09:23:25 -08:00
markspanbroek
b62ccf5a8a
Update Request: remove PoR, add merkle root (#590)
* [build] update codex-contracts-eth

* [contracts] update request parameters

- Remove PoR parameters and totalChunks
- Add merkleRoot
2023-11-27 12:25:01 +00:00
Slava
a805af02c1
Run C-Tests on master commits (#636)
https://github.com/codex-storage/cs-codex-dist-tests/pull/84
2023-11-27 13:32:13 +02:00
Adam Uhlíř
79fce39dbf
fix: require expiry from api (#629)
Co-authored-by: markspanbroek <mark@spanbroek.net>
2023-11-22 11:35:26 +00:00
Adam Uhlíř
8681a40ee7
feat: update expiry when data downloaded and slot filled (#619)
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
Co-authored-by: markspanbroek <mark@spanbroek.net>
2023-11-22 10:09:12 +00:00
Tomasz Bekas
4d546f9ace
Bump questionable and test for tree init failure (#630) 2023-11-21 19:30:14 +01:00
Dmitriy Ryajov
ec8d0c98b2
Fix REST endpoints semantics (#612)
* Fix REST endpoints semantics

* update endpoint description

* update, operation id

* Adding enum support

* make enum descerializer public

* add support for listing manifests

* test `/data` endpoint to list local manifests

* debug leftovers

* remove commented out line
2023-11-20 16:14:06 -08:00
Tomasz Bekas
70efd13c73
Fix block delete by treeCid and index (#623) 2023-11-20 14:48:56 +01:00
markspanbroek
374e6b645b
Add poseidon2 dependency (#625)
* [build] add nim-poseidon2

* [build] Workaround incompatibility constantine and secp256k1

Constantine sets compiler flag that is incompatible
with the assembly code in secp256k1:
5f7ba18f2e/constantine/platforms/isa/macro_assembler_x86.nim (L19)

---------

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2023-11-20 09:51:36 +01:00
Ben Bierens
750fe2392f
Adds option to automatically announce codex by its public IP (#621)
* Adds option to automatically announce codex by its public IP

* Review comments by Slava

* Make NAT_PUBLIC_IP_AUTO more verbose and error resistant

Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>

---------

Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
Co-authored-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2023-11-17 13:48:25 +00:00
Ben Bierens
bece1b88a1
Feat/bump questionable (#627)
* Bumps questionable version to 0.10.12

* removes unnecessary questionnable bindings.

* Fixes tests

* unnecessary whitespaces
2023-11-17 13:49:45 +01:00
Ben Bierens
40d7714a67
Fixes json formatting for NodeId (#614) 2023-11-15 13:29:17 +00:00
Dmitriy Ryajov
4d50ecf504
handle exceptions gracefuly in rest api (#617) 2023-11-14 11:05:43 -08:00
Dmitriy Ryajov
8d61391073
Light cleanup (#616)
mostly whitespaces cleanup
2023-11-14 09:52:27 -08:00
Dmitriy Ryajov
06bb21bfc7
backport changes from fix-erasure-tests (#618) 2023-11-14 08:53:06 -08:00
Dmitriy Ryajov
a1725594a8
Fix json serialization for cid and enums (#615)
* use str on JString types, `$` will preserve `"`

* Adding enum support

* deserialize cid test

* make enum descerializer public

* unify fromJson for objects and refs

* add enum descerialization testing
2023-11-14 07:23:50 -08:00
Tomasz Bekas
2396c4d76d
Blockexchange uses merkle root and index to fetch blocks (#566)
* Blockexchange uses merkle root and index to fetch blocks

* Links the network store getTree to the local store.

* Update codex/stores/repostore.nim

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Signed-off-by: Tomasz Bekas <tomasz.bekas@gmail.com>

* Rework erasure.nim to include recent cleanup

* Revert accidential changes to lib versions

* Addressing review comments

* Storing proofs instead of trees

* Fix a comment

* Fix broken tests

* Fix for broken testerasure.nim

* Addressing PR comments

---------

Signed-off-by: Tomasz Bekas <tomasz.bekas@gmail.com>
Co-authored-by: benbierens <thatbenbierens@gmail.com>
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2023-11-14 13:02:17 +01:00
Adam Uhlíř
778093d12e
docs(api): add note about nodes in request storage creation (#547) 2023-11-13 14:39:22 +00:00
Ben Bierens
34a9c5e88d
Storage/purchases endpoint (#609)
* adds list of purchase ids to storage API

* adds list of purchase ids to storage API

* Updates openAPI

* fixes openAPI type
2023-11-13 11:30:27 +00:00
Ben Bierens
80e2ef4eb5
Debug/info shows announce addresses (#610)
* Shows announce addresses in debug/info endpoint.

* Shows announce addresses in debug/info endpoint.
2023-11-12 09:58:40 +01:00
Ben Bierens
cb02962231
Adds endpoint for listing files (manifests) in node. Useful for demo UI. (#599)
* Adds endpoint for listing files (manifests) in node. Useful for demo UI.

* Moves upload/download/files into content API calls.

* Cleans up json serialization for manifest

* Cleans up some more json serialization

* Moves block iteration and decoding to node.nim

* Moves api methods into their own init procs.

* Applies RestContent api object.

* Replaces format methods with Rest objects in json.nim

* Unused import

* Review comments by Adam

* Fixes issue where content/local endpoint clashes with content/cid.

* faulty merge resolution

* Renames content API to data.

* Fixes faulty rebase

* Adds test for data/local API

* Renames local and download api.
2023-11-09 08:47:09 +00:00
Eric
7d4ea878d2
Support logging to file (#558)
* Support logging to file

* Log the entire config and fix build error

* Downgrade log level for "starting codex node" config output

* bump ethers to prevent nonce gaps

* fix tests
2023-11-09 16:35:55 +11:00
Adam Uhlíř
c0bec2f899
feat: ensure block expiry (#597)
* feat: update block expiry

* chore: feedback implementation

* chore: feedback implementation

* chore: feedback implementation
2023-11-06 08:10:30 +00:00
Slava
11cd2c46ad
Rename metrics (#478) (#603) 2023-11-03 17:21:54 +02:00
Adam Uhlíř
0014ffdef5
test: integration tests listen on localhost (#596) 2023-10-24 16:52:06 +02:00
Adam Uhlíř
2fc71cf81b
feat: partial payouts for cancelled requests (#561) 2023-10-24 10:12:54 +00:00
Dmitriy Ryajov
8a7d74e6b2
removing old por proofs implementation (#593) 2023-10-23 07:58:07 -07:00
markspanbroek
a77d0cdcec
[sales] Fix intermittently failing test (#591) 2023-10-19 15:46:21 +02:00
Adam Uhlíř
c28627d16f
chore(logging): purchases logging (#567) 2023-10-19 10:12:49 +02:00
Slava
5b30bfd427
Use token for coverage data upload to Codecov (#583)
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2023-10-18 14:35:46 +03:00
Adam Uhlíř
5f4218ba1c
fix: use increaseAllowance for token approving (#565) 2023-10-16 13:09:54 +02:00
Ben Bierens
040434aa19
renames log properties named 'count' (#560) 2023-10-03 13:33:01 +02:00
Ben Bierens
fede106118
Numbered log entries (#559)
* add numbered log entries (hack)

* Changed from CLI argument to compiler define

* Use uint64 for counting

---------

Co-authored-by: gmega <giuliano.mega@gmail.com>
2023-10-03 11:53:58 +02:00