Commit Graph

729 Commits

Author SHA1 Message Date
Ben 4b5c35534d
Merge branch 'master' into feature/blkexc-peer-selection
# Conflicts:
#	tests/integration/testsales.nim
2024-12-17 15:13:59 +01:00
Ben Bierens 20bb5e5a38
Applies multinodessuite in twonodessuite (#1031)
* Sets timeout for codexclient httpClient. Adds reliable transfer test.

* disable new test to check timeout setting in CI

* restores new test

* adds heartbeat log and logfile to ci output

* fixes suite

* fixes blocked output stream by switching to multinode fixture

* new twonodessuite based on multinodesuite

* Applies updated twonodessuite

* removes heartbeat log

* applies multinodesuite in testsales

* applies multinodesuite in testmarketplace

* fixes account fetch for host and client in testmarketplace

* adds waitTillNextPeriod at end of marketplace test

* Uses marketplacesuite in testmarketplace
2024-12-17 13:01:41 +00:00
Ben Bierens f0b1e098fc
Merge branch 'master' into feature/blkexc-peer-selection 2024-12-17 12:56:13 +01:00
Eric c498e2f53b
fix(nodeprocess): asyncspawn capture output (#1045)
- Ensures no exceptions are raised from `captureOutput`
- Asyncspawns the future to ensure errors are not silently swallowed
2024-12-17 09:51:38 +00:00
Eric f0f04ddf1d
refactor(then): removes then util (#1047)
- removes then util as it is no longer being used in the codebase
2024-12-17 09:51:05 +00:00
Ben Bierens 1ecd784d7b
Merge branch 'master' into feature/blkexc-peer-selection 2024-12-16 16:15:18 +01:00
Eric bef1160799
fix(validation): asyncSpawns run (#1039)
- annotates run with raises: []
- asyncSpawns run, to ensure there are no escaping exceptions
2024-12-16 06:55:19 +00:00
Eric b0cc27f563
fix(blockexchange): ensures futures are asyncSpawned (#1037)
* fix(blockexchange): asyncSpawn advertising of local store blocks

* fix(blockexchange): asyncSpawn discoveryQueueLoop

- prevents silently swallowing async errors

* fix(blockexchange): asyncSpawns block exchange tasks

- prevents silently swallow future exceptions
2024-12-16 06:01:49 +00:00
Eric 5f2ba14281
fix(codexnode): ensure timer loop is asyncSpawned (#1038)
* fix(codexnode): stop clock after validator stops

* fix(timer): ensure timer loop is asyncSpawned
2024-12-16 04:24:57 +00:00
Slava 6d415b0ace
ci: split nim-matrix workflow (#1041) 2024-12-15 23:31:55 +00:00
Eric 01fb685bf6
fix(sales): replaces `then` with `asyncSpawn` (#1036)
- ensures `addSlotToQueue` does not raise exceptions as it is now asyncSpawned
2024-12-15 23:19:31 +00:00
Marcin Czenko 92a0eda79a
Validator historical state restoration (#922)
* adds a new overload of queryPastEvents allowing to query past events based on timestamp in the past

* adds state restoration to validator

* refactors a bit to get the tests back to work

* replaces deprecated generic methods from Market with methods for specific event types

* Refactors binary search

* adds market tests for querying past SlotFilled events and binary search

* Takes into account that <<earliest>> block available is not necessarily the genesis block

* Adds more logging and makes testing earliest block boundary more reliable

* adds validation tests for historical state restoration

* adds mockprovider to simplify and improve testing of the edge conditions

* adds slot reservation to the new tests after rebasing

* adds validation groups and group index in logs of validator

* adds integration test with two validators

* adds comment on how to enable logging in integration test executable itself

* testIntegration: makes list is running nodes injected and available in the body of the test

* validation: adds integration test for historical state

* adds more logging to validator

* integration test: validator only looks 30 days back for historical state

* adds logging of the slotState when removing slots during validation

* review and refactor validator integration tests

* adds validation to the set of integration tests

* Fixes mistyped name of the mock provider module in testMarket

* Fixes a typo in the name of the validation suite in integration tests

* Makes validation unit test a bit easier to follow

* better use of logScopes to reduce duplication

* improves timing and clarifies the test conditions

* uses http as default RPC provider for nodes running in integration tests as a workaround for dropped subscriptions

* simplifies the validation integration tests by waiting for failed request instead of tracking slots

* adds config option allowing selectively to set different provider url

* Brings back the default settings for RPC provider in integration tests

* use http RPC provider for clients in validation integration tests

* fine-tune the tests

* Makes validator integration test more robust - adds extra tracking

* brings tracking of marketplace event back to validator integration test

* refactors integration tests

* deletes tmp file

* adds <<return>> after forcing integration test to fail preliminarily

* re-enables all integration tests and matrix

* stops debug output in CI

* allows to choose a different RPC provider for a given integration test suite

* fixes signature of <<getBlock>> method in mockProvider

* adds missing import which seem to be braking integration tests on windows

* makes sure that clients, SPs, and validators use the same provider url

* makes validator integration tests using http at 127.0.0.1:8545

* testvalidator: stop resubscribing as we are now using http polling as rpc provider

* applying review comments

* groups queryPastStorage overrides together (review comment)

* groups the historical validation tests into a sub suite

* removes the temporary extensions in marketplacesuite and multinodesuite allowing to specify provider url

* simplifies validation integration tests

* Removes debug logs when waiting for request to fail

* Renaming waitForRequestFailed => waitForRequestToFail

* renames blockNumberForBlocksAgo to pastBlockTag and makes it private

* removes redundant debugging logs

* refines logging in validation

* removes dev logging from mockmarket

* improves exception handling in provider helper procs and prepares for extraction to a separate module

* Uses chronos instead of std/times for Duration

* extracts provider and binary search helpers to a separate module

* removes redundant log entry params from validator

* unifies the notation to consistently use method call syntax

* reuses ProviderError from nim-ethers in the provider extension

* clarifies the comment in multinodesuite

* uses == operator to check the predefined tags and raises exception when `BlockTag.pending` is requested.

* when waiting for request to fail, we break on any request state that is not Started

* removes tests that were moved to testProvider from testMarket

* extracts tests that use MockProvider to a separate async suite

* improves performance of the historical state restoration

* removing redundant log messages in validator (groupIndex and groups)

* adds testProvider to testContracts group

* removes unused import in testMarket
2024-12-14 05:07:55 +00:00
Ben Bierens 9c2f3faa4d
Merge branch 'master' into feature/blkexc-peer-selection 2024-12-13 08:31:00 +01:00
Eric 1f49f86131
fix(slotqueue): asyncSpawns futures correctly (#1034)
- asyncSpawns `run` and worker `dispatch` in slotqueue.
- removes usage of `then` from slotqueue.
2024-12-13 06:42:05 +00:00
Eric 7c804b0ec9
fix(asyncstatemachine): fixes not awaiting or asyncSpawning futures (#1033)
- adds a break in scheduler when CancelledError is caught
- tracks asyncSpawned state.run, so that it can be cancelled during stop
- removes usages of `then`
- ensures that no exceptions are leaked from async procs
2024-12-13 02:35:39 +00:00
Adam Uhlíř 19af79786e
feat: repair is rewarded (#1022)
* feat: repair is rewarded

* chore: update contracts repo

* feat: proving loop handles repair case

* test: assert repair state

* chore: update contracts repo

* fix: upon unknown state of repair go to error
2024-12-12 20:19:56 +00:00
Adam Uhlíř d10072bf67
refactor: marketplace configuration is cached (#1029) 2024-12-12 12:57:34 +00:00
Eric da234d503b
fix(trackedfutures): removes usage of `then` from tracked futures (#1032)
- removes usage of `then`, simplifying the logic, and allowing `then` to be removed completely
- updates annotations to reflect that all procs (sync and async) raise no exceptions
2024-12-12 12:31:51 +00:00
Vaclav Pavlin 855b973811
chore: fix inconsistent metric naming (#1027) 2024-12-12 10:45:47 +00:00
Ben 960c0107f0
fixtest: testsales enable logging 2024-12-10 11:05:59 +01:00
Ben Bierens cc1121c9f0
Merge branch 'master' into feature/blkexc-peer-selection 2024-12-10 09:30:14 +01:00
Marko Burčul 0c6784da7e
nix: make derivation and update shell (#1003)
* nix: make derivation and update shell

Create a structure for nix files. Add the derivation file which is using
system Nim to compile Codex.

Referenced issue: https://github.com/codex-storage/nim-codex/issues/940

Signed-off-by: markoburcul <marko@status.im>

* nim-circom-compat: update

Include commit which allows building circom-compat-ffi using Nix(doesn't
affect current usage of the submodule).

Referenced issue: https://github.com/codex-storage/nim-codex/issues/940

Signed-off-by: markoburcul <marko@status.im>

* makefile: fix for detecting linux arch

Signed-off-by: markoburcul <marko@status.im>

---------

Signed-off-by: markoburcul <marko@status.im>
2024-12-09 17:07:01 +00:00
Ben c3c04e84df
Marks cancel-presence situation with todo comment. 2024-12-09 09:14:49 +01:00
Ben Bierens 7e4182ef12
Merge branch 'master' into feature/blkexc-peer-selection 2024-12-07 13:09:49 +01:00
benbierens dcf43d3227
fast 2024-12-06 12:41:17 +01:00
Eric fb4577f25c
chore(tests): fix unneeded async (#1021)
Removes an unneeded `check eventually` inside a helper proc, removing the need for the proc to be async.
2024-12-06 08:02:57 +00:00
benbierens 1f063fe163
prevents duplicate scheduling. Fixes cancellation 2024-12-05 19:52:57 +01:00
benbierens 532b83a52b
consistent logging of addresses 2024-12-05 15:20:25 +01:00
Ben Bierens d32e052deb
Merge branch 'master' into feature/blkexc-peer-selection 2024-12-05 14:40:31 +01:00
Ben Bierens f51eae30fe
Fixes raceconditions in testadvertiser (#1008) 2024-12-05 09:30:01 +00:00
benbierens ed3e91cbde
Review comments by Dmitriy 2024-12-05 10:21:32 +01:00
Ben 64e691b824
Fixes issue where peerWants are only stored for type wantBlock. 2024-12-04 16:04:08 +01:00
Ben 7c2bff79c0
Cleanup cheapestPeer. Fixes test for peers lists 2024-12-04 16:04:08 +01:00
Ben 3607b88e9d
sends wantBlock to peers with block. wantHave to everyone else 2024-12-04 16:04:08 +01:00
Ben Bierens 8e29939cf8
Send pluralized wantBlock messages (#1016)
* don't unroll wantCids when sending wantBlock message in blockPresenceHandler

* workaround logging

* Fixes logformatting upraises for sequences.

* Applies upraises rule for setProperty of textmode for sequences.

* Replaces upraises with raises

* Removes redundant log in sendWantHave
2024-12-04 13:33:48 +00:00
Slava 921159f87f
Run release tests for docker images (#1017)
* Refactor Docker reusable workflow and add release tests support

https://github.com/codex-storage/cs-codex-dist-tests/issues/108

* Add an option to run release tests for Docker images

https://github.com/codex-storage/cs-codex-dist-tests/issues/108

* Use bigger instance for arm builds

https://github.com/codex-storage/cs-codex-dist-tests/issues/108

* Pass repository and branch to release tests workflow

https://github.com/codex-storage/cs-codex-dist-tests/issues/108

* Do not use computation job because run_release_tests is a string

https://github.com/codex-storage/cs-codex-dist-tests/issues/108
2024-12-04 11:52:51 +00:00
Ben Bierens 63e54d135c
Fixes race condition in initial-proving tests (#1007) 2024-12-04 08:56:53 +00:00
Adam Uhlíř 0707446cdd
feat: expose underlying nim-ethers errors to logs (#985)
* feat: expose underlying nim-ethers errors to logs

* chore: bump nim-ethers

* test: fix testproof compilation

* test: raise defects on results error
2024-12-03 11:16:24 +00:00
Marcin Czenko 3f510eb501
fixes incomplete async annotation (#1015) 2024-12-02 15:07:43 +00:00
Ben Bierens ab019a08ae
Enables stacktrace in docker images (#1013)
* Enables libbacktrace in docker images

* Make libbacktrace configurable in docker via build-arg

---------

Co-authored-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2024-12-02 09:32:53 +00:00
Ben Bierens 21249968d4
Requesting the same CID sometimes causes a worker to discard the request if it's already inflight by another worker. (#1002) 2024-11-27 07:13:34 +00:00
Ben Bierens d47ce38894
Fixes race in testsales (#995)
* Adds isWaiting to mockClock to remove sleep in testsales

* Review comments by Eric. Also replaced two more sleeps with check-eventually
2024-11-26 10:48:52 +00:00
Slava 5c6bbb0cee
chore: update testnet marketplace address (#1000) (#1001)
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2024-11-26 08:16:41 +00:00
Ben Bierens 024c75e4f9
Bumps leveldbstatic to v0.1.6 (#998) 2024-11-25 17:15:05 +00:00
Slava d12de20868
ci: use rust 1.7.9 for release workflow and dockerfile (#999)
* ci: use rust 1.7.9 for release workflow (#996)

* docker: use rust 1.7.9 for docker (#996)
2024-11-25 16:13:14 +00:00
markspanbroek 29433bad9a
Fix concurrency issues (#993)
* Use http subscriptions instead of websocket for tests

To work around this issue when subscriptions are
inactive for more than 5 minutes:
https://github.com/NomicFoundation/hardhat/issues/2053

Use 100 millisecond polling; default polling interval
of 4 seconds is too close to the 5 second timeout for
`check eventually`.

* use .confirm(1) instead of confirm(0)

confirm(0) doesn't wait at all, confirm(1) waits
for the transaction to be mined

* speed up partial payout integration test

* update nim-ethers to version 0.10.0

includes fixes for http polling and .confirm()

* fix timing of marketplace tests

allow for a bit more time to withdraw funds

* use .confirm(1) in marketplace tests

to ensure that the transaction has been processed
before continuing with the test

* fix timing issue in validation unit test

* fix proof integration test

there were two logic errors in this test:
- a slot is freed anyway at the end of the contract
- when starting the request takes a long time, the
  first slot can already be freed because there were
  too many missing proofs

* fix intermittent error in contract tests

currentTime() doesn't always correctly reflect
the time of the next transaction

* reduce number of slots in integration test

otherwise the windows runner in the CI won't
be able to start the request before it expires

* fix timing in purchasing test

allow for a bit more time for a request to
be submitted

* fix timing of request submission in test

windows ci is so slow, it can take up to 40 seconds
just to submit a storage request to hardhat

* increase proof period to 90 seconds

* adjust timing of integration tests

reason: with the increased period length of 90 seconds, it
can take longer to wait for a stable challenge at the
beginning of a period.

* increase CI timeout to 2 hours

* Fix slow builds on windows

apparently it takes windows 2-3 seconds to
resolve "localhost" to 127.0.0.1 for every
json-rpc connection that we make 🤦
2024-11-25 11:23:04 +00:00
Slava 6038fb456e
ci: split linux and macos tests (#997)
* Split Linux and macOS tests

* Make jobs names more readable
2024-11-22 12:05:00 +00:00
Slava 71b8a95d12
ci: install rust 1.7.9 as required by packages (#996)
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2024-11-22 06:34:10 +00:00
Arnaud 9b7f3f4aaf
chore(openapi): update the openapi spec for the manifest object (#972)
* Update the manifest openapi spec

* Make metadata nullable

Signed-off-by: Arnaud <arnaud@status.im>

---------

Signed-off-by: Arnaud <arnaud@status.im>
Co-authored-by: Giuliano Mega <giuliano.mega@gmail.com>
2024-11-15 13:21:18 +00:00
Slava d7ae8b734a
makefile: Use do not use -mssse3 instructions on arm (#940) (#990) 2024-11-09 11:39:39 +00:00