734 Commits

Author SHA1 Message Date
Adam Uhlíř
e5df8c50d3
style: nph formatting (#1067)
* style: nph setup

* chore: formates codex/ and tests/ folder with nph 0.6.1
2025-01-21 20:54:46 +00:00
markoburcul
d114e6e942 nix: build nim compiler
Using nimbus build system since the codex now uses version of Nim >2.
Tested with nim versions(>2) in nixpkgs but none of them work thus the
removal of USE_SYSTEM_NIM and its logic.
Signed-off-by: markoburcul <marko@status.im>
2025-01-21 16:45:33 +01:00
markoburcul
893f6d02ab jenkins: create initial pipeline
Referenced issue: https://github.com/codex-storage/nim-codex/issues/940

Signed-off-by: markoburcul <marko@status.im>
2025-01-21 16:45:33 +01:00
Arnaud
389ab59aa7
Use custom error message (#1079) 2025-01-20 16:04:50 +00:00
Giuliano Mega
ac12de37b2
chore: delete leftover code from old threaded erasure coding backend (#1080) 2025-01-20 14:50:33 +00:00
Slava
833e253baa
feat: switch to github arm runners (#1077)
* feat: switch to github arm runners

* chore: fabiocaccamo/create-matrix-action
2025-01-20 14:42:01 +00:00
Arnaud
2ad7c31c85
Remove duplicate header (#971)
Co-authored-by: Giuliano Mega <giuliano.mega@gmail.com>
2025-01-16 13:51:07 +00:00
Arnaud
4606726e27
Add default Content-Disposition header for download (#981) 2025-01-16 13:25:26 +00:00
Adam Uhlíř
68ad804f9e
refactor: marketplace custom errors handling (#1061)
* refactor: marketplace custom errors handling

* chore: update contracts repo

* chore: update contracts submodule
2025-01-16 09:34:44 +00:00
Marko Burčul
39e8e6e6fa
nix: update readme (#1064)
Include the instructions for running nim-codex as a systemd service on
NixOS.

Signed-off-by: markoburcul <marko@status.im>
2025-01-14 08:21:11 +00:00
markoburcul
0cffa02748
nix: add codex service definition
Referenced issue: https://github.com/codex-storage/nim-codex/issues/940

Signed-off-by: markoburcul <marko@status.im>
2025-01-10 17:49:20 +01:00
markoburcul
3dc7224330
nix: update nix packages and dependencies in flake
Signed-off-by: markoburcul <marko@status.im>
2025-01-10 17:49:18 +01:00
Arnaud
f25c555d59
Chore/update nim version (#1052)
* Move to version 2.0.6

* Update nim-confutils submodule to latest version

* Update dependencies

* Update Nim version to 2.0.12

* Add gcsafe pragma

* Add missing import

* Update specific conf for Nim 2.x

* Fix method signatures

* Revert erasure coding attempt to fix bug

* More gcsafe pragma

* Duplicate code from libp2p because it is not exported anymore

* Fix camelcase function names

* Use alreadySeen because need is not a bool anymore

* newLPStreamReadError does not exist anymore so use another error

* Replace ValidIpAddress by IpAddress

* Add gcsafe pragma

* Restore maintenance parameter deleted by mistake when removing esasure coding fix attempt code

* Update method signatures

* Copy LPStreamReadError code from libp2p which was removed

* Fix camel case

* Fix enums in tests

* Fix camel case

* Extract node components to a variable to make Nim 2 happy

* Update the tests using ValidIpAddress to IpAddress

* Fix cast for value which is already an option

* Set nim version to 2.0.x for CI

* Set nim version to 2.0.x for CI

* Move to miniupnp version 2.2.4 to avoid symlink error

* Set core.symlinks to false for Windows for miniupnp >= 2.2.5 support

* Update to Nim 2.0.14

* Update CI nim versions to 2.0.14

* Try with GCC 14

* Replace apt-fast by apt-get

* Update ubuntu runner to latest

* Use Ubuntu 20.04 for coverage

* Disable CI cache for coverage

* Add coverage property description

* Remove commented test

* Check the node value of seen instead of using alreadySeen

* Fix the merge. The taskpool work was reverted.

* Update nim-ethers submodule

* Remove deprecated ValidIpAddress. Fix missing case and imports.

* Fix a weird issue where nim-confutils cannot find NatAny

* Fix tests and remove useless static keyword
2025-01-10 14:12:37 +00:00
Ben Bierens
caed3c07a3
Fix sending of WantBlocks messages and tracking of peerWants (#1019)
* sends wantBlock to peers with block. wantHave to everyone else

* Cleanup cheapestPeer. Fixes test for peers lists

* Fixes issue where peerWants are only stored for type wantBlock.

* Review comments by Dmitriy

* consistent logging of addresses

* prevents duplicate scheduling. Fixes cancellation

* fast

* Marks cancel-presence situation with todo comment.

* fixtest: testsales enable logging

* Review by Dmitriy: Remember peerWants only if we don't have them.

* rework `wantListHandler` handling

---------

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2025-01-09 22:44:02 +00:00
munna0908
74c46b3651
network: support for uPnP and PMP nat traversal (#1050)
* add support for uPnP and PMP nat traversal

* update cli flags

* merge with master

* use listener address from switch

* update build script to rename version file

* fix styling issues

* update docker files

- remove Disc_ip env
- update CODE_NAT parsing logic

* code cleanup

* move nat flag parsing logic to conf.nim
2025-01-09 18:11:22 +00:00
Adam Uhlíř
407f77871f
chore: warning cleanup (#1055)
* chore: warning cleanup

* chore: fix proper disabling of warning

* chore: ignore the import when not needed
2025-01-08 11:30:54 +00:00
Arnaud
145aa5d84a
chore: remove old fixme (#1060)
* Remove FIXME

* Fix indentation
2025-01-08 07:45:15 +00:00
Arnaud
0badcb662a
chore(openapi): add announceAddresses for debug (#994)
Co-authored-by: Adam Uhlíř <adam@uhlir.dev>
2025-01-08 07:34:54 +00:00
Adam Uhlíř
4b99b58645
fix: unhide log-format cli flag (#1049) 2024-12-19 12:03:50 +00:00
Eric
6ff4d30b43
fix(tests/integration): fix bootstrap node spr (#1048)
After a change in PR #1031, bootstrap node sprs may not work when Hardhat nodes are started with the tests. This fixes it by appending all started client's and provider's SPR to a sequence, and using that sequence of SPRs to start the next node. This means all subsequently started nodes will be connected to its previously started peers.

This also fixes the case when bootstrap SPRs would not be present if no clients were started.
2024-12-18 08:10:36 +00:00
Eric
8645d336ff
refactor(trackedfutures): remove return of future from tracked futures api (#1046)
- cleans up all instances of `.track` to use the `module.trackedfutures.track(future)` procedure, for better readability
- removes the `track` override that is no longer used in the codebase
2024-12-18 07:39:03 +00:00
Ben Bierens
20bb5e5a38
Applies multinodessuite in twonodessuite (#1031)
* Sets timeout for codexclient httpClient. Adds reliable transfer test.

* disable new test to check timeout setting in CI

* restores new test

* adds heartbeat log and logfile to ci output

* fixes suite

* fixes blocked output stream by switching to multinode fixture

* new twonodessuite based on multinodesuite

* Applies updated twonodessuite

* removes heartbeat log

* applies multinodesuite in testsales

* applies multinodesuite in testmarketplace

* fixes account fetch for host and client in testmarketplace

* adds waitTillNextPeriod at end of marketplace test

* Uses marketplacesuite in testmarketplace
2024-12-17 13:01:41 +00:00
Eric
c498e2f53b
fix(nodeprocess): asyncspawn capture output (#1045)
- Ensures no exceptions are raised from `captureOutput`
- Asyncspawns the future to ensure errors are not silently swallowed
2024-12-17 09:51:38 +00:00
Eric
f0f04ddf1d
refactor(then): removes then util (#1047)
- removes then util as it is no longer being used in the codebase
2024-12-17 09:51:05 +00:00
Eric
bef1160799
fix(validation): asyncSpawns run (#1039)
- annotates run with raises: []
- asyncSpawns run, to ensure there are no escaping exceptions
2024-12-16 06:55:19 +00:00
Eric
b0cc27f563
fix(blockexchange): ensures futures are asyncSpawned (#1037)
* fix(blockexchange): asyncSpawn advertising of local store blocks

* fix(blockexchange): asyncSpawn discoveryQueueLoop

- prevents silently swallowing async errors

* fix(blockexchange): asyncSpawns block exchange tasks

- prevents silently swallow future exceptions
2024-12-16 06:01:49 +00:00
Eric
5f2ba14281
fix(codexnode): ensure timer loop is asyncSpawned (#1038)
* fix(codexnode): stop clock after validator stops

* fix(timer): ensure timer loop is asyncSpawned
2024-12-16 04:24:57 +00:00
Slava
6d415b0ace
ci: split nim-matrix workflow (#1041) 2024-12-15 23:31:55 +00:00
Eric
01fb685bf6
fix(sales): replaces then with asyncSpawn (#1036)
- ensures `addSlotToQueue` does not raise exceptions as it is now asyncSpawned
2024-12-15 23:19:31 +00:00
Marcin Czenko
92a0eda79a
Validator historical state restoration (#922)
* adds a new overload of queryPastEvents allowing to query past events based on timestamp in the past

* adds state restoration to validator

* refactors a bit to get the tests back to work

* replaces deprecated generic methods from Market with methods for specific event types

* Refactors binary search

* adds market tests for querying past SlotFilled events and binary search

* Takes into account that <<earliest>> block available is not necessarily the genesis block

* Adds more logging and makes testing earliest block boundary more reliable

* adds validation tests for historical state restoration

* adds mockprovider to simplify and improve testing of the edge conditions

* adds slot reservation to the new tests after rebasing

* adds validation groups and group index in logs of validator

* adds integration test with two validators

* adds comment on how to enable logging in integration test executable itself

* testIntegration: makes list is running nodes injected and available in the body of the test

* validation: adds integration test for historical state

* adds more logging to validator

* integration test: validator only looks 30 days back for historical state

* adds logging of the slotState when removing slots during validation

* review and refactor validator integration tests

* adds validation to the set of integration tests

* Fixes mistyped name of the mock provider module in testMarket

* Fixes a typo in the name of the validation suite in integration tests

* Makes validation unit test a bit easier to follow

* better use of logScopes to reduce duplication

* improves timing and clarifies the test conditions

* uses http as default RPC provider for nodes running in integration tests as a workaround for dropped subscriptions

* simplifies the validation integration tests by waiting for failed request instead of tracking slots

* adds config option allowing selectively to set different provider url

* Brings back the default settings for RPC provider in integration tests

* use http RPC provider for clients in validation integration tests

* fine-tune the tests

* Makes validator integration test more robust - adds extra tracking

* brings tracking of marketplace event back to validator integration test

* refactors integration tests

* deletes tmp file

* adds <<return>> after forcing integration test to fail preliminarily

* re-enables all integration tests and matrix

* stops debug output in CI

* allows to choose a different RPC provider for a given integration test suite

* fixes signature of <<getBlock>> method in mockProvider

* adds missing import which seem to be braking integration tests on windows

* makes sure that clients, SPs, and validators use the same provider url

* makes validator integration tests using http at 127.0.0.1:8545

* testvalidator: stop resubscribing as we are now using http polling as rpc provider

* applying review comments

* groups queryPastStorage overrides together (review comment)

* groups the historical validation tests into a sub suite

* removes the temporary extensions in marketplacesuite and multinodesuite allowing to specify provider url

* simplifies validation integration tests

* Removes debug logs when waiting for request to fail

* Renaming waitForRequestFailed => waitForRequestToFail

* renames blockNumberForBlocksAgo to pastBlockTag and makes it private

* removes redundant debugging logs

* refines logging in validation

* removes dev logging from mockmarket

* improves exception handling in provider helper procs and prepares for extraction to a separate module

* Uses chronos instead of std/times for Duration

* extracts provider and binary search helpers to a separate module

* removes redundant log entry params from validator

* unifies the notation to consistently use method call syntax

* reuses ProviderError from nim-ethers in the provider extension

* clarifies the comment in multinodesuite

* uses == operator to check the predefined tags and raises exception when `BlockTag.pending` is requested.

* when waiting for request to fail, we break on any request state that is not Started

* removes tests that were moved to testProvider from testMarket

* extracts tests that use MockProvider to a separate async suite

* improves performance of the historical state restoration

* removing redundant log messages in validator (groupIndex and groups)

* adds testProvider to testContracts group

* removes unused import in testMarket
2024-12-14 05:07:55 +00:00
Eric
1f49f86131
fix(slotqueue): asyncSpawns futures correctly (#1034)
- asyncSpawns `run` and worker `dispatch` in slotqueue.
- removes usage of `then` from slotqueue.
2024-12-13 06:42:05 +00:00
Eric
7c804b0ec9
fix(asyncstatemachine): fixes not awaiting or asyncSpawning futures (#1033)
- adds a break in scheduler when CancelledError is caught
- tracks asyncSpawned state.run, so that it can be cancelled during stop
- removes usages of `then`
- ensures that no exceptions are leaked from async procs
2024-12-13 02:35:39 +00:00
Adam Uhlíř
19af79786e
feat: repair is rewarded (#1022)
* feat: repair is rewarded

* chore: update contracts repo

* feat: proving loop handles repair case

* test: assert repair state

* chore: update contracts repo

* fix: upon unknown state of repair go to error
2024-12-12 20:19:56 +00:00
Adam Uhlíř
d10072bf67
refactor: marketplace configuration is cached (#1029) 2024-12-12 12:57:34 +00:00
Eric
da234d503b
fix(trackedfutures): removes usage of then from tracked futures (#1032)
- removes usage of `then`, simplifying the logic, and allowing `then` to be removed completely
- updates annotations to reflect that all procs (sync and async) raise no exceptions
2024-12-12 12:31:51 +00:00
Vaclav Pavlin
855b973811
chore: fix inconsistent metric naming (#1027) 2024-12-12 10:45:47 +00:00
Marko Burčul
0c6784da7e
nix: make derivation and update shell (#1003)
* nix: make derivation and update shell

Create a structure for nix files. Add the derivation file which is using
system Nim to compile Codex.

Referenced issue: https://github.com/codex-storage/nim-codex/issues/940

Signed-off-by: markoburcul <marko@status.im>

* nim-circom-compat: update

Include commit which allows building circom-compat-ffi using Nix(doesn't
affect current usage of the submodule).

Referenced issue: https://github.com/codex-storage/nim-codex/issues/940

Signed-off-by: markoburcul <marko@status.im>

* makefile: fix for detecting linux arch

Signed-off-by: markoburcul <marko@status.im>

---------

Signed-off-by: markoburcul <marko@status.im>
2024-12-09 17:07:01 +00:00
Eric
fb4577f25c
chore(tests): fix unneeded async (#1021)
Removes an unneeded `check eventually` inside a helper proc, removing the need for the proc to be async.
2024-12-06 08:02:57 +00:00
Ben Bierens
f51eae30fe
Fixes raceconditions in testadvertiser (#1008) 2024-12-05 09:30:01 +00:00
Ben Bierens
8e29939cf8
Send pluralized wantBlock messages (#1016)
* don't unroll wantCids when sending wantBlock message in blockPresenceHandler

* workaround logging

* Fixes logformatting upraises for sequences.

* Applies upraises rule for setProperty of textmode for sequences.

* Replaces upraises with raises

* Removes redundant log in sendWantHave
2024-12-04 13:33:48 +00:00
Slava
921159f87f
Run release tests for docker images (#1017)
* Refactor Docker reusable workflow and add release tests support

https://github.com/codex-storage/cs-codex-dist-tests/issues/108

* Add an option to run release tests for Docker images

https://github.com/codex-storage/cs-codex-dist-tests/issues/108

* Use bigger instance for arm builds

https://github.com/codex-storage/cs-codex-dist-tests/issues/108

* Pass repository and branch to release tests workflow

https://github.com/codex-storage/cs-codex-dist-tests/issues/108

* Do not use computation job because run_release_tests is a string

https://github.com/codex-storage/cs-codex-dist-tests/issues/108
2024-12-04 11:52:51 +00:00
Ben Bierens
63e54d135c
Fixes race condition in initial-proving tests (#1007) 2024-12-04 08:56:53 +00:00
Adam Uhlíř
0707446cdd
feat: expose underlying nim-ethers errors to logs (#985)
* feat: expose underlying nim-ethers errors to logs

* chore: bump nim-ethers

* test: fix testproof compilation

* test: raise defects on results error
2024-12-03 11:16:24 +00:00
Marcin Czenko
3f510eb501
fixes incomplete async annotation (#1015) 2024-12-02 15:07:43 +00:00
Ben Bierens
ab019a08ae
Enables stacktrace in docker images (#1013)
* Enables libbacktrace in docker images

* Make libbacktrace configurable in docker via build-arg

---------

Co-authored-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2024-12-02 09:32:53 +00:00
Ben Bierens
21249968d4
Requesting the same CID sometimes causes a worker to discard the request if it's already inflight by another worker. (#1002) 2024-11-27 07:13:34 +00:00
Ben Bierens
d47ce38894
Fixes race in testsales (#995)
* Adds isWaiting to mockClock to remove sleep in testsales

* Review comments by Eric. Also replaced two more sleeps with check-eventually
2024-11-26 10:48:52 +00:00
Slava
5c6bbb0cee
chore: update testnet marketplace address (#1000) (#1001)
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2024-11-26 08:16:41 +00:00
Ben Bierens
024c75e4f9
Bumps leveldbstatic to v0.1.6 (#998) 2024-11-25 17:15:05 +00:00
Slava
d12de20868
ci: use rust 1.7.9 for release workflow and dockerfile (#999)
* ci: use rust 1.7.9 for release workflow (#996)

* docker: use rust 1.7.9 for docker (#996)
2024-11-25 16:13:14 +00:00