Commit Graph

410 Commits

Author SHA1 Message Date
Dmitriy Ryajov 5331dc9606
sync up with master 2022-11-25 12:58:08 -06:00
Dmitriy Ryajov 3a53bb1338
wip 2022-11-25 12:43:46 -06:00
Tanguy 499890d44c
Revert "[build] track nim-libp2p's unstable branch"
This reverts commit 8899b68d1a.
2022-11-25 12:43:20 -06:00
Michael Bradley, Jr 61b18bbe93
[build] track nim-libp2p's unstable branch 2022-11-25 12:41:23 -06:00
Dmitriy Ryajov 7b80e24962
sync with master 2022-11-25 12:06:13 -06:00
Shaun Orssaud 5264ac34dc
BUMP testground sdk (#320) 2022-11-25 11:15:07 -06:00
Tanguy 11e02b50dd
Update testground example 2022-11-25 11:15:06 -06:00
Tanguy b02a18b0db
Revert "[build] track nim-libp2p's unstable branch"
This reverts commit 8899b68d1a.
2022-11-25 11:15:04 -06:00
Michael Bradley, Jr d958785d5c
[wip] basic testground/README.md 2022-11-25 11:11:15 -06:00
Michael Bradley, Jr ecbef6dcd1
[wip] slightly better defaults 2022-11-25 11:11:15 -06:00
Michael Bradley, Jr 2b0d4af94f
[wip] rename hello_codex to simple_libp2p 2022-11-25 11:11:15 -06:00
Michael Bradley, Jr 8a3024119c
[wip] rm commented lines from testground/hello_codex/manifest.toml 2022-11-25 11:11:15 -06:00
Michael Bradley, Jr 3930c2744a
[wip] make simple_tcp_ping the default testground plan (for now) 2022-11-25 11:11:15 -06:00
Michael Bradley, Jr 95dea7ec52
[wip] bump vendor/testground-nim-sdk 2022-11-25 11:11:15 -06:00
Michael Bradley, Jr 8e2bad3055
[wip] git don't ignore Dockerfile 2022-11-25 11:11:14 -06:00
Michael Bradley, Jr ed389b75cb
[wip] specify --allow-empty option when applying scratch.patch 2022-11-25 11:11:14 -06:00
Michael Bradley, Jr c210f54b58
[wip] revise testground/hello_codex/Dockerfile 2022-11-25 11:11:14 -06:00
Michael Bradley, Jr 8d42b960c5
[wip] append contents of testground/hello_codex/config.nims to scratch/nim-codex/config.nims 2022-11-25 11:11:14 -06:00
Michael Bradley, Jr 339d1cc8cf
[wip] limit chronicles_sinks to textlines in testground/hello_codex/config.nims 2022-11-25 11:11:14 -06:00
Michael Bradley, Jr 70f0d87ac0
[wip] `make testground` should work when working tree is clean 2022-11-25 11:11:14 -06:00
Michael Bradley, Jr 93270e0649
[wip] more flexibility 2022-11-25 11:11:13 -06:00
Michael Bradley, Jr 9d5e9cf0e1
[wip] refactor Makefile and .nimble support to be more flexible 2022-11-25 11:11:13 -06:00
Michael Bradley, Jr 71301c74e8
[wip] don't specify chronicles_log_level=TRACE in nimble task for hello_codex 2022-11-25 11:11:13 -06:00
Michael Bradley, Jr 2b4b5bed5f
[wip] add config.nims to testground/hello_codex 2022-11-25 11:11:13 -06:00
Michael Bradley, Jr 30caf7f197
[wip] code from uncommitted testground-nim-sdk example 2022-11-25 11:11:13 -06:00
Michael Bradley, Jr 48e18ecf31
[wip] support in Makefile, codex.nimble; git ignore scratch/ 2022-11-25 11:11:13 -06:00
Michael Bradley, Jr 2a76583c57
[wip] testground/hello_codex/ 2022-11-25 11:11:12 -06:00
Michael Bradley, Jr 75af9ee41f
[build] add github.com/status-im/testground-nim-sdk to vendor 2022-11-25 11:11:12 -06:00
Michael Bradley, Jr 644a8f84de
[build] track nim-libp2p's unstable branch 2022-11-25 11:11:10 -06:00
markspanbroek 9143a25fa1
[ci] Add tips for maintaining CI (#311)
* [ci] Add tips for maintaining CI

Lessons learned from speeding up the CI in
https://github.com/status-im/nim-codex/pull/303

* [ci] Readme: add tip about timing of every test

As suggested by @Menduist

* [ci] Readme: add link to github actions usage limits

As suggested by @Menduist
2022-11-16 14:58:49 +01:00
markspanbroek cdd175279c
Fix nimble requirements (#305)
* [build] Update nim-ethers to 0.2.4

* [build] set correct lrucache requirement in nimble file
2022-11-16 13:10:23 +01:00
Dmitriy Ryajov 88f1be388a
fix `help`, `info`, add description for `loglevel` (#316)
* fix `help`, `info`, add description for `loglevel`

* add abbreviations for discovery params
2022-11-15 13:16:02 -06:00
Dmitriy Ryajov 5abf80cc69
Block download (#304)
* track inflight requests

* preperly handle precense updates

* trace number of of scheduled blocks

* invoke `payForBlocks` at the correct time

* reduntant block info on want list updates

* don't update prices in task handler

* PeerID -> PeerId

* cleanup

* proper log topic

* better chronicles topic filtering

* more trace logging

* sort want blocks

* wip - fix tests

* wip - fix tests, presence changes

* fix small test issue

* return price

* payment related changes

* misc

* re-enable payment tests

* fix warn wording

* fix `u256` conversion

* minor misc changes

* don't idle for so long on `encode`

* logging

* move buff

* disable cache by default

* disable cache by default

* fix streamOneBlock

* log node stopping/exiting

* trace logging

* don't stringify cid

* use `self`

* quick cleanup

* rename enums

* rename enums

* turns out we don't needs this test

* fix wording
2022-11-15 09:46:21 -06:00
markspanbroek 456e675b8a
[build] Update nim-leopard to version 0.1.0 (#310)
Reduces the time required to build nim-leopard.
2022-11-15 09:05:52 +01:00
Dmitriy Ryajov 5f9507cfcd
Blocks leak and inflight for pending blocks (#315)
* attach `wait` to handle fut

* fix crash when can't find a CID and timeout hits

the exception was not raised, thus a Nil was returned that was
than wrapped in an option, leading to crash.


Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* add inFlight flag

* adding pending blocks tests

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
Co-authored-by: Csaba Kiraly <csaba.kiraly@gmail.com>
2022-11-15 00:12:05 -06:00
Dmitriy Ryajov 978e2f09e4
bumping chronos to latest master (#314) 2022-11-14 22:15:37 -06:00
Dmitriy Ryajov f235cc6621
don't spawn queue requests (#313)
* don't spawn queue requests

* adjust list blocks idle timer

* increase timeout on failing test...
2022-11-14 18:01:05 -06:00
Dmitriy Ryajov f36dc54eda
Log topic filtering (#312)
* comment

* split out coders and add log level endpoin

* adjust info url

* add log filtering test

* don't stringify cid

* add log level test
2022-11-14 17:42:57 -06:00
markspanbroek ccf3d04dc8
Speed up development (#303)
* [ci] Extract setup of nimbus build system into separate file

* [ci] Cleanup NodeJS setup

* [ci] Use amd64 by default

* [ci] Separate steps for unit, contract and integration tests

* [ci] Separate build job for coverage that is allowed to fail

* [ci] Separate build job for Nim 1.2

* [ci] Calculate coverage on unit tests only

This enables faster development cycles at
the expense of code coverage accuracy.

* [PoR] Test proofs of retrievability with a small block size

To get a reasonable test execution time.

* [ci] Set NIM_COMMIT environment variable

To prevent subsequent `make` command to build a
different version of Nim.

* [ci] Checkout submodules recursively

So that nimbus-build-system doesn't have to
do it afterwards.

* [ci] Update caches on every run

Will automatically update caches when a new minor version
of Nim is released.

See also:
https://github.com/actions/cache/blob/main/workarounds.md#update-a-cache

* [ci] Set cache_nonce for all jobs, not just those in the matrix

* [build] update to latest nimbus-build-system

Requires an update to latest nim-nat-traversal

* [erasure] Test erasure coding with a small block size

To get a reasonable test execution time.

* [erasure] fix typo

* [PoR] Test PoR store with a small block size

To get a reasonable test execution time.

* [PoR] Test PoR network with a small block size

To get a reasonable test execution time.

* [ci] Ensure that unit tests are not dependent on Ethereum

Start Ethereum node *after* the unit tests are run.

* [ci] Cancel all other jobs when one fails

Allows for faster feedback loops; a next attempt to run
the CI will start sooner.
2022-11-14 14:50:00 +01:00
markspanbroek 4175689745
Load purchase state from chain (#283)
* [purchasing] Simplify test

* [utils] Move StorageRequest.example up one level

* [purchasing] Load purchases from market

* [purchasing] load purchase states

* Implement myRequest() and getState() methods for OnChainMarket

* [proofs] Fix intermittently failing tests

Ensures that examples of proofs in tests are never of length 0;
these are considered invalid proofs by the smart contract logic.

* [contracts] Fix failing test

With the new solidity contracts update, a contract can only
be paid out after it started.

* [market] Add method to get request end time

* [purchasing] wait until purchase is finished

Purchase.wait() would previously wait until purchase
was started, now we wait until it is finished.

* [purchasing] Handle 'finished' and 'failed' states

* [marketplace] move to failed state once request fails

- Add support for subscribing to request failure events.
- Add supporting contract tests for subscribing to request failure events.
- Allow the PurchaseStarted state to move to PurchaseFailure once a request failure event is emitted
- Add supporting tests for moving from PurchaseStarted to PurchaseFailure
- Add state transition tests for PurchaseUnknown.

* [marketplace] Fix test with longer sleepAsync

* [integration] Add function to restart a codex node

* [purchasing] Set client address before requesting storage

To prevent the purchase id (which equals the request id)
from changing once it's been submitted.

* [contracts] Fix: OnChainMarket.getState()

Had the wrong method signature before

* [purchasing] Load purchases on node start

* [purchasing] Rename state 'PurchaseError' to 'PurchaseErrored'

Allows for an exception type called 'PurchaseError'

* [purchasing] Load purchases in background

No longer calls market.getRequest() for every purchase
on node start.

* [contracts] Add `$` for RequestId, SlotId and Nonce

To aid with debugging

* [purchasing] Add Purchasing.stop()

To ensure that all contract interactions have both a
start() and a stop() for

* [tests] Remove sleepAsync where possible

Use `eventually` loop instead, to make sure that we're
not waiting unnecessarily.

* [integration] Fix: handle non-json response in test

* [purchasing] Add purchase state to json

* [integration] Ensure that purchase is submitted before restart

Fixes test failure on slower CI

* [purchasing] re-implement `description` as method

Allows description to be set in the same module where the
state type is defined.

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>

* [contracts] fix typo

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>

* [market] Use more generic error type

Should we decide to change the provider type later

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
2022-11-08 08:10:17 +01:00
markspanbroek be32b9619b
Fixes loading of private key on Windows (#299)
Unix permissions don't work on Windows; 
adds check for correct ACL settings in Windows.
2022-11-07 15:54:24 +01:00
Michael Bradley c893b1f0cb
track latest nim-libp2p's master branch (#248)
* [build] track nim-libp2p's unstable branch

* cid formatIt change

* track nim-libp2p-unstable

* don't stringify cid

* Fixed testblockexc.nim

1. Assign price to blocks
2. Delete on node1 cached blocks from node2 before buying them again

* add trace logging

* init pricing info

* remove duplicate price setting

* cid serialization in trace logs

* bumping dht to latest main

* bump

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Co-authored-by: Bulat-Ziganshin <bulat.ziganshin@gmail.com>
2022-11-02 11:40:28 -06:00
Dmitriy Ryajov 0ecbfcec9f
setup and persist private key (#292)
* setup and persist private key

* return dht record spr

* helper to remap multiaddr ip and port

* set/update discovery and announce addrs

* add nat and discovery IPs

* allow for announce and DHT addresses separatelly

* update tests

* check for nat or discoveryIp

* fix integration tests

* misc align

* don't share data dirs and and set bootstrap node

* add log scope

* remap announceAddrs after node start

* simplify discovery initialization

* make nat and disc-ip required

* add log scope don't init dht spr in constructor

* bump dht

* dissallow `0.0.0.0` for `--nat`
2022-11-01 18:58:41 -06:00
Dmitriy Ryajov ae46f4dc2f
add support for logging topics (#293)
* add support for logging topics

* add top level codex topic
2022-11-01 14:05:40 -06:00
Dmitriy Ryajov 6e4a8b86ab
Rework discovery (#288)
* use multiaddrs instead of ip/port

* rework to support updating spr

* fix tests

* fix compilation

* use base for base methods
2022-10-27 07:44:56 -06:00
Dmitriy Ryajov e50ea88411
Remove protobuf serialization (#289)
* add format for cid

* cid formatIt change

* track nim-libp2p-unstable

* rework probuf serialization for por

* add missing include

* removing nim protobuf serialization

* rollback to dht to main

* remove protobuf serialization import
2022-10-27 07:41:34 -06:00
Mark Spanbroek 92eecb0702 [purchasing] Rework purchasing as a state machine 2022-10-25 15:10:35 +11:00
Mark Spanbroek 7841a9692c [utils] Add AsyncState to state machine 2022-10-25 15:10:35 +11:00
Mark Spanbroek b52d291785 [utils] Add state machine implementation 2022-10-25 15:10:35 +11:00
Mark Spanbroek e648c26340 [utils] Add operator `as` for optional casting 2022-10-25 15:10:35 +11:00