Commit Graph

120 Commits

Author SHA1 Message Date
Eugene Kabanov 18409a69e1
Light forward sync mechanism (#6515)
* Initial commit.

* Add hybrid syncing.

* Compilation fixes.

* Cast custom event for our purposes.

* Instantiate AsyncEventQueue properly.

* Fix mistype.

* Further research on optimistic updates.

* Fixing circular deps.

* Add backfilling.

* Add block download feature.

* Add block store.

* Update backfill information before storing block.

* Use custom block verifier for backfilling sync.

* Skip signature verification in backfilling.

* Add one more generic reload to storeBackfillBlock().

* Add block verification debugging statements.

* Add more debugging

* Do not use database for backfilling, part 1.

* Fix for stash.

* Stash fixes part 2.

* Prepare for testing.

* Fix assertion.

* Fix post-restart syncing process.

* Update backfill loading log statement.
Use proper backfill slot callback for sync manager.

* Add handling of Duplicates.

* Fix store duration and block backfilled log statements.

* Add proper syncing state log statement.

* Add snappy compression to beaconchain_file.
Format syncing speed properly.

* Add blobs verification.

* Add `slot` number to file structure for easy navigation over stream of compressed objects.

* Change database filename.

* Fix structure size.

* Add more consistency properties.

* Fix checkRepair() issues.

* Preparation to state rebuild process.

* Add plain & compressed size.

* Debugging snappy encode process.

* Add one more debugging line.

* Dump blocks.

* One more filedump.

* Fix chunk corruption code.

* Fix detection issue.

* Some fixes in state rebuilding process.

* Add more clearance steps.

* Move updateHead() back to block_processor.

* Fix compilation issues.

* Make code more async friendly.

* Fix async issues.
Add more information when proposer verification failed.

* Fix 8192 slots issue.

* Fix Future double completion issue.

* Pass updateFlags to some of the core procedures.

* Fix tests.

* Improve initial sync handling mechanism.

* Fix checkStateTransition() performance improvements.

* Add some performance tuning and meters.

* Light client performance tuning.

* Remove debugging statement.

* Use single file descriptor for blockchain file.

* Attempt to fix LC.

* Fix timeleft calculation when untrusted sync backfilling started right after LC block received.

* Workaround for `chronicles` + `results` `error` issue.
Remove some compilation warnings.
Fix `CatchableError` leaks on Windows.

* Address review comments.

* Address review comments part 2.

* Address review comments part 1.

* Rebase and fix the issues.

* Address review comments part 3.

* Add tests and fix some issues in auto-repair mechanism.

* Add tests to all_tests.

* Rename binary test file to pass restrictions.

* Add `bin` extension to excluded list.
Recover binary test data.

* Rename fixture file to .bin again.

* Update AllTests.

* Address review comments part 4.

* Address review comments part 5 and fix tests.

* Address review comments part 6.

* Eliminate foldl and combine from blobs processing.
Add some tests to ensure that checkResponse() also checks for correct order.

* Fix forgotten place.

* Post rebase fixes.

* Add unique slots tests.

* Optimize updateHead() code.

* Add forgotten changes.

* Address review comments on state as argument.
2024-10-30 05:38:53 +00:00
tersec b4efaaa573
test with macOS ARM64 (#6635)
* test with macOS ARM64

* assign platform variable
2024-10-11 11:08:46 +02:00
tersec 409023b95c
use macos-13 builders; macos-12 EOL (#6602) 2024-10-06 17:11:46 +02:00
tersec 72c7398f85
Revert "stop testing with broken upstream/version-2-0 (#6597)" (#6599)
This reverts commit ac321707e0.
2024-10-06 03:02:19 +02:00
tersec ac321707e0
stop testing with broken upstream/version-2-0 (#6597) 2024-10-05 05:27:24 +02:00
tersec 1258fd3592
Revert "stop testing with broken upstream/version-2-0" (#6573)
* Revert "stop testing with broken upstream/version-2-0 (#6554)"

This reverts commit 205dff3378.

* stop using template which Nim 2.0.10 will apparently make recursive
2024-10-03 01:00:39 +00:00
tersec 205dff3378
stop testing with broken upstream/version-2-0 (#6554) 2024-09-17 01:14:29 +02:00
Jacek Sieka 42c4ea109c
Revert "stop testing broken upstream version-2-0" (#6506)
* Revert "stop testing broken upstream version-2-0 (#6499)"

This reverts commit 2be7eba25a.

* fix field warning template
2024-08-23 12:18:06 +02:00
tersec 2be7eba25a
stop testing broken upstream version-2-0 (#6499) 2024-08-20 14:33:54 +02:00
tersec 2d4ece0c3b
bump nimbus-build-system to use Nim v2.0.6 (#6386)
* bump nimbus-build-system to use Nim v2.0.6

* fix: update name and hash for csources of Nim v2

Otherwise we get errors like:
```
Building: Nim compiler
/build/source/vendor/nimbus-build-system/vendor/Nim /build/source
cmd: git clone -q --depth 1 -b master https://github.com/nim-lang/csources_v2.git csources_v2
24.6.0-dirty
cmd: cd csources_v2
ci/funs.sh: line 10: cd: csources_v2: No such file or directory
make[1]: *** [vendor/nimbus-build-system/makefiles/targets.mk:81: build-nim] Error 1
```
Also need to add source for `checksums` repository.

Signed-off-by: Jakub Sokołowski <jakub@status.im>

---------

Signed-off-by: Jakub Sokołowski <jakub@status.im>
Co-authored-by: Jakub Sokołowski <jakub@status.im>
2024-06-25 00:02:16 +02:00
Etan Kissling e4efe9ed75
ensure that rarely ran files are compiled on PR (#6379)
* ensure that rarely ran files are compiled on PR

Add some missing files to `isMainModule` developer internal builds CI.

* fix fork choice compilation

* fix `rest_api_benchmark` compilation

* skip linking

* fix loop

* fix `mock_genesis`

* fix signedness
2024-06-23 23:35:56 +00:00
Miran 597f47317f
use Nim 2.0.6 (#6366) 2024-06-18 03:54:15 +02:00
Jakub Sokołowski 90cf1753bd
use self-hosted runners for Developer builds GH action
Otherwise we get:
```
Error: The operation was canceled.
```
Which is most probably due to OOM error.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
2024-05-29 15:45:44 +02:00
tersec 1051c3c9c5
use upstream/version-2-0 of Nim in CI (#6277) 2024-05-09 02:22:28 +02:00
Jakub Sokołowski 4a1a2c8196
add packages to Nix flake config, move to repo root
This way we can actually build and run a node using just:
```sh
nix run 'github:status-im/nimbus-eth2?submodules=1'
```
The `?submodules=1` part should eventually not be necessary.
For more details see:
https://github.com/NixOS/nix/issues/4423

Signed-off-by: Jakub Sokołowski <jakub@status.im>
2024-05-06 16:57:23 +02:00
tersec fc3fad2433
revert to v2.0.4 2024-05-04 08:39:33 +00:00
Jacek Sieka ab30efaf80
Integrate `hashtree` for better hashing performance (#5188)
status-im/nim-ssz-serialization#35 brings in https://github.com/prysmaticlabs/hashtree as a supported backend for SHA256, giving a nice little performance boost to all hash_tree_root calls on supported platforms / compilers.

Expected gains are on the order of 30% which in the case of a replayed state nets us 0.2-0.3s improvement.

More about this design here: https://hackmd.io/@potuz/BJyrx9DOF - kudos to @potuz for this excellent library!
2024-05-02 08:54:32 +02:00
Etan Kissling 9e8d2e70bd
block pull requests to `stable` (#6001)
* block pull requests to `stable`

Pull requests should be opened against the `unstable` branch.
See https://nimbus.guide/contribute.html#build-and-deploy

* avoid deleting branch to mimic manual close

* avoid triggering on dependabot PRs
2024-05-01 19:30:03 +00:00
tersec 90ac2def31
switch back to version-2-0 (#6233) 2024-04-24 13:49:05 +00:00
tersec eb997f4e1f
add 10 sub-test-fixtures for electra epoch transitions; use v2.0.4 for Nim 2.x (#6229) 2024-04-23 01:15:55 +00:00
tersec 41f8400f97
fix Electra light client objects; use version-2-0 for Nim again (#6222) 2024-04-19 19:20:45 +00:00
tersec 73738c702c
use consensus spec v1.5.0-alpha.0 test vectors; use Nim v2.0.4 specifically for 2.0.x (#6221) 2024-04-19 02:55:58 +00:00
Etan Kissling ecf175856f
bump `actions/checkout` and `actions/cache` to `v4` (#5971)
Typically, mixing different versions of GitHub actions may lead to hard
to fix issues. As many libraries have been updated to use actions v4,
also bump our own `actions/checkout@v2`, `actions/checkout@v3`,
`actions/cache@v2` and `actions/cache@v3` to their `@v4` equivalents.
2024-02-28 19:04:34 +01:00
Etan Kissling 702f4c6a45
add `{.push raises: [].}` to more files (#5943)
Extend `{.push raises: [].}` to all files with spec references that are
periodically updated.

- https://github.com/status-im/nimbus-eth2/pull/5942
2024-02-22 11:03:09 +01:00
Etan Kissling 698a802168
extend linter to enforce exception tracking is turned on (#5909)
Status Nim style mandates `{.push raises: []}.` at start of modules.
Add a CI task to ensure exceptions keep getting properly tracked.

- https://status-im.github.io/nim-style-guide/errors.exceptions.html
- https://github.com/status-im/nim-eth/pull/614#discussion_r1220906149
2024-02-19 11:02:36 +01:00
Etan Kissling 30b7c6153f
handle `Exception` during `EraFile.verify` (#5900)
`Taskpool.new()` is marked as `{.raises: [Exception].}`. Catch this.
2024-02-17 18:19:30 +01:00
tersec 7b5815358e
test macOS with Nim 2.0 in CI and use non-EOL macOS version (#5890) 2024-02-16 04:53:13 +00:00
Etan Kissling ed8743b986
fix standalone compilation of `trusted_node_sync.nim` (#5861)
#5544 contained a regression that broke standalone compilation of
`trusted_node_sync` as a main module. Fix it, and add to CI.
2024-02-07 19:26:29 +00:00
Etan Kissling 388b865aac
add `-d:has_deposit_root_checks` build to CI (#5856)
To prevent accidental breakage of builds with non-standard developer
flags, add the `-d:has_deposit_root_checks` build to CI. This is only
a quick compilation check, and is only run on Linux.
2024-02-07 00:11:51 +00:00
andri lim a45609c4a3
Switch to llvm-mingw for faster Windows CI (#5729)
* Switch to llvm-mingw for faster Windows CI

* Enable ubsan of Windows CI
2024-01-13 11:45:20 +02:00
jangko a5297a352c
Use GITHUB_OUTPUT to replace set output in main CI 2024-01-11 10:14:32 +07:00
Jacek Sieka 3f525acb87
chronos: bump (#5684)
This PR causes a few new warnings to appear - these are harmless but
will need addressing separately as they span several libraries.

* new asyncraises syntax
* asyncraises support in several modules
* `sink` usage reduces spurious copying
* `?` compatiblity for `async` + `results`
* remove `-d:chronosStrictException` (obsolete)
2024-01-05 10:08:38 +01:00
Etan Kissling 0639eaafd0
in lint, fix timestamp comparison when timezone does not match (#5688)
Git by defaults returns commit timestamp according to the committer's
time zone instead of the local one, breaking the simple alphanumeric
comparison for timestamps that we use in lint. Force the timezone to
UTC so that comparison is correct regardless of committer timezone.
2024-01-03 00:34:39 +00:00
tersec aa6c13cd67
use newer geth testnet binary directory names (#5575)
* use newer geth testnet binary directory names

* linting; Geth requires merged genesis
2023-11-07 16:52:30 +01:00
tersec 95b22e3a4f
Revert "Revert "ci: adding self-hosted tag for github-runner (#4886)" (#5494)"
This reverts commit 722deedd2b.
2023-10-13 13:45:07 +02:00
tersec 722deedd2b
Revert "ci: adding self-hosted tag for github-runner (#4886)" (#5494)
This reverts commit acbfef9852.
2023-10-11 20:27:45 +00:00
apentori acbfef9852
ci: adding self-hosted tag for github-runner (#4886)
Signed-off-by: Alexis Pentori <alexis@status.im>
2023-10-11 20:26:18 +00:00
Etan Kissling a261384efc
exclude `*.service` files from lint check (#5440)
These `*.service` files are installed on user installations, so
referring to a license file outside their installation doesn't make
too much sense.
2023-09-18 21:54:06 +00:00
tersec a06c0c550f
enable Windows GitHub Actions with Nim 20 (#5413) 2023-09-12 04:57:20 +00:00
tersec 59ccd25d8d
disable GitHub Actions Windows CI harder 2023-09-08 16:35:39 +00:00
tersec b424d214c2
Windows GitHub Action builds OOM; disable (#5410) 2023-09-08 16:13:25 +00:00
Etan Kissling b8db44d761
update tested Nim versions (#5142)
Add `version-2-0` testing, and make upstream version testing Linux only.

Co-authored-by: tersec <tersec@users.noreply.github.com>
2023-09-06 19:00:15 +00:00
Etan Kissling f420f09ac1
less confusing lint error msg when bumping to very recent commit (#5400)
When bumping to a more recent commit than the configured `branch`,
currently the lint error message is confusing:

```
fatal: error processing shallow info: 4
Submodule 'vendor/nim-chronos': Failed to fetch 'master':
```

This happens when the selected commit is more recent than the latest
one on the `branch`. Comparing the commit dates allows a better message.
2023-09-06 18:52:31 +00:00
Etan Kissling 4d2fd8c2b6
fix branch check when bumping to commit outside of shallow range (#5349)
CI Lint check failed when bumping to a commit outside default shallow
range. Deepen the checkout through the bumped commit date to ensure
history is available for the ancestry check.
2023-08-24 18:23:05 +00:00
Etan Kissling bee0342f35
fix branch check in CI (#5326)
Use explicit refs to avoid occasional issues with branch check reporting
incorrectly that commit is not an ancestor of upstream head.
2023-08-19 20:49:52 +00:00
Etan Kissling 1fa3941aa5
set `WITH_UBSAN` correctly when running on GA (#5299)
GitHub actions script was not setting `WITH_UBSAN` correctly. Fix it.
2023-08-16 16:56:05 +00:00
Etan Kissling c26e8132fc
fix ubsan for `test_libnimbus_lc` (#5258)
Pass `-fsanitize=undefined` flag when compiling `test_libnimbus_lc`
when building in ubsan mode.
2023-08-07 07:57:56 +02:00
Etan Kissling 722b7f6f6b
move lint checks to separate job (#5200)
Running the lint checks separately allows running tests to check code
correctness even when targeting non-master branches or having outdated
copyright headers.
2023-07-19 13:55:23 +02:00
Etan Kissling 971b4483c5
add `libnimbus_lc.a` C library (#5122)
Add a new C library for processing light client data based on the Nimbus
implementation. This can be used from other, non-Nimbus components.
2023-07-19 09:48:39 +02:00
Etan Kissling 3c3a4ff3f3
run tests against both submodule and latest Nim (#5140)
Currently CI only tests against status `version-1-6` branch.
Update to test against the selected commit through submodule lock,
as well as the latest upstream `version-1-6` instead.
2023-06-28 23:24:23 +02:00