status-im
/
nimbus-eth2
mirror of https://github.com/status-im/nimbus-eth2.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
7,068 Commits 109 Branches 116 Tags 222 MiB
Commit Graph

179 Commits

Author SHA1 Message Date
Eugene Kabanov 18409a69e1
Light forward sync mechanism (#6515) 
* Initial commit.

* Add hybrid syncing.

* Compilation fixes.

* Cast custom event for our purposes.

* Instantiate AsyncEventQueue properly.

* Fix mistype.

* Further research on optimistic updates.

* Fixing circular deps.

* Add backfilling.

* Add block download feature.

* Add block store.

* Update backfill information before storing block.

* Use custom block verifier for backfilling sync.

* Skip signature verification in backfilling.

* Add one more generic reload to storeBackfillBlock().

* Add block verification debugging statements.

* Add more debugging

* Do not use database for backfilling, part 1.

* Fix for stash.

* Stash fixes part 2.

* Prepare for testing.

* Fix assertion.

* Fix post-restart syncing process.

* Update backfill loading log statement.
Use proper backfill slot callback for sync manager.

* Add handling of Duplicates.

* Fix store duration and block backfilled log statements.

* Add proper syncing state log statement.

* Add snappy compression to beaconchain_file.
Format syncing speed properly.

* Add blobs verification.

* Add `slot` number to file structure for easy navigation over stream of compressed objects.

* Change database filename.

* Fix structure size.

* Add more consistency properties.

* Fix checkRepair() issues.

* Preparation to state rebuild process.

* Add plain & compressed size.

* Debugging snappy encode process.

* Add one more debugging line.

* Dump blocks.

* One more filedump.

* Fix chunk corruption code.

* Fix detection issue.

* Some fixes in state rebuilding process.

* Add more clearance steps.

* Move updateHead() back to block_processor.

* Fix compilation issues.

* Make code more async friendly.

* Fix async issues.
Add more information when proposer verification failed.

* Fix 8192 slots issue.

* Fix Future double completion issue.

* Pass updateFlags to some of the core procedures.

* Fix tests.

* Improve initial sync handling mechanism.

* Fix checkStateTransition() performance improvements.

* Add some performance tuning and meters.

* Light client performance tuning.

* Remove debugging statement.

* Use single file descriptor for blockchain file.

* Attempt to fix LC.

* Fix timeleft calculation when untrusted sync backfilling started right after LC block received.

* Workaround for `chronicles` + `results` `error` issue.
Remove some compilation warnings.
Fix `CatchableError` leaks on Windows.

* Address review comments.

* Address review comments part 2.

* Address review comments part 1.

* Rebase and fix the issues.

* Address review comments part 3.

* Add tests and fix some issues in auto-repair mechanism.

* Add tests to all_tests.

* Rename binary test file to pass restrictions.

* Add `bin` extension to excluded list.
Recover binary test data.

* Rename fixture file to .bin again.

* Update AllTests.

* Address review comments part 4.

* Address review comments part 5 and fix tests.

* Address review comments part 6.

* Eliminate foldl and combine from blobs processing.
Add some tests to ensure that checkResponse() also checks for correct order.

* Fix forgotten place.

* Post rebase fixes.

* Add unique slots tests.

* Optimize updateHead() code.

* Add forgotten changes.

* Address review comments on state as argument.
 2024-10-30 05:38:53 +00:00
tersec a39dd8e874
remove remaining references to logtrace (#6684) 2024-10-29 11:02:44 +01:00
Etan Kissling 03369d9c07
Replace `apt-fast` with `apt-get` (#6644) 
`apt-fast` was removed from GitHub with Ubuntu 24.04:

- https://github.com/actions/runner-images/issues/10003

For compatibility, switch back to `apt-get`.
 2024-10-15 21:45:28 +00:00
tersec b4efaaa573
test with macOS ARM64 (#6635) 
* test with macOS ARM64

* assign platform variable
 2024-10-11 11:08:46 +02:00
tersec 409023b95c
use macos-13 builders; macos-12 EOL (#6602) 2024-10-06 17:11:46 +02:00
tersec 72c7398f85
Revert "stop testing with broken upstream/version-2-0 (#6597)" (#6599) 
This reverts commit ac321707e0.
 2024-10-06 03:02:19 +02:00
tersec ac321707e0
stop testing with broken upstream/version-2-0 (#6597) 2024-10-05 05:27:24 +02:00
tersec 1258fd3592
Revert "stop testing with broken upstream/version-2-0" (#6573) 
* Revert "stop testing with broken upstream/version-2-0 (#6554)"

This reverts commit 205dff3378.

* stop using template which Nim 2.0.10 will apparently make recursive
 2024-10-03 01:00:39 +00:00
tersec f54a0366ab
use separate upload/download names for v4 migration 2024-09-30 11:36:15 +00:00
tersec 7a91815014
update actions/upload-artifact@v3 to actions/upload-artifact@v4 2024-09-30 10:47:46 +00:00
tersec 205dff3378
stop testing with broken upstream/version-2-0 (#6554) 2024-09-17 01:14:29 +02:00
dependabot[bot] e6ebefc364
Bump actions/download-artifact from 3 to 4.1.7 in /.github/workflows (#6533) 
Bumps [actions/download-artifact](https://github.com/actions/download-artifact) from 3 to 4.1.7.
- [Release notes](https://github.com/actions/download-artifact/releases)
- [Commits](https://github.com/actions/download-artifact/compare/v3...v4.1.7)

---
updated-dependencies:
- dependency-name: actions/download-artifact
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-09-04 03:13:33 +00:00
Jacek Sieka 42c4ea109c
Revert "stop testing broken upstream version-2-0" (#6506) 
* Revert "stop testing broken upstream version-2-0 (#6499)"

This reverts commit 2be7eba25a.

* fix field warning template
 2024-08-23 12:18:06 +02:00
tersec 2be7eba25a
stop testing broken upstream version-2-0 (#6499) 2024-08-20 14:33:54 +02:00
Etan Kissling cac63a3a82
use `pull_request_target` to run PR block action (#6376) 
To avoid requiring authorization to run the PR block action for new
contributors, use `pull_request_target`. Running the workflow file from
the destination branch does not require approval.

- https://stackoverflow.com/questions/74957218/what-is-the-difference-between-pull-request-and-pull-request-target-event-in-git
 2024-07-03 13:00:39 +02:00
tersec 2d4ece0c3b
bump nimbus-build-system to use Nim v2.0.6 (#6386) 
* bump nimbus-build-system to use Nim v2.0.6

* fix: update name and hash for csources of Nim v2

Otherwise we get errors like:
```
Building: Nim compiler
/build/source/vendor/nimbus-build-system/vendor/Nim /build/source
cmd: git clone -q --depth 1 -b master https://github.com/nim-lang/csources_v2.git csources_v2
24.6.0-dirty
cmd: cd csources_v2
ci/funs.sh: line 10: cd: csources_v2: No such file or directory
make[1]: *** [vendor/nimbus-build-system/makefiles/targets.mk:81: build-nim] Error 1
```
Also need to add source for `checksums` repository.

Signed-off-by: Jakub Sokołowski <jakub@status.im>

---------

Signed-off-by: Jakub Sokołowski <jakub@status.im>
Co-authored-by: Jakub Sokołowski <jakub@status.im>
 2024-06-25 00:02:16 +02:00
Etan Kissling e4efe9ed75
ensure that rarely ran files are compiled on PR (#6379) 
* ensure that rarely ran files are compiled on PR

Add some missing files to `isMainModule` developer internal builds CI.

* fix fork choice compilation

* fix `rest_api_benchmark` compilation

* skip linking

* fix loop

* fix `mock_genesis`

* fix signedness
 2024-06-23 23:35:56 +00:00
Miran 597f47317f
use Nim 2.0.6 (#6366) 2024-06-18 03:54:15 +02:00
Jakub Sokołowski 90cf1753bd
use self-hosted runners for Developer builds GH action 
Otherwise we get:
```
Error: The operation was canceled.
```
Which is most probably due to OOM error.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2024-05-29 15:45:44 +02:00
tersec 1051c3c9c5
use upstream/version-2-0 of Nim in CI (#6277) 2024-05-09 02:22:28 +02:00
Jakub Sokołowski 4a1a2c8196
add packages to Nix flake config, move to repo root 
This way we can actually build and run a node using just:
```sh
nix run 'github:status-im/nimbus-eth2?submodules=1'
```
The `?submodules=1` part should eventually not be necessary.
For more details see:
https://github.com/NixOS/nix/issues/4423

Signed-off-by: Jakub Sokołowski <jakub@status.im>
 2024-05-06 16:57:23 +02:00
tersec fc3fad2433
revert to v2.0.4 2024-05-04 08:39:33 +00:00
Jacek Sieka ab30efaf80
Integrate `hashtree` for better hashing performance (#5188) 
status-im/nim-ssz-serialization#35 brings in https://github.com/prysmaticlabs/hashtree as a supported backend for SHA256, giving a nice little performance boost to all hash_tree_root calls on supported platforms / compilers.

Expected gains are on the order of 30% which in the case of a replayed state nets us 0.2-0.3s improvement.

More about this design here: https://hackmd.io/@potuz/BJyrx9DOF - kudos to @potuz for this excellent library!
 2024-05-02 08:54:32 +02:00
Etan Kissling 9e8d2e70bd
block pull requests to `stable` (#6001) 
* block pull requests to `stable`

Pull requests should be opened against the `unstable` branch.
See https://nimbus.guide/contribute.html#build-and-deploy

* avoid deleting branch to mimic manual close

* avoid triggering on dependabot PRs
 2024-05-01 19:30:03 +00:00
tersec 90ac2def31
switch back to version-2-0 (#6233) 2024-04-24 13:49:05 +00:00
tersec eb997f4e1f
add 10 sub-test-fixtures for electra epoch transitions; use v2.0.4 for Nim 2.x (#6229) 2024-04-23 01:15:55 +00:00
tersec 41f8400f97
fix Electra light client objects; use version-2-0 for Nim again (#6222) 2024-04-19 19:20:45 +00:00
tersec 73738c702c
use consensus spec v1.5.0-alpha.0 test vectors; use Nim v2.0.4 specifically for 2.0.x (#6221) 2024-04-19 02:55:58 +00:00
Etan Kissling a18c396d9d
use correct Nim version in daily job, and `--mm:refc` on `devel` (#6023) 
Daily CI uses an outdated Nim 1.6 because it uses `origin/version-1-6`
which is not maintained very regularly. Pull from `upstream/version-1-6`
instead, same as in `ci.yml`, and also make sure that `--mm:refc` is
turned on for `upstream/devel`.
 2024-03-04 15:21:28 +01:00
Etan Kissling ecf175856f
bump `actions/checkout` and `actions/cache` to `v4` (#5971) 
Typically, mixing different versions of GitHub actions may lead to hard
to fix issues. As many libraries have been updated to use actions v4,
also bump our own `actions/checkout@v2`, `actions/checkout@v3`,
`actions/cache@v2` and `actions/cache@v3` to their `@v4` equivalents.
 2024-02-28 19:04:34 +01:00
Etan Kissling 702f4c6a45
add `{.push raises: [].}` to more files (#5943) 
Extend `{.push raises: [].}` to all files with spec references that are
periodically updated.

- https://github.com/status-im/nimbus-eth2/pull/5942
 2024-02-22 11:03:09 +01:00
Etan Kissling 698a802168
extend linter to enforce exception tracking is turned on (#5909) 
Status Nim style mandates `{.push raises: []}.` at start of modules.
Add a CI task to ensure exceptions keep getting properly tracked.

- https://status-im.github.io/nim-style-guide/errors.exceptions.html
- https://github.com/status-im/nim-eth/pull/614#discussion_r1220906149
 2024-02-19 11:02:36 +01:00
Etan Kissling 30b7c6153f
handle `Exception` during `EraFile.verify` (#5900) 
`Taskpool.new()` is marked as `{.raises: [Exception].}`. Catch this.
 2024-02-17 18:19:30 +01:00
tersec 7b5815358e
test macOS with Nim 2.0 in CI and use non-EOL macOS version (#5890) 2024-02-16 04:53:13 +00:00
Etan Kissling ed8743b986
fix standalone compilation of `trusted_node_sync.nim` (#5861) 
#5544 contained a regression that broke standalone compilation of
`trusted_node_sync` as a main module. Fix it, and add to CI.
 2024-02-07 19:26:29 +00:00
Etan Kissling a9ca59e074
avoid mixing versions across `upload-artifact`/`download-artifact` (#5853) 
The various major versions of `action/upload-artifact` and
`action/download-artifact` are not necessarily compatible.
Align all the uploads / downloads to `v3`.
`v4` exists but is not currently supported on GHES yet.
 2024-02-07 09:54:58 +01:00
Etan Kissling 388b865aac
add `-d:has_deposit_root_checks` build to CI (#5856) 
To prevent accidental breakage of builds with non-standard developer
flags, add the `-d:has_deposit_root_checks` build to CI. This is only
a quick compilation check, and is only run on Linux.
 2024-02-07 00:11:51 +00:00
andri lim a45609c4a3
Switch to llvm-mingw for faster Windows CI (#5729) 
* Switch to llvm-mingw for faster Windows CI

* Enable ubsan of Windows CI
 2024-01-13 11:45:20 +02:00
jangko a5297a352c
Use GITHUB_OUTPUT to replace set output in main CI 2024-01-11 10:14:32 +07:00
Jacek Sieka 3f525acb87
chronos: bump (#5684) 
This PR causes a few new warnings to appear - these are harmless but
will need addressing separately as they span several libraries.

* new asyncraises syntax
* asyncraises support in several modules
* `sink` usage reduces spurious copying
* `?` compatiblity for `async` + `results`
* remove `-d:chronosStrictException` (obsolete)
 2024-01-05 10:08:38 +01:00
Etan Kissling 0639eaafd0
in lint, fix timestamp comparison when timezone does not match (#5688) 
Git by defaults returns commit timestamp according to the committer's
time zone instead of the local one, breaking the simple alphanumeric
comparison for timestamps that we use in lint. Force the timezone to
UTC so that comparison is correct regardless of committer timezone.
 2024-01-03 00:34:39 +00:00
tersec aa6c13cd67
use newer geth testnet binary directory names (#5575) 
* use newer geth testnet binary directory names

* linting; Geth requires merged genesis
 2023-11-07 16:52:30 +01:00
tersec 95b22e3a4f
Revert "Revert "ci: adding self-hosted tag for github-runner (#4886)" (#5494)" 
This reverts commit 722deedd2b.
 2023-10-13 13:45:07 +02:00
tersec 722deedd2b
Revert "ci: adding self-hosted tag for github-runner (#4886)" (#5494) 
This reverts commit acbfef9852.
 2023-10-11 20:27:45 +00:00
apentori acbfef9852
ci: adding self-hosted tag for github-runner (#4886) 
Signed-off-by: Alexis Pentori <alexis@status.im>
 2023-10-11 20:26:18 +00:00
Etan Kissling a261384efc
exclude `*.service` files from lint check (#5440) 
These `*.service` files are installed on user installations, so
referring to a license file outside their installation doesn't make
too much sense.
 2023-09-18 21:54:06 +00:00
tersec a06c0c550f
enable Windows GitHub Actions with Nim 20 (#5413) 2023-09-12 04:57:20 +00:00
tersec 59ccd25d8d
disable GitHub Actions Windows CI harder 2023-09-08 16:35:39 +00:00
tersec b424d214c2
Windows GitHub Action builds OOM; disable (#5410) 2023-09-08 16:13:25 +00:00
Etan Kissling b8db44d761
update tested Nim versions (#5142) 
Add `version-2-0` testing, and make upstream version testing Linux only.

Co-authored-by: tersec <tersec@users.noreply.github.com>
 2023-09-06 19:00:15 +00:00