* Bump nimbus-eth2 and fix beacon LC related changes
* Disable local_testnet test in CI
Latest version is no longer compatible with the old nightlies that are being downloaded for the test
* Updated to the latest nim-eth, nim-rocksdb, nim-web3
* Bump nimbus-eth2 module and fix related issues
Temporarily disabling Portal beacon light client network as it is
a lot of copy pasted code that did not yet take into account
forks. This will require a bigger rework and was not yet tested
in an actual network anyhow.
* More nimbus fixes after module bumps
---------
Co-authored-by: Adam Spitz <adamspitz@status.im>
Co-authored-by: jangko <jangko128@gmail.com>
Two unresolved items currently:
- Three tests that are temporarily disabled as they fail in the
macro_assembler code, which seems to be due to an ambigious
identifier Stop (Ops and chronos ServerCommand enum).
- i386 CI disabled as it fails at Nim compilation already. Failed
tests where already ignored for this target.
* Move BlockHeaderWithProof content to content key selector 0
- Remove as content type with content key selector 4
- Replace regular block header with BlockHeaderWithProof at
content key selector 0
* Apply blockHeader content key also to bridge
* Add tests for header with proof generation and verification
The `BlockHeader` structure in `nim-eth` was updated with support for
EIP-4844 (danksharding). To enable the `nim-eth` bump, the ingress of
`BlockHeader` structures has been hardened to reject headers that have
the new `excessDataGas` field until proper EIP4844 support exists.
https://github.com/status-im/nim-eth/pull/570
The `BlockHeader` structure in `nim-eth` was updated with support for
EIP-4895 (withdrawals). To enable the `nim-eth` bump, the ingress of
`BlockHeader` structures has been hardened to reject headers that have
the new `withdrawalsRoot` field until proper withdrawals support exists.
https://github.com/status-im/nim-eth/pull/562
* Add headers with proof content type and use it for verification
- Add BlockHeaderWithProof content type & content
- Use BlockHeaderWithProof content to verify if chain data is
part of the canonical chain
- Adjust parser & seeder code to be able to seed these headers
with proof
- Adjust eth_data_exporter to be able to export custom header
ranges for which to build proofs (mostly for testing)
There is currently quite some ugliness & clean-up needed for which
a big part is due tos upporting both BlockHeader and
BlockHeaderWithProof on the network.
* Change accumulator proof to array / SSZ vector type
- Change accumulator proof to SSZ vector instead of SSZ list.
- Add and use general buildProof and buildHeaderWithProof func.
* Make the BlockHeaderWithProof an SSZ Union with None option
* Update portal-spec-tests to master commit
Portal master accumulator was removed from the network specs as a
content type shared on the network, as since the merge this is
a finite accumulator (pre-merge only).
So in this PR the accumulator gets removed as network type and
gets instead baked into the library. Building it is done by
seperate tooling (eth_data_exporter).
Because of this a lot of extra code can be removed that was
located in history_network, content_db, portal_protocol, etc.
Also removed to option to build the accumulator at start-up
of fluffy as this takes several minutes making it not viable.
It can still be loaded from a provided file however.
The ssz accumulator file is for now stored in the recently
created portal-spec-tests repository.
In preparation of our migration to the new Nimble-based setup
and [nim-workspace][1], we switch to a new model where the vendor
packages are no longer imported through a locally generated Nimble
dir, but rather through an auto-generated `nimbus-build-system.paths`
file that features regular `--path:` statements.
This file will be imported only within the nimbus-build-system
environment in order to avoid any unwanted interference in working
copies based on the new Nimble setup.
[1]: https://github.com/status-im/nim-workspace
* Bump nim-stew
why:
Need fixed interval set
* Keep track of accumulated account ranges over all state roots
* Added comments and explanations to unit tests
* typo
* Provided common scheduler API, applied to `full` sync
* Use hexary trie as storage for proofs_db records
also:
+ Store metadata with account for keeping track of account state
+ add iterator over accounts
* Common scheduler API applied to `snap` sync
* Prepare for accounts bulk import
details:
+ Added some ad-hoc checks for proving accounts data received from the
snap/1 (will be replaced by proper database version when ready)
+ Added code that dumps some of the received snap/1 data into a file
(turned of by default, see `worker_desc.nim`)
* Error return in `persistBlocks()` on initial `VmState` roblem
why:
previously threw an exception
* Updated sync mode option
why:
using enum rather than bool => space for more
* Added sync mode `full`, re-factued legacy sync
also:
rebased
* Fix typo (crashes `pesistBlocks()` otherwise)
also:
rebase to master
* Reduce log ticker noise by suppressing duplicate messages
* Clarify staged queue overflow handling
why:
backtrack/re-org mode in `stageItem()` should be detected by both,
the global indicator or the work item where it might have moved into.
also:
rebased
* Relocated `IntervalSets` to nim-stew repo
* Accumulate accounts on temporary kv-DB
why:
Explore the data as returned from snap/1. Will be converted to a
`eth/db` next.
details:
Verify and accumulate per/state-root accounts downloaded via snap.
also:
Some unit tests
* Replace `Table` by `TrieDatabaseRef` for accounts accumulator
* update ticker statistics
details:
mean/variance based counter update
* allow persistent db for proved accounts
* rebase, and globally activate unit test
* fix statistics
* Reorg SnapPeerBase descriptor, notably start/stop flags
details:
Instead of using three boolean flags startedFetch, stopped, and
stopThisState a single enum type is used with values SyncRunningOk,
SyncStopRequest, and SyncStopped.
* Restricting snap to eth66 and later
why:
Id-tracked request/response wire protocol can handle overlapped
responses when requests are sent in row.
* Align function names with source code file names
why:
Easier to reconcile when following the implemented logic.
* Update trace logging (want file locations)
why:
The macros previously used hid the relevant file location (when
`chroniclesLineNumbers` turned on.) It rather printed the file
location of the template that was wrapping `trace`.
* Use KeyedQueue table instead of sequence
why:
Quick access, easy configuration as LRU or FIFO with max entries
(currently LRU.)
* Dissolve `SnapPeerEx` object extension into `SnapPeer`
why;
It is logically cleaner and more obvious not to inherit from
`SnapPeerBase` but to specify opaque field object references of the
merged `SnapPeer` object. These can then be locally inherited.
* Dissolve `SnapSyncEx` object extension into `SnapSync`
why;
It is logically cleaner and more obvious not to inherit from
`SnapSyncEx` but to specify opaque field object references of
the `SnapPeer` object. These can then be locally inherited.
Also, in the re-factored code here the interface descriptor
`SnapSyncCtx` inherited `SnapSyncEx` which was sub-optimal (OO
inheritance makes it easier to work with call back functions.)
* Update exception tracinig
* Use lazy JSON parser
why:
Uint246 type is not directly supported by the JSON serializer
* Json parser update for stringified UInt256 integers
why:
Now available
* update sub-module branch reference
Currently re-using http connections for the json-rpc requests
seems to slow down the requests considerably. Avoid this by
forcing a close after each requests in the blockwalk tool.
* Enable JWT authentication for websockets
details:
Currently, this is optional and only enabled when the jwtsecret option
is set.
There is a default mechanism to generate a JWT secret if it is not
explicitly stated. This mechanism is currently unused.
* Make JWT authentication compulsory for websockets
* Fix unit test entry point + cosmetics
* Update JSON-RPC link
* Improvements as suggested by Mamy
* Enable optional chunked RLPx messages
why:
Legacy feature used by Nethermind
details:
Disable with make flag: ENABLE_CHUNKED_RLPX=0
* Rebase & bump nim-eth
* Fix default behaviour
why:
Got lost somehow. Comments do not match GNU make code directives.
* dist: precompiled binaries and Docker images
The builds are reproducible, the binaries are portable and statically link librocksdb.
This took some patching. Upstream PR: https://github.com/facebook/rocksdb/pull/9752
32-bit ARM is missing as a target because two different GCC versions
fail with an ICE when trying to cross-compile RocksDB. Using Clang
instead is too much trouble for a platform that nobody should be using
anyway.
(Clang doesn't come with its own target headers and libraries, can't be
easily convinced to use the ones from GCC, so it needs an fs image from
a 32-bit ARM distro - at which point I stopped caring).
* CI: disable reproducibility test
* Improve the tests of the local testnet
The local testnet test was rather flaky and would occasionally
fail. It has been made more robust by adding the ENRs directly
to the routing table instead of doing some random lookups.
Additionally, the amount of nodes were increased (=64), ip limits
configuration was added, and the bits-per-hop value was set to 1
in order to make the lookups more likely to hit the network
instead of only the local routing table.
Failure is obviously still possible to happen when sufficient
packets get lost. If this turns out to be the case with the current
amount of nodes, we might have to revise the testing strategy here.
* Disable lookup test for State network
Disable lookup test for State network due to issue with custom
distance function causing the lookup to not always converging
towards the target.
Includes a simple test harness for the merge interop M1 milestone
This aims to enable connecting nimbus-eth2 to nimbus-eth1 within
the testing protocol described here:
https://github.com/status-im/nimbus-eth2/blob/amphora-merge-interop/docs/interop_merge.md
To execute the work-in-progress test, please run:
In terminal 1:
tests/amphora/launch-nimbus.sh
In terminal 2:
tests/amphora/check-merge-test-vectors.sh
* crash test scenario
details:
Example code for inspecting nested block chain and accounts cache
database transaction framework. There seems to be a pathological
case where the system crashes after a rollback (as appeared in the
tx-pool packer code.)
* simplified crash scenario
* Workable solution (as suggested by Andri)
details:
Avoiding db.rollback() (db.commit() is OK) while vmState.stateDB is
alive.
* Rename text_txcrash => test_accounts_cache
why:
Unit tests covers part of accounts_cache handling
* comment update
Currently bootstrap nodes for discv5 and for the Portal nodes
were provided through separate cli arguments. This is however
confusing and cumbersome as typically when (currently) testing
a node will have both discv5 and the Portal networks enabled.
We merge them into one argument.
If a node happens not to support a Portal network, it will be
removed after message request failure.
Commit also contains additional clean-up and nim-eth bump.
bugfix and features:
- Switch to the Chronos HTTP client (adds support for HTTPS)
- Allow dynamic RPC method names in the 'rpc' macro
- Restore the support for using the news package
- Add basic discv5 and portal json-rpc calls and activate them in
fluffy
- Renames in the rpc folder
- Add local testnet script and run this script in CI
- bump nim-eth
* Add SSZ Unions through case objects
* Add connection id content response test and improve other test vectors
* Implement content keys and ids for state network as per spec
Content keys case object is used so that it can be serialized and
deserialized as an SSZ Union.
* Let message Union in Portal wire protocol start at 0 as per new spec
features:
- playground: add support to run inside docker
- add secure mode to graphql http server
- add secure mode to graphql http client
- httpclient: add unzip dan chunked transfer of HTTP 1.1
bugfixes:
- fix accept-encoding bug in httpserver
websock:
- "Host" header override (#87)
- Use caps for log-levels in tests
- Perform utf-8 validation at message boundaries (#90)
json-rpc:
- fix both ws rpc server and client due to websock breaking changes
- nim-graphql from v0.2.16 to v0.2.18:
- replace nim-miniz with nim-zlib
- replace unittest with unittest2
- nim-http-utils:
- Fix to handle HTTP response with `<space>CRLF` after response status code.
- nim-json-rpc:
- rename ws to websocket
- nim-websock:
- rename ws to websock
- fix mutual closing bug
- fix compile time error when using chronicles sink:json
- nim-chronicles:
- add isLogFormatUsed, used by nim-websock
- nim-json-rpc already replace news with nim-ws
- we also remove news from dependency
- we already have include nim-ws as dependency
- new chronos features is needed by nim-ws
- we replace nim-miniz with nim-zlib because nim-ws requires
features not available in miniz
- nim-graphql also switch to nim-zlib
- nim-ws depends on nim-zlib and we need it for upcoming
websocket rpc
- fixes non repeatable directive error message
- refactor nim caching in github action ci
- refactor github action ci matrix
- extract miniz to it's own repo
This bump is required by `TransactionHost` coming next.
There have been many changes to `nim-evmc` since earlier developments on
`nimbus-eth1`. The types in the API have changed, and there is no longer a
need to cast the function types.
This commit bumps the submodule while keeping with EVMC API 7.5 for now.
`.gitmodules` is updated to follow the module branch `api-version-7.5`.
Submodule changes:
> Nim: Function casts no longer required, type-checking finally works
> Nim: Make the test host use exactly the type signatures in the API
> add github action script
> Nim: Export `evmc_create_vm_name_fn`
> Nim: Use `var` instead of `ptr` for arguments that are pass-by-ref
> Nim: Fix missing `ptr` in signature of `execute`
> Nim: Use `ptr` in `release` for API consistency
> Nim: Disable "Imported and not used" warning in test program
> Nim: Remove unnecessary exports from the Nim example host/VM
> Nim: Simplify Nim C++ codegen bug workaround
> Nim: Add type and documentation for `evmc_create_example_fn`
> Nim: Workaround Nim compiler bug with Nim 1.2.10
> Nim: Workaround Nim compiler bug with `uint32`-sized bitset
> Nim: Use a more varied bit pattern in `get_capabilities` test
> Nim: Convert `evmc_capabilities` to Nim bitset
> Nim: Convert `evmc_flags` to Nim bitset
> Nim: Compile time verify that target C enums are `int` sized
> Nim: Recognise that target C enums are not always `cint`
> Nim: Make `evmc_status_code` a normal enum
> Nim: Use `csize_t` in test program and Nim example host/VM
> Nim: Use `csize_t` in the VM support API
> Nim: Use `csize_t` in appropriate places to match EVMC API
> Nim: Use binary-compatible C99 `bool` without target assumptions
> Version (README): Announce we support EVMC 7.5.0 and Istanbul fork
> Doc: Tweak GitHub badges on README
> Doc: Make title fit on GitHub page without wrapping
> Doc: README edits
> Doc: Update the README for 2021 - the year of Nimbus!
> License: Add licensing info about third-party files
> License: Update license text in README
> License: Fix incorrect name of Apache v2 license file and tweak
> License: Add missing MIT license file
> Nim: Add and update copyright headers
> Nim: Update Nim API to follow EVMC API 7.5.0
> Nim: Update doc comments and fix formatting
> Make the tests work with the new C++ mini-EVM in EVMC 7.5.0
> Change C++ `example_host` to workaround Nim -> C call type mismatch
> The EVMC package expects C++14 now
> Upgrade C++ `example_host` in `tests/evmc_c` to EVMC 7.5.0
> Upgrade C++ `example_vm` in `tests/evmc_c` to EVMC 7.5.0
> Upgrade C/C++ headers in `tests/evmc_c` to EVMC 7.5.0
Signed-off-by: Jamie Lokier <jamie@shareable.org>
enhancements:
- scalar `Float` now have proper validation
- seqs are not copied anymore in some places, but using system.move
bugfixes:
- `removeWhitespaces` in test_common no longer eat white spaces
from quoted string.
- `skipWhitespace` in lexer.nim now preserve \r\n instead of replace it with \n
new features:
- add custom instrument infrastructure
- add Query Complexity instrumen
- enhancements:
- response stream reform:
- allow for more response stream implementation easily
- features:
- add NodeRespStream:
- return AST produced by execution engine instead of serialize
it into json or something else
- add store flag to parseSchema and parseQuery family:
- with this additional flag, both query and schema can be marked
with stored flag
- purge also become selective when purging queries
- dog fooding the introspection schema loader using this flag
bugfixes:
- next fields will continue execution if there is error:
- if one field failed, it will not terminate subsequent fields
execution, but errors are kept.
- propagate inner field error to it's parent
- Since Non-Null type fields cannot be null, field errors are propagated to be
handled by the parent field. If the parent field may be null then it resolves
to null, otherwise if it is a Non-Null type, the field error is further
propagated to it’s parent field.
- If a List type wraps a Non-Null type, and one of the elements of that list
resolves to null, then the entire list must resolve to null. If the List type
is also wrapped in a Non-Null, the field error continues to propagate upwards.
- If all fields from the root of the request to the source of the field error
return Non-Null types, then the "data" entry in the response should be null.
nim-graphql v0.2.2 have numerous bugfixes, but notable ones are:
- only one non-introspection field are allowed in subscription root
- @skip and @include should not allowed for subscription root
- more descriptive error message for playground ethapi, fixes
- fixes GraphiQL client complaint about our instrospection system is old
- graphql http server enhancements: gzip encoding and chunked transfer
- fixes http server response status code
- fixes `__schema.types` and `__schema.directives` implementation
- fixes 'getOperation' in executor.nim
- web ui(graphiql) for http server
a crash bug is found in nim-graphql validator for
input object value coercion. the bug already fixed.
another non critical bug is about builtin 'specifiedByURL'
directive changed to 'specifiedBy' following the newest
graphql spec.
recent nim-serialization API changes already made it's way into
toml-serialization. and with the addition of newest toml-serialization
along with it's bugfixes, those submodules mentioned above
become out of date because toml-serialization need newer version of
nim-serialization.
* switch to chronos metrics, remove insecure
See https://github.com/status-im/nimbus-eth2/pull/2468
also fixes pcre linking for real, and adds some random build flags that
help nimbus-eth2 stay afloat
* fix help
* don't omit frame pointers on windows
Prior to this commit, when using discv4 (Kademlia) to find peers, there is a
crash after a few minutes. It occurs for most of us on Eth1 mainnet, and
everyone on Ropsten.
Tested for about 4 days constant operation on Ropsten. The crash which occured
every few minutes no longer occurs, and discv4 keeps working.
Signed-off-by: Jamie Lokier <jamie@shareable.org>
* bump vendor/nimbus-build-system
- add the Nim compiler header to the Nimbus header
- also support the USE_LIBBACKTRACE env var
* "go-checks" target no longer available
Kim De Mey
jangko
Adam Spitz
Etan Kissling
KonradStaniec
Zahary Karadjov
Jacek Sieka
Nikolay Mitev
Jordan Hrycaj
Ștefan Talpalaru
Zahary Karadjov
Jamie Lokier