status-im/nim-ssz-serialization#35 brings in https://github.com/prysmaticlabs/hashtree as a supported backend for SHA256, giving a nice little performance boost to all hash_tree_root calls on supported platforms / compilers.
Expected gains are on the order of 30% which in the case of a replayed state nets us 0.2-0.3s improvement.
More about this design here: https://hackmd.io/@potuz/BJyrx9DOF - kudos to @potuz for this excellent library!
* Make listen-address default to use dualstack.
* Use correct newProtocol().
* Bump nim-eth.
* Bump nim-eth one more time.
* Use `*` instead of IPv6 address for dualstack sockets.
* Bump chronos and nim-eth.
* Use new constructor.
* Fix listenAddress should be Opt[T] not Option[T].
* Fix options.md.
* chore: change nim-libp2p branch from unstable to master
* chore: dummy bump of nim-libp2p version to please linter
---------
Co-authored-by: ksr <kaiserd@users.noreply.github.com>
- fix: reset accept fut in stop
- always allow new data be received if the recvWindow is > 0
- Revert "always allow new data be received if the recvWindow is > 0"
- Add custom ring buffer into chronos streams and transports.
- Add `join()` operation to wait for future completion.
- Ensure that all buffers used inside HTTP client will follow original buffer size.
- Add automatic constructors for TCP and UDP transports.
- add support for setting protocol handlers with `{.raises.}` annotation
- fix: valueOr and withValue utilities
- fix: remove explicit param from GossipSubParams constructor
- fix miniupnpc build on Windows by not escaping PATH
- allow overriding CC for libimiupnpc.a and libnatpmp.a targets
- Use -fPIC when building nat traversal
- default `MultiAddress` param for `newStandardSwitch` does not raise
- clean up triple lookup and avoid `KeyError` when adding muxer
- `{.async: (raises).}` for `relay/utils.nim`
- `{.async: (raises).}` annotations for `protocols/secure`
- avoid pointless exception raising in `dcutr/server`
- fix: allFuturesThrowing compilation issue on daily
- feat: Yamux timeout
- Fix empty path crash issue for MultiAddresses `unix`, `ip6zone`, `dns***`.
- fix: move transport interop tests to nim-libp2p repo
- use chronos 4.0.0
- fix typo in `ProtoMessage.toString()`
- increase tolerance of `simple heartbeat` test
- remove unused `MultiBase.encode(..., Cid)` function
- move `allFutureThrowing` helper to tests
- use new Chronos `trackCounter` APIs for leaks checks in tests
- remove unused `LPStreamError` types
- change `SecioError` and `NoiseError` to descendants of `LPStreamError`
- do not log yamux buffers without sanitization (trace log level)
- don't forget closing the stream when final `{Fin}` fails in yamux
- use `race` instead of `or` to avoid lockup
- in yamux, do not write `{Rst}` packet to stream that's in use
- avoid `KeyError` in edge case of yamux handler
- avoid triple lookup in `m.flushed` yamux table
- catch `WebSocketError` in `wstransport`
- define proper parent error type for `YamuxError`
- document known `--mm:orc` crash
- add `{.async: (raises).}` to `libp2p/stream` modules
Fix the `/eth/v1/beacon/deposit_snapshot` API to produce proper EIP-4881
compatible `DepositTreeSnapshot` responses. The endpoint used to expose
a Nimbus-specific database internal format.
Also fix trusted node sync to consume properly formatted EIP-4881 data
with `--with-deposit-snapshot`, and `--finalized-deposit-tree-snapshot`
beacon node launch option to use the EIP-4881 data. Further ensure that
`ncli_testnet` produces EIP-4881 formatted data for interoperability.
- fix conversion error with `or` on futures with `{.async: (raises: []).}`
- Fix `or` deadlock issue.
- Ensure that `OwnCancelSchedule` flag will not be removed from `wait()` and `withTimeout()`.
- Add missing parts of defaults buffer size increase.
- Avoid `ValueError` effect in varargs `race`/`one`
- fix circular reference in timer
- Ensure `transp.reader` is reset to `nil` on error
This reverts commit 21601f59e2.
Reason: `/eth/v1/beacon/light_client/updates?start_period=0&count=128`
provides malformed data. Reverting fixes it locally.
Download JSON and SSZ, then use this script to validate (Holesky).
Need to export some symbols for parsing.
```nim
import
stew/io2,
./beacon_chain/spec/[digest, forks, helpers, presets],
./beacon_chain/spec/eth2_apis/[eth2_rest_serialization, rest_light_client_calls]
let jsonBytes = io2.readAllBytes("/Users/etan/Downloads/json_updates.json").get()
let
(cfg, unknowns) = readRuntimeConfig("/Users/etan/Documents/Repos/nimbus-eth2/vendor/holesky/custom_config_data/config.yaml")
genesis_validators_root = Eth2Digest.fromHex("0x9143aa7c615a7f7115e2b6aac319c03529df8242ae705fba9df39b79c59fa8b1")
forkDigests = newClone ForkDigests.init(cfg, genesis_validators_root)
let updates = seq[ForkedLightClientUpdate].decodeJsonLightClientObjects(jsonBytes, cfg, forkDigests)
var entries = newSeqOfCap[RestVersioned[ForkedLightClientUpdate]](updates.len)
for update in updates:
let
contextEpoch = withForkyUpdate(update):
when lcDataFork > LightClientDataFork.None:
forkyUpdate.contextEpoch
else:
continue
contextFork = cfg.consensusForkAtEpoch(contextEpoch)
entries.add RestVersioned[ForkedLightClientUpdate](
data: update,
jsonVersion: contextFork,
sszContext: forkDigests[].atconsensusFork(contextFork))
let res =
block:
var default: seq[byte]
try:
var stream = memoryOutput()
for e in entries:
withForkyUpdate(e.data):
when lcDataFork > LightClientDataFork.None:
var cursor = stream.delayFixedSizeWrite(sizeof(uint64))
let initPos = stream.pos
stream.write e.sszContext.data
var writer = SszWriter.init(stream)
writer.writeValue forkyUpdate
cursor.finalWrite (stream.pos - initPos).uint64.toBytesLE()
stream.getOutput(seq[byte])
except SerializationError:
default
except IOError:
default
let sszBytes = io2.readAllBytes("/Users/etan/Downloads/ssz_updates.ssz").get()
let sszUpdates = seq[ForkedLightClientUpdate].decodeSszLightClientObjects(res, cfg, forkDigests)
doAssert updates.len == sszUpdates.len
for i in 0 ..< updates.len:
doAssert updates[i].kind == sszUpdates[i].kind
withForkyUpdate(updates[i]):
when lcDataFork > LightClientDataFork.None:
doAssert forkyUpdate == sszUpdates[i].forky(lcDataFork)
doAssert sszBytes == res
```
- Replace eth_getBlockReceipts return type with Opt[T] instead of Option[T]
- use non-EOL macOS version and fix spelling
- test refc in CI in Nim 2.0 and later
- unittest2.nim: ensure the testTeardownIMPL is performed at the end
- Add test for PR #35
- only run `teardown` if `setup` completed
- Fix allow break in test body regression
- Upgrade github actions to v4
- use non-EOL macOS version for GitHub Actions CI
This PR allows sharing the pubkey data between validators by using a
thread-local cache for pubkey data, netting about a 400mb mem usage
reduction on holesky due to us keeping 3 permanent + several ephemeral
state copies in memory at all times and each state copy holding a full
validator.
The PR also introduces a hash cache for the key which gives ~14% speedup
for a full state `hash_tree_root` - the key makes up for a large part of
the `Validator` htr time.
Finally, the time it takes to copy a state goes down as well from ~80m
ms to ~60, for reasons similar to htr.
We use a `ptr` even if a `ref` could in theory have been used - there is
not much practical benefit to a `ref` (given it's mutable) while a `ptr`
is cheaper and easier to copy (when copying temporary states).
We could go further and cache a cooked pubkey but it turns out this is
quite intrusive - in all the relevant places, we're already using a
cooked key from the immutable validator data so there are no immediate
performance gains of doing so while managing the compressed -> cooked
key mapping would become more difficult - something for a future PR
perhaps.
Co-authored-by: Etan Kissling <etan@status.im>
- Refactor p2pProtocol internals
- Fix chronos related deprecated warnings in discv5 and uTP
- normalise nimble, ci
- Fix chronos related deprecated warnings in uTP code part II
- Remove usage of stew/shim/net
- Upgrade github actions to v4
- Fix two bugs in Receipts RLP encoding/decoding
- Move Ethereum specific RLP encoding tests under tests/common
- Reduce compiler warnings in rlp
- test refc in CI in Nim 2.0 and later
- test Nim 2.0 in CI and use non-EOL macOS version
- Use asyncraises in p2p
- Don't auto write p2pProtocol macro expansion to file
- Fix improper yield usage in rlpx and refine exception handling
- Client also handle error message if id is null
- Client pass meaningful error to newFut when processMessage failed
- Refactoring: extract rpc handler from HTTP and WebSocket server
- Use pragma push/pop pair to disable warning
- http server better exception handling
- Fix CI badge url
- Upgrade github actions to v4
- Revert "Fix CI badge url"
- HttpAuthHook use async raises
- Move CancelledError handling to outer try/except of RpcWebsocketServer
- Implement RPC batch call both in servers and clients
- v0.4.0
- Should compile if chronicles log turned on
- Add framework to support more optional types
- v0.4.2
- test refc in CI in Nim 2.0 and later
- use non-EOL macOS version for GitHub Actions CI
- avoid failing uninitialized `Future`
- Improve batch call example and wrapper comments
- Fix ws and socket client error handling and add test to #212
- Add build test with chronicles to json enabled
- update instructions for tracking upstream MIRACL Core
- bump `bls12-381-tests` to `v0.1.2`
- bump `miracl-core` to `0f67878bee7c4108405deb2b0b5e4e58d1ae30fc`
- test refc in CI in Nim 2.0 and later
- rename `milagro.nims` -> `miracl.nims`
- rename `milagro.nim` -> `miracl.nim`
- rename `milagro(Path|_func)` -> `miracl(Path|_func)`
- rename `milagro` references -> `miracl` in documentation
- Synchronize order of `blobGasUsed` / `excessBlobGas` in test
- Reduce compiler warnings
- Upgrade github actions to v4
- v0.3.2
- re-add support for `fromBlock`, `toBlock`, `blockHash` in `eth_getLogs`
- Feature flag for raises support
- HTTP server middleware implementation
- Fix examples documentation
- check leaks after every test
- deprecate `callback=`, UDP fixes
- Improve parseList and parseInlineTable strictness
- v0.2.10
- Switch to llvm-mingw for faster Windows CI
- Add configureTomlDeserialization to README.md
- Put array/inline table nonstandard behavior behind flag
- Unify parseList and parseArray implementation
- v0.2.12
- Fix a bunch of compiler hints and warnings in uTP and discv5
- Fix missing std/times import for the metrics 0.0.1 case
- Fix for uTP issues with latest chronos
- Clean-up, correct and clarify utp_protocol tests
- better async timeout wait
- Adjust test names and comments for `blobGasUsed` field
- Add data over multiple sockets uTP test
- Add uTP over discv5 test and small uTP performance improvements
Etan Kissling (2):
remove unused `skip0xPrefix`
keep the internal count helper
Will (1):
Bugfix/nully values (#61)
Yuriy Glukhov (5):
Contract constructor support
Fixed compilation error in exec function
Added string encoding
Fixed source->from field of EthCall
More flexibility to contract DSL, Async contract caller
jangko (5):
Reduce compiler warnings when using Nim v2
Migrate to json-serialization
Add tests of json rpc marshalled types
Resolve contract_dsl ambiguity
Event handler passing around JsonString instead of JsonNode
Share encoder between json-rpc and chronicles (#119)
Simplify generic constraint of rpc and chronicles encoders
Feature/execution api spec (#69)
v0.3.0
bump nim-json-rpc to a6475e49b26d3afc58aaa3d67621c94eafef8efb
coffeepots (1):
Use nim-json-serialization for RPCs (#172)
jangko (10):
Add copyright to source file
Remove StringOfJson
Fix optional parameter parsing fails in rpc macro with generics
Rename jrpc_sys module back to jsonmarshal
Reenable test hhtps
Add test for createRpcSigsFromNim and createSingleRpcSig
Let the OS choose the port for tests
Add onProcessMessage hook to client
Fix example in the README.md
Move errors module back to json_rpc folder
Upgrade rpc router internals (#178)
RPC server handle null return value correctly
v0.3.0
kdeme (1):
Add example test case that currently fails the Option parsing
- use still-supported versions of Ubuntu and macOS in GitHub Actions CI
- try py3 for autobahn
- Remove Nim 1.2
- Handle cancellation in close
- make nimble file consistent with CI setup
- replace `utils` module with direct `generate` calls
- allow manually triggering CI
- Fix grammatical errors of README.md
- Fix breaking changes introduces by upstream nim-serialization
- Remove unreachable code
- Fix cache key of Nim binaries
- Increase stack size of Windows
- Add note to increase stack size on Windows
- add `{.raises.}` annotation to `writeValue`
- add more missing `{.raises.}` annotations
- version 0.2.8
- Add table of contents to README.md
- Add JsonVoid, JsonNumber, and JsonValueRef to JsonWriter
- Add JsonWriter helper procs to README.md
- v0.2.2
- Improve nested structure depth limit check
- Fix JsonValueRef nil comparison
- Reduce declared but not used warning
- Restore accidentally deleted raises pragma
- Switch to mingw-llvm for faster Windows CI
- Increase stack size on Windows when running CI
This PR causes a few new warnings to appear - these are harmless but
will need addressing separately as they span several libraries.
* new asyncraises syntax
* asyncraises support in several modules
* `sink` usage reduces spurious copying
* `?` compatiblity for `async` + `results`
* remove `-d:chronosStrictException` (obsolete)
The `eth2-networks` repo often receives metadata updates with a delay.
Switch to `goerli` repo to obtain the latest config (Dencun scheduling)
when it is updated. This is in line with how Sepolia / Holesky work.
- https://github.com/eth-clients/goerli/pull/178
This requires all object types to be explicitly white-listed for
default serialization. The PR makes the minimal changes, although
a number of similar mechanisms in eth2_rest_serialization can now
be removed.
* Bump nim-eth: Change block timestamp from std.Time to distinct uint64
Also change tx.maxFeePerBlobGas from GasInt to UInt256
following Cancun latest spec
* Fix EthTime.now from func to proc due to sideeffects
- Add IP and location information to mainnet bootnodes
- Change IP and location of a Teku bootstrap node
- Add lighthouse's ipv6 compatible bootnodes
- Adding new Teku mainnet bootnodes
- Remove the ENRs from eth2 networks
* implement EIP-7514 for Deneb: Add Max Epoch Churn Limit
Cap activations per epoch according to EIP-7514:
- https://eips.ethereum.org/EIPS/eip-7514
- https://github.com/ethereum/consensus-specs/pull/3499
* apply proposer boost to first block in case of equivocation
Implement spec changes to fork choice; this only affects equivocation
when multiple blocks are signed for the same slot. Regular operation
is not changed.
- https://github.com/ethereum/consensus-specs/pull/3352
* bump test vectors to v1.4.0-beta.2-hotfix
---------
Co-authored-by: tersec <tersec@users.noreply.github.com>
The 'vendor/holesky/public-keys/all.txt' file is not used by Nimbus,
but sometimes fails to download due to payment issues in `@eth-clients`.
Skipping the download of that file should reduce that risk.
```
Downloading public-keys/all.txt (240 MB)
Error downloading object: public-keys/all.txt (ba977b5): Smudge error: Error downloading public-keys/all.txt (ba977b5b1da180914c934623fce187860e1b54cff939e6208533b2cb5f589e07): batch response: This repository is over its data quota. Account responsible for LFS bandwidth should purchase more data packs to restore access.
```
* bump nbs: Patch libnatpmp.a rule to support llvm-mingw in targets.mk
* bump nbs: Allow passing config to submodule initialization in `update-common`
* Add metadata for the Holesky network
* Add copyright banner to the new Nim module
* Working version
* Bump Chronos to fix downloading from Github
* Add checksum check of the downloaded file
* Clean up debugging code and obsolete imports
Suite names were not being used because `test` has to have access to it
during instantiation - this PR cleans things up a little while at the
same time upgrading unittest2.
tersec
Etan Kissling
Jacek Sieka
Eugene Kabanov
kaiserd
diegomrsantos
andri lim
Kim De Mey
zah
4rgon4ut
Tanguy