status-im/nim-ssz-serialization#35 brings in https://github.com/prysmaticlabs/hashtree as a supported backend for SHA256, giving a nice little performance boost to all hash_tree_root calls on supported platforms / compilers.
Expected gains are on the order of 30% which in the case of a replayed state nets us 0.2-0.3s improvement.
More about this design here: https://hackmd.io/@potuz/BJyrx9DOF - kudos to @potuz for this excellent library!
* Make listen-address default to use dualstack.
* Use correct newProtocol().
* Bump nim-eth.
* Bump nim-eth one more time.
* Use `*` instead of IPv6 address for dualstack sockets.
* Bump chronos and nim-eth.
* Use new constructor.
* Fix listenAddress should be Opt[T] not Option[T].
* Fix options.md.
* chore: change nim-libp2p branch from unstable to master
* chore: dummy bump of nim-libp2p version to please linter
---------
Co-authored-by: ksr <kaiserd@users.noreply.github.com>
- fix: reset accept fut in stop
- always allow new data be received if the recvWindow is > 0
- Revert "always allow new data be received if the recvWindow is > 0"
- Add custom ring buffer into chronos streams and transports.
- Add `join()` operation to wait for future completion.
- Ensure that all buffers used inside HTTP client will follow original buffer size.
- Add automatic constructors for TCP and UDP transports.
- add support for setting protocol handlers with `{.raises.}` annotation
- fix: valueOr and withValue utilities
- fix: remove explicit param from GossipSubParams constructor
- fix miniupnpc build on Windows by not escaping PATH
- allow overriding CC for libimiupnpc.a and libnatpmp.a targets
- Use -fPIC when building nat traversal
- default `MultiAddress` param for `newStandardSwitch` does not raise
- clean up triple lookup and avoid `KeyError` when adding muxer
- `{.async: (raises).}` for `relay/utils.nim`
- `{.async: (raises).}` annotations for `protocols/secure`
- avoid pointless exception raising in `dcutr/server`
- fix: allFuturesThrowing compilation issue on daily
- feat: Yamux timeout
- Fix empty path crash issue for MultiAddresses `unix`, `ip6zone`, `dns***`.
- fix: move transport interop tests to nim-libp2p repo
- use chronos 4.0.0
- fix typo in `ProtoMessage.toString()`
- increase tolerance of `simple heartbeat` test
- remove unused `MultiBase.encode(..., Cid)` function
- move `allFutureThrowing` helper to tests
- use new Chronos `trackCounter` APIs for leaks checks in tests
- remove unused `LPStreamError` types
- change `SecioError` and `NoiseError` to descendants of `LPStreamError`
- do not log yamux buffers without sanitization (trace log level)
- don't forget closing the stream when final `{Fin}` fails in yamux
- use `race` instead of `or` to avoid lockup
- in yamux, do not write `{Rst}` packet to stream that's in use
- avoid `KeyError` in edge case of yamux handler
- avoid triple lookup in `m.flushed` yamux table
- catch `WebSocketError` in `wstransport`
- define proper parent error type for `YamuxError`
- document known `--mm:orc` crash
- add `{.async: (raises).}` to `libp2p/stream` modules
Fix the `/eth/v1/beacon/deposit_snapshot` API to produce proper EIP-4881
compatible `DepositTreeSnapshot` responses. The endpoint used to expose
a Nimbus-specific database internal format.
Also fix trusted node sync to consume properly formatted EIP-4881 data
with `--with-deposit-snapshot`, and `--finalized-deposit-tree-snapshot`
beacon node launch option to use the EIP-4881 data. Further ensure that
`ncli_testnet` produces EIP-4881 formatted data for interoperability.
- fix conversion error with `or` on futures with `{.async: (raises: []).}`
- Fix `or` deadlock issue.
- Ensure that `OwnCancelSchedule` flag will not be removed from `wait()` and `withTimeout()`.
- Add missing parts of defaults buffer size increase.
- Avoid `ValueError` effect in varargs `race`/`one`
- fix circular reference in timer
- Ensure `transp.reader` is reset to `nil` on error
This reverts commit 21601f59e2.
Reason: `/eth/v1/beacon/light_client/updates?start_period=0&count=128`
provides malformed data. Reverting fixes it locally.
Download JSON and SSZ, then use this script to validate (Holesky).
Need to export some symbols for parsing.
```nim
import
stew/io2,
./beacon_chain/spec/[digest, forks, helpers, presets],
./beacon_chain/spec/eth2_apis/[eth2_rest_serialization, rest_light_client_calls]
let jsonBytes = io2.readAllBytes("/Users/etan/Downloads/json_updates.json").get()
let
(cfg, unknowns) = readRuntimeConfig("/Users/etan/Documents/Repos/nimbus-eth2/vendor/holesky/custom_config_data/config.yaml")
genesis_validators_root = Eth2Digest.fromHex("0x9143aa7c615a7f7115e2b6aac319c03529df8242ae705fba9df39b79c59fa8b1")
forkDigests = newClone ForkDigests.init(cfg, genesis_validators_root)
let updates = seq[ForkedLightClientUpdate].decodeJsonLightClientObjects(jsonBytes, cfg, forkDigests)
var entries = newSeqOfCap[RestVersioned[ForkedLightClientUpdate]](updates.len)
for update in updates:
let
contextEpoch = withForkyUpdate(update):
when lcDataFork > LightClientDataFork.None:
forkyUpdate.contextEpoch
else:
continue
contextFork = cfg.consensusForkAtEpoch(contextEpoch)
entries.add RestVersioned[ForkedLightClientUpdate](
data: update,
jsonVersion: contextFork,
sszContext: forkDigests[].atconsensusFork(contextFork))
let res =
block:
var default: seq[byte]
try:
var stream = memoryOutput()
for e in entries:
withForkyUpdate(e.data):
when lcDataFork > LightClientDataFork.None:
var cursor = stream.delayFixedSizeWrite(sizeof(uint64))
let initPos = stream.pos
stream.write e.sszContext.data
var writer = SszWriter.init(stream)
writer.writeValue forkyUpdate
cursor.finalWrite (stream.pos - initPos).uint64.toBytesLE()
stream.getOutput(seq[byte])
except SerializationError:
default
except IOError:
default
let sszBytes = io2.readAllBytes("/Users/etan/Downloads/ssz_updates.ssz").get()
let sszUpdates = seq[ForkedLightClientUpdate].decodeSszLightClientObjects(res, cfg, forkDigests)
doAssert updates.len == sszUpdates.len
for i in 0 ..< updates.len:
doAssert updates[i].kind == sszUpdates[i].kind
withForkyUpdate(updates[i]):
when lcDataFork > LightClientDataFork.None:
doAssert forkyUpdate == sszUpdates[i].forky(lcDataFork)
doAssert sszBytes == res
```
- Replace eth_getBlockReceipts return type with Opt[T] instead of Option[T]
- use non-EOL macOS version and fix spelling
- test refc in CI in Nim 2.0 and later
- unittest2.nim: ensure the testTeardownIMPL is performed at the end
- Add test for PR #35
- only run `teardown` if `setup` completed
- Fix allow break in test body regression
- Upgrade github actions to v4
- use non-EOL macOS version for GitHub Actions CI
This PR allows sharing the pubkey data between validators by using a
thread-local cache for pubkey data, netting about a 400mb mem usage
reduction on holesky due to us keeping 3 permanent + several ephemeral
state copies in memory at all times and each state copy holding a full
validator.
The PR also introduces a hash cache for the key which gives ~14% speedup
for a full state `hash_tree_root` - the key makes up for a large part of
the `Validator` htr time.
Finally, the time it takes to copy a state goes down as well from ~80m
ms to ~60, for reasons similar to htr.
We use a `ptr` even if a `ref` could in theory have been used - there is
not much practical benefit to a `ref` (given it's mutable) while a `ptr`
is cheaper and easier to copy (when copying temporary states).
We could go further and cache a cooked pubkey but it turns out this is
quite intrusive - in all the relevant places, we're already using a
cooked key from the immutable validator data so there are no immediate
performance gains of doing so while managing the compressed -> cooked
key mapping would become more difficult - something for a future PR
perhaps.
Co-authored-by: Etan Kissling <etan@status.im>
- Refactor p2pProtocol internals
- Fix chronos related deprecated warnings in discv5 and uTP
- normalise nimble, ci
- Fix chronos related deprecated warnings in uTP code part II
- Remove usage of stew/shim/net
- Upgrade github actions to v4
- Fix two bugs in Receipts RLP encoding/decoding
- Move Ethereum specific RLP encoding tests under tests/common
- Reduce compiler warnings in rlp
- test refc in CI in Nim 2.0 and later
- test Nim 2.0 in CI and use non-EOL macOS version
- Use asyncraises in p2p
- Don't auto write p2pProtocol macro expansion to file
- Fix improper yield usage in rlpx and refine exception handling
- Client also handle error message if id is null
- Client pass meaningful error to newFut when processMessage failed
- Refactoring: extract rpc handler from HTTP and WebSocket server
- Use pragma push/pop pair to disable warning
- http server better exception handling
- Fix CI badge url
- Upgrade github actions to v4
- Revert "Fix CI badge url"
- HttpAuthHook use async raises
- Move CancelledError handling to outer try/except of RpcWebsocketServer
- Implement RPC batch call both in servers and clients
- v0.4.0
- Should compile if chronicles log turned on
- Add framework to support more optional types
- v0.4.2
- test refc in CI in Nim 2.0 and later
- use non-EOL macOS version for GitHub Actions CI
- avoid failing uninitialized `Future`
- Improve batch call example and wrapper comments
- Fix ws and socket client error handling and add test to #212
- Add build test with chronicles to json enabled
- update instructions for tracking upstream MIRACL Core
- bump `bls12-381-tests` to `v0.1.2`
- bump `miracl-core` to `0f67878bee7c4108405deb2b0b5e4e58d1ae30fc`
- test refc in CI in Nim 2.0 and later
- rename `milagro.nims` -> `miracl.nims`
- rename `milagro.nim` -> `miracl.nim`
- rename `milagro(Path|_func)` -> `miracl(Path|_func)`
- rename `milagro` references -> `miracl` in documentation
- Synchronize order of `blobGasUsed` / `excessBlobGas` in test
- Reduce compiler warnings
- Upgrade github actions to v4
- v0.3.2
- re-add support for `fromBlock`, `toBlock`, `blockHash` in `eth_getLogs`
- Feature flag for raises support
- HTTP server middleware implementation
- Fix examples documentation
- check leaks after every test
- deprecate `callback=`, UDP fixes
- Improve parseList and parseInlineTable strictness
- v0.2.10
- Switch to llvm-mingw for faster Windows CI
- Add configureTomlDeserialization to README.md
- Put array/inline table nonstandard behavior behind flag
- Unify parseList and parseArray implementation
- v0.2.12
- Fix a bunch of compiler hints and warnings in uTP and discv5
- Fix missing std/times import for the metrics 0.0.1 case
- Fix for uTP issues with latest chronos
- Clean-up, correct and clarify utp_protocol tests
- better async timeout wait
- Adjust test names and comments for `blobGasUsed` field
- Add data over multiple sockets uTP test
- Add uTP over discv5 test and small uTP performance improvements
tersec
Etan Kissling
Jacek Sieka
Eugene Kabanov
kaiserd
diegomrsantos
andri lim
Kim De Mey