Commit Graph

2432 Commits

Author SHA1 Message Date
jangko 77092641b5
add websocket rpc server 2021-08-06 07:32:19 +07:00
jangko 5e87624315
config: copy chainId to networkid if networkid not set in cli
although they are technically different, but in reality,
many networks are using the same id for ChainId dan NetworkId.
in this commit, we set networkid from config file's chainId.
2021-08-06 07:31:02 +07:00
jangko 4459fc3519
add more networkid cli test 2021-08-06 07:31:01 +07:00
jangko 1da4346295
config: fixes bug networkid parser
previously it mistakenly parse into the `result`
now it correctly parse networkId into res.
2021-08-06 07:31:01 +07:00
jangko c69e57e5d4
config: fix config file parser
- allow clique period and epoch to be configured via config file
- this also activate poaEngine mode
- remove clique period configuration from cli to reduce confusion
- fix #786
2021-08-06 07:30:53 +07:00
KonradStaniec 63fa6d0865
Remove duplicated code (#783) 2021-08-05 16:04:29 +02:00
jangko e357a4ea5d
vendor: update websock and json-rpc
websock:
- "Host" header override (#87)
- Use caps for log-levels in tests
- Perform utf-8 validation at message boundaries (#90)

json-rpc:
- fix both ws rpc server and client due to websock breaking changes
2021-08-05 17:30:49 +07:00
Jamie Lokier 20e1831e6f
vm2: Use `ContractSalt` type for `CREATE2` salt
As this branch of vm2 doesn't support EVMC, this EVMC-motivated change is only
required here for internal compatibility.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-08-05 11:07:10 +01:00
Jamie Lokier 11f03a1846
Transaction: EVMC fix, `CREATE2` salt is a 256-bit blob not a number
This changes fixes a bug in `CREATE2` ops when used with EVMC.
Because it changes the salt type, it affects non-EVMC code as well.

The salt was passed through EVMC with the wrong byte order, although this went
unnoticed as the Nimbus host flipped the byte order before using it.

This was found when running Nimbus with third-party EVM,
["evmone"](https://github.com/ethereum/evmone).

There are different ways to remedy this.

If treated as a number, Nimbus EVM would byte-flip the value when calling EVMC,
then Nimbus host would flip the received value.  Finally, it would be flipped a
third time when generating the address in `generateSafeAddress`.  The first two
flips can be eliminated by negotiation (like other numbers), but there would
always be one flip.

As a bit pattern, Nimbus EVM would flip the same way it does when dealing with
hashes on the stack (e.g. with `getBlockHash`).  Nimbus host wouldn't flip at
all - and when using third-party EVMs there would be no flips in Nimbus.

Because this value is not for arithmetic, any bit pattern is valid, and there
shouldn't be any flips when using a third-party EVM, the bit-pattern
interpretation is favoured.  The only flip is done in Nimbus EVM (and might be
eliminated in an optimised version).

As suggested, we'll define a new "opaque 256 bits" type to hold this value.
(Similar to `Hash256`, but the salt isn't necessarily a hash.)

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-08-05 10:35:52 +01:00
Jamie Lokier a1fab0e918
Rename `ZERO_HASH32` to `ZERO_HASH256` to match `Hash256` type
Nimbus types generally use the bit count not the byte count, e.g. `UInt256`,
`Hash256`, so make `ZERO_HASH256` (which has type `Hash256`) fit this pattern.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-08-05 10:21:11 +01:00
jangko 7213d3e67a
Merge branch 'master' of https://github.com/status-im/nimbus-eth1 2021-08-05 14:11:14 +07:00
KonradStaniec 4b1fa050e9
Use websockets in proxy (#779) 2021-08-05 08:14:25 +02:00
Jordan Hrycaj dc8ef09727
fix CI failing 2021-08-05 12:27:14 +07:00
Jordan Hrycaj 4713bd4cf4
#768 Moved/re-implemented ecRecover() from Clique sources to utils/ec_recover
why:
  The same functionality was differently implemented in one or the
  other form.

details:
  Caching and non-caching variants available
2021-08-05 12:27:10 +07:00
Jamie Lokier 9aea669363
Tests: Add loopMul.json and loopExp.json to skipped, slow tests
Add these to the list of specially disabled tests during `make test`,
as they are exceptionally slow.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-08-03 15:03:05 +01:00
Ștefan Talpalaru e43e1cc932
bump nimbus-build-system (#775) 2021-08-03 14:13:05 +02:00
Jordan Hrycaj ec9354d2d0
Jordan/poa voting header (#782)
* Provide PoA voting header generator

why:
  Handy for hive/smoke test

details:
  Header generator is a re-implementation of the generator previously
  used for the canonical reference tests.

* try fixing ci out-of-mem condition

why:
  for some reason, the ci began behaving like a real win7/i386 machine
  where gcc is limited to 64k optimiser space

* fix comments, typos ..
2021-08-03 08:15:32 +01:00
jangko cc89d40a41
hive: use new command line options in nimbus.sh
they are:
- ws
- wsapi
- wsbind
- clique-period
- import-key
- engine-signer
2021-08-02 19:41:39 +07:00
jangko c142119487
add new command line options: ws, wsbind, and wsapi
new command line options:
  --ws                    Enable the Websocket JSON-RPC server
  --wsbind:<value>        Set address:port pair(s) (comma-separated) Websocket JSON-RPC server will bind to (default: localhost:8546)
  --wsapi:<value>         Enable specific set of Websocket RPC API from list (comma-separated) (available: eth, debug)

fixes #770
2021-07-31 19:27:13 +07:00
jangko 6015a4e029
add new command line options: clique-period, engine-signer, and import-key
new command line options:
  --clique-period:<value> Enables clique support. value is block time in seconds
  --engine-signer:<value> Enables mining. value is EthAddress in hex
  --import-key:<path>     Import unencrypted 32 bytes hex private key file

fixes #771
2021-07-31 19:16:30 +07:00
Kim De Mey e2b50d2339
Add routing table and required lookup code to Portal protocol (#773)
* Add routing table and required lookup code to Portal protocol

* Enable Portal lookup and validate loops + minor adjustments
2021-07-30 21:19:03 +02:00
Jordan Hrycaj ca07c40a48
Fearture/poa clique tuning (#765)
* Provide API

details:
  API is bundled via clique.nim.

* Set extraValidation as default for PoA chains

why:
  This triggers consensus verification and an update of the list
  of authorised signers. These signers are integral part of the
  PoA block chain.

todo:
  Option argument to control validation for the nimbus binary.

* Fix snapshot state block number

why:
  Using sub-sequence here, so the len() function was wrong.

* Optional start where block verification begins

why:
  Can speed up time building loading initial parts of block chain. For
  PoA, this allows to prove & test that authorised signers can be
  (correctly) calculated starting at any point on the block chain.

todo:
  On Goerli around blocks #193537..#197568, processing time increases
  disproportionally -- needs to be understand

* For Clique test, get old grouping back (7 transactions per log entry)

why:
  Forgot to change back after troubleshooting

* Fix field/function/module-name misunderstanding

why:
  Make compilation work

* Use eth_types.blockHash() rather than utils.hash() in Clique modules

why:
  Prefer lib module

* Dissolve snapshot_misc.nim

details:
  .. into clique_verify.nim (the other source file clique_unused.nim
  is inactive)

* Hide unused AsyncLock in Clique descriptor

details:
  Unused here but was part of the Go reference implementation

* Remove fakeDiff flag from Clique descriptor

details:
  This flag was a kludge in the Go reference implementation used for the
  canonical tests. The tests have been adapted so there is no need for
  the fakeDiff flag and its implementation.

* Not observing minimum distance from epoch sync point

why:
  For compiling PoA state, the go implementation will walk back to the
  epoch header with at least 90000 blocks apart from the current header
  in the absence of other synchronisation points.

  Here just the nearest epoch header is used. The assumption is that all
  the checkpoints before have been vetted already regardless of the
  current branch.

details:
  The behaviour of using the nearest vs the minimum distance epoch is
  controlled by a flag and can be changed at run time.

* Analysing processing time (patch adds some debugging/visualisation support)

why:
  At the first half million blocks of the Goerli replay, blocks on the
  interval #194854..#196224 take exceptionally long to process, but not
  due to PoA processing.

details:
  It turns out that much time is spent in p2p/excecutor.processBlock()
  where the elapsed transaction execution time is significantly greater
  for many of these blocks.

  Between the 1371 blocks #194854..#196224 there are 223 blocks with more
  than 1/2 seconds execution time whereas there are only 4 such blocks
  before and 13 such after this range up to #504192.

* fix debugging symbol in clique_desc (causes CI failing)

* Fixing canonical reference tests

why:
  Two errors were introduced earlier but ovelooked:
   1. "Remove fakeDiff flag .." patch was incomplete
   2. "Not observing minimum distance .." introduced problem w/tests 23/24

details:
  Fixing 2. needed to revert the behaviour by setting the
  applySnapsMinBacklog flag for the Clique descriptor. Also a new
  test was added to lock the new behaviour.

* Remove cruft

why:
  Clique/PoA processing was intended to take place somewhere in
  executor/process_block.processBlock() but was decided later to run
  from chain/persist_block.persistBlock() instead.

* Update API comment

* ditto
2021-07-30 15:06:51 +01:00
Kim De Mey e9bfcedcaf
Bump nimbus-build-system and nim-eth vendor modules (#774) 2021-07-30 13:11:25 +02:00
Jamie Lokier 83b15c83d2
Sync: Add messages about `eth/65` handshake parameters
Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-07-27 14:12:59 +01:00
Jamie Lokier 57de56bab6
Sync: Add packet tracing to `blockchain_sync` network calls
Using the same packet tracing format to match `protocol_eth65`.
There aren't many calls, and this makes them clear.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-07-27 14:12:57 +01:00
Jamie Lokier b28396d10d
Sync: Add packet tracing to `eth/65` in a consistent format
The format is reasonably useful and not too large, when looking at the
behaviour of sync processes.  It doesn't try to show all the details of
packets, just something at a useful level of detail to see what's going on.
The consistent presentation has proven helpful too, e.g. when grepping.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-07-27 14:12:56 +01:00
Jamie Lokier ab9067133c
Tracing: Remove some trace messages that occur a lot during sync
Disable some trace messages which appeared a lot in the output and probably
aren't so useful any more, when block processing is functioning well at high
speed.

Turning on the trace level globally is useful to get a feel for what's
happening, but only if each category is kept to a reasonable amount.

As well as overwhelming the output so that it's hard to see general activity,
some of these messages happen so much they severely slow down processing.  Ones
called every time an EVM opcode uses some gas are particularly extreme.

These messages have all been chosen as things which are probably not useful any
more (the relevant functionality has been debugged and is tested plenty).

These have been commented out rather than removed.  It may be that turning
trace topics on/off, or other selection, is a better longer term solution, but
that will require better command line options and good defaults for sure.
(I think higher levels `tracev` and `tracevv` levels (extra verbose) would be
more useful for this sort of deep tracing on request.)

For now, enabling `--log-level:TRACE` on the command line is quite useful as
long as we keep each category reasonable, and this patch tries to keep that
balance.

- Don't show "has transactions" on virtually every block imported.
- Don't show "Sender" and "txHash" lines on every transaction processed.
- Don't show "GAS CONSUMPTION" on every opcode executed", this is way too much.
- Don't show "GAS RETURNED" and "GAS REFUND" on each contract call.
- Don't show "op: Stop" on every Stop opcode, which means every transaction.
- Don't show "Insufficient funds" whenever a contract can't call another.
- Don't show "ECRecover", "SHA256 precompile", "RIPEMD160", "Identity"
  or even "Call precompile" every time a precompile is called.  These are
  very well tested now.
- Don't show "executeOpcodes error" whenever a contract returns an error.
  (This is changed to `trace` too, it's a normal event that is well tested.)

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-07-27 14:12:55 +01:00
Jamie Lokier c435409292
Sync: Move `blockchain_sync` code and use it with `eth/65`
Move `blockchain_sync.nim` from `nim-eth` to `nimbus-eth1`.

This lets `blockchain_sync` use the `eth/65` protocol to synchronise with more
modern peers than before.

Practically, the effect is the sync process runs more quickly and reliably than
before.  It finds usable peers, and they are up to date.

Note, this is mostly old code, and it mostly performs "classic sync", the
original Ethereum method.  Here's a summary of this code:

- It decides on a blockchain canonical head by sampling a few peers.
- Starting from block 0 (genesis), it downloads each block header and
  block, mostly in order.
- After it downloads each block, it executes the EVM transactions in that block
  and updates state trie from that, before going to the next block.
- This way the database state is updated by EVM executions in block order,
  and new state is persisted to the trie database after each block.

Even though it mentions Geth "fast sync" (comments near end of file), and has
some elements, it isn't really.  The most obvious missing part is this code
_doesn't download a state trie_, it calculates all state from block 0.
Geth "fast sync" has several parts:

1. Find an agreed common chain among several peers to treat as probably secure,
   and a sufficiently long suffix to provide "statistical economic consensus"
   when it is validated.
2. Perform a subset of PoW calculations, skipping forward over a segment to
   verify some of the PoWs according to a pattern in the relevant paper.
3. Download the state trie from the block at the start of that last segment.
4. Execute only the blocks/transactions in that last segment, using the
   downloaded state trie, to fill out the later states and properly validate the
   blocks in the last segment.

Some other issues with `blockchain_sync` code:

- If it ever reaches the head of the chain, it doesn't follow new blocks with
  increasing block numbers, at least not rapidly.
- If the chain undergoes a reorg, this code won't fetch a block number it has
  already fetched, so it can't accept the reorg.  It will end up conflicted
  with peers. This hasn't mattered because the development focus has been on
  the bulk of the catching up process, not the real-time head and reorgs.
- So it probably doesn't work correctly when it gets close to the head due to
  many small reorgs, though it might for subtle reasons.
- Some of the network message handling isn't sufficiently robust, and it
  discards some replies that have valid data according to specification.
- On rare occasions the initial query mapping block hash to number can
  fail (because the peer's state changes).
- It makes some assumptions about the state of peers based on their responses
  which may not be valid (I'm not convinced they are).  The method for working
  out "trusted" peers that agree a common chain prefix is clever.  It compares
  peers by asking each peer if it has the header matching another peer's
  canonical head block by hash.  But it's not clear that merely knowing about a
  block constitutes agreement about the canonical chain.  (If it did, query by
  block number would give the same answer more authoritatively.)

Nonetheless, being able to run this sync process on `eth/65` is useful.

<# interactive rebase in progress; onto 66532e8a

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-07-27 14:12:53 +01:00
Jamie Lokier 936a18b4f4
Eth65: Rename and stop exporting `protocolVersion`
This constant shouldn't be used outside `protocol_eth65`.

When we support multiple `eth/NN` versions side by side, or even just have
multiple code files, there's a risk some code would import just one of the
files (e.g. `protocol_eth65`), use `protocolVersion`, and incorrectly act as
though that version is the one active on the node.

In fact that happened, and now it can't happen.  Other code needs to query the
`EtheruemNode` to find what versions are really active.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-07-27 14:12:52 +01:00
Jamie Lokier 5ddfd7dd05
RPC: Calculate `eth_protocolVersion` from live protocol
In RPC `eth_protocolVersion`, look at the live `EthereumNode` to find which
version of `eth/NN` protocol is active, instead of trusting a compile-time
constant.  It's better to check dynamically.  GraphQL already does this.

As a result, the RPC code doesn't depend on `eth_protocol` any more.

To make sure there are no more accidental users of the old constant,
`protocolVersion` is no longer exported from `protocol_eth65`.

(The simplest way to support `eth/65` was to make `eth_protocolVersion` use
`protocol_eth65.protocolVersion`, to get 65.  But that's silly.  More
seriously, when we add another version (`eth/66`) running alongside `eth/65`,
that expression would still compile ok yet return the wrong value, while still
passing the RPC test suite.)

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-07-27 14:12:51 +01:00
Jamie Lokier bb282a5348
RPC: Add comment about `eth_protocolVersion` expectations
While looking at the RPC `eth_protocolVersion` to see exactly what it should
report when there are multiple `eth/NN` versions supported by a node, I found
some differences in the documentation:

- The old Ethereum wiki documents it as returning a decimal string.

- But Infura documents it as returning 0x-prefixed hex string.

- Geth 1.10.0 has removed this call entirely "as it makes no sense".

https://eth.wiki/json-rpc/API#eth_protocolversion
https://infura.io/docs/ethereum/json-rpc/eth-protocolVersion
https://blog.ethereum.org/2021/03/03/geth-v1-10-0/#compatibility
Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-07-27 14:12:49 +01:00
Jamie Lokier 22241f3cff
GraphQL: Update `protocolVersion` expected result to hard-coded 65
Use a hard-coded number instead of the same expression as the client, so that
bugs introduced via that expression are detected.  Using the same expression as
the client can hide issues when the value is wrong in both places.

When the expected value genuinely changes, it'll be obvious.
Just change this number.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-07-27 14:12:48 +01:00
Jamie Lokier b3cf42ed28
RPC: Update `eth_protocolVersion` expected result to hard-coded 65
Use a hard-coded number instead of the same expression as the client, so that
bugs introduced via that expression are detected.  Using the same expression as
the client can hide issues when the value is wrong in both places.

When the expected value genuinely changes, it'll be obvious.
Just change this number.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-07-27 14:12:46 +01:00
Jamie Lokier 00647b373c
Sync: Switch other modules over to the `eth/65` module
Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-07-27 14:12:41 +01:00
Jamie Lokier 889a796b72
Sync: Fix `NewBlockAnnounce` RLP decoding errors
Turns out `{.rlpInline.}` doesn't do anything.
It's documented but not implemented.

Due to this, whenever a peer sent us a `NewBlock` message, we had an RLP
decoding error processing it, and disconnected the peer thinking it was the
peer's error.

These messages are sent often by good peers, so whenever we connected to a
really good peer, we'd end up disconnecting from it due to this.

Because a block body is a list of transactions, the parse errors looked
suspiciously like EIP-2718/2976/2930/1559 typed transaction RLP errors.
But it was a failure to parse `BlockBody` inline.

Conveniently, the `EthBlock` type defined for another reason is encoded exactly
the way `NewBlockAnnounce` needs to be, so we can reuse that type.

This didn't stand out before updating to `eth/65`, because with old protocols
we tend to only connect to old peers, which may be out of date themselves and
have no new blocks to send.  Also, we didn't really investigate occasional
disconnects before, we assumed they're just part of P2P life.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-07-27 14:12:38 +01:00
Jamie Lokier 3161d395a6
Sync: Support for `eth/65` protocol
This patch adds the `eth/65` protocol, documented at
https://github.com/ethereum/devp2p/blob/master/caps/eth.md.

This is an intentionally simple patch, designed to not break, change or depend
on other functionality much, so that the "_old_ sync" methods can be run
usefully again and observed.  This patch isn't "new sync" (a different set of
sync algorithms), but it is one of the foundations.

For a while now Nimbus Eth1 only supported protocol `eth/63`.  But that's
obsolete, and very few nodes still support it.  This meant Nimbus Eth1 would
make slow progress trying to sync, as most up to date clients rejected it.

The current specification is `eth/66`, and the functionality we really need is
in `eth/64`.

So why `eth/65`?

- `eth/64` is essential because of the `forkId` feature.  But `eth/64` is on
  its way out as well.  Many clients, especially the most up to date Geth
  running the current hard-forks (Berlin/London) don't talk `eth/64` any more.

- `eth/66` is the current specification, but some clients don't talk `eth/66`
  yet.  We'd like to have the best peer connectivity during tests, and
  currently everything that talks `eth/66` also talks `eth/65`.

- Nimbus Eth1 RLPx only talks one version at a time.  (Without changes to the
  RLPx module.  When those go in we'll add `eth/64..eth/66` for greater peer
  reach and testing the `eth/66` behaviour.  For simplicity and decoupling,
  this patch contains just one version, the most useful.)

What are `eth/64` and `eth/65`?

- `eth/64` (EIP-2364) added `forkId` which allows nodes to distinguish between
  Ethereum (ETH) and Ethereum Classic (ETC) blockchains, which share the same
  genesis block.  `forkId` also protects the system when a new hard fork is
  going to be rolled out, by blocking interaction with out of date nodes.  The
  feature is required nowadays.

  We send the right details to allow connection (this has been tested a lot),
  but don't apply the full validation rules of EIP-2124/EIP-2364 in this patch.
  It's to keep this patch simple (in its effects) and because those rules have
  consequences best tested separately.  In practice the other node will reject
  us when we would reject it, so this is ok for testing, as long as it doesn't
  get seriously deployed.

- `eth/65` added more efficient transaction pool methods.

- Then a later version of `eth/65` (without a new number) added typed
  transactions, described in [EIP-2976](https://eips.ethereum.org/EIPS/eip-2976).

Why it's moved to `nimbus-eth1`:

- Mainly because `eth/64` onwards depend on the current state of block
  synchronisation, as well as the blockchain's sequence of hard-fork block
  numbers, both of which are part of `nimbus-eth1` run-time state.  These
  aren't available to pure `nim-eth` code.  Although it would be possible to
  add an API to let `nimbus-eth1` set these numbers, there isn't any point
  because the protocol would still only be useful to `nimbus-eth1`.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-07-27 14:12:35 +01:00
Jordan Hrycaj a0d0e35a70
Renamed source file clique_utils => clique_helpers (#762)
* Renamed source file clique_utils => clique_helpers

why:
  New name is more in line with other modules where local libraries
  are named similarly.

* re-implemented PoA verification module as clique_verify.nim

details:
  The verification code was ported from the go sources and provisionally
  stored in the clique_misc.nim source file.

todo:
  Bring it to life.

* re-design Snapshot descriptor as: ref object

why:
  Avoids some copying descriptor objects

details:
  The snapshot management in clique_snapshot.nim has been cleaned up.

todo:
  There is a lot of unnecessary copying & sub-list manipulation of
  seq[BlockHeader] lists which needs to be simplified by managing
  index intervals.

* optimised sequence handling for Clique/PoA

why:
  To much ado about nothing

details:
  * Working with shallow sequences inside PoA processing avoids
    unnecessary copying.
  * Using degenerate lists in the cliqueVerify() batch where only the
    parent (and no other ancestor) is needed.

todo:
  Expose only functions that are needed, shallow sequences should be
  handles with care.

* fix var-parameter function argument

* Activate PoA engine -- currently proof of concept

details:
  PoA engine is activated with newChain(extraValidation = true) applied
  to a PoA network.

status and todo:
  The extraValidation flag on the Chain object can be set at a later
  state which allows to pre-load parts of the block chain without
  verification. Setting it later will only go back the block chain to
  the latest epoch checkpoint. This is inherent to the Clique protocol,
  needs testing though.

  PoA engine works in fine weather mode on Goerli replay. With the
  canonical eip-225 tests, there are quite a few fringe conditions
  that fail. These can easily fudged over to make things work but need
  some more work to understand and correct properly.

* Make the last offending verification header available

why:
  Makes some fringe case tests work.

details:
  Within a failed transaction comprising several blocks, this
  feature help to identify the offending block if there was a
  PoA verification error.

* Make PoA header verifier store the final snapshot

why:
  The last snapshot needed by the verifier is the one of the parent but
  the list of authorised signer is derived from the current snapshot. So
  updating to the latest snapshot provides the latest signers list.

details:
  Also, PoA processing has been implemented as transaction in
  persistBlocks() with Clique state rollback.

  Clique tests succeed now.

* Avoiding double yields in iterator => replaced by template

why:
  Tanks to Andri who observed it (see #762)

* Calibrate logging interval and fix logging event detection

why:
  Logging interval as copied from Go implementation was too large and
  needed re-calibration. Elapsed time calculation was bonkers, negative
  the wrong way round.
2021-07-21 14:31:52 +01:00
Kim De Mey 66532e8ad5
Use ssz vectors for the hashes and add enrs in the tests (#763) 2021-07-20 14:04:57 +02:00
Kim De Mey e37bafd47e
Add a first, simple, content network test (#760)
* Add a first, simple, content network test

* Use sqlite as nimbus_db_backend for fluffy tests

* Fix backend selection for sqlite
2021-07-15 15:12:33 +02:00
Jordan Hrycaj cfe955c962
Feature/implement poa processing (#748)
* re-shuffled Clique functions

why:
  Due to the port from the go-sources, the interface logic is not optimal
  for nimbus. The main visible function is currently snapshot() and most
  of the _procurement_ of this function result has been moved to a
  sub-directory.

* run eip-225 Clique test against p2p/chain.persistBlocks()

why:
  Previously, loading the test block chains was fugdged with the purpose
  only to fill the database. As it is now clear how nimbus works on
  Goerli, the same can be achieved with a more realistic scenario.

details:
  Eventually these tests will be pre-cursor to the reply tests for the
  Goerli chain supporting TDD approach with more simple cases.

* fix exception annotations for executor module

why:
  needed for exception tracking

details:
  main annoyance are vmState methods (in state.nim) which potentially
  throw a base level Exception (a proc would only throws CatchableError)

* split p2p/chain into sub-modules and fix exception annotations

why:
  make space for implementing PoA stuff

* provide over-loadable Clique PRNG

why:
  There is a PRNG provided for generating reproducible number sequences.
  The functions which employ the PRNG to generate time slots were ported
  ported from the go-implementation. They are currently unused.

* implement trusted signer assembly in p2p/chain.persistBlocks()

details:
  * PoA processing moved there at the end of a transaction. Currently,
   there is no action (eg. transaction rollback) if this fails.
  * The unit tests with staged blocks work ok. In particular, there should
    be tests with to-be-rejected blocks.
  * TODO: 1.Optimise throughput/cache handling; 2.Verify headers

* fix statement cast in pool.nim

* added table features to LRU cache

why:
  Clique uses the LRU cache using a mixture of volatile online items
  from the LRU cache and database checkpoints for hard synchronisation.
  For performance, Clique needs more table like features.

details:
  First, last, and query key added, as well as efficient random delete
  added. Also key-item pair iterator added for debugging.

* re-factored LRU snapshot caching

why:
  Caching was sub-optimal (aka. bonkers) in that it skipped over memory
  caches in many cases and so mostly rebuild the snapshot from the
  last on-disk checkpoint.

details;
  The LRU snapshot toValue() handler has been moved into the module
  clique_snapshot. This is for the fact that toValue() is not supposed
  to see the whole LRU cache database. So there must be a higher layer
  working with the the whole LRU cache and the on-disk checkpoint
  database.

also:
  some clean up

todo:
  The code still assumes that the block headers are valid in itself. This
  is particular important when an epoch header (aka re-sync header) is
  processed as it must contain the PoA result of all previous headers.

  So blocks need to be verified when they come in before used for PoA
  processing.

* fix some snapshot cache fringe cases

why:
  Must not index empty sequences in clique_snapshot module
2021-07-14 16:13:27 +01:00
KonradStaniec 0fb0fc4680
[Minor] use proper constructor for proxy in fluffy (#759) 2021-07-13 15:49:53 +02:00
Kim De Mey 25522b2c24
Further implement Portal Nodes and FoundContent messages (#757) 2021-07-13 15:15:33 +02:00
KonradStaniec bd862cd649
[BUMP] bump json rpc module (#758) 2021-07-13 13:34:52 +02:00
Kim De Mey 117f744f52
Bump nim-eth to test after removing portal code (#755) 2021-07-09 23:55:02 +02:00
Kim De Mey c7569c2247
Add/update ssz calls for fluffy (#750) 2021-07-09 16:15:10 +02:00
Kim De Mey 424fd405e9
Move Portal wire protocol from nim-eth to fluffy (#749)
* Move Portal wire protocol from nim-eth to fluffy

* Rename fluffy make targets

* Move all_fluffy_tests a folder down
2021-07-09 13:34:16 +02:00
KonradStaniec 1c5c2cba8b
[FEATURE] Use rpc proxy (#747) 2021-07-07 14:13:27 +02:00
KonradStaniec a59a2c61b6
[FEATURE] Add support for handling experimental api call (#746)
* [FEATURE] Add support for handling experimental api call
2021-07-07 11:04:18 +02:00
Jordan Hrycaj fbff3aea68
Feature/goerli replay clique poa (#743)
* extract unused clique/mining support into separate file

why:
  mining is currently unsupported by nimbus

* Replay first 51840 transactions from Goerli block chain

why:
  Currently Goerli is loaded but the block headers are not verified.
  Replaying allows real data PoA development.

details:
  Simple stupid gzipped dump/undump layer for debugging based on
  the zlib module (no nim-faststream support.)

  This is a replay running against p2p/chain.persistBlocks() where
  the data were captured from.

* prepare stubs for PoA engine

* split executor source into sup-modules

why:
  make room for updates, clique integration should go into
  executor/update_poastate.nim

* Simplify p2p/executor.processBlock() function prototype

why:
  vmState argument always wraps basicChainDB

* split processBlock() into sub-functions

why:
  isolate the part where it will support clique/poa

* provided additional processTransaction() function prototype without _fork_ argument

why:
  with the exception of some tests, the _fork_ argument is always derived
  from the other prototype argument _vmState_

details:
  similar situation with makeReceipt()

* provide new processBlock() version explicitly supporting PoA

details:
  The new processBlock() version supporting PoA is the general one also
  supporting non-PoA networks, it needs an additional _Clique_ descriptor
  function argument for PoA state (if any.)
  The old processBlock() function without the _Clique_ descriptor argument
  retorns an error on PoA networgs (e.g. Goerli.)

* re-implemented Clique descriptor as _ref object_

why:
  gives more flexibility when moving around the descriptor object

details:
  also cleaned up a bit the clique sources

* comments for clarifying handling of Clique/PoA state descriptor
2021-07-06 14:14:45 +01:00
KonradStaniec aaab0d7bfd
[BUMP] bump json rpc (#745) 2021-07-02 16:45:53 +02:00