How to write tests for the consensus layer (#2700)
* Create README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Signed my name * Update tests/README.md Co-authored-by: Danny Ryan <dannyjryan@gmail.com> * Update tests/README.md Co-authored-by: Danny Ryan <dannyjryan@gmail.com> * Update tests/README.md Co-authored-by: Danny Ryan <dannyjryan@gmail.com> * Update tests/README.md Co-authored-by: Danny Ryan <dannyjryan@gmail.com> * Update tests/README.md Co-authored-by: Danny Ryan <dannyjryan@gmail.com> * Update tests/README.md Co-authored-by: Danny Ryan <dannyjryan@gmail.com> * Update README.md * Update README.md Co-authored-by: Danny Ryan <dannyjryan@gmail.com>
This commit is contained in:
Ori Pomerantz
GitHub
parent
bbdb0d8fba
commit
9999331f81
|
|
@ -0,0 +1,473 @@
|
|||
# Getting Started with Consensus Spec Tests
|
||||
|
||||
## Getting Started
|
||||
|
||||
### Creating the environment
|
||||
|
||||
Use an OS that has Python 3.8 or above. For example, Debian 11 (bullseye)
|
||||
|
||||
1. Install the packages you need:
|
||||
```sh
|
||||
sudo apt install -y make git wget python3-venv gcc python3-dev
|
||||
```
|
||||
1. Download the latest [consensus specs](https://github.com/ethereum/consensus-specs)
|
||||
```sh
|
||||
git clone https://github.com/ethereum/consensus-specs.git
|
||||
cd consensus-specs
|
||||
```
|
||||
1. Create the specifications and tests:
|
||||
```sh
|
||||
make install_test
|
||||
make pyspec
|
||||
```
|
||||
|
||||
To read more about creating the environment, [see here](core/pyspec/README.md).
|
||||
|
||||
### Running your first test
|
||||
|
||||
|
||||
1. Enter the virtual Python environment:
|
||||
```sh
|
||||
cd ~/consensus-specs
|
||||
. venv/bin/activate
|
||||
```
|
||||
1. Run a sanity check test:
|
||||
```sh
|
||||
cd tests/core/pyspec
|
||||
python -m pytest -k test_empty_block_transition --fork Merge eth2spec
|
||||
```
|
||||
1. The output should be similar to:
|
||||
```
|
||||
============================= test session starts ==============================
|
||||
platform linux -- Python 3.9.2, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
|
||||
rootdir: /home/qbzzt1/consensus-specs
|
||||
plugins: cov-2.12.1, forked-1.3.0, xdist-2.3.0
|
||||
collected 629 items / 626 deselected / 3 selected
|
||||
|
||||
eth2spec/test/merge/sanity/test_blocks.py . [ 33%]
|
||||
eth2spec/test/phase0/sanity/test_blocks.py .. [100%]
|
||||
|
||||
=============================== warnings summary ===============================
|
||||
../../../venv/lib/python3.9/site-packages/cytoolz/compatibility.py:2
|
||||
/home/qbzzt1/consensus-specs/venv/lib/python3.9/site-packages/cytoolz/compatibility.py:2:
|
||||
DeprecationWarning: The toolz.compatibility module is no longer needed in Python 3 and has
|
||||
been deprecated. Please import these utilities directly from the standard library. This
|
||||
module will be removed in a future release.
|
||||
warnings.warn("The toolz.compatibility module is no longer "
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/warnings.html
|
||||
================ 3 passed, 626 deselected, 1 warning in 16.81s =================
|
||||
```
|
||||
|
||||
|
||||
## The "Hello, World" of Consensus Spec Tests
|
||||
|
||||
One of the `test_empty_block_transition` tests is implemented by a function with the same
|
||||
name located in
|
||||
[`~/consensus-specs/tests/core/pyspec/eth2spec/test/phase0/sanity/test_blocks.py`](https://github.com/ethereum/consensus-specs/blob/dev/tests/core/pyspec/eth2spec/test/phase0/sanity/test_blocks.py).
|
||||
To learn how consensus spec tests are written, let's go over the code:
|
||||
|
||||
```python
|
||||
@with_all_phases
|
||||
```
|
||||
|
||||
This [decorator](https://book.pythontips.com/en/latest/decorators.html) specifies that this test
|
||||
is applicable to all the phases of consensus layer development. These phases are similar to forks (Istanbul,
|
||||
Berlin, London, etc.) in the execution blockchain. If you are interested, [you can see the definition of
|
||||
this decorator here](https://github.com/ethereum/consensus-specs/blob/dev/tests/core/pyspec/eth2spec/test/context.py#L331-L335).
|
||||
|
||||
```python
|
||||
@spec_state_test
|
||||
```
|
||||
|
||||
[This decorator](https://github.com/qbzzt/consensus-specs/blob/dev/tests/core/pyspec/eth2spec/test/context.py#L232-L234) specifies
|
||||
that this test is a state transition test, and that it does not include a transition between different forks.
|
||||
|
||||
```python
|
||||
def test_empty_block_transition(spec, state):
|
||||
```
|
||||
|
||||
This type of test receives two parameters:
|
||||
|
||||
* `specs`: The protocol specifications
|
||||
* `state`: The genesis state before the test
|
||||
|
||||
```python
|
||||
pre_slot = state.slot
|
||||
```
|
||||
|
||||
A slot is a unit of time (every 12 seconds in mainnet), for which a specific validator (selected randomly but in a
|
||||
deterministic manner) is a proposer. The proposer can propose a block during that slot.
|
||||
|
||||
```python
|
||||
pre_eth1_votes = len(state.eth1_data_votes)
|
||||
pre_mix = spec.get_randao_mix(state, spec.get_current_epoch(state))
|
||||
```
|
||||
|
||||
Store some values to check later that certain updates happened.
|
||||
|
||||
```python
|
||||
yield 'pre', state
|
||||
```
|
||||
|
||||
In Python `yield` is used by [generators](https://wiki.python.org/moin/Generators). However, for our purposes
|
||||
we can treat it as a partial return statement that doesn't stop the function's processing, only adds to a list
|
||||
of return values. Here we add two values, the string `'pre'` and the initial state, to the list of return values.
|
||||
|
||||
[You can read more about test generators and how the are used here](generators).
|
||||
|
||||
```python
|
||||
block = build_empty_block_for_next_slot(spec, state)
|
||||
```
|
||||
|
||||
The state contains the last block, which is necessary for building up the next block (every block needs to
|
||||
have the hash of the previous one in a blockchain).
|
||||
|
||||
```python
|
||||
signed_block = state_transition_and_sign_block(spec, state, block)
|
||||
```
|
||||
|
||||
Create a block signed by the appropriate proposer and advance the state.
|
||||
|
||||
```python
|
||||
yield 'blocks', [signed_block]
|
||||
yield 'post', state
|
||||
```
|
||||
|
||||
More `yield` statements. The output of a consensus test is:
|
||||
|
||||
1. `'pre'`
|
||||
2. The state before the test was run
|
||||
3. `'blocks'`
|
||||
4. A list of signed blocks
|
||||
5. `'post'`
|
||||
6. The state after the test
|
||||
|
||||
|
||||
|
||||
```python
|
||||
# One vote for the eth1
|
||||
assert len(state.eth1_data_votes) == pre_eth1_votes + 1
|
||||
|
||||
# Check that the new parent root is correct
|
||||
assert spec.get_block_root_at_slot(state, pre_slot) == signed_block.message.parent_root
|
||||
|
||||
# Random data changed
|
||||
assert spec.get_randao_mix(state, spec.get_current_epoch(state)) != pre_mix
|
||||
```
|
||||
|
||||
Finally we assertions that test the transition was legitimate. In this case we have three assertions:
|
||||
|
||||
1. One item was added to `eth1_data_votes`
|
||||
2. The new block's `parent_root` is the same as the block in the previous location
|
||||
3. The random data that every block includes was changed.
|
||||
|
||||
|
||||
## New Tests
|
||||
|
||||
The easiest way to write a new test is to copy and modify an existing one. For example,
|
||||
lets write a test where the first slot of the beacon chain is empty (because the assigned
|
||||
proposer is offline, for example), and then there's an empty block in the second slot.
|
||||
|
||||
We already know how to accomplish most of what we need for this test, but the only way we know
|
||||
to advance the state is `state_transition_and_sign_block`, a function that also puts a block
|
||||
into the slot. So let's see if the function's definition tells us how to advance the state without
|
||||
a block.
|
||||
|
||||
First, we need to find out where the function is located. Run:
|
||||
|
||||
```sh
|
||||
find . -name '*.py' -exec grep 'def state_transition_and_sign_block' {} \; -print
|
||||
```
|
||||
|
||||
And you'll find that the function is defined in
|
||||
[`eth2spec/test/helpers/state.py`](https://github.com/ethereum/consensus-specs/blob/dev/tests/core/pyspec/eth2spec/test/helpers/state.py). Looking
|
||||
in that file, we see that the second function is:
|
||||
|
||||
```python
|
||||
def next_slot(spec, state):
|
||||
"""
|
||||
Transition to the next slot.
|
||||
"""
|
||||
spec.process_slots(state, state.slot + 1)
|
||||
```
|
||||
|
||||
This looks like exactly what we need. So we add this call before we create the empty block:
|
||||
|
||||
|
||||
```python
|
||||
.
|
||||
.
|
||||
.
|
||||
yield 'pre', state
|
||||
|
||||
next_slot(spec, state)
|
||||
|
||||
block = build_empty_block_for_next_slot(spec, state)
|
||||
.
|
||||
.
|
||||
.
|
||||
```
|
||||
|
||||
That's it. Our new test works (copy `test_empty_block_transition`, rename it, add the `next_slot` call, and then run it to
|
||||
verify this).
|
||||
|
||||
|
||||
|
||||
## Tests Designed to Fail
|
||||
|
||||
It is important to make sure that the system rejects invalid input, so our next step is to deal with cases where the protocol
|
||||
is supposed to reject something. To see such a test, look at `test_prev_slot_block_transition` (in the same
|
||||
file we used previously,
|
||||
[`~/consensus-specs/tests/core/pyspec/eth2spec/test/phase0/sanity/test_blocks.py`](https://github.com/ethereum/consensus-specs/blob/dev/tests/core/pyspec/eth2spec/test/phase0/sanity/test_blocks.py)).
|
||||
|
||||
```python
|
||||
@with_all_phases
|
||||
@spec_state_test
|
||||
def test_prev_slot_block_transition(spec, state):
|
||||
spec.process_slots(state, state.slot + 1)
|
||||
block = build_empty_block(spec, state, slot=state.slot)
|
||||
```
|
||||
|
||||
Build an empty block for the current slot.
|
||||
|
||||
```python
|
||||
proposer_index = spec.get_beacon_proposer_index(state)
|
||||
```
|
||||
|
||||
Get the identity of the current proposer, the one for *this* slot.
|
||||
|
||||
```python
|
||||
spec.process_slots(state, state.slot + 1)
|
||||
```
|
||||
|
||||
Transition to the new slot, which naturally has a different proposer.
|
||||
|
||||
```python
|
||||
yield 'pre', state
|
||||
expect_assertion_error(lambda: transition_unsigned_block(spec, state, block))
|
||||
```
|
||||
|
||||
Specify that the function `transition_unsigned_block` will cause an assertion error.
|
||||
You can see this function in
|
||||
[`~/consensus-specs/tests/core/pyspec/eth2spec/test/helpers/block.py`](https://github.com/ethereum/consensus-specs/blob/dev/tests/core/pyspec/eth2spec/test/helpers/block.py),
|
||||
and one of the tests is that the block must be for this slot:
|
||||
> ```python
|
||||
> assert state.slot == block.slot
|
||||
> ```
|
||||
|
||||
Because we use [lambda notation](https://www.w3schools.com/python/python_lambda.asp), the test
|
||||
does not call `transition_unsigned_block` here. Instead, this is a function parameter that can
|
||||
be called later.
|
||||
|
||||
```python
|
||||
block.state_root = state.hash_tree_root()
|
||||
```
|
||||
|
||||
Set the block's state root to the current state hash tree root, which identifies this block as
|
||||
belonging to this slot (even though it was created for the previous slot).
|
||||
|
||||
```python
|
||||
signed_block = sign_block(spec, state, block, proposer_index=proposer_index)
|
||||
```
|
||||
|
||||
Notice that `proposer_index` is the variable we set earlier, *before* we advanced
|
||||
the slot with `spec.process_slots(state, state.slot + 1)`. It is not the proposer
|
||||
for the current state.
|
||||
|
||||
```python
|
||||
yield 'blocks', [signed_block]
|
||||
yield 'post', None # No post state, signifying it errors out
|
||||
```
|
||||
|
||||
This is the way we specify that a test is designed to fail - failed tests have no post state,
|
||||
because the processing mechanism errors out before creating it.
|
||||
|
||||
|
||||
## Attestation Tests
|
||||
|
||||
The consensus layer doesn't provide any direct functionality to end users. It does
|
||||
not execute EVM programs or store user data. It exists to provide a secure source of
|
||||
information about the latest verified block hash of the execution layer.
|
||||
|
||||
For every slot a validator is randomly selected as the proposer. The proposer proposes a block
|
||||
for the current head of the consensus layer chain (built on the previous block). That block
|
||||
includes the hash of the proposed new head of the execution layer.
|
||||
|
||||
For every slot there is also a randomly selected committee of validators that needs to vote whether
|
||||
the new consensus layer block is valid, which requires the proposed head of the execution chain to
|
||||
also be a valid block. These votes are called [attestations](https://notes.ethereum.org/@hww/aggregation#112-Attestation),
|
||||
and they are sent as independent messages. The proposer for a block is able to include attestations from previous slots,
|
||||
which is how they get on chain to form consensus, reward honest validators, etc.
|
||||
|
||||
[You can see a simple successful attestation test here](https://github.com/ethereum/consensus-specs/blob/926e5a3d722df973b9a12f12c015783de35cafa9/tests/core/pyspec/eth2spec/test/phase0/block_processing/test_process_attestation.py#L26-L30):
|
||||
Lets go over it line by line.
|
||||
|
||||
|
||||
```python
|
||||
@with_all_phases
|
||||
@spec_state_test
|
||||
def test_success(spec, state):
|
||||
attestation = get_valid_attestation(spec, state, signed=True)
|
||||
```
|
||||
|
||||
[This function](https://github.com/ethereum/consensus-specs/blob/30fe7ba1107d976100eb0c3252ca7637b791e43a/tests/core/pyspec/eth2spec/test/helpers/attestations.py#L88-L120)
|
||||
creates a valid attestation (which can then be modified to make it invalid if needed).
|
||||
To see an attestion "from the inside" we need to follow it.
|
||||
|
||||
|
||||
> ```python
|
||||
> def get_valid_attestation(spec,
|
||||
> state,
|
||||
> slot=None,
|
||||
> index=None,
|
||||
> filter_participant_set=None,
|
||||
> signed=False):
|
||||
> ```
|
||||
>
|
||||
> Only two parameters, `spec` and `state` are required. However, there are four other parameters that can affect
|
||||
> the attestation created by this function.
|
||||
>
|
||||
>
|
||||
> ```python
|
||||
> # If filter_participant_set filters everything, the attestation has 0 participants, and cannot be signed.
|
||||
> # Thus strictly speaking invalid when no participant is added later.
|
||||
> if slot is None:
|
||||
> slot = state.slot
|
||||
> if index is None:
|
||||
> index = 0
|
||||
> ```
|
||||
>
|
||||
> Default values. Normally we want to choose the current slot, and out of the proposers and committees that it can have,
|
||||
> we want the first one.
|
||||
>
|
||||
> ```python
|
||||
> attestation_data = build_attestation_data(
|
||||
> spec, state, slot=slot, index=index
|
||||
> )
|
||||
> ```
|
||||
>
|
||||
> Build the actual attestation. You can see this function
|
||||
> [here](https://github.com/ethereum/consensus-specs/blob/30fe7ba1107d976100eb0c3252ca7637b791e43a/tests/core/pyspec/eth2spec/test/helpers/attestations.py#L53-L85)
|
||||
> to see the exact data in an attestation.
|
||||
>
|
||||
> ```python
|
||||
> beacon_committee = spec.get_beacon_committee(
|
||||
> state,
|
||||
> attestation_data.slot,
|
||||
> attestation_data.index,
|
||||
> )
|
||||
> ```
|
||||
>
|
||||
> This is the committee that is supposed to approve or reject the proposed block.
|
||||
>
|
||||
> ```python
|
||||
>
|
||||
> committee_size = len(beacon_committee)
|
||||
> aggregation_bits = Bitlist[spec.MAX_VALIDATORS_PER_COMMITTEE](*([0] * committee_size))
|
||||
> ```
|
||||
>
|
||||
> There's a bit for every committee member to see if it approves or not.
|
||||
>
|
||||
> ```python
|
||||
> attestation = spec.Attestation(
|
||||
> aggregation_bits=aggregation_bits,
|
||||
> data=attestation_data,
|
||||
> )
|
||||
> # fill the attestation with (optionally filtered) participants, and optionally sign it
|
||||
> fill_aggregate_attestation(spec, state, attestation, signed=signed, filter_participant_set=filter_participant_set)
|
||||
>
|
||||
> return attestation
|
||||
> ```
|
||||
|
||||
```python
|
||||
next_slots(spec, state, spec.MIN_ATTESTATION_INCLUSION_DELAY)
|
||||
```
|
||||
|
||||
Attestations have to appear after the block they attest for, so we advance
|
||||
`spec.MIN_ATTESTATION_INCLUSION_DELAY` slots before creating the block that includes the attestation.
|
||||
Currently a single block is sufficient, but that may change in the future.
|
||||
|
||||
```python
|
||||
yield from run_attestation_processing(spec, state, attestation)
|
||||
```
|
||||
|
||||
[This function](https://github.com/ethereum/consensus-specs/blob/30fe7ba1107d976100eb0c3252ca7637b791e43a/tests/core/pyspec/eth2spec/test/helpers/attestations.py#L13-L50)
|
||||
processes the attestation and returns the result.
|
||||
|
||||
|
||||
### Adding an Attestation Test
|
||||
|
||||
Attestations can't happen in the same block as the one about which they are attesting, or in a block that is
|
||||
after the block is finalized. This is specified as part of the specs, in the `process_attestation` function
|
||||
(which is created from the spec by the `make pyspec` command you ran earlier). Here is the relevant code
|
||||
fragment:
|
||||
|
||||
|
||||
```python
|
||||
def process_attestation(state: BeaconState, attestation: Attestation) -> None:
|
||||
data = attestation.data
|
||||
assert data.target.epoch in (get_previous_epoch(state), get_current_epoch(state))
|
||||
assert data.target.epoch == compute_epoch_at_slot(data.slot)
|
||||
assert data.slot + MIN_ATTESTATION_INCLUSION_DELAY <= state.slot <= data.slot + SLOTS_PER_EPOCH
|
||||
...
|
||||
```
|
||||
|
||||
In the last line you can see two conditions being asserted:
|
||||
|
||||
1. `data.slot + MIN_ATTESTATION_INCLUSION_DELAY <= state.slot` which verifies that the attestation doesn't
|
||||
arrive too early.
|
||||
1. `state.slot <= data.slot + SLOTS_PER_EPOCH` which verifies that the attestation doesn't
|
||||
arrive too late.
|
||||
|
||||
This is how the consensus layer tests deal with edge cases, by asserting the conditions required for the
|
||||
values to be legitimate. In the case of these particular conditions, they are tested
|
||||
[here](https://github.com/ethereum/consensus-specs/blob/926e5a3d722df973b9a12f12c015783de35cafa9/tests/core/pyspec/eth2spec/test/phase0/block_processing/test_process_attestation.py#L87-L104).
|
||||
One test checks what happens if the attestation is too early, and another if it is too late.
|
||||
|
||||
However, it is not enough to ensure we reject invalid blocks. It is also necessary to ensure we accept all valid blocks. You saw earlier
|
||||
a test (`test_success`) that tested that being `MIN_ATTESTATION_INCLUSION_DELAY` after the data for which we attest is enough.
|
||||
Now we'll write a similar test that verifies that being `SLOTS_PER_EPOCH` away is still valid. To do this, we modify the
|
||||
`test_after_epoch_slots` function. We need two changes:
|
||||
|
||||
1. Call `transition_to_slot_via_block` with one less slot to advance
|
||||
1. Don't tell `run_attestation_processing` to return an empty post state.
|
||||
|
||||
The modified function is:
|
||||
|
||||
```python
|
||||
@with_all_phases
|
||||
@spec_state_test
|
||||
def test_almost_after_epoch_slots(spec, state):
|
||||
attestation = get_valid_attestation(spec, state, signed=True)
|
||||
|
||||
# increment to latest inclusion slot (not beyond it)
|
||||
transition_to_slot_via_block(spec, state, state.slot + spec.SLOTS_PER_EPOCH)
|
||||
|
||||
yield from run_attestation_processing(spec, state, attestation)
|
||||
```
|
||||
|
||||
Add this function to the file `consensus-specs/tests/core/pyspec/eth2spec/test/phase0/block_processing/test_process_attestation.py`,
|
||||
and run the test:
|
||||
|
||||
```sh
|
||||
cd ~/consensus-specs
|
||||
. venv/bin/activate
|
||||
cd tests/core/pyspec
|
||||
python -m pytest -k almost_after --fork Merge eth2spec
|
||||
```
|
||||
|
||||
You should see it ran successfully (although you might get a warning, you can ignore it)
|
||||
|
||||
## How are These Tests Used?
|
||||
|
||||
So far we've ran tests against the formal specifications. This is a way to check the specifications
|
||||
are what we expect, but it doesn't actually check the beacon chain clients. The way these tests get applied
|
||||
by clients is that every few weeks
|
||||
[new test specifications are released](https://github.com/ethereum/consensus-spec-tests/releases),
|
||||
in a format [documented here](https://github.com/ethereum/consensus-specs/tree/dev/tests/formats).
|
||||
All the consensus layer clients implement test-runners that consume the test vectors in this standard format.
|
||||
|
||||
---
|
||||
|
||||
Original version by [Ori Pomerantz](mailto:qbzzt1@gmail.com)
|
||||
Loading…
Reference in New Issue