logos-blockchain/logos-blockchain-specs
1
0
Fork 0
mirror of https://github.com/logos-blockchain/logos-blockchain-specs.git synced 2026-01-05 14:43:11 +00:00
Code Issues Packages Projects Releases Wiki Activity

Runnable Python implementation (#7)

Browse Source 
* use coarse grained events

* Add timeout handler

* Added runnable carnot implementation bare bones

* Removed rusty_results dependency

* Make easy tests

* Python impl tests (#9)

* Make easy tests

* Made sure an old aggregatedQC is not used.

* Test when a block has an old qc

* adding highest voted view so that a node doesn't vote twice.

* adding highest voted view so that a node doesn't vote twice.

* Tests for voting

* Tests for voting

* Tests for voting

* Tests for voting

* Update test_happy_path.py

---------

Co-authored-by: mjalalzai <33738574+MForensic@users.noreply.github.com>

* Fix commit grandparent

* Add tests assertions

* Added base vote test

* Tests for updating latest_committed_view and high_qc.

* Vote tests

* Vote tests

* Get max timeout by highQC

* Received Votes tests

* Receive timeout msgs

* Receive timeout msgs

* Remove local files

* Stylish, adjustments and fixes

* Start unhappy path and update tests (#10)

* Tests for updating latest_committed_view and high_qc.

* Vote tests

* Vote tests

* Get max timeout by highQC

* Received Votes tests

* Receive timeout msgs

* Receive timeout msgs

* Remove local files

* Stylish, adjustments and fixes

---------

Co-authored-by: mjalalzai <33738574+MForensic@users.noreply.github.com>

* Update build timeout qc test

* Added block content

* Fix tests with block content and comments

* Fix all timeouts are from the same view in timeout call

* Add check for double view seen different block

* Store just highest qc and aggregated views in AggregatedQc,

* Fix timeout preconditions

* Happy + Unhappy path implementation and tests (#15)

* Implement timeouts unhappy path

* views are sequential or consecutive.

* Make sure view changes are incrementally done in ascending order.

* Make sure view changes are incrementally done in ascending order.

* Make sure view changes are incrementally done in ascending order.

* Cleanup

* refactor

* Overlay based happy tests (#11)

* Cleanup

* Leaf committee member vote.

* Leaf committee member vote, test.

* Leaf committee member vote, test.

* Description

* Description

* Description-refactoring

* Cleanup

* Fix leaf votes test

* Clean overlay

* Test single committee advances

* Remove unhappy path test file

* Update carnot description

* Refactor local_timeout for NewView

* Fix unhappy path conditions and added broadcasting

* Unhappy path tests description

* Commit all grandparents of a block from latest_committed view

* Cleanup docs

* Add unhappy path test vector

* Remove block content

---------

Co-authored-by: mjalalzai <33738574+MForensic@users.noreply.github.com>

* Extract MockCarnot in happy path tests

* Add is_safe_to_timeout checks on unhappy path methods

* Fix happy math mockoverlay missing methods

* Fix unhappy path double increments
Fill missing qc building implementations

* Implement first case of unhappy path with simple overlay
Missing final assertions

* Unhappy path base test assertions (#14)

* Unhappy path tests replacing timeout msgs with NewView. Also revising the conditions to enter into the unhappy path.

* Unhappy path tests assertions.

* Fix assertions on test

* Cleanup

---------

Co-authored-by: mjalalzai <33738574+MForensic@users.noreply.github.com>

* Remove redundant code and fix commit parent (#16)

* Fix unhappy path tests (#17)

* fix test

* get highest qc from new view votes

* Add missing assertion

* Cleanup approve new view function

* Remove calls to safe_timeout invariant

* Fix approve new view

* Make unhappy test continuously timeout

* Increment current view on reciving a new qc
Refactor reset last timeout qc

* Fix happy path current view after receiving block qc

* Complete mix succeed fails unhappy test

* Refactor timeout_high_qc test to use fail method

* Add block assertions on mixed unhappy test

* Simplify approve block

* add spec tests to ci

* Extract implicit information from safe blocks (#19)

* extract implicit information from safe blocks

* fix test

* Refactor last_view_timeout_qc and update calls on unhappy path

* Update view upon reception of timeout qc (#20)

* update view upon reception of timeout qc

* only increase highest_voted_view

* fix comments

* [WIP] Use events instead of send/broadcast methods (#21)

* Remove send and broadcast, use events

* Adjust tests to use events

* Adjust unhappy path tests to use events

* Fix missing wrongly optional return types

* Extrac common assert on propose_block

* add informative comments and remove panic (#22)

* add disclaimer (#23)

* fix approve_new_view preconditions

---------

Co-authored-by: mjalalzai <33738574+MForensic@users.noreply.github.com>
Co-authored-by: Giacomo Pasini <g.pasini98@gmail.com>
Co-authored-by: Giacomo Pasini <Zeegomo@users.noreply.github.com>

---------

Co-authored-by: Giacomo Pasini <g.pasini98@gmail.com>
Co-authored-by: mjalalzai <33738574+MForensic@users.noreply.github.com>
Co-authored-by: Giacomo Pasini <Zeegomo@users.noreply.github.com>
This commit is contained in:
Daniel Sanchez 2023-05-05 09:58:58 +02:00 committed by GitHub
parent 26501440b1
commit c2e05a48c5
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
7 changed files with 1431 additions and 196 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

17
.github/workflows/ci.yml vendored Normal file
View File

@ -0,0 +1,17 @@
name: Spec tests
on: [pull_request]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Python 3.x
uses: actions/setup-python@v4
with:
# Semantic version range syntax or exact version of a Python version
python-version: '3.x'
- name: Run tests
run: cd carnot && python -m unittest

79
.gitignore vendored
View File

@ -1,79 +0,0 @@
# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio, WebStorm and Rider
# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839
# User-specific stuff
.idea/**/workspace.xml
.idea/**/tasks.xml
.idea/**/usage.statistics.xml
.idea/**/dictionaries
.idea/**/shelf
# AWS User-specific
.idea/**/aws.xml
# Generated files
.idea/**/contentModel.xml
# Sensitive or high-churn files
.idea/**/dataSources/
.idea/**/dataSources.ids
.idea/**/dataSources.local.xml
.idea/**/sqlDataSources.xml
.idea/**/dynamic.xml
.idea/**/uiDesigner.xml
.idea/**/dbnavigator.xml
# Gradle
.idea/**/gradle.xml
.idea/**/libraries
# Gradle and Maven with auto-import
# When using Gradle or Maven with auto-import, you should exclude module files,
# since they will be recreated, and may cause churn. Uncomment if using
# auto-import.
# .idea/artifacts
# .idea/compiler.xml
# .idea/jarRepositories.xml
# .idea/modules.xml
# .idea/*.iml
# .idea/modules
# *.iml
# *.ipr
# CMake
cmake-build-*/
# Mongo Explorer plugin
.idea/**/mongoSettings.xml
# File-based project format
*.iws
# IntelliJ
out/
# mpeltonen/sbt-idea plugin
.idea_modules/
# JIRA plugin
atlassian-ide-plugin.xml
# Cursive Clojure plugin
.idea/replstate.xml
# SonarLint plugin
.idea/sonarlint/
# Crashlytics plugin (for Android Studio and IntelliJ)
com_crashlytics_export_strings.xml
crashlytics.properties
crashlytics-build.properties
fabric.properties
# Editor-based Rest Client
.idea/httpRequests
# Android studio 3.1+ serialized cache file
.idea/caches/build_file_checksums.serpython
python
venv

1
carnot/__init__.py Normal file
View File

@ -0,0 +1 @@
from .carnot import *

594
carnot/carnot.py Normal file
View File

@ -0,0 +1,594 @@
# The Carnot protocol is designed to be elastic, responsive, and provide fast finality
# Elastic scalability allows the protocol to operate effectively with both small and large networks
# All nodes in the Carnot network participate in the consensus of a block
# Optimistic responsiveness enables the protocol to operate quickly during periods of synchrony and honest leadership
# There is no block generation time in Carnot, allowing for fast finality
# Carnot avoids the chain reorg problem, making it compatible with PoS schemes
# This enhances the robustness of the protocol, making it a valuable addition to the ecosystem of consensus protocols
# The protocol in Carnot operates in two modes: the happy path and the unhappy path.
#
# In Carnot, nodes are arranged in a binary tree overlay committee structure. Moreover, Carnot is a
# pipelined consensus protocol where a block contains the proof of attestation of its parent. In happy path the
# leader proposes a block that contains a quorum certificate (QC) with votes from more than two-thirds of the root
# committee and its child committee/ committees. The voting process begins at the leaf committee where nodes verify
# the proposal and send their votes to the parent committee. Once a node in the parent committee receives more than
# two-thirds of the votes from its child committee members, it sends its votes to its parent. This process continues
# recursively until the root committee members collect votes from its child committee/ committees. The root committee
# member builds a QC from the votes and sends it to the next leader. The leader builds a QC and proposes the next block
# upon receiving more than two-thirds of votes.
# In the unhappy path, if a node does not receive a message within a timeout interval, it will timeout. Only nodes at
# the root committee and its child committee/ committees send their timeout messages to the root committee. The root
# committee builds a timeout QC from more than two-thirds of messages, recalculates the new overlay, and broadcasts it
# to the network. Similar to the happy path, the timeout message moves from leaves to the root. Each parent waits for
# more than two-thirds of timeout messages from its child committees and sends its timeout to the parent committee once
# the threshold is reached. A node in the root committee builds a QC from timeout messages received from its
# child committee/committees and forwards it to the next leader. Upon receiving more than two-thirds of timeout
# messages, the next leader builds an aggregated QC and proposes the next block containing the aggregated QC.
# It should be noted that while receiving timeout messages, each node also updates its high_qc (the most recent QC)
# and passes it to its parent through the timeout message. In this way, the aggregated QC will include the high_qc seen
# by the majority of honest nodes. Hence, after the view change, the protocol safety is preserved.
# Please note this is still a work in progress
from dataclasses import dataclass
from typing import TypeAlias, List, Set, Self, Optional, Dict, FrozenSet
from abc import abstractmethod
Id: TypeAlias = bytes
View: TypeAlias = int
Committee: TypeAlias = Set[Id]
def int_to_id(i: int) -> Id:
return bytes(str(i), encoding="utf8")
@dataclass(unsafe_hash=True)
class StandardQc:
block: Id
view: View
def view(self) -> View:
return self.view
@dataclass
class AggregateQc:
qcs: List[View]
highest_qc: StandardQc
view: View
def view(self) -> View:
return self.view
def high_qc(self) -> StandardQc:
assert self.highest_qc.view == max(self.qcs)
return self.highest_qc
Qc: TypeAlias = StandardQc | AggregateQc
@dataclass
class Block:
view: View
qc: Qc
_id: Id # this is an abstration over the block id, which should be the hash of the contents
def extends(self, ancestor: Self) -> bool:
"""
:param ancestor:
:return: true if block is descendant of the ancestor in the chain
"""
return self.view > ancestor.view
def parent(self) -> Id:
match self.qc:
case StandardQc(block):
return block
case AggregateQc() as aqc:
return aqc.high_qc().block
def id(self) -> Id:
return self._id
@dataclass(unsafe_hash=True)
class Vote:
block: Id
view: View
voter: Id
qc: Optional[Qc]
@dataclass
class TimeoutQc:
view: View
high_qc: Qc
qc_views: List[View]
sender_ids: Set[Id]
sender: Id
@dataclass
class Timeout:
"""
Local timeout field is only used by the root committee and its children when they timeout. The timeout_qc is built
from local_timeouts. Leaf nodes when receive timeout_qc build their timeout msg and includes the timeout_qc in it.
The timeout_qc is indicator that the root committee and its child committees (if exist) have failed to collect votes.
"""
view: View
high_qc: Qc
sender: Id
timeout_qc: TimeoutQc
# Timeout has been detected, nodes agree on it and gather high qc
@dataclass
class NewView:
view: View
high_qc: Qc
sender: Id
timeout_qc: TimeoutQc
Quorum: TypeAlias = Set[Vote] | Set[NewView]
Payload: TypeAlias = Block | Vote | Timeout | NewView | TimeoutQc
@dataclass
class BroadCast:
payload: Payload
@dataclass
class Send:
to: [Id]
payload: Payload
Event: TypeAlias = BroadCast | Send
class Overlay:
"""
Overlay structure for a View
"""
@abstractmethod
def is_leader(self, _id: Id):
"""
:param _id: Node id to be checked
:return: true if node is the leader of the current view
"""
pass
@abstractmethod
def leader(self, view: View) -> Id:
"""
:param view:
:return: the leader Id of the specified view
"""
pass
@abstractmethod
def is_member_of_leaf_committee(self, _id: Id) -> bool:
"""
:param _id: Node id to be checked
:return: true if the participant with Id _id is in the leaf committee of the committee overlay
"""
pass
@abstractmethod
def is_member_of_root_committee(self, _id: Id) -> bool:
"""
:param _id:
:return: true if the participant with Id _id is member of the root committee withing the tree overlay
"""
pass
@abstractmethod
def is_member_of_child_committee(self, parent: Id, child: Id) -> bool:
"""
:param parent:
:param child:
:return: true if participant with Id child is member of the child committee of the participant with Id parent
"""
pass
@abstractmethod
def parent_committee(self, _id: Id) -> Optional[Committee]:
"""
:param _id:
:return: Some(parent committee) of the participant with Id _id withing the committee tree overlay
or Empty if the member with Id _id is a participant of the root committee
"""
pass
@abstractmethod
def leaf_committees(self) -> Set[Committee]:
pass
@abstractmethod
def root_committee(self) -> Committee:
"""
:return: returns root committee
"""
pass
@abstractmethod
def is_child_of_root_committee(self, _id: Id) -> bool:
"""
:return: returns child committee/s of root committee if present
"""
pass
@abstractmethod
def leader_super_majority_threshold(self, _id: Id) -> int:
"""
Amount of distinct number of messages for a node with Id _id member of a committee
The return value may change depending on which committee the node is member of, including the leader
:return:
"""
pass
@abstractmethod
def super_majority_threshold(self, _id: Id) -> int:
pass
def download(view) -> Block:
raise NotImplementedError
class Carnot:
def __init__(self, _id: Id):
self.id: Id = _id
# Current View counter
# It is the view currently being processed by the node. Once a Qc is received, the view is considered completed
# and the current view is updated to qc.view+1
self.current_view: View = 0
# Highest voted view counter. This is used to prevent a node from voting twice or vote after timeout.
self.highest_voted_view: View = -1
# This is most recent (in terms of view) Standard QC that has been received by the node
self.local_high_qc: Optional[Qc] = None
# Validated blocks with their validated QCs are included here. If commit conditions are satisfied for
# each one of these blocks it will be committed.
self.safe_blocks: Dict[Id, Block] = dict()
# Whether the node timeed out in the last view and corresponding qc
self.last_view_timeout_qc: Optional[TimeoutQc] = None
self.overlay: Overlay = Overlay() # TODO: integrate overlay
# Committing conditions for a block
# TODO: explain the conditions in comment
def can_commit_grandparent(self, block) -> bool:
parent = self.safe_blocks.get(block.parent())
grand_parent = self.safe_blocks.get(parent.parent())
# this case should just trigger on genesis_case,
# as the preconditions on outer calls should check on block validity
if not parent or not grand_parent:
return False
return (
parent.view == (grand_parent.view + 1) and
isinstance(block.qc, (StandardQc,)) and
isinstance(parent.qc, (StandardQc,))
)
# The latest committed view is implicit in the safe blocks tree given
# the committing conditions.
# For convenience, this is an helper method to retrieve that value.
def latest_committed_view(self) -> View:
return self.latest_committed_block().view
# Return the list of blocks received by a node for a specific view.
# It will return more than one block only in case of a malicious leader
def blocks_in_view(self, view: View) -> List[Block]:
return [block for block in self.safe_blocks.values() if block.view == view]
def genesis_block(self) -> Block:
return self.blocks_in_view(0)[0]
def latest_committed_block(self) -> Block:
for view in range(self.current_view, 0, -1):
for block in self.blocks_in_view(view):
if self.can_commit_grandparent(block):
return self.safe_blocks.get(self.safe_blocks.get(block.parent()).parent())
# genesis blocks is always considered committed
return self.genesis_block()
# Given committing conditions, the set of committed blocks is implicit
# in the safe blocks tree. For convenience, this is an helper method to
# retrieve that set.
def committed_blocks(self) -> Dict[Id, Block]:
tip = self.latest_committed_block()
committed_blocks = {tip.id(): tip, self.genesis_block().id: self.genesis_block()}
while tip.view > 0:
committed_blocks[tip.id()] = tip
tip = self.safe_blocks.get(tip.parent())
return committed_blocks
def block_is_safe(self, block: Block) -> bool:
return (
block.view >= self.current_view and
block.view == block.qc.view + 1
)
# Ask Dani
def update_high_qc(self, qc: Qc):
match (self.local_high_qc, qc):
case (None, StandardQc() as new_qc):
self.local_high_qc = new_qc
case (None, AggregateQc() as new_qc):
self.local_high_qc = new_qc.high_qc()
case (old_qc, StandardQc() as new_qc) if new_qc.view > old_qc.view:
self.local_high_qc = new_qc
case (old_qc, AggregateQc() as new_qc) if new_qc.high_qc().view != old_qc.view:
self.local_high_qc = new_qc.high_qc()
# if my view is not updated I update it when I see a qc for that view
if qc.view == self.current_view:
self.current_view = self.current_view + 1
def update_timeout_qc(self, timeout_qc: TimeoutQc):
match (self.last_view_timeout_qc, timeout_qc):
case (None, timeout_qc):
self.last_view_timeout_qc = timeout_qc
case (self.last_view_timeout_qc, timeout_qc) if timeout_qc.view > self.last_view_timeout_qc.view:
self.last_view_timeout_qc = timeout_qc
def receive_block(self, block: Block):
assert block.parent() in self.safe_blocks
if block.id() in self.safe_blocks:
return
if self.blocks_in_view(block.view) != [] or block.view <= self.latest_committed_view():
# TODO: Report malicious leader
# TODO: it could be possible that a malicious leader send a block to a node and another one to
# the rest of the network. The node should be able to catch up with the rest of the network after having
# validated that the history of the block is correct and diverged from its fork.
# By rejecting any other blocks except the first one received for a view this code does NOT do that.
return
# TODO: check the proposer of the block is indeed leader for that view
if self.block_is_safe(block):
self.safe_blocks[block.id()] = block
self.update_high_qc(block.qc)
def approve_block(self, block: Block, votes: Set[Vote]) -> Event:
assert block.id() in self.safe_blocks
assert len(votes) == self.overlay.super_majority_threshold(self.id)
assert all(self.overlay.is_member_of_child_committee(self.id, vote.voter) for vote in votes)
assert all(vote.block == block.id() for vote in votes)
assert self.highest_voted_view < block.view
if self.overlay.is_member_of_root_committee(self.id):
qc = self.build_qc(block.view, block, None)
else:
qc = None
vote: Vote = Vote(
block=block.id(),
voter=self.id,
view=block.view,
qc=qc
)
self.highest_voted_view = max(self.highest_voted_view, block.view)
if self.overlay.is_member_of_root_committee(self.id):
return Send(to=self.overlay.leader(block.view + 1), payload=vote)
return Send(to=self.overlay.parent_committee(self.id), payload=vote)
def forward_vote(self, vote: Vote) -> Optional[Event]:
assert vote.block in self.safe_blocks
assert self.overlay.is_member_of_child_committee(self.id, vote.voter)
# we only forward votes after we've voted ourselves
assert self.highest_voted_view == vote.view
if self.overlay.is_member_of_root_committee(self.id):
return Send(to=self.overlay.leader(self.current_view + 1), payload=vote)
def forward_new_view(self, msg: NewView) -> Optional[Event]:
assert msg.view == self.current_view
assert self.overlay.is_member_of_child_committee(self.id, msg.sender)
# we only forward votes after we've voted ourselves
assert self.highest_voted_view == msg.view
if self.overlay.is_member_of_root_committee(self.id):
return Send(to=self.overlay.leader(self.current_view + 1), payload=msg)
def build_qc(self, view: View, block: Optional[Block], new_views: Optional[Set[NewView]]) -> Qc:
# unhappy path
if new_views:
new_views = list(new_views)
return AggregateQc(
qcs=[msg.high_qc.view for msg in new_views],
highest_qc=max(new_views, key=lambda x: x.high_qc.view).high_qc,
view=new_views[0].view
)
# happy path
return StandardQc(
view=view,
block=block.id()
)
def propose_block(self, view: View, quorum: Quorum) -> Event:
assert self.overlay.is_leader(self.id)
assert len(quorum) >= self.overlay.leader_super_majority_threshold(self.id)
qc = None
quorum = list(quorum)
# happy path
if isinstance(quorum[0], Vote):
vote = quorum[0]
qc = self.build_qc(vote.view, self.safe_blocks[vote.block], None)
# unhappy path
elif isinstance(quorum[0], NewView):
new_view = quorum[0]
qc = self.build_qc(new_view.view, None, quorum)
block = Block(
view=view,
qc=qc,
# Dummy id for proposing next block
_id=int_to_id(hash(
(
bytes(f"{view}".encode(encoding="utf8")),
bytes(f"{qc.view}".encode(encoding="utf8"))
)
))
)
return BroadCast(payload=block)
def is_safe_to_timeout_invariant(
self,
):
"""
Local timeout is different for the root and its child committees. If other committees timeout, they only
stop taking part in consensus. If a member of root or its child committees timeout it sends its timeout message
to all members of root to build the timeout qc. Using this qc we assume that the new
overlay can be built. Hence, by building the new overlay members of root committee can send the timeout qc
to the leaf committee of the new overlay. Upon receipt of the timeout qc the leaf committee members update
their local_high_qc, last_timeout_view_qc and last_voted_view if the view of qcs
(local_high_qc, last_timeout_view_qc) received is higher than their local view. Similarly last_voted_view is
updated if it is greater than the current last_voted_view. When parent committee member receives more than two
third of timeout messages from its children it also updates its local_high_qc, last_timeout_view_qc and
last_voted_view if needed and then send its timeout message upward. In this way the latest qcs move upward
that makes it possible for the next leader to propose a block with the latest local_high_qcs in aggregated qc
from more than two third members of root committee and its children.
"""
# Make sure the node doesn't time out continuously without finishing the step to increment the current view.
# Make sure current view is always higher than the local_high_qc so that the node won't timeout unnecessary
# for a previous view.
assert self.current_view > max(self.highest_voted_view - 1, self.local_high_qc.view)
# This condition makes sure a node waits for timeout_qc from root committee to change increment its view with
# a view change.
# A node must change its view after making sure it has the high_Qc or last_timeout_view_qc
# from previous view.
return (
self.current_view == self.local_high_qc.view + 1 or
self.current_view == self.last_view_timeout_qc.view + 1 or
(self.current_view == self.last_view_timeout_qc.view)
)
def local_timeout(self) -> Optional[Event]:
"""
Root committee changes for each failure, so repeated failure will be handled by different
root committees
"""
# avoid voting after we timeout
self.highest_voted_view = self.current_view
if self.overlay.is_member_of_root_committee(self.id) or self.overlay.is_child_of_root_committee(self.id):
timeout_msg: Timeout = Timeout(
view=self.current_view,
high_qc=self.local_high_qc,
# local_timeout is only true for the root committee or members of its children
# root committee or its children can trigger the timeout.
timeout_qc=self.last_view_timeout_qc,
sender=self.id
)
return Send(payload=timeout_msg, to=self.overlay.root_committee())
def timeout_detected(self, msgs: Set[Timeout]) -> Event:
"""
Root committee detected that supermajority of root + its children has timed out
The view has failed and this information is sent to all participants along with the information
necessary to reconstruct the new overlay
"""
assert len(msgs) == self.overlay.leader_super_majority_threshold(self.id)
assert all(msg.view >= self.current_view for msg in msgs)
assert len(set(msg.view for msg in msgs)) == 1
assert self.overlay.is_member_of_root_committee(self.id)
timeout_qc = self.build_timeout_qc(msgs, self.id)
return BroadCast(payload=timeout_qc) # we broadcast so all nodes can get ready for voting on a new view
# Note that receive_timeout qc should be called for root nodes as well
# noinspection PyTypeChecker
def approve_new_view(self, timeout_qc: TimeoutQc, new_views: Set[NewView]) -> Event:
"""
We will always need for timeout_qc to have been preprocessed by the received_timeout_qc method when the event
happens before approve_new_view is processed.
"""
# newView.view == self.last_timeout_view_qc.view for member of root committee and its children because
# they have already created the timeout_qc. For other nodes newView.view > self.last_timeout_view_qc.view.
if self.last_view_timeout_qc is not None:
assert all(new_view.view > self.last_view_timeout_qc.view for new_view in new_views)
assert all(new_view.timeout_qc.view == timeout_qc.view for new_view in new_views)
assert len(new_views) == self.overlay.super_majority_threshold(self.id)
assert all(self.overlay.is_member_of_child_committee(self.id, new_view.sender) for new_view in new_views)
# the new view should be for the view successive to the timeout
assert all(timeout_qc.view + 1 == new_view.view for new_view in new_views)
view = timeout_qc.view + 1
assert self.highest_voted_view < view
# get the highest qc from the new views
messages_high_qc = (new_view.high_qc for new_view in new_views)
high_qc = max(
[timeout_qc.high_qc, *messages_high_qc],
key=lambda qc: qc.view
)
self.update_high_qc(high_qc)
timeout_msg = NewView(
view=view,
# TODO: even if this event is processed "later", we should not allow high_qc.view to be >= timeout_qc.view
high_qc=self.local_high_qc,
sender=self.id,
timeout_qc=timeout_qc,
)
# This checks if a node has already incremented its voted view by local_timeout. If not then it should
# do it now to avoid voting in this view.
self.highest_voted_view = max(self.highest_voted_view, view)
if self.overlay.is_member_of_root_committee(self.id):
return Send(payload=timeout_msg, to=[self.overlay.leader(self.current_view + 1)])
return Send(payload=timeout_msg, to=self.overlay.parent_committee(self.id))
# Just a suggestion that received_timeout_qc can be reused by each node when the process timeout_qc of the NewView msg.
# TODO: check that receiving (and processing) a timeout qc "in the future" allows to process old(er) blocks
# e.g. we might still need access to the old leader schedule to validate qcs
def receive_timeout_qc(self, timeout_qc: TimeoutQc):
if timeout_qc.view < self.current_view:
return
new_high_qc = timeout_qc.high_qc
self.update_high_qc(new_high_qc)
self.update_timeout_qc(timeout_qc)
# Update our current view and go ahead with the next step
self.update_current_view_from_timeout_qc(timeout_qc)
self.rebuild_overlay_from_timeout_qc(timeout_qc)
def rebuild_overlay_from_timeout_qc(self, timeout_qc: TimeoutQc):
assert timeout_qc.view >= self.current_view
self.overlay = Overlay()
@staticmethod
def build_timeout_qc(msgs: Set[Timeout], sender: Id) -> TimeoutQc:
msgs = list(msgs)
return TimeoutQc(
view=msgs[0].view,
high_qc=max(msgs, key=lambda x: x.high_qc.view).high_qc,
qc_views=[msg.view for msg in msgs],
sender_ids={msg.sender for msg in msgs},
sender=sender,
)
def update_current_view_from_timeout_qc(self, timeout_qc: TimeoutQc):
self.current_view = timeout_qc.view + 1
if __name__ == "__main__":
pass

230
carnot/spec.md
View File

@ -6,6 +6,10 @@ In addition, all types can be expected to have their invariants checked by the t
'Q:' is used to indicate unresolved questions.
Notation is loosely based on CDDL.
Similar to the Carnot algorithm, this specification will be event-based, prescribing the actions to perform in response to relevant events in the Carnot consensus.
Events should be processed one at a time, picking any from the available ones.
As for ordering between events, there are some constraints (e.g. can't process a proposal before it's parent) which will likely form a DAG of relations. The expectation is that an implementation will respect these requirements by processing only events which have all preconditions satisfied.
## Messages
A critical piece in the protocol, these are the different kind of messages used by participants during the protocol execution.
* `Block`: propose a new block
@ -93,7 +97,6 @@ class Timeout:
view: View
high_qc: AggregateQc
```
## Local Variables
Participants in the protocol are expected to mainting the following data in addition to the DAG of received proposal:
* `current_view`
@ -105,7 +108,8 @@ Participants in the protocol are expected to mainting the following data in addi
CURRENT_VIEW: View
LOCAL_HIGH_QC: Qc
LATEST_COMMITTED_VIEW: View
COLLECTION: Q?
SAFE_BLOCKS: Set[Block]
LAST_VIEW_TIMEOUT_QC: TimeoutQc
```
@ -140,38 +144,42 @@ The following functions are expected to be available to participants during the
## Core functions
## Core events
These are the core events necessary for the Carnot consensus protocol. In response to such events a participant is expected to execute the corresponding handler action.
* receive block b -> `receive_block(b)`
Preconditions:
* `b.parent() in SAFE_BLOCKS`
* receive a supermajority of votes for block b -> `vote(b, votes)`
Preconditions:
* `b in SAFE_BLOCKS`
* `local_timeout(b.view)` never called
* receive a vote v for block b when a supermajority of votes already exists -> `forward_votes(b, v)`
Preconditions:
* `b in SAFE_BLOCKS`
* `vote(b, some_votes)` already called and `v not in some_votes`
* `local_timeout(b.view)` never called
* `current_time() - time(last view update) > TIMEOUT` and received new overlay -> `local_timeout(last view, new_overlay)`
* leader for view v and leader supermajority for previous proposal -> `propose_block(v, votes)`
* receive a supermajority of timeouts for view v -> `timeout(v, timeouts)`
Preconditions:
* `local_timeout(v)` already called
These are the core functions necessary for the Carnot consensus protocol, to be executed in response of incoming messages, except for `timeout` which is triggered by a participant configurable timer.
### Receive block
```python3
def receive_block(block: Block):
if block.id() is known or block.view <=LATEST_COMMITTED_VIEW:
return
# checking preconditions
assert block.parent() in SAFE_BLOCKS
# Recursively make sure that we process blocks in order
if block.parent() is missing:
parent: Block = download(block.parent())
receive(parent)
if block.id() in SAFE_BLOCKS or block.view <= LATEST_COMMITTED_VIEW:
return
if safe_block(block):
# This is not in the original spec, but
# let's validate I have this clear.
assert block.view == current_view
SAFE_BLOCKS.add(block)
update_high_qc(block.qc)
vote = create_vote()
if member_of_leaf_committee():
if member_of_root_committee():
send(vote, leader(current_view + 1))
else:
send(vote, parent_commitee())
current_view += 1
reset_timer()
try_to_commit_grandparent(block)
```
##### Auxiliary functions
@ -198,6 +206,8 @@ def safe_block(block: Block):
```
```python
# FIX_ME: Don't think we need to specify this as a function if we don't use
# LAST_COMMITTED_VIEW
# Commit a grand parent if the grandparent and
# the parent have been added during two consecutive views.
def try_to_commit_grand_parent(block: Block):
@ -231,110 +241,96 @@ def update_high_qc(qc: Qc):
# Q: same thing about missing views
```
### Receive Vote
Q: this whole function needs to be revised
### Vote
```python
def vote(block: Block, votes: Set[Vote]):
# check preconditions
assert block in SAFE_BLOCKS
assert supermajority(votes)
assert all(child_committee(vote.id) for vote in votes)
assert all(vote.block == block for vote in votes)
def receive_vote(vote: Vote):
if vote.block is missing:
block = download(vote.block)
receive(block)
vote = create_vote(votes)
# Q: we should probably return if we already received this vote
if member_of_internal_com() and not_member_of_root():
if child_commitee(vote.voter):
COLLECTION[vote.block].append(vote)
else:
# Q: not returning here would mean it's extremely easy to
# trigger building a new vote in the following branches
return
if supermajority(COLLECTION[vote.block]):
# Q: should we send it to everyone in the committee?
self_vote = build_vote()
send(self_vote, parent_committee)
# Q: why here?
current_view += 1
reset_timer()
# Q: why do we do this here?
try_to_commit_grand_parent(block)
if member_of_root():
if child_commitee(vote.voter):
COLLECTION[vote.block].append(vote)
else:
# Q: not returning here would mean it's extremely easy to
# trigger building a new vote in the following branches
return
if supermajority(COLLECTION[vote.block]):
# Q: The vote to send is not the one received but
# the one build by this participant, right?
self_vote = build_vote();
qc = build_qc(collection[vote.block])
self_vote.qc=qc
send(self_vote, leader(current_view + 1))
# Q: why here?
current_view += 1
reset_timer()
# Q: why here?
try_to_commit_grandparent(block)
# Q: this means that we send a message for every incoming
# message after the threshold has been reached, i.e. a vote
# from a node in the leaf committee can trigger
# at least height(tree) messages.
if morethanSsupermajority(collection[vote.block]):
# just forward the vote to the leader
# Q: But then childcommitte(vote.voter) would return false
# in the leader, as it's a granchild, not a child
send(vote, leader(current_view + 1))
if leader(view): # Q? Which view? CURRENT_VIEW or vote.view?
if vote.view < CURRENT_VIEW - 1:
return
# Q: No filtering? I can just create a key and vote?
COLLECTION[vote.block].append(vote)
if supermajority(collection[vote.block]):
qc = build_qc(collection[vote.block])
block = build_block(qc)
broadcast(block)
vote.qc = build_qc(votes)
send(vote, leader(CURRENT_VIEW + 1))
else:
send(vote, parent_committee())
# Q: what about a node that is joining later and does not
# have access to votes? how does it commit blocks?
current_view += 1
reset_timer()
try_to_commit_grandparent(block)
```
### Receive Timeout
### Forward vote
```python
# Failure Case
def receive(timeout: Timeout):
# download the missing block
if timeout.high_qc().block is missing:
block = download(timeout.high_qc.block)
receive(block)
def forward_vote(vote: Vote):
assert vote.block in SAFE_BLOCKS
assert child_committe(vote.id)
# already supermajority
# It's an old message. Ignore it.
if timeout.view < CURRENT_VIEW:
return
update_high_qc(timeout.high_qc())
if member_of_internal_com():
COLLECTION[timeout.view].append(timeout)
if supermajority(timeout.view):
new_view_qc = build_qc(COLLECTION[timeout.view])
if member_of_root():
send(new_view_qc, leader(CURRENT_VIEW +1))
CURRENT_VIEW += 1
else:
send(new_view_qc, parent_committee())
if member_of_root():
# just forward the vote to the leader
# Q: But then childcommitte(vote.voter) would return false
# in the leader, as it's a granchild, not a child
send(vote, leader(vote.block.view + 1))
```
### Propose block
```python
def propose_block(view: View, quorum: Set[Vote] | Set[TimeoutMsg]):
assert leader(view)
assert leader_supermajority(quorum)
qc = build_qc(votes)
block = build_block(qc)
broadcast(block)
```
### Timeout
```python
def timeout():
raise NotImplementedError
```
def local_timeout(new_overlay: Overlay):
# make it so we don't vote or forward any more vote after this
LAST_TIMEOUT_VIEW = CURRENT_VIEW
# TODO: change overlay
if member_of_leaf():
timeout_msg = create_timeout(CURRENT_VIEW, LOCAL_HIGH_QC, LAST_TIMEOUT_VIEW_QC)
send(timeout_msg, parent_committee())
```
### Receive
this is called *after* local_timeout
```python
def timeout(view: View, msgs: Set[TimeoutMsg]):
assert supermajority(msgs)
assert all(child_committee(msg.id) for msg in msgs)
assert all(timeout.view == view for timeout in msgs)
if CURRENT_VIEW > view:
return
if view <= LAST_VIEW_TIMEOUT_QC.view:
return
if view > LOCAL_HIGH_QC.view:
LOCAL_HIGH_QC = timeout_Msg.high_qc
timeout_qc = create_timeout_qc(msgs)
increment_view_timeout_qc(timeout_qc.view)
LAST_VIEW_TIMEOUT_QC = timeout_qc
send(timeout_qc, own_committee()) ####helps nodes to sync quicker but not required
if member_of_root():
send(timeout_qc, leader(view+1))
else:
send(timeout_qc, parent_committee())
```
We need to make sure that qcs can't be removed from aggQc when going up the tree

441
carnot/test_happy_path.py Normal file
View File

@ -0,0 +1,441 @@
from carnot import *
from unittest import TestCase
class TestCarnotHappyPath(TestCase):
@staticmethod
def add_genesis_block(carnot: Carnot) -> Block:
genesis_block = Block(view=0, qc=StandardQc(block=b"", view=0), _id=b"")
carnot.safe_blocks[genesis_block.id()] = genesis_block
return genesis_block
def test_receive_block(self):
carnot = Carnot(int_to_id(0))
genesis_block = self.add_genesis_block(carnot)
block = Block(view=1, qc=StandardQc(block=genesis_block.id(), view=0), _id=b"1")
carnot.receive_block(block)
def test_receive_multiple_blocks_for_the_same_view(self):
carnot = Carnot(int_to_id(0))
genesis_block = self.add_genesis_block(carnot)
# 1
block1 = Block(view=1, qc=StandardQc(block=genesis_block.id(), view=0), _id=b"1")
carnot.receive_block(block1)
# 2
block2 = Block(view=2, qc=StandardQc(block=block1.id(), view=1), _id=b"2")
carnot.receive_block(block2)
# 3
block3 = Block(view=3, qc=StandardQc(block=block2.id(), view=2), _id=b"3")
carnot.receive_block(block3)
# 4
block4 = Block(view=4, qc=StandardQc(block=block3.id(), view=3), _id=b"4")
carnot.receive_block(block4)
self.assertEqual(len(carnot.safe_blocks), 5)
# next block is duplicated and as it is already processed should be skipped
block5 = Block(view=4, qc=StandardQc(block=block3.id(), view=3), _id=b"4")
carnot.receive_block(block5)
self.assertEqual(len(carnot.safe_blocks), 5)
# next block has a different view but is duplicated and as it is already processed should be skipped
block5 = Block(view=5, qc=StandardQc(block=block3.id(), view=4), _id=b"4")
carnot.receive_block(block5)
self.assertEqual(len(carnot.safe_blocks), 5)
def test_receive_block_has_old_view_number(self):
carnot = Carnot(int_to_id(0))
genesis_block = self.add_genesis_block(carnot)
# 1
block1 = Block(view=1, qc=StandardQc(block=genesis_block.id(), view=0), _id=b"1")
carnot.receive_block(block1)
# 2
block2 = Block(view=2, qc=StandardQc(block=block1.id(), view=1), _id=b"2")
carnot.receive_block(block2)
# 3
block3 = Block(view=3, qc=StandardQc(block=block2.id(), view=2), _id=b"3")
carnot.receive_block(block3)
# 4
block4 = Block(view=4, qc=StandardQc(block=block3.id(), view=3), _id=b"4")
carnot.receive_block(block4)
self.assertEqual(len(carnot.safe_blocks), 5)
# This block should be rejected based on the condition below in block_is_safe().
# block.view >= self.latest_committed_view and block.view == (standard.view + 1)
# block_is_safe() should return false.
block5 = Block(view=3, qc=StandardQc(block=block4.id(), view=4), _id=b"5")
self.assertFalse(carnot.block_is_safe(block5))
carnot.receive_block(block5)
self.assertEqual(len(carnot.safe_blocks), 5)
def test_receive_block_has_an_old_qc(self):
carnot = Carnot(int_to_id(0))
genesis_block = self.add_genesis_block(carnot)
# 1
block1 = Block(view=1, qc=StandardQc(block=genesis_block.id(), view=0), _id=b"1")
carnot.receive_block(block1)
# 2
block2 = Block(view=2, qc=StandardQc(block=block1.id(), view=1), _id=b"2")
carnot.receive_block(block2)
# 3
block3 = Block(view=3, qc=StandardQc(block=block2.id(), view=2), _id=b"3")
carnot.receive_block(block3)
# 4
block4 = Block(view=4, qc=StandardQc(block=block3.id(), view=3), _id=b"4")
carnot.receive_block(block4)
self.assertEqual(len(carnot.safe_blocks), 5)
# 5 This is the old standard qc of block number 3. For standard QC we must always have qc.view==block.view-1.
# This block should be rejected based on the condition below in block_is_safe().
# block.view >= self.latest_committed_view and block.view == (standard.view + 1)
# block_is_safe() should return false.
block5 = Block(view=5, qc=StandardQc(block=block3.id(), view=3), _id=b"5")
self.assertFalse(carnot.block_is_safe(block5))
carnot.receive_block(block5)
self.assertEqual(len(carnot.safe_blocks), 5)
def test_receive_block_and_commit_its_grand_parent_chain(self):
"""
Any block with block.view < 4 must be committed
"""
carnot = Carnot(int_to_id(0))
genesis_block = self.add_genesis_block(carnot)
# 1
block1 = Block(view=1, qc=StandardQc(block=genesis_block.id(), view=0), _id=b"1")
carnot.receive_block(block1)
# 2
block2 = Block(view=2, qc=StandardQc(block=block1.id(), view=1), _id=b"2")
carnot.receive_block(block2)
# 3
block3 = Block(view=3, qc=StandardQc(block=block2.id(), view=2), _id=b"3")
carnot.receive_block(block3)
# 4
block4 = Block(view=4, qc=StandardQc(block=block3.id(), view=3), _id=b"4")
carnot.receive_block(block4)
block5 = Block(view=5, qc=StandardQc(block=block4.id(), view=4), _id=b"5")
carnot.receive_block(block5)
for block in (block1, block2, block3):
self.assertIn(block.id(), carnot.committed_blocks())
def test_receive_block_has_an_old_qc_and_tries_to_revert_a_committed_block(self):
"""
Block3 must be committed as it is the grandparent of block5. Hence, it should not be possible
to avert it.
"""
carnot = Carnot(int_to_id(0))
genesis_block = self.add_genesis_block(carnot)
# 1
block1 = Block(view=1, qc=StandardQc(block=genesis_block.id(), view=0), _id=b"1")
carnot.receive_block(block1)
# 2
block2 = Block(view=2, qc=StandardQc(block=block1.id(), view=1), _id=b"2")
carnot.receive_block(block2)
# 3
block3 = Block(view=3, qc=StandardQc(block=block2.id(), view=2), _id=b"3")
carnot.receive_block(block3)
# 4
block4 = Block(view=4, qc=StandardQc(block=block3.id(), view=3), _id=b"4")
carnot.receive_block(block4)
self.assertEqual(len(carnot.safe_blocks), 5)
# 5 This is the old standard qc of block number 2. By using the QC for block2, block5 tries to form a fork
# to avert block3 and block b4. Block3 is a committed block
# block_is_safe() should return false.
block5 = Block(view=5, qc=StandardQc(block=block2.id(), view=2), _id=b"5")
self.assertFalse(carnot.block_is_safe(block5))
carnot.receive_block(block5)
self.assertEqual(len(carnot.safe_blocks), 5)
def test_receive_block_and_verify_if_latest_committed_block_and_high_qc_is_updated(self):
carnot = Carnot(int_to_id(0))
genesis_block = self.add_genesis_block(carnot)
# 1
block1 = Block(view=1, qc=StandardQc(block=genesis_block.id(), view=0), _id=b"1")
carnot.receive_block(block1)
# 2
block2 = Block(view=2, qc=StandardQc(block=block1.id(), view=1), _id=b"2")
carnot.receive_block(block2)
# 3
block3 = Block(view=3, qc=StandardQc(block=block2.id(), view=2), _id=b"3")
carnot.receive_block(block3)
# 4
block4 = Block(view=4, qc=StandardQc(block=block3.id(), view=3), _id=b"4")
carnot.receive_block(block4)
self.assertEqual(len(carnot.safe_blocks), 5)
block5 = Block(view=5, qc=StandardQc(block=block4.id(), view=4), _id=b"5")
carnot.receive_block(block5)
self.assertEqual(carnot.latest_committed_view(), 3)
self.assertEqual(carnot.local_high_qc.view, 4)
# Test cases for vote:
def test_vote_for_received_block(self):
"""
1: Votes received should increment highest_voted_view and current_view but should not change
latest_committed_view and last_timeout_view
"""
class MockOverlay(Overlay):
def is_member_of_root_committee(self, _id: Id) -> bool:
return False
def is_member_of_child_committee(self, parent: Id, child: Id) -> bool:
return True
def super_majority_threshold(self, _id: Id) -> int:
return 10
def parent_committee(self, _id: Id) -> Optional[Committee]:
return set()
carnot = Carnot(int_to_id(0))
carnot.overlay = MockOverlay()
genesis_block = self.add_genesis_block(carnot)
# 1
block1 = Block(view=1, qc=StandardQc(block=genesis_block.id(), view=0), _id=b"1")
carnot.receive_block(block1)
votes = set(
Vote(
voter=int_to_id(i),
view=1,
block=block1.id(),
qc=StandardQc(block=block1.id(), view=1)
) for i in range(10)
)
carnot.approve_block(block1, votes)
self.assertEqual(carnot.highest_voted_view, 1)
self.assertEqual(carnot.current_view, 1)
self.assertEqual(carnot.latest_committed_view(), 0)
self.assertEqual(carnot.last_view_timeout_qc, None)
def test_vote_for_received_block_if_threshold_votes_has_not_reached(self):
"""
2 If last_voted_view is incremented after calling vote with votes lower than.
"""
class MockOverlay(Overlay):
def is_member_of_root_committee(self, _id: Id) -> bool:
return False
def is_member_of_child_committee(self, parent: Id, child: Id) -> bool:
return True
def super_majority_threshold(self, _id: Id) -> int:
return 10
def parent_committee(self, _id: Id) -> Optional[Committee]:
return set()
carnot = Carnot(int_to_id(0))
carnot.overlay = MockOverlay()
genesis_block = self.add_genesis_block(carnot)
# 1
block1 = Block(view=1, qc=StandardQc(block=genesis_block.id(), view=0), _id=b"1")
carnot.receive_block(block1)
votes = set(
Vote(
voter=int_to_id(i),
view=1,
block=block1.id(),
qc=StandardQc(block=block1.id(), view=1)
) for i in range(3)
)
with self.assertRaises((AssertionError, )):
carnot.approve_block(block1, votes)
# The test passes as asserting fails in len(votes) == self.overlay.super_majority_threshold(self.id)
# when number of votes are < 9
self.assertEqual(carnot.highest_voted_view, -1)
self.assertEqual(carnot.current_view, 1)
def test_initial_leader_proposes_and_advance(self):
class MockOverlay(Overlay):
def is_leader(self, _id: Id):
return True
def is_member_of_root_committee(self, _id: Id):
return True
def is_member_of_leaf_committee(self, _id: Id):
return True
def leader(self, view: View) -> Id:
return int_to_id(0)
def is_member_of_child_committee(self, parent: Id, child: Id) -> bool:
return True
def leader_super_majority_threshold(self, _id: Id) -> int:
return 10
def super_majority_threshold(self, _id: Id) -> int:
return 10
def parent_committee(self, _id: Id) -> Optional[Committee]:
return set()
carnot = Carnot(int_to_id(0))
carnot.overlay = MockOverlay()
genesis_block = self.add_genesis_block(carnot)
# votes for genesis block
votes = set(
Vote(
block=genesis_block.id(),
view=0,
voter=int_to_id(i),
qc=StandardQc(
block=genesis_block.id(),
view=0
),
) for i in range(10)
)
# propose a new block
proposed_block = carnot.propose_block(view=1, quorum=votes).payload
# process the proposed block as member of a committee
carnot.receive_block(proposed_block)
child_votes = set(
Vote(
block=proposed_block.id(),
view=1,
voter=int_to_id(i),
qc=StandardQc(
block=genesis_block.id(),
view=0
),
) for i in range(10)
)
# vote with votes from child committee
carnot.approve_block(proposed_block, child_votes)
# check carnot state advanced
self.assertTrue(carnot.current_view, 1)
self.assertEqual(carnot.highest_voted_view, 1)
self.assertEqual(carnot.local_high_qc.view, 0)
self.assertIn(proposed_block.id(), carnot.safe_blocks)
def test_leaf_member_advance(self):
"""
Leaf committees do not collect votes as they don't have any child. Therefore, leaf committees in happy
path votes and updates state after receipt of a block
"""
class MockOverlay(Overlay):
def is_leader(self, _id: Id):
return False
def leader(self, view: View) -> Id:
return int_to_id(0)
def parent_committee(self, _id: Id) -> Optional[Committee]:
return set()
def is_member_of_leaf_committee(self, _id: Id) -> bool:
return True
def super_majority_threshold(self, _id: Id) -> int:
return 0
carnot = Carnot(int_to_id(0))
carnot.overlay = MockOverlay()
genesis_block = self.add_genesis_block(carnot)
proposed_block = Block(view=1, qc=StandardQc(block=genesis_block.id(), view=0), _id=b"1")
# Receive the proposed block as a member of the leaf committee
carnot.receive_block(proposed_block)
carnot.approve_block(proposed_block, set())
proposed_block = Block(view=2, qc=StandardQc(block=genesis_block.id(), view=1), _id=b"2")
carnot.receive_block(proposed_block)
carnot.approve_block(proposed_block, set())
# Assert that the current view, highest voted view, and local high QC have all been updated correctly
self.assertEqual(carnot.current_view, 2)
self.assertEqual(carnot.highest_voted_view, 2)
self.assertEqual(carnot.local_high_qc.view, 1)
# Assert that the proposed block has been added to the set of safe blocks
self.assertIn(proposed_block.id(), carnot.safe_blocks)
def test_single_committee_advance(self):
"""
Test that having a single committee (both root and leaf) and a leader is able to advance
"""
nodes = [Carnot(int_to_id(i)) for i in range(4)]
leader = nodes[0]
class MockOverlay(Overlay):
def is_member_of_child_committee(self, parent: Id, child: Id) -> bool:
return False
def leader_super_majority_threshold(self, _id: Id) -> int:
return 3
def is_leader(self, _id: Id):
# Leader is the node with id 0, otherwise not
return {
int_to_id(0): True
}.get(_id, False)
def is_member_of_root_committee(self, _id: Id):
return True
def leader(self, view: View) -> Id:
return int_to_id(0)
def parent_committee(self, _id: Id) -> Optional[Committee]:
return None
def is_member_of_leaf_committee(self, _id: Id) -> bool:
return True
def super_majority_threshold(self, _id: Id) -> int:
return 0
overlay = MockOverlay()
# inject overlay
genesis_block = None
for node in nodes:
node.overlay = overlay
genesis_block = self.add_genesis_block(node)
# votes for genesis block
votes = set(
Vote(
block=genesis_block.id(),
view=0,
voter=int_to_id(i),
qc=StandardQc(
block=genesis_block.id(),
view=0
),
) for i in range(3)
)
proposed_block = leader.propose_block(1, votes).payload
votes = []
for node in nodes:
node.receive_block(proposed_block)
vote = node.approve_block(proposed_block, set())
votes.append(vote.payload)
next_proposed_block = leader.propose_block(2, set(votes)).payload
for node in nodes:
# A node receives the second proposed block
node.receive_block(next_proposed_block)
# it hasn't voted for the view 2, so its state is linked to view 1 still
self.assertEqual(node.highest_voted_view, 1)
self.assertEqual(node.current_view, 2)
# when the node approves the vote we update the current view
# and the local high qc, so they need to be increased
node.approve_block(next_proposed_block, set())
self.assertEqual(node.current_view, 2)
self.assertEqual(node.local_high_qc.view, 1)
self.assertEqual(node.highest_voted_view, 2)

265
carnot/test_unhappy_path.py Normal file
View File

@ -0,0 +1,265 @@
from carnot import *
from unittest import TestCase
from itertools import chain
class MockCarnot(Carnot):
def __init__(self, id):
super(MockCarnot, self).__init__(id)
def rebuild_overlay_from_timeout_qc(self, timeout_qc: TimeoutQc):
pass
class MockOverlay(Overlay):
"""
Overlay for 5 nodes where the leader is the single member of the root committee
0
│
1◄──┴──►2
│
3◄─┴─►4
"""
def __init__(self):
self.parents = {
int_to_id(1): {int_to_id(0)},
int_to_id(2): {int_to_id(0)},
int_to_id(3): {int_to_id(1)},
int_to_id(4): {int_to_id(1)}
}
self.childs = {
int_to_id(0): {
int_to_id(1), int_to_id(2)
},
int_to_id(1): {
int_to_id(3), int_to_id(4)
}
}
self.leafs = {
int_to_id(2), int_to_id(3), int_to_id(4)
}
def leaf_committees(self) -> Set[Committee]:
return [[leaf] for leaf in self.leafs]
def root_committee(self) -> Committee:
return {int_to_id(0)}
def is_child_of_root_committee(self, _id: Id) -> bool:
return _id in {int_to_id(1), int_to_id(2)}
def is_member_of_child_committee(self, parent: Id, child: Id) -> bool:
return child in childs if (childs := self.childs.get(parent)) else False
def leader_super_majority_threshold(self, _id: Id) -> int:
return 3
def is_leader(self, _id: Id):
# Leader is the node with id 0, otherwise not
return _id == int_to_id(0)
def is_member_of_root_committee(self, _id: Id):
return _id == int_to_id(0)
def leader(self, view: View) -> Id:
return int_to_id(0)
def parent_committee(self, _id: Id) -> Optional[Committee]:
return self.parents.get(_id)
def is_member_of_leaf_committee(self, _id: Id) -> bool:
return _id in self.leafs
def super_majority_threshold(self, _id: Id) -> int:
thresholds = {
int_to_id(0): 2,
int_to_id(1): 2,
}
return thresholds.get(_id, 0)
def add_genesis_block(carnot: Carnot) -> Block:
genesis_block = Block(view=0, qc=StandardQc(block=b"", view=0), _id=b"")
carnot.safe_blocks[genesis_block.id()] = genesis_block
carnot.receive_block(genesis_block)
carnot.local_high_qc = genesis_block.qc
carnot.current_view = 1
return genesis_block
def setup_initial_setup(test_case: TestCase, overlay: MockOverlay, size: int) -> (Dict[Id, Carnot], MockCarnot, Block):
nodes = {int_to_id(i): MockCarnot(int_to_id(i)) for i in range(size)}
# add overlay
for node in nodes.values():
node.overlay = overlay
leader: MockCarnot = nodes[overlay.leader(0)]
genesis_block = None
for node in nodes.values():
genesis_block = add_genesis_block(node)
# votes for genesis block
genesis_votes = set(
Vote(
block=genesis_block.id(),
view=0,
voter=int_to_id(i),
qc=StandardQc(
block=genesis_block.id(),
view=0
),
) for i in range(5)
)
proposed_block = leader.propose_block(1, genesis_votes).payload
test_case.assertIsNotNone(proposed_block)
return nodes, leader, proposed_block
def parents_from_childs(overlay: MockOverlay, childs: List[Id]) -> Set[Id]:
if len(childs) == 0:
return set()
possible_parents = filter(
lambda x: x is not None,
chain.from_iterable(parent for _id in childs if (parent := overlay.parent_committee(_id)))
)
return set(possible_parents) if possible_parents else set()
def succeed(test_case: TestCase, overlay: MockOverlay, nodes: Dict[Id, MockCarnot], proposed_block: Block) -> List[Vote]:
# broadcast the block
for node in nodes.values():
node.receive_block(proposed_block)
votes = {}
childs_ids = list(chain.from_iterable(overlay.leaf_committees()))
leafs = [nodes[_id] for _id in childs_ids]
for node in leafs:
vote = node.approve_block(proposed_block, set()).payload
votes[node.id] = vote
while len(parents := parents_from_childs(overlay, childs_ids)) != 0:
for node_id in parents:
node = nodes[node_id]
child_votes = [votes[_id] for _id in votes.keys() if overlay.is_member_of_child_committee(node_id, _id)]
if len(child_votes) == overlay.super_majority_threshold(node_id) and node_id not in votes:
vote = node.approve_block(proposed_block, child_votes).payload
votes[node_id] = vote
childs_ids = list(set(parents))
root_votes = [
votes[node_id]
for node_id in nodes
if overlay.is_member_of_root_committee(node_id) or overlay.is_child_of_root_committee(node_id)
]
return root_votes
def fail(test_case: TestCase, overlay: MockOverlay, nodes: Dict[Id, MockCarnot], proposed_block: Block) -> List[NewView]:
# broadcast the block
for node in nodes.values():
node.receive_block(proposed_block)
node: MockCarnot
timeouts = []
for node in (nodes[_id] for _id in nodes if overlay.is_member_of_root_committee(_id) or overlay.is_child_of_root_committee(_id)):
timeout = node.local_timeout().payload
timeouts.append(timeout)
root_member = next(nodes[_id] for _id in nodes if overlay.is_member_of_root_committee(_id))
timeout_qc = root_member.timeout_detected(timeouts).payload
for node in nodes.values():
node.receive_timeout_qc(timeout_qc)
votes = {}
childs_ids = list(chain.from_iterable(overlay.leaf_committees()))
leafs = [nodes[_id] for _id in childs_ids]
for node in leafs:
vote = node.approve_new_view(timeout_qc, set()).payload
votes[node.id] = vote
while len(parents := parents_from_childs(overlay, childs_ids)) != 0:
for node_id in parents:
node = nodes[node_id]
child_votes = [votes[_id] for _id in votes.keys() if overlay.is_member_of_child_committee(node_id, _id)]
if len(child_votes) == overlay.super_majority_threshold(node_id) and node_id not in votes:
vote = node.approve_new_view(timeout_qc, child_votes).payload
votes[node_id] = vote
childs_ids = list(set(parents))
root_votes = [
votes[node_id]
for node_id in nodes
if overlay.is_member_of_root_committee(node_id) or overlay.is_child_of_root_committee(node_id)
]
return root_votes
class TestCarnotUnhappyPath(TestCase):
def test_timeout_high_qc(self):
"""
At the end of the timeout the highQC in the next leader's aggregatedQC should be the highestQC held by the
majority of nodes or a qc higher than th highestQC held by the majority of nodes.
Majority means more than two thirds of total number of nodes, randomly assigned to committees.
"""
overlay = MockOverlay()
nodes, leader, proposed_block = setup_initial_setup(self, overlay, 5)
# In this loop 'view' is the view that fails
for view in range(1, 4, 2):
# When view v fails, a timeout qc is built for view v and nodes jump to view v + 1
# while aggregating votes for the high qc. Those votes are then forwarded to the leader of view v + 2
# which can propose a block with those aggregate votes as proof of the previous round completion.
root_votes = fail(self, overlay, nodes, proposed_block)
proposed_block = leader.propose_block(view+2, root_votes).payload
# Add final assertions on nodes
# Thus, the first block that can be proposed is 2 views after the timeout
self.assertEqual(proposed_block.view, view + 2)
# Its qc is always for the view before the block is proposed for
self.assertEqual(proposed_block.qc.view, view + 1)
# The high qc is 0, since we never had a successful round
self.assertEqual(proposed_block.qc.high_qc().view, 0)
self.assertEqual(leader.last_view_timeout_qc.view, view)
self.assertEqual(leader.local_high_qc.view, 0)
self.assertEqual(leader.highest_voted_view, view+1)
for node in nodes.values():
self.assertEqual(node.latest_committed_view(), 0)
def test_interleave_success_fails(self):
"""
At the end of the timeout the highQC in the next leader's aggregatedQC should be the highestQC held by the
majority of nodes or a qc higher than th highestQC held by the majority of nodes.
Majority means more than two thirds of total number of nodes, randomly assigned to committees.
"""
overlay = MockOverlay()
leader: MockCarnot
nodes, leader, proposed_block = setup_initial_setup(self, overlay, 5)
for view in range(2, 5):
root_votes = succeed(self, overlay, nodes, proposed_block)
proposed_block = leader.propose_block(view, root_votes).payload
root_votes = fail(self, overlay, nodes, proposed_block)
proposed_block = leader.propose_block(6, root_votes).payload
for view in range(7, 8):
root_votes = succeed(self, overlay, nodes, proposed_block)
proposed_block = leader.propose_block(view, root_votes).payload
root_votes = fail(self, overlay, nodes, proposed_block)
proposed_block = leader.propose_block(9, root_votes).payload
for view in range(10, 15):
root_votes = succeed(self, overlay, nodes, proposed_block)
proposed_block = leader.propose_block(view, root_votes).payload
committed_blocks = [view for view in range(1, 11) if view not in (4, 5, 7, 8)]
for node in nodes.values():
for view in committed_blocks:
self.assertIn(view, [block.view for block in node.committed_blocks().values()])