status-im
/
nim-raft
mirror of https://github.com/status-im/nim-raft.git
2
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Refactor the types and API to accomodate user defined Log Entry Data Type and State Machine State type + user defined State Machine initialization/application functions

This commit is contained in:
Raycho Mukelov 2023-08-09 10:17:03 +03:00
parent ede7e0b75a
commit b20ba9fac8
12 changed files with 112 additions and 64 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

BIN
doc/RAFT_Library_Roadmap_Proposal.odt
View File

Binary file not shown.

33
doc/road-map.md
View File

@ -0,0 +1,33 @@
# RAFT Consensus Nim library Road-map
## Proposed milestones during the library development
1. Create Nim library package. Implement basic functionality: fully functional RAFT Node and it’s API. The RAFT Node should be abstract working without network communication by the means of API calls and Callback calls only. The RAFT Node should cover all the functionality described in the RAFT Paper excluding Dynamic Adding/Removing of RAFT Node Peers and Log Compaction. Create appropriate tests.
*Duration: 3 weeks (2 weeks for implementation, 1 week for test creation/testing)*
2. Implement advanced functionality: Log Compaction and Dynamic Adding/Removing of RAFT Node Peers and the corresponding tests. Implement Anti Entropy measures observed in other projects (if appropriate).
*Duration: 3 weeks (2 weeks for implementation, 1 week for test creation/testing)*
3. Integrate the RAFT library in the Nimbus project - define p2p networking deal with serialization etc. Create relevant tests. I guess it is a good idea to add some kind of basic RAFT Node metrics. Optionally implement some of the following enhancements (if relevant):
- Optimistic pipelining to reduce log replication latency
- Writing to leader's disk in parallel
- Automatic stepping down when the leader loses quorum
- Leadership transfer extension
- Pre-vote protocol
*Duration: 1+ week (?)[^note]*
4. Final testing of the solution. Fix unexpected bugs.
*Duration: 1 week (?)[^note]*
5. Implement any new requirements aroused after milestone 4 completion.
*Duration: 0+ week(s) (?)[^note]*
6. End
---
[^note] Durations marked with an (?) means I am not pretty sure how much this will take.

6
raft_consensus.nim → raft.nim
View File

@ -1,4 +1,4 @@
# nim-raft-consesnsus
# nim-raft
# Copyright (c) 2023 Status Research & Development GmbH
# Licensed under either of
# * Apache License, version 2.0, ([LICENSE-APACHE](LICENSE-APACHE))
@ -8,7 +8,7 @@
# those terms.
import
raft_consensus/raft_consensus_api
raft/raft_api
export
raft_consensus_api, types, protocol
raft_api, types, protocol

9
raft_consensus.nimble → raft.nimble
View File

@ -1,4 +1,4 @@
# nim-raft-consensus
# nim-raft
# Copyright (c) 2023 Status Research & Development GmbH
# Licensed under either of
# * Apache License, version 2.0, ([LICENSE-APACHE](LICENSE-APACHE))
@ -9,7 +9,7 @@
mode = ScriptMode.Verbose
packageName = "raft_consensus"
packageName = "raft"
version = "0.0.1"
author = "Status Research & Development GmbH"
description = "raft consensus in nim"
@ -21,6 +21,7 @@ requires "stew >= 0.1.0"
requires "nimcrypto >= 0.5.4"
requires "unittest2 >= 0.0.4"
requires "chronicles >= 0.10.2"
requires "nim-eth >= 1.0.0"
requires "eth >= 1.0.0"
requires "chronos >= 3.2.0"
# Helper functions
# Helper functions

2
raft_consensus/anti_entropy.nim → raft/anti_entropy.nim
View File

@ -1,4 +1,4 @@
# nim-raft-consesnsus
# nim-raft
# Copyright (c) 2023 Status Research & Development GmbH
# Licensed under either of
# * Apache License, version 2.0, ([LICENSE-APACHE](LICENSE-APACHE))

2
raft_consensus/consensus_module.nim → raft/consensus_module.nim
View File

@ -1,4 +1,4 @@
# nim-raft-consesnsus
# nim-raft
# Copyright (c) 2023 Status Research & Development GmbH
# Licensed under either of
# * Apache License, version 2.0, ([LICENSE-APACHE](LICENSE-APACHE))

2
raft_consensus/log_compaction_module.nim → raft/log_compaction_module.nim
View File

@ -1,4 +1,4 @@
# nim-raft-consesnsus
# nim-raft
# Copyright (c) 2023 Status Research & Development GmbH
# Licensed under either of
# * Apache License, version 2.0, ([LICENSE-APACHE](LICENSE-APACHE))

2
raft_consensus/membersip_change_module.nim → raft/membersip_change_module.nim
View File

@ -1,4 +1,4 @@
# nim-raft-consesnsus
# nim-raft
# Copyright (c) 2023 Status Research & Development GmbH
# Licensed under either of
# * Apache License, version 2.0, ([LICENSE-APACHE](LICENSE-APACHE))

27
raft_consensus/protocol.nim → raft/protocol.nim
View File

@ -1,4 +1,4 @@
# nim-raft-consesnsus
# nim-raft
# Copyright (c) 2023 Status Research & Development GmbH
# Licensed under either of
# * Apache License, version 2.0, ([LICENSE-APACHE](LICENSE-APACHE))
@ -10,36 +10,39 @@
# #
# RAFT Messages Protocol definition #
# #
import options
import types
type
# RAFT Node Messages definitions
# RAFT Node Messages OPs
RAFTMessageOps* = enum
REQUEST_VOTE = 0,
APPEND_LOG_ENTRY = 1,
INSTALL_SNAPSHOT = 2 # For dynamic adding of new RAFT Nodes
RAFTMessagePayloadChecksum* = object # Checksum probably will be a SHA3 hash not sure about this at this point
RAFTMessagePayload* = ref object
data*: RAFTNodeLogEntry
RAFTMessagePayload*[LogEntryDataType] = ref object
data*: RAFTNodeLogEntry[LogEntryDataType]
checksum*: RAFTMessagePayloadChecksum
RAFTMessage* = ref object of RAFTMessageBase
RAFTMessage*[LogEntryDataType] = ref object of RAFTMessageBase
op*: RAFTMessageOps # Message Op - Ask For Votes, Append Entry(ies) or Install Snapshot
payload*: seq[RAFTMessagePayload] # Message Payload(s) - e.g. log entry(ies) etc. Will be empty for a Heart-Beat # Heart-Beat will be a message with Append Entry(ies) Op and empty payload
payload*: Option[seq[RAFTMessagePayload[LogEntryDataType]]] # Optional Message Payload(s) - e.g. log entry(ies). Will be empty for a Heart-Beat # Heart-Beat will be a message with Append Entry(ies) Op and empty payload
RAFTMessageResponse* = ref object of RAFTMessageBase
RAFTMessageResponse*[SMStateType] = ref object of RAFTMessageBase
success*: bool # Indicates success/failure
state*: Option[SMStateType] # RAFT Abstract State Machine State
# RAFT Node Client Request/Response definitions
RAFTNodeClientRequestOps = enum
REQUEST_STATE = 0,
APPEND_NEW_ENTRY = 1
RAFTNodeClientRequest* = ref object
RAFTNodeClientRequest*[LogEntryDataType] = ref object
op*: RAFTNodeClientRequestOps
payload*: RAFTNodeLogEntry
payload*: Option[RAFTMessagePayload[LogEntryDataType]] # Optional RAFTMessagePayload carrying a Log Entry
RAFTNodeClientResponse* = ref object
success*: bool # Indicate succcess
raft_node_redirect_id*: RAFTNodeId # RAFT Node ID to redirect the request to in case of failure
RAFTNodeClientResponse*[SMStateType] = ref object
success*: bool # Indicate succcess
state*: Option[SMStateType] # Optional RAFT Abstract State Machine State
raft_node_redirect_id*: Option[RAFTNodeId] # Optional RAFT Node ID to redirect the request to in case of failure

33
raft_consensus/raft_consensus_api.nim → raft/raft_api.nim
View File

@ -1,4 +1,4 @@
# nim-raft-consesnsus
# nim-raft
# Copyright (c) 2023 Status Research & Development GmbH
# Licensed under either of
# * Apache License, version 2.0, ([LICENSE-APACHE](LICENSE-APACHE))
@ -13,19 +13,21 @@ import protocol
export types, protocol
# RAFT Node Public API procedures / functions
proc RAFTNodeCreateNew*(id: RAFTNodeId, peers: RAFTNodePeers, state_machine: RAFTNodeStateMachine, # Create New RAFT Node
log: RAFTNodeLog, persistent_storage: RAFTNodePersistentStorage,
msg_send_callback: RAFTMessageSendCallback): RAFTNode =
proc RAFTNodeCreateNew*[LogEntryDataType, SMStateType]( # Create New RAFT Node
id: RAFTNodeId, peers: RAFTNodePeers,
persistent_storage: RAFTNodePersistentStorage,
msg_send_callback: RAFTMessageSendCallback): RAFTNode[LogEntryDataType, SMStateType] =
discard
proc RAFTNodeLoad*(state_machine: RAFTNodeStateMachine, log: RAFTNodeLog, # Load RAFT Node From Storage
persistent_storage: RAFTNodePersistentStorage, msg_send_callback: RAFTMessageSendCallback): Result[RAFTNode, string] =
proc RAFTNodeLoad*[LogEntryDataType, SMStateType](
persistent_storage: RAFTNodePersistentStorage, # Load RAFT Node From Storage
msg_send_callback: RAFTMessageSendCallback): Result[RAFTNode[LogEntryDataType, SMStateType], string] =
discard
proc RAFTNodeStop*(node: RAFTNode) =
discard
proc RAFTNodeStart*(node: RaftNode) =
proc RAFTNodeStart*(node: RAFTNode) =
discard
func RAFTNodeIdGet*(node: RAFTNode): RAFTNodeId = # Get RAFT Node ID
@ -49,12 +51,21 @@ proc RAFTNodeMessageDeliver*(node: RAFTNode, raft_message: RAFTMessageBase): RAF
proc RAFTNodeRequest*(node: RAFTNode, req: RAFTNodeClientRequest): RAFTNodeClientResponse = # Process RAFTNodeClientRequest
discard
proc RAFTNodeLogLenGet*(node: RAFTNode): RAFTLogIndex =
proc RAFTNodeLogIndexGet*(node: RAFTNode): RAFTLogIndex =
node.log_index
discard
proc RAFTNodeLogEntryGet*(node: RAFTLogIndex): Result[RAFTNodeLogEntry, string] =
proc RAFTNodeLogEntryGet*(node: RAFTNode, log_index: RAFTLogIndex): Result[RAFTNodeLogEntry, string] =
discard
proc RAFTNodeStateMachineStateGet*(node: RAFTNode): RAFTNodeStateMachineState =
discard
# Abstract State Machine Ops
func RAFTNodeSMStateGet*[LogEntryDataType, SMStateType](node: RAFTNode[LogEntryDataType, SMStateType]): SMStateType =
node.state_machine.state
proc RAFTNodeSMInit[LogEntryDataType, SMStateType](state_machine: var RAFTNodeStateMachine[LogEntryDataType, SMStateType]) =
mixin RAFTSMInit
RAFTSMInit(state_machine)
proc RAFTNodeSMApply[LogEntryDataType, SMStateType](state_machine: RAFTNodeStateMachine[LogEntryDataType, SMStateType], log_entry: LogEntryDataType) =
mixin RAFTSMApply
RAFTSMApply(state_machine, log_entry)

58
raft_consensus/types.nim → raft/types.nim
View File

@ -1,4 +1,4 @@
# nim-raft-consesnsus
# nim-raft
# Copyright (c) 2023 Status Research & Development GmbH
# Licensed under either of
# * Apache License, version 2.0, ([LICENSE-APACHE](LICENSE-APACHE))
@ -30,53 +30,54 @@ type
RAFTNodeTerm* = uint64 # RAFT Node Term Type
RAFTLogIndex* = uint64 # RAFT Node Log Index Type
# RAFT Node State Machine basic definitions
RAFTNodeStateMachineState* = object # State Machine State
RAFTNodeStateMachine* = ref object # Some probably opaque State Machine Impelementation to be used by the RAFT Node
# RAFT Node Abstract State Machine type
RAFTNodeStateMachine*[LogEntryDataType, SMStateType] = ref object # Some probably opaque State Machine Impelementation to be used by the RAFT Node
# providing at minimum operations for initialization, querying the current state
# and RAFTNodeLogEntry application
state: RAFTNodeStateMachineState
state: SMStateType
# RAFT Node Persistent Storage basic definition
RAFTNodePersistentStorage* = ref object # Should be some kind of Persistent Transactional Store Wrapper
# Basic modules (algos) definitions
RAFTNodeAccessCallback = proc: RAFTNode {.nimcall, gcsafe.} # This should be implementes as a closure holding the RAFTNode
RAFTConsensusModule* = object of RootObj
RAFTNodeAccessCallback[LogEntryDataType] = proc: RAFTNode[LogEntryDataType] {.nimcall, gcsafe.} # This should be implementes as a closure holding the RAFTNode
RAFTConsensusModule*[LogEntryDataType] = object of RootObj
state_transitions_fsm: seq[byte] # I plan to use nim.fsm https://github.com/ba0f3/fsm.nim
raft_node_access_callback: RAFTNodeAccessCallback
raft_node_access_callback: RAFTNodeAccessCallback[LogEntryDataType]
RAFTLogCompactionModule* = object of RootObj
raft_node_access_callback: RAFTNodeAccessCallback
RAFTLogCompactionModule*[LogEntryDataType] = object of RootObj
raft_node_access_callback: RAFTNodeAccessCallback[LogEntryDataType]
RAFTMembershipChangeModule* = object of RootObj
raft_node_access_callback: RAFTNodeAccessCallback
RAFTMembershipChangeModule*[LogEntryDataType] = object of RootObj
raft_node_access_callback: RAFTNodeAccessCallback[LogEntryDataType]
# Callback for sending messages out of this RAFT Node
RAFTMessageId* = UUID # UUID assigned to every RAFT Node Message,
# so it can be matched with it's coresponding response etc.
# so it can be matched with it's corresponding response etc.
RAFTMessageSendCallback* = proc (raft_message: RAFTMessageBase) {.nimcall, gcsafe.} # Callback for Sending RAFT Node Messages
# out of this RAFT Node. Can be used for broadcasting
# (a Heart-Beat for example)
# RAFT Node basic Log definitions
RAFTNodeLogEntry* = ref object # Abstarct RAFT Node Log entry containing opaque binary data (Blob)
RAFTNodeLogEntry*[LogEntryDataType] = ref object # Abstarct RAFT Node Log entry containing opaque binary data (Blob etc.)
term*: RAFTNodeTerm
data*: Blob
data*: LogEntryDataType
RAFTNodeLog* = ref object # Needs more elaborate definition. Probably this will be a RocksDB/MDBX/SQLite Store Wrapper etc.
log_data*: seq[RAFTNodeLogEntry] # RAFT Node Log Data
RAFTNodeLog*[LogEntryDataType] = ref object # Needs more elaborate definition.
# Probably this will be a RocksDB/MDBX/SQLite Store Wrapper etc.
log_data*: seq[RAFTNodeLogEntry[LogEntryDataType]] # RAFT Node Log Data
# Base typoe for RAFT message objects
# Base type for RAFT message objects
RAFTMessageBase* = ref object of RootObj # Base Type for RAFT Node Messages
msg_id*: RAFTMessageId # Message UUID
sender_id*: RAFTNodeId # Sender RAFT Node ID
sender_term*: RAFTNodeTerm # Sender RAFT Node Term
peers*: RAFTNodePeers # List of RAFT Node IDs, which should receive this message
# RAFT Node Object definitions
RAFTNode* = object
# RAFT Node Object type
RAFTNode*[LogEntryDataType, SMStateType] = ref object
# Timers
voting_timout: uint64
heart_beat_timeout: uint64
@ -89,9 +90,9 @@ type
raft_comm_mutex_client_response: Lock
# Modules (Algos)
consensus_module: RAFTConsensusModule
log_compaction_module: RAFTLogCompactionModule
membership_change_module: RAFTMembershipChangeModule
consensus_module: RAFTConsensusModule[LogEntryDataType]
log_compaction_module: RAFTLogCompactionModule[LogEntryDataType]
membership_change_module: RAFTMembershipChangeModule[LogEntryDataType]
# Misc
msg_send_callback: RAFTMessageSendCallback
@ -101,17 +102,16 @@ type
id: RAFTNodeId # This RAFT Node ID
state: RAFTNodeState # This RAFT Node State
current_term: RAFTNodeTerm # Latest term this RAFT Node has seen (initialized to 0 on first boot, increases monotonically)
log: RAFTNodeLog # This RAFT Node Log
voted_for: RAFTNodeId # Candidate RAFTNodeId that received vote in current term (or nil/zero if none)
log: RAFTNodeLog[LogEntryDataType] # This RAFT Node Log
voted_for: RAFTNodeId # Candidate RAFTNodeId that received vote in current term (or nil/zero if none),
# also used to redirect Client Requests in case this RAFT Node is not the leader
peers: RAFTNodePeers # This RAFT Node Peers IDs. I am not sure if this must be persistent or volatile but making it persistent
# makes sense for the moment
state_machine: RAFTNodeStateMachine # Not sure for now putting it here. I assume that persisting the State Machine's state is enough
# to consider it 'persisted'
state_machine: RAFTNodeStateMachine[LogEntryDataType, SMStateType] # Not sure for now putting it here. I assume that persisting the State Machine's
# state is enough to consider it 'persisted'
# Volatile state
commit_index: RAFTLogIndex # Index of highest log entry known to be committed (initialized to 0, increases monotonically)
last_applied: RAFTLogIndex # Index of highest log entry applied to state machine (initialized to 0, increases monotonically)
current_leader_id: RAFTNodeId # Current RAFT Node Leader ID (used to redirect Client Requests in case this RAFT Node is not the leader)
# Volatile state on leaders
next_index: seq[RAFTLogIndex] # For each peer RAFT Node, index of the next log entry to send to that Node

2
tests/all_tests.nim
View File

@ -1,4 +1,4 @@
# nim-raft-consesnsus
# nim-raft
# Copyright (c) 2023 Status Research & Development GmbH
# Licensed under either of
# * Apache License, version 2.0, ([LICENSE-APACHE](LICENSE-APACHE))